Observational Trial Methodology: A New Facet

By Douglas A. Colquhoun, MB ChB, from the IARS, AUA and SOCCA 2019 Annual Meetings*

The Sunday morning panel, Understanding Observational Sciences: Updates from Perioperative Epidemiologists, was moderated by Karthik Raghunathan, MD, MPH, Associate Professor, Duke University and featured presentations from W. Scott Beattie, MD, PhD, Professor of Anesthesiology, University of Toronto, Duminda N. Wijeysundera, MD, MPH, Associate Professor, University of Toronto and Vijay Krishnamoorthy, MD, PhD, Assistant Professor, Duke University. Each presentation explored a different facet of observational trial methodology.

Dr. Beattie in his presentation, “Using Propensity Scores in Observational Research,” explored the rationale for the use propensity scores by exploring the challenges of RCT by likening this to attempting to balance an unknown number of magnitude confounders. He went on to demonstrate, via the use of standardized mean difference, using published RCTs that in practice these require extremely high numbers of subjects to (in excess of 2000 subjects per group) achieve natural balance. He subsequently described the basics of performing a propensity score analysis, in score generation (by logistic regression or classification and regression trees [CART]) and subsequent adjustment (by stratification, matching, covariate/regression adjustment, weighting) and describe methods for ensuring the propensity score works as designed (using standardized differences of the covariates).

Dr. Beattie identified characteristics of a good propensity score match that the model balances covariates with standardized differences in the order of 2-5%. It adjusts for selection bias and difference across numerous variables and includes a sensitivity analysis (recommended the calculation of the e-value to achieve an estimate of the magnitude of an unmeasured confounder which would be required to influence the result). Using practical examples from his own work and the published literature he demonstrated the benefits.

The concept of “Hospital Level Analyses” was discussed by Dr. Wijeysundera. He described that hospital level analyses are traditionally thought to be relevant to questions, which are based on the summation of concepts of hospital care such as treatment volume, teaching status, organizational culture and complex processes of care. However, in some cases, hospital level variables may actually disclose differences in patient level practices. The hospital became a proxy for a patient level intervention. Given that unmeasured confounders can never be fully excluded, truly undertaking a propensity analysis becomes challenging.

However, based on the understanding that the differences in the baseline patient populations are likely smaller than differences in the utilization of care patterns between hospitals, the approach of comparing high and low utilization hospitals for specific care patterns may offer insights into the impact of care patterns on patient outcome. He described an example of instrumental variable analysis. The applicability of this type of analysis is that it reveals the impact on the marginal patient (i.e., the patient for whom decisions to receive particular care patterns is one of clinical judgment, not patients who will “always” or “never” receive a particular treatment). Dr. Wijeysundera noted some of the pitfalls of drawing conclusions from these types of study in his closing remarks in particular around patient level comparisons.

In the final presentation of the panel, “Natural Experiments in Perioperative Medicine and Critical Care,” Dr. Krishnamoorthy described utilization of exogenous events (such as natural disasters, changes in healthcare policy or drug shortages causing shifts in practice) that are out of the control of the investigators to give understanding of how particular factors impact patient outcome. This may allow the side-stepping of unmeasured confounding present in some observational trials. All natural experiments require clearly defined pre-intervention, intervention and postintervention time periods and may benefit from a true control group not impacted by the exogenous event. Dr. Krishnamoorthy focused on different techniques and interrupted time series analysis in describing methods for analyzing the effect of these exogenous events on patient outcomes but cautioned about potential sources of bias in this analysis and the requirement for a biologically plausible association between these events.

Finally, the presentation closed with a discussion for methods for strengthening causal inferences by using multiple types of experimental design including the addition of qualitative methods and ensuring results are robust, different analytic approaches.

*Coverage from the Understanding Observational Sciences: Updates from Perioperative Epidemiologists during the IARS 2019 Annual Meeting