The Value and Challenges of Using Multi-Center Data

Allison M. Janda, MD

Investigators and experts in statistics and data-analysis, Robert Freundlich, MD, MS, MSCI, Ira Hofer, MD, and Amy Shanks, PhD, delved deep into the value add of multicenter data when conducting trials. Their session, “Moving Past the Single Center Trial: Opportunities and Challenges in Multi-Center Data Collaborations,” held on Sunday, March 20 at the IARS 2022 Annual Meeting covered methods for utilizing multi-center data, the challenges surrounding sharing those data, using multi-center administration, electronic health records and registry data for quality improvement.

First panelist, Robert Freundlich, MD, MS, MSCI, Chief, Informatics Research Division, Associate Professor of Anesthesiology and Biomedical Informatics, Anesthesiology Critical Care Medicine, Informatics Research Division, and Director, Vanderbilt Anesthesiology & Perioperative Informatics Research at Vanderbilt University Medical Center, discussed “External Validation – Applying Results from Other Centers to Your Practice.” First, he focused on how we can make models using our data more useful. Dr. Freundlich emphasized the key to creating a useful model is to create a trustworthy one which is generalizable and reproducible. External validation is crucial to help establish trust, including temporal (same population, different time), geographical (different population, similar characteristics), and domain validation. As an example of this strategy, Dr. Freundlich discussed his paper, “A Predictive Model of Reintubation After Cardiac Surgery Using the Electronic Health Record” (Freundlich et al., 2021). Model performance can be presented, and model accuracy can be assessed through calibration and discrimination, but the findings or coefficients of each model may not be extrapolatable to other patient populations or institutions. Dr. Freundlich explained that this study was limited by the single-center scope, so his group has engaged other institutions and they are currently performing geographic validation and shared some of the preliminary data regarding this study, involving geographic validation of the model. He closed by emphasizing the importance of external validation to generate trustworthy and generalizable models.

Ira Hofer, MD, a clinical informaticist and Associate Professor, Department of Anesthesiology and Medicine at Icahn School of Medicine at Mount Sinai as well as the Director of the Division of Bioinformatics and Analytics at University of California, Los Angeles, discussed the challenges with multicenter studies and focused on how federated learning can be a possible tool in his talk, “Federated Learning – Multi-Center Results Without Sharing Data.” Dr. Hofer started by providing an overview of machine learning. He emphasized that the size of the data needed is proportional to the number of features, therefore single-center data is limited by its smaller sample size and overfitting. Also, generalizability of models trained on single-center data may be limited and may not perform similarly on data from other institutions. Federated learning can be used to help protect data security since each site trains the model on their own data and only sends the coefficients to the main site, which are then combined and sent back to the individual sites. Some challenges with federated learning are that, although no data is shared, the sites must all have the data in the same format and must have the technical infrastructure to train models at each site.

Amy Shanks, PhD, Acting Director of Analytics and Insights of Healthcare Value, Blue Cross/Blue Shield of Michigan, Detroit, discussed analyzing the methodology and reproducibility of multicenter data and modeling in her talk, “Multi-Center Quality Improvement – Using Different Kinds of Data to Get a Better Picture.” The goal of multicenter modeling is to improve outcomes, decrease unfounded variation in treatment patterns and lower total cost of care. Dr. Shanks also emphasized how cross collaboration across clinicians, payers and quality champions is key to success. She discussed the pros and cons of using payer data including the pros of the ability to assess episode-of-care costs to track use and utilization, track pharmacy spending and quantify medication adherence, and use population health measurement to guide decision-making to eliminate healthcare disparities. The cons include inability to quantify the severity of a patient’s illness, limitations regarding the granularity of data regarding demographics, laboratory and biometric data, and a lack of patient outcomes. Dr. Shanks outlined how data integration requires appropriate modeling using multilevel modeling adjusting for random and nested effects, and modeling adjusting for selection bias using methods such as propensity score matching, covariate adjustment and weighting.

She then discussed the methods of a paper by Brescia et al. (2020) which used payer data from the Michigan Value Collaborative and Society of Thoracic Surgeons Cardiac registry data to provide an excellent example of using integrated administrative and registry data. Dr. Shanks also noted that collaboration can also drive quality of care to identify best practices, implement the best practices, and measure compliance with best practice guidelines. Participation in Collaborative Quality Initiatives (CQIs) have lowered complications and substantial cost avoidance. Dr. Shanks also stated that participation in CQIs including ASPIRE, the anesthesia CQI in the state of Michigan and quality arm of the Multicenter Perioperative Outcomes group (MPOG), and surgical quality collaboratives (MBSC and MARCQI) have demonstrated an improvement in care and decreases in variation of care that have been shown to contribute to poor outcomes. Dr. Shanks concluded her presentation by emphasizing that using appropriate data integration, statistical methodology can model many different research questions and hypotheses across a diverse and broad patient population. CQI participation has enhanced healthcare quality and reduced poor outcomes and costs.