Making Research Matter: The Reproducibility Crisis in the Era of Precision Medicine

By Christian S. Guay, MD, from the IARS, AUA and SOCCA 2019 Annual Meetings*

Renowned experts in the continually evolving field of precision medicine, Drs. Regina Nuzzo, Philippe Payne, David Mazer and Bruce Biccard, shared their research on tailoring disease diagnosis, treatment, and prevention in the AUA President’s Panel during the AUA 2019 Annual Meeting.

Regina Nuzzo, PhD, outlined four fundamental problems and associated solutions for “How We Scientists Can Stop Fooling Ourselves” in her presentation. She provided attendees with a framework to improve the science that we use to ultimately guide patient care.

The first problem came in the form of a question: “Do our stats tools always line up with our intuition?” In short, the answer is no. In fact, our intuitive sense of what is due to chance and what can be considered a true signal varies depending on context. By using striking and relatable examples, Dr. Nuzzo demonstrated how using a single universal threshold for statistical significance (p<0.05) across contexts is absurd. Some solutions that she offered included likelihood ratios, Bayesian approaches, multiple lines of evidence and pre-publication replication.

The second problem asked “does statistical cheating even feel like cheating in the current system?” Here, Dr. Nuzzo unpacked the problem of “p-hacking” and the solution of blind data analysis, whereby a preliminary set of “jittered data” is used to develop the analysis algorithms, which are then applied to the true data. This strategy prevents investigators from modifying their analysis in seeking statistical significance. Problem three addressed the human tendency to passionately pursue a hypothesis, often at the expense of objective, dispassionate analysis. Proposed solutions included adversarial collaboration and independent discussions, a strategy recently implemented by the British Journal of Anesthesia.

Finally, Dr. Nuzzo asked “are we rewarding the right thing?” Through a series of biases, including publication, outcome reporting, spin and citation bias, positive results are more likely to enter the mind of a clinician than negative results. Registered reports are one mechanism through which investigators can submit their methodology to peer-review and publication prior to data acquisition. Encouraging investigators to develop and publish their methodologies before collecting data may help increase strict adherence to the published protocol and publishing results regardless of outcomes.   

Philippe Payne, PhD, FACMI, from Washington University School of Medicine, followed Dr. Nuzzo with a discussion of precision medicine and informatics. The concept of deep phenotyping, which describes a patient from their molecular make-up to their social and ecological contexts, will be central to the implementation of precision medicine in the 21^st century. However, this level of individual detail will require a shift in the way we view clinical trials. The randomized clinical trial (RCT) is still widely considered as a gold standard for evidence meant to inform clinical care. However, powering an RCT in the context of precision medicine and deep phenotyping quickly becomes impractical, with study size calculations often reaching into the millions. One solution to this problem is the adaptive trial design, whereby a series of interim analyses refine and optimize a clinical trial “on-the-go.” Dr. Payne also proposed a more generalized approach to the problem of RCT’s in the age of machine learning and EHR’s: building a smart healthcare system where research and practice become indistinguishable from one another. In other words, every clinical encounter would feed into the system’s research enterprise, and likewise every clinical encounter would be informed by the ongoing research.  

David Mazer, MD, from University of Toronto, then discussed the gaps that exist between evidence and practice. A major theme in Dr. Mazer’s presentation was that guidelines can truly help change practice and enhance safety, but that the process of integrating evidence into guidelines is often delayed, and once the guidelines are ultimately published physicians may not follow them for a whole host of reasons. As a result, a third of patients don’t receive treatment of proven effectiveness and a quarter of patients receive care that is not needed or may cause harm. Improvements in the flow of evidence into guidelines and communication of these guidelines to clinicians are needed to close these gaps.

Bruce Biccard, MBBCh, from University of Cape Town and Groote Schuur Hospital, closed the panel with his presentation on the African Surgical Outcomes Study (ASOS). This ambitious, multicenter observational study collected post-operative outcome data from 247 African hospitals, spread across 25 countries. Compared to the International Surgical Outcomes Study (ISOS), African patients were younger, had lower ASA scores, and had relatively few comorbidities. Nonetheless, ASOS reported higher complication and mortality rates compared to ISOS. Strikingly, there were 2.5 times more obstetric complications, and a 50-fold increase in maternal mortality. This phenomenon of a proportionally higher increase in mortality than complications has been termed “failure to rescue.” A follow up interventional study, ASOS 2, will aim to reduce failure to rescue rates by increasing physiological surveillance of identified high-risk patients. 

*Coverage from the AUA President’s Panel: The Reproducibility Crisis in the Era of Precision Medicine, presented by Michael Avidan, MBBCh, Regina Nuzzo, PhD, Philip R.O. Payne, PhD, David Mazer, MD, and Bruce Biccard, MBBCh, at the AUA 2019 Annual Meeting