Value of Information Analysis for External Validation of Risk Prediction Models

作     者：Sadatsafavi, Mohsen Lee, Tae Yoon Wynants, Laure Vickers, Andrew Gustafson, Paul 

作者机构：Respiratory Evaluation Sciences Program Faculty of Pharmaceutical Sciences University of British Columbia VancouverBC Canada Department of Epidemiology CAPHRI Care and Public Health Research Institute Maastricht University Maastricht Netherlands Department of Development and Regeneration KU Leuven Leuven Belgium Department of Epidemiology and Biostatistics Memorial Sloan Kettering Cancer Center New YorkNY United States Department of Statistics University of British Columbia VancouverBC Canada 

出 版 物：《arXiv》 (arXiv)

年 卷 期：2022年

核心收录：

主　　题：Decision theory 

摘      要：Background: Before being used to inform patient care, a risk prediction model needs to be validated in a representative sample from the target population. The finite size of the validation sample entails that there is uncertainty with respect to estimates of model performance. We apply value-of-information methodology as a framework to quantify the consequence of such uncertainty in terms of NB. Methods: We define the Expected Value of Perfect Information (EVPI) for model validation as the expected loss in NB due to not confidently knowing which of the alternative decisions confers the highest NB at a given risk threshold. We propose methods for EVPI calculations based on Bayesian or ordinary bootstrapping of NBs, as well as an asymptotic approach supported by the central limit theorem. We conducted brief simulation studies to compare the performance of these methods, and used subsets of data from an international clinical trial for predicting mortality after myocardial infarction as a case study. Results: The three computation methods generated similar EVPI values in simulation studies. In the case study, at the pre-specified threshold of 0.02, the best decision with current information would be to use the model, with an expected incremental NB of 0.0020 over treating all. At this threshold, EVPI was 0.0005 (a relative EVPI of 25%). When scaled to the annual number of heart attacks in the US, this corresponds to a loss of 400 true positives, or extra 19,600 false positives per year, indicating the value of further model validation. As expected, the validation EVPI generally declined with larger samples. Conclusion: Value-of-information methods can be applied to the NB calculated during external validation of clinical prediction models to provide a decision-theoretic perspective to the consequences of uncertainty. © 2022, CC BY.