Causal Machine Learning Methods for Estimating Personalised Treatment Effects — Insights on validity from two large trials

[Submitted on 7 Jan 2025]

View a PDF of the paper titled Causal Machine Learning Methods for Estimating Personalised Treatment Effects — Insights on validity from two large trials, by Hongruyu Chen and 3 other authors

View PDF

Abstract:Causal machine learning (ML) methods hold great promise for advancing precision medicine by estimating personalized treatment effects. However, their reliability remains largely unvalidated in empirical settings. In this study, we assessed the internal and external validity of 17 mainstream causal heterogeneity ML methods — including metalearners, tree-based methods, and deep learning methods — using data from two large randomized controlled trials: the International Stroke Trial (N=19,435) and the Chinese Acute Stroke Trial (N=21,106). Our findings reveal that none of the ML methods reliably validated their performance, neither internal nor external, showing significant discrepancies between training and test data on the proposed evaluation metrics. The individualized treatment effects estimated from training data failed to generalize to the test data, even in the absence of distribution shifts. These results raise concerns about the current applicability of causal ML models in precision medicine, and highlight the need for more robust validation techniques to ensure generalizability.