Laure Wynants has been working with prediction models since receiving her PhD. “To predict whether ovarian tumours are benign or malignant, for example, or the risk of bloodstream infection after inserting a catheter.”
One morning last March, she had an idea. “We know from the literature that prediction models can help predict other diseases, so COVID-19 would be no different. I wondered if any models had already been developed, for example in China, where the epidemic began. They could help make important decisions. When there was a shortage of coronavirus tests: does this person need to be tested? In the doctor’s office: can this person recover at home or do they need hospital care? In the hospital: can this patient be admitted to the COVID ward or do they need intensive care? And in the intensive care unit: what are this patient’s prospects? How long should we continue to provide care?”
The basic principle of a prediction model is simple. You enter factors into it that may influence a certain disease, COVID-19 in this case. “You assign a weight to each factor. For example, a patient’s age may play a greater role than whether or not they have a fever.” The algorithm then calculates the probability of a certain outcome for the patient. “We not only looked at traditional prediction models, but also, for example, at Artificial Intelligence to automatically interpret CT scans.” But in order for the model to be reliable, it has to use the right factors.
“I thought, what if we just appraise every paper about a COVID-19 prediction model to see how valid and useful it is. A systematic collection and review of all published research.” Wynants called her colleague Maarten van Smeden at Utrecht University. He loved the idea. “It all happened very quickly after that. We asked a few people in our network to join us. They immediately said yes, including some big names in our field.”
The quality of the research they have reviewed is disappointing. “The majority of the models claim to work very well, but we’ve only found four models with a low risk of systematic errors. In fact, several models were worse at predicting disease progression than only looking at age as a factor.”
How is that possible? “Over-optimism – people overestimating the accuracy of their model – and overfitting: the model is designed in such a way that it works very well with the data they feed it. But that doesn’t say anything about how and whether it will work in other situations, like with new patients or in another country. Also, a lot of researchers really live on their own data island. In the beginning, we had to make do with what we had: the available data were limited. But even now, there are still people working with fairly small data sets, despite the fact that these kinds of models need a lot of data, preferably from different hospitals and countries. You have to work together.”
The BMJ, a renowned journal, agreed to open a living review in which Wynants and her fellow researchers can keep updating their findings. By now, the team has read a total of 40 thousand articles and reviewed 236 models. “Always by at least two people.” It’s quite an effort, which they are mainly carrying out in their spare time. “We didn’t receive any funding for this project, so everyone is working on it in the evenings and weekends. There have been quite a few late-night videocalls between Maarten and me.”
Wynants and her team are focusing on three things. Firstly, they want to give pointers to people looking to develop their own model. What should they pay attention to? “The paper is being cited a lot, so we can see that people are actually using it.” Secondly, they want to use the data for international validation. “This is when you check whether a particular model works equally well in different circumstances. We’re still working on this. It’s a big project involving a lot of sensitive personal data.” Thirdly, they are developing a model for general practitioners to use. “I’m working on this together with Jochen Cals [a general practitioner and professor at UM]. There still isn’t a lot of information available for primary care providers, even though they are also making important decisions.”
The BMJ wants to keep the living review open until April 2022. “But hopefully, the models will be redundant earlier”, says Wynants. “Although we may continue to see small outbreaks of COVID-19 in the future.” She wants to use the Hustinx Prize money (15 thousand euros) for the project. “We already have a new batch of articles waiting for us.”