Which model can predict how COVID-19 will progress?

Edmond Hustinx Prize 2020 goes to statistician Laure Wynants

02-03-2021

How can you predict which COVID-19 patients will become seriously ill? And what are the chances of survival in patients who have been admitted to the ICU? From the beginning of the pandemic, doctors have been trying to find answers to these kinds of questions. Prediction models can help with this. But which model works well? Statistician Laure Wynants (Care and Public Health Research Institute CAPHRI), together with a team of forty researchers from the Netherlands and abroad, started a living review to review and appraise all published papers on this topic. She was awarded the Edmond Hustinx Prize 2020, that goes to a Maastricht researcher every year, for her work.

Laure Wynants has been working with prediction models since receiving her PhD. “To predict whether ovarian tumours are benign or malignant, for example, or the risk of bloodstream infection after inserting a catheter.”

One morning last March, she had an idea. “We know from the literature that prediction models can help predict other diseases, so COVID-19 would be no different. I wondered if any models had already been developed, for example in China, where the epidemic began. They could help make important decisions. When there was a shortage of coronavirus tests: does this person need to be tested? In the doctor’s office: can this person recover at home or do they need hospital care? In the hospital: can this patient be admitted to the COVID ward or do they need intensive care? And in the intensive care unit: what are this patient’s prospects? How long should we continue to provide care?”

The basic principle of a prediction model is simple. You enter factors into it that may influence a certain disease, COVID-19 in this case. “You assign a weight to each factor. For example, a patient’s age may play a greater role than whether or not they have a fever.” The algorithm then calculates the probability of a certain outcome for the patient. “We not only looked at traditional prediction models, but also, for example, at Artificial Intelligence to automatically interpret CT scans.” But in order for the model to be reliable, it has to use the right factors.

“I thought, what if we just appraise every paper about a COVID-19 prediction model to see how valid and useful it is. A systematic collection and review of all published research.” Wynants called her colleague Maarten van Smeden at Utrecht University. He loved the idea. “It all happened very quickly after that. We asked a few people in our network to join us. They immediately said yes, including some big names in our field.”

The quality of the research they have reviewed is disappointing. “The majority of the models claim to work very well, but we’ve only found four models with a low risk of systematic errors. In fact, several models were worse at predicting disease progression than only looking at age as a factor.”

How is that possible? “Over-optimism – people overestimating the accuracy of their model – and overfitting: the model is designed in such a way that it works very well with the data they feed it. But that doesn’t say anything about how and whether it will work in other situations, like with new patients or in another country. Also, a lot of researchers really live on their own data island. In the beginning, we had to make do with what we had: the available data were limited. But even now, there are still people working with fairly small data sets, despite the fact that these kinds of models need a lot of data, preferably from different hospitals and countries. You have to work together.”

The BMJ, a renowned journal, agreed to open a living review in which Wynants and her fellow researchers can keep updating their findings. By now, the team has read a total of 40 thousand articles and reviewed 236 models. “Always by at least two people.” It’s quite an effort, which they are mainly carrying out in their spare time. “We didn’t receive any funding for this project, so everyone is working on it in the evenings and weekends. There have been quite a few late-night videocalls between Maarten and me.”

Wynants and her team are focusing on three things. Firstly, they want to give pointers to people looking to develop their own model. What should they pay attention to? “The paper is being cited a lot, so we can see that people are actually using it.” Secondly, they want to use the data for international validation. “This is when you check whether a particular model works equally well in different circumstances. We’re still working on this. It’s a big project involving a lot of sensitive personal data.” Thirdly, they are developing a model for general practitioners to use. “I’m working on this together with Jochen Cals [a general practitioner and professor at UM]. There still isn’t a lot of information available for primary care providers, even though they are also making important decisions.”

The BMJ wants to keep the living review open until April 2022. “But hopefully, the models will be redundant earlier”, says Wynants. “Although we may continue to see small outbreaks of COVID-19 in the future.” She wants to use the Hustinx Prize money (15 thousand euros) for the project. “We already have a new batch of articles waiting for us.”

The Edmond Hustinx Prize

The Edmond Hustinx Prize for Science exists to emphasise the importance of the practical application of research as well as the importance of UM to Limburg. The cash prize of 15,000 euros is awarded to a different faculty each year. The annual prize is normally awarded during the Opening of the Academic Year in September. This year, the jury needed more time due to COVID-19, so the Edmond Hustinx Prize was supposed to be awarded during the Dies Natalis celebration at the end of January instead. When that also was cancelled, the ceremony was postponed to the Opining of the Academic Year in September 2021, but they did announce the winner.

A summary of the research, written in cooperation with laymen and severe COVID-19 survivors is available on www.covprecise.org/living-review/.

The results of the research can by found in Wynants Laure, Van Calster Ben, Collins Gary S, Riley Richard D, Heinze Georg, Schuit Ewoud et al. Prediction models for diagnosis and prognosis of covid-19: systematic review and critical appraisal BMJ 2020; 369 :m1328.

Which model can predict how COVID-19 will progress?
Laure Wynants_300720_KH_01
Author: Cleo Freriks

Archive LW

Categories: Science
Tags: statistics,research,Covid-19,instagram,hustinx prize

Add Response