Own picture Mark Spigt
Chic conferences, polished presentations and glossy journals. The scientific world seems like one big success story. “Seems”, because reality, of course, is different. In this series, Observant will look for the mistakes, the setbacks, the slip-ups, the unexpected turns. Because those too, or maybe even especially, are science. Today: Mark Spigt, researcher at Family Medicine.
Sunday evening, 7 PM. Mark Spigt, researcher at Family Medicine, has just cycled to his office on Debyeplein. He’s sitting at his computer now. He is tense; the results of his PhD research may show up in his email inbox any second now. They’ll be sent by the man, the partner of a colleague, who has designed the database and entered all the results. In his office, Spigt has access to the software he can use to analyse the data.
He shifts restlessly in his chair. This is the final part of a comprehensive study that took five years. The key question: is drinking water a remedy for urinary problems in older men? These are caused by a weak urine stream, resulting in incomplete emptying of the bladder. Patients have to go to the bathroom frequently and suffer from urinary incontinence. Doctors believe an enlarged prostate is the problem and usually recommend surgery.
Spigt suspected that training the bladder by drinking one and a half litres of water every day would relieve the symptoms. It’s an elegant solution, and definitely a lot cheaper than surgical intervention. He recruited 142 men via general practices, half of whom drank water every day and the other half of whom took a spoonful of placebo syrup daily. Specially trained research assistants and Spigt himself carried out urine measurements in all participants for months on end. It was an enormous task.
Ah, the email with all the data has arrived. Now all Spigt has to do is calculate the average. His heart is pounding as he waits for the final result. His spirits sink: there is no difference between the two groups. Spigt is incredibly disappointed. Is it really true? Does drinking water not make a difference at all?
“Your mind just starts racing right away. You can forget about getting a flying start to your academic career. If the outcome had been positive, I could’ve continued with other target groups, women for example. From there, I might have landed a Veni grant, the perfect start to an academic career. Now, I had reached a dead end.”
The article was published in Urology – a good journal, but no Lancet. “A well-conducted randomised controlled trial with that many participants will always get accepted somewhere, though not necessarily by a top journal.”
At the same time, the experiment had an unexpected effect: quite a few participants reported fewer headaches. “I set up a pilot study about it after my PhD. Ultimately, it was a modest effect.”
This disappointing end to a study happened in 2004, but a similar letdown followed in 2017. Spigt was co-author of a comprehensive study on smoking cessation. Three hundred participants, heavy smokers, took an anti-smoking pill; half of them also received intensive support from a GP assistant, whereas the other half only received general advice from their GP. But the number of people who quit smoking turned out to be the same in both groups. In other words, intensive support had no added effect whatsoever.
“If you read that study and see how many GP assistants we trained, how many participants were involved… I was genuinely surprised that the experimental group showed no advantage. But the outcome was still relevant, because GP assistants were already providing additional support. So our message was clear: they could stop doing it.”
These “negative” results didn’t hurt Spigt’s academic career at all. By now, he is a senior lecturer and has conducted many other experiments, for example on training prisoners to recognise tuberculosis, published in The Lancet Global Health. Or on the efficacy of handwashing by school-aged children in preventing intestinal parasites, published in Plos Medicine. Both studies took place in Ethiopia.
Back to that Sunday evening. Spigt falls back in his chair, dazed by the disappointing result. “We always tell students that disappointing results don’t exist, not from an objective, scientific point of view. But at the same time...”
He’s the only person in the building. The thought crosses his mind for just a second: he could easily change some of the results. No one would know.
It was just a fleeting thought he would never have acted on. In fact, Spigt had taken precautions after discussions about integrity in the department. He and an epidemiologist from another department had made agreements about the interpretation of the results. To be able to speak of a positive result, the force of the urinary stream would’ve had to decrease by a certain percentage. And it hadn’t.
“That’s pretty much standard practice nowadays; researchers record their studies in a register beforehand so that they can’t change the design. You could say I was an early adopter.”
That’s the main thing he learnt from his PhD research: the importance of integrity. “If I’d manipulated the results, researchers at other universities might have built on them. Before you know it, it’s twenty years later and you have a line of research based on a lie that has probably attracted quite a bit of funding. That’s what we call research waste.”
It’s quite astonishing, actually, how easy it is for researchers to manipulate results themselves. Spigt was reminded of it just this week. “Because of an ongoing study in which two students are entering all the data and managing the database. It’s a safe procedure, as neither person can change anything without the other noticing it.”
They could commit research fraud together, though.
“Yes, they could, but you don’t really start a conversation saying, ‘Hey, let’s deceive everyone.’ The danger is really in the seclusion in which researchers can operate.”
Journal of “failed” science
Many things go wrong in science, as they do everywhere. But why are the failures, setbacks or dead ends in research rarely exposed? Is it because of the tremendous amount of pressure on researchers to be successful? Is that why failure is a taboo in science?
“We have unrealistically high expectations of researchers”, says recently graduated historian of science Martijn van der Meer. “If failure was a little more accepted in the scientific world, the work environment would immediately be a lot healthier and more pleasant. Sometimes failure is necessary to achieve something beautiful.”
Van der Meer is one of the master’s students from Utrecht University who founded the Journal of Trial and Error (JOTE). This open-access journal embraces negative, non-significant results rather than shying away from them.
The point of the journal, which first appeared in November, isn’t to glorify “sloppy science”, says Van der Meer. Papers with incorrect statistics, improper data collection or sloppy writing are rejected. All articles go through a rigorous peer-review process and first appear online in preprint.
The journal isn’t receiving a lot of submissions yet. “People have plenty of articles in their desk drawers”, suspects Van der Meer, “but they have to be brave enough to submit them. Some might be worried that being published in a journal of “failed” science wouldn’t look good on their CV.” And that’s exactly the problem JOTE aims to address. (HOP)