1
Missing Data
Sponsors get the answer they want
Before we get going, we need to establish one thing beyond any doubt: industry-funded trials are more likely to produce a positive, flattering result than independently-funded trials. This is our core premise, and you're about to read a very short chapter, because this is one of the most well-documented phenomena in the growing field of ‘research about research'. It has also become much easier to study in recent years, because the rules on declaring industry funding have become a little clearer.
We can begin with some recent work: in 2010, three researchers from Harvard and Toronto found all the trials looking at five major cla**es of drug – antidepressants, ulcer d** and so on – then measured two key features: were they positive, and were they funded by industry? 1 They found over five hundred trials in total: 85 per cent of the industry-funded studies were positive, but only 50 per cent of the government-funded trials were. That's a very significant difference.
In 2007, researchers looked at every published trial that set out to explore the benefit of a statin. 2 These are cholesterol-lowering d** which reduce your risk of having a heart attack, they are prescribed in very large quantities, and they will loom large in this book. This study found 192 trials in total, either comparing one statin against another, or comparing a statin against a different kind of treatment. Once the researchers controlled for other factors (we'll delve into what this means later), they found that industry-funded trials were twenty times more likely to give results favouring the test drug. Again, that's a very big difference.
We'll do one more. In 2006, researchers looked into every trial of psychiatric d** in four academic journals over a ten-year period, finding 542 trial outcomes in total. Industry sponsors got favourable outcomes for their own drug 78 per cent of the time, while independently-funded trials only gave a positive result in 48 per cent of cases. If you were a competing drug put up against the sponsor's drug in a trial, you were in for a pretty rough ride: you would only win a measly 28 per cent of the time. 3
These are dismal, frightening results, but they come from individual studies. When there has been lots of research in a field, it's always possible that someone – like me, for example – could cherry-pick the results, and give a partial view. I could, in essence, be doing exactly what I accuse the pharmaceutical industry of doing, and only telling you about the studies that support my case, while hiding the rea**uring ones from you.
To guard against this risk, researchers invented the systematic review. We'll explore this in more detail soon (p.14), since it's at the core of modern medicine, but in essence a systematic review is simple: instead of just mooching through the research literature, consciously or unconsciously picking out papers here and there that support your pre-existing beliefs, you take a scientific, systematic approach to the very process of looking for scientific evidence, ensuring that your evidence is as complete and representative as possible of all the research that has ever been done.
Systematic reviews are very, very onerous. In 2003, by coincidence, two were published, both looking specifically at the question we're interested in. They took all the studies ever published that looked at whether industry funding is a**ociated with pro-industry results. Each took a slightly different approach to finding research papers, and both found that industry-funded trials were, overall, about four times more likely to report positive results. 4 A further review in 2007 looked at the new studies that had been published in the four years after these two earlier reviews: it found twenty more pieces of work, and all but two showed that industry-sponsored trials were more likely to report flattering results. 5
I am setting out this evidence at length because I want to be absolutely clear that there is no doubt on the issue. Industry-sponsored trials give favourable results, and that is not my opinion, or a hunch from the occasional pa**ing study. This is a very well-documented problem, and it has been researched extensively, without anybody stepping out to take effective action, as we shall see.
There is one last study I'd like to tell you about. It turns out that this pattern of industry-funded trials being vastly more likely to give positive results persists even when you move away from published academic papers, and look instead at trial reports from academic conferences, where data often appears for the first time (in fact, as we shall see, sometimes trial results only appear at an academic conference, with very little information on how the study was conducted).
Fries and Krishnan studied all the research abstracts presented at the 2001 American College of Rheumatology meetings which reported any kind of trial, and acknowledged industry sponsorship, in order to find out what proportion had results that favoured the sponsor's drug. There is a small punch-line coming, and to understand it we need to cover a little of what an academic paper looks like. In general, the results section is extensive: the raw numbers are given for each outcome, and for each possible causal factor, but not just as raw figures. The ‘ranges' are given, subgroups are perhaps explored, statistical tests are conducted, and each detail of the result is described in table form, and in shorter narrative form in the text, explaining the most important results. This lengthy process is usually spread over several pages.
In Fries and Krishnan [2004] this level of detail was unnecessary. The results section is a single, simple, and – I like to imagine – fairly pa**ive-aggressive sentence:
The results from every RCT (45 out of 45) favored the drug of the sponsor.
This extreme finding has a very interesting side effect, for those interested in time-saving shortcuts. Since every industry-sponsored trial had a positive result, that's all you'd need to know about a piece of work to predict its outcome: if it was funded by industry, you could know with absolute certainty that the trial found the drug was great.
How does this happen? How do industry-sponsored trials almost always manage to get a positive result? It is, as far as anyone can be certain, a combination of factors. It may be that companies are more likely to run trials when they're more confident their treatment is going to ‘win'; this sounds reasonable, although even this conflicts with the ethical principle that you should only do a trial when there's genuine uncertainty about which treatment is best (otherwise you're exposing half of your participants to a treatment you already know to be inferior). Sometimes the chances of one treatment winning can be increased with outright design flaws. You can compare your new drug with something you know to be rubbish – an existing drug at an inadequate dose, perhaps, or a placebo sugar pill that does almost nothing. You can choose your patients very carefully, so they are more likely to get better on your treatment. You can peek at the results halfway through, and stop your trial early if they look good (which is – for interesting reasons we shall discuss – statistical poison). And so on.
But before we get to these fascinating methodological twists and quirks, these nudges and bumps that stop a trial from being a fair test of whether a treatment works or not, there is something very much simpler at hand.
Sometimes drug companies conduct lots of trials, and when they see that the results are unflattering, they simply fail to publish them. This is not a new problem, and it's not limited to medicine. In fact, this issue of negative results that go missing in action cuts into almost every corner of science. It distorts findings in fields as diverse as brain imaging and economics, it makes a mockery of all our efforts to exclude bias from our studies, and despite everything that regulators, drug companies and even some academics will tell you, it is a problem that has been left unfixed for decades.
In fact, it is so deep-rooted that even if we fixed it today – right now, for good, forever, without any flaws or loopholes in our legislation – that still wouldn't help, because we would still be practising medicine, cheerfully making decisions about which treatment is best, on the basis of decades of medical evidence which is – as you've now seen – fundamentally distorted.
But there is a way ahead.
Why missing data matters
Reboxetine is a drug I myself have prescribed. Other d** had done nothing for this particular patient, so we wanted to try something new. I'd read the trial data before I wrote the prescription, and found only well-designed, fair tests, with overwhelmingly positive results. Reboxetine was better than placebo, and as good as any other antidepressant in head-to-head comparisons. It's approved for use by the Medicines and Healthcare products Regulatory Agency (the MHRA), which governs all d** in the UK. Millions of doses are prescribed every year, around the world. Reboxetine was clearly a safe and effective treatment. The patient and I discussed the evidence briefly, and agreed it was the right treatment to try next. I signed a piece of paper, a prescription, saying I wanted my patient to have this drug.
But we had both been misled. In October 2010 a group of researchers were finally able to bring together all the trials that had ever been conducted on reboxetine. 6 Through a long process of investigation – searching in academic journals, but also arduously requesting data from the manufacturers and gathering documents from regulators – they were able to a**emble all the data, both from trials that were published, and from those that had never appeared in academic papers.
When all this trial data was put together it produced a shocking picture. Seven trials had been conducted comparing reboxetine against placebo. Only one, conducted in 254 patients, had a neat, positive result, and that one was published in an academic journal, for doctors and researchers to read. But six more trials were conducted, in almost ten times as many patients. All of them showed that reboxetine was no better than a dummy sugar pill. None of these trials was published. I had no idea they existed.
It got worse. The trials comparing reboxetine against other d** showed exactly the same picture: three small studies, 507 patients in total, showed that reboxetine was just as good as any other drug. They were all published. But 1,657 patients' worth of data was left unpublished, and this unpublished data showed that patients on reboxetine did worse than those on other d**. If all this wasn't bad enough, there was also the side-effects data. The drug looked fine in the trials which appeared in the academic literature: but when we saw the unpublished studies, it turned out that patients were more likely to have side effects, more likely to drop out of taking the drug, and more likely to withdraw from the trial because of side effects, if they were taking reboxetine rather than one of its competitors.
If you're ever in any doubt about whether the stories in this book make me angry – and I promise you, whatever happens, I will keep to the data, and strive to give a fair picture of everything we know – you need only look at this story. I did everything a doctor is supposed to do. I read all the papers, I critically appraised them, I understood them, I discussed them with the patient, and we made a decision together, based on the evidence. In the published data, reboxetine was a safe and effective drug. In reality, it was no better than a sugar pill, and worse, it does more harm than good. As a doctor I did something which, on the balance of all the evidence, harmed my patient, simply because unflattering data was left unpublished.
If you find that amazing, or outrageous, your journey is just beginning. Because nobody broke any law in that situation, reboxetine is still on the market, and the system that allowed all this to happen is still in play, for all d**, in all countries in the world. Negative data goes missing, for all treatments, in all areas of science. The regulators and professional bodies we would reasonably expect to stamp out such practices have failed us.
In a few pages, we will walk through the literature that demonstrates all of this beyond any doubt, showing that ‘publication bias' – the process whereby negative results go unpublished – is endemic throughout the whole of medicine and academia; and that regulators have failed to do anything about it, despite decades of data showing the size of the problem. But before we get to that research, I need you to feel its implications, so we need to think about why missing data matters.
Evidence is the only way we can possibly know if something works – or doesn't work – in medicine. We proceed by testing things, as cautiously as we can, in head-to-head trials, and gathering together all of the evidence. This last step is crucial: if I withhold half the data from you, it's very easy for me to convince you of something that isn't true. If I toss a coin a hundred times, for example, but only tell you about the results when it lands heads-up, I can convince you that this is a two-headed coin. But that doesn't mean I really do have a two-headed coin: it means I'm misleading you, and you're a fool for letting me get away with it. This is exactly the situation we tolerate in medicine, and always have. Researchers are free to do as many trials as they wish, and then choose which ones to publish.
The repercussions of this go way beyond simply misleading doctors about the benefits and harms of interventions for patients, and way beyond trials. Medical research isn't an abstract academic pursuit: it's about people, so every time we fail to publish a piece of research we expose real, living people to unnecessary, avoidable suffering.
TGN1412
In March 2006, six volunteers arrived at a London hospital to take place in a trial. It was the first time a new drug called TGN1412 had ever been given to humans, and they were paid £2,000 each. 7 Within an hour these six men developed headaches, muscle aches, and a feeling of unease. Then things got worse: high temperatures, restlessness, periods of forgetting who and where they were. Soon they were shivering, flushed, their pulses racing, their blood pressure falling. Then, a cliff: one went into respiratory failure, the oxygen levels in his blood falling rapidly as his lungs filled with fluid. Nobody knew why. Another dropped his blood pressure to just 65/40, stopped breathing properly, and was rushed to an intensive care unit, knocked out, intubated, mechanically ventilated. Within a day all six were disastrously unwell: fluid on their lungs, struggling to breathe, their kidneys failing, their blood clotting uncontrollably throughout their bodies, and their white blood cells disappearing. Doctors threw everything they could at them: steroids, antihistamines, immune-system receptor blockers. All six were ventilated on intensive care. They stopped producing urine; they were all put on dialysis; their blood was replaced, first slowly, then rapidly; they needed plasma, red cells, platelets. The fevers continued. One developed pneumonia. And then the blood stopped getting to their peripheries. Their fingers and toes went flushed, then brown, then black, and then began to rot and die. With heroic effort, all escaped, at least, with their lives.
The Department of Health convened an Expert Scientific Group to try to understand what had happened, and from this two concerns were raised. 8 Firstly: can we stop things like this from happening again? It's plainly foolish, for example, to give a new experimental treatment to all six participants in a ‘first-in-man' trial at the same time, if that treatment is a completely unknown quantity. New d** should be given to participants in a staggered process, slowly, over a day. This idea received considerable attention from regulators and the media.
Less noted was a second concern: could we have foreseen this disaster? TGN1412 is a molecule that attaches to a receptor called CD28 on the white blood cells of the immune system. It was a new and experimental treatment, and it interfered with the immune system in ways that are poorly understood, and hard to model in animals (unlike, say, blood pressure, because immune systems are very variable between different species). But as the final report found, there was experience with a similar intervention: it had simply not been published. One researcher presented the inquiry with unpublished data on a study he had conducted in a single human subject a full ten years earlier, using an antibody that attached to the CD3, CD2 and CD28 receptors. The effects of this antibody had parallels with those of TGN1412, and the subject on whom it was tested had become unwell. But nobody could possibly have known that, because these results were never shared with the scientific community. They sat unpublished, unknown, when they could have helped save six men from a terrifying, destructive, avoidable ordeal.
That original researcher could not foresee the specific harm he contributed to, and it's hard to blame him as an individual, because he operated in an academic culture where leaving data unpublished was regarded as completely normal. The same culture exists today. The final report on TGN1412 concluded that sharing the results of all first-in-man studies was essential: they should be published, every last one, as a matter of routine. But phase 1 trial results weren't published then, and they're still not published now. In 2009, for the first time, a study was published looking specifically at how many of these first-in-man trials get published, and how many remain hidden. 9 They took all such trials approved by one ethics committee over a year. After four years, nine out of ten remain unpublished; after eight years, four out of five were still unpublished.
In medicine, as we shall see time and again, research is not abstract: it relates directly to life, d**h, suffering and pain. With every one of these unpublished studies we are potentially exposed, quite unnecessarily, to another TGN1412. Even a huge international news story, with horrific images of young men brandishing blackened feet and hands from hospital beds, wasn't enough to get movement, because the issue of missing data is too complicated to fit in one sentence.
When we don't share the results of basic research, such as a small first-in-man study, we expose people to unnecessary risks in the future. Was this an extreme case? Is the problem limited to early, experimental, new d**, in small groups of trial participants? No.
In the 1980s, doctors began giving anti-arrhythmic d** to all patients who'd had a heart attack. This practice made perfect sense on paper: we knew that anti-arrhythmic d** helped prevent abnormal heart rhythms; we also knew that people who've had a heart attack are quite likely to have abnormal heart rhythms; we also knew that often these went unnoticed, undiagnosed and untreated. Giving anti-arrhythmic d** to everyone who'd had a heart attack was a simple, sensible, preventive measure.
Unfortunately, it turned out that we were wrong. This prescribing practice, with the best of intentions, on the best of principles, actually k**ed people. And because heart attacks are very common, it k**ed them in very large numbers: well over 100,000 people died unnecessarily before it was realised that the fine balance between benefit and risk was completely different for patients without a proven abnormal heart rhythm.
Could anyone have predicted this? Sadly, yes, they could have. A trial in 1980 tested a new anti-arrhythmic drug, lorcainide, in a small number of men who'd had a heart attack – less than a hundred – to see if it was any use. Nine out of forty-eight men on lorcainide died, compared with one out of forty-seven on placebo. The drug was early in its development cycle, and not long after this study it was dropped for commercial reasons. Because it wasn't on the market, nobody even thought to publish the trial. The researchers a**umed it was an idiosyncrasy of their molecule, and gave it no further thought. If they had published, we would have been much more cautious about trying other anti-arrhythmic d** on people with heart attacks, and the phenomenal d**h toll – over 100,000 people in their graves prematurely – might have been stopped sooner. More than a decade later, the researchers finally did publish their results, with a mea culpa, recognising the harm they had done by not sharing them earlier:
When we carried out our study in 1980, we thought that the increased d**h rate that occurred in the lorcainide group was an effect of chance. The development of lorcainide was abandoned for commercial reasons, and this study was therefore never published; it is now a good example of ‘publication bias'. The results described here might have provided an early warning of trouble ahead. 10
As we shall shortly see, this problem of unpublished data is widespread throughout medicine, and indeed the whole of academia, even though the scale of the problem, and the harm it causes, have been documented beyond any doubt. We will see stories on basic cancer research, Tamiflu, cholesterol blockbusters, obesity d**, antidepressants and more, with evidence that goes from the dawn of medicine to the present day, and data that is still being withheld, right now, as I write, on widely used d** which many of you reading this book will have taken this morning. We will also see how regulators and academic bodies have repeatedly failed to address the problem.
Because researchers are free to bury any result they please, patients are exposed to harm on a staggering scale throughout the whole of medicine, from research to practice. Doctors can have no idea about the true effects of the treatments they give. Does this drug really work best, or have I simply been deprived of half the data? Nobody can tell. Is this expensive drug worth the money, or have the data simply been ma**aged? No one can tell. Will this drug k** patients? Is there any evidence that it's dangerous? No one can tell.
This is a bizarre situation to arise in medicine, a discipline where everything is supposed to be based on evidence, and where everyday practice is bound up in medico-legal anxiety. In one of the most regulated corners of human conduct we've taken our eyes off the ball, and allowed the evidence driving practice to be polluted and distorted. It seems unimaginable. We will now see how deep this problem goes.
Why we summarise data
Missing data has been studied extensively in medicine. But before I lay out that evidence, we need to understand exactly why it matters, from a scientific perspective. And for that we need to understand systematic reviews and ‘meta-an*lysis'. Between them, these are two of the most powerful ideas in modern medicine. They are incredibly simple, but they were invented shockingly late.
When we want to find out if something works or not, we do a trial. This is a very simple process, and the first recorded attempt at some kind of trial was in the Bible (Daniel 1:12, if you're interested). Firstly, you need an unanswered question: for example, ‘Does giving steroids to a woman delivering a premature baby increase the chances of that baby surviving?' Then you find some relevant participants, in this case, mothers about to deliver a premature baby. You'll need a reasonable number of them, let's say two hundred for this trial. Then you divide them into two groups at random, give the mothers in one group the current best treatment (whatever that is in your town), while the mothers in the other group get current best treatment plus some steroids. Finally, when all two hundred women have gone through your trial, you count up how many babies survived in each group.
This is a real-world question, and lots of trials were done on this topic, starting from 1972 onwards: two trials showed that steroids saved lives, but five showed no significant benefit. Now, you will often hear that doctors disagree when the evidence is mixed, and this is exactly that kind of situation. A doctor with a strong pre-existing belief that steroids work – perhaps preoccupied with some theoretical molecular mechanism, by which the drug might do something useful in the body – could come along and say: ‘Look at these two positive trials! Of course we must give steroids!' A doctor with a strong prior intuition that steroids were rubbish might point at the five negative trials and say: ‘Overall the evidence shows no benefit. Why take a risk?'
Up until very recently, this was basically how medicine progressed. People would write long, languorous review articles – essays surveying the literature – in which they would cite the trial data they'd come across in a completely unsystematic fashion, often reflecting their own prejudices and values. Then, in the 1980s, people began to do something called a ‘systematic review'. This is a clear, systematic survey of the literature, with the intention of getting all the trial data you can possibly find on one topic, without being biased towards any particular set of findings. In a systematic review, you describe exactly how you looked for data: which databases you searched, which search engines and indexes you used, even what words you searched for. You pre-specify the kinds of studies that can be included in your review, and then you present everything you've found, including the papers you rejected, with an explanation of why. By doing this, you ensure that your methods are fully transparent, replicable and open to criticism, providing the reader with a clear and complete picture of the evidence. It may sound like a simple idea, but systematic reviews are extremely rare outside clinical medicine, and are quietly one of the most important and transgressive ideas of the past forty years.
When you've got all the trial data in one place, you can conduct something called a meta-an*lysis, where you bring all the results together in one giant spreadsheet, pool all the data and get one single, summary figure, the most accurate summary of all the data on one clinical question. The output of this is called a ‘blobbogram', and you can see one on the opposite page, in the logo of the Cochrane Collaboration, a global, non-profit academic organisation that has been producing gold-standard reviews of evidence on important questions in medicine since the 1980s.
This blobbogram shows the results of all the trials done on giving steroids to help premature babies survive. Each horizontal line is a trial: if that line is further to the left, then the trial showed steroids were beneficial and saved lives. The central, vertical line is the ‘line of no effect': and if the horizontal line of the trial touches the line of no effect, then that trial showed no statistically significant benefit. Some trials are represented by longer horizontal lines: these were smaller trials, with fewer participants, which means they are prone to more error, so the estimate of the benefit has more uncertainty, and therefore the horizontal line is longer. Finally, the diamond at the bottom shows the ‘summary effect': this is the overall benefit of the intervention, pooling together the results of all the individual trials. These are much narrower than the lines for individual trials, because the estimate is much more accurate: it is summarising the effect of the drug in many more patients. On this blobbogram you can see – because the diamond is a long way from the line of no effect – that giving steroids is hugely beneficial. In fact, it reduces the chances of a premature baby dying by almost half.
The amazing thing about this blobbogram is that it had to be invented, and this happened very late in medicine's history. For many years we had all the information we needed to know that steroids saved lives, but nobody knew they were effective, because nobody did a systematic review until 1989. As a result, the treatment wasn't given widely, and huge numbers of babies died unnecessarily; not because we didn't have the information, but simply because we didn't synthesise it together properly.
In case you think this is an isolated case, it's worth examining exactly how broken medicine was until frighteningly recent times. The diagram on the opposite page contains two blobbograms, or ‘forest plots', showing all the trials ever conducted to see whether giving streptokinase, a clot-busting drug, improves survival in patients who have had a heart attack. 11
Look first only at the forest plot on the left. This is a conventional forest plot, from an academic journal, so it's a little busier than the stylised one in the Cochrane logo. The principles, however, are exactly the same. Each horizontal line is a trial, and you can see that there is a hodgepodge of results, with some trials showing a benefit (they don't touch the vertical line of no effect, headed ‘1') and some showing no benefit (they do cross that line). At the bottom, however, you can see the summary effect – a dot on this old-fashioned blobbogram, rather than a diamond. And you can see very clearly that overall, streptokinase saves lives.
So what's that on the right? It's something called a cumulative meta-an*lysis. If you look at the list of studies on the left of the diagram, you can see that they are arranged in order of date. The cumulative meta-an*lysis on the right adds in each new trial's results, as they arrived over history, to the previous trials' results. This gives the best possible running estimate, each year, of how the evidence would have looked at that time, if anyone had bothered to do a meta-an*lysis on all the data available to them. From this cumulative blobbogram you can see that the horizontal lines, the ‘summary effects', narrow over time as more and more data is collected, and the estimate of the overall benefit of this treatment becomes more accurate. You can also see that these horizontal lines stopped touching the vertical line of no effect a very long time ago – and crucially, they do so a long time before we started giving streptokinase to everyone with a heart attack.
In case you haven't spotted it for yourself already – to be fair, the entire medical profession was slow to catch on – this chart has devastating implications. Heart attacks are an incredibly common cause of d**h. We had a treatment that worked, and we had all the information we needed to know that it worked, but once again we didn't bring it together systematically to get that correct answer. Half of the people in those trials at the bottom of the blobbogram were randomly a**igned to receive no streptokinase, I think unethically, because we had all the information we needed to know that streptokinase worked: they were deprived of effective treatments. But they weren't alone, because so were most of the rest of the people in the world at the time.
These stories illustrate, I hope, why systematic reviews and meta-an*lyses are so important: we need to bring together all of the evidence on a question, not just cherry-pick the bits that we stumble upon, or intuitively like the look of. Mercifully the medical profession has come to recognise this over the past couple of decades, and systematic reviews with meta-an*lyses are now used almost universally, to ensure that we have the most accurate possible summary of all the trials that have been done on a particular medical question.
But these stories also demonstrate why missing trial results are so dangerous. If one researcher or doctor ‘cherry-picks', when summarising the existing evidence, and looks only at the trials that support their hunch, then they can produce a misleading picture of the research. That is a problem for that one individual (and for anyone who is unwise or unlucky enough to be influenced by them). But if we are all missing the negative trials, the entire medical and academic community, around the world, then when we pool the evidence to get the best possible view of what works – as we must do – we are all completely misled. We get a misleading impression of the treatment's effectiveness: we incorrectly exaggerate its benefits; or perhaps even find incorrectly that an intervention was beneficial, when in reality it did harm.
Now that you understand the importance of systematic reviews, you can see why missing data matters. But you can also appreciate that when I explain how much trial data is missing, I am giving you a clean overview of the literature, because I will be explaining that evidence using systematic reviews.
How much data is missing?
If you want to prove that trials have been left unpublished, you have an interesting problem: you need to prove the existence of studies you don't have access to. To work around this, people have developed a simple approach: you identify a group of trials you know have been conducted and completed, then check to see if they have been published. Finding a list of completed trials is the tricky part of this job, and to achieve it people have used various strategies: trawling the lists of trials that have been approved by ethics committees (or ‘institutional review boards' in the USA), for example; or chasing up the trials discussed by researchers at conferences.
In 2008 a group of researchers decided to check for publication of every trial that had ever been reported to the US Food and Drug Administration for all the antidepressants that came onto the market between 1987 and 2004. 12 This was no small task. The FDA archives contain a reasonable amount of information on all the trials that were submitted to the regulator in order to get a licence for a new drug. But that's not all the trials, by any means, because those conducted after the drug has come onto the market will not appear there; and the information that is provided by the FDA is hard to search, and often scanty. But it is an important subset of the trials, and more than enough for us to begin exploring how often trials go missing, and why. It's also a representative slice of trials from all the major drug companies.
The researchers found seventy-four studies in total, representing 12,500 patients' worth of data. Thirty-eight of these trials had positive results, and found that the new drug worked; thirty-six were negative. The results were therefore an even split between success and failure for the d**, in reality. Then the researchers set about looking for these trials in the published academic literature, the material available to doctors and patients. This provided a very different picture. Thirty-seven of the positive trials – all but one – were published in full, often with much fanfare. But the trials with negative results had a very different fate: only three were published. Twenty-two were simply lost to history, never appearing anywhere other than in those dusty, disorganised, thin FDA files. The remaining eleven which had negative results in the FDA summaries did appear in the academic literature, but were written up as if the drug was a success. If you think this sounds absurd, I agree: we will see in Chapter 4, on ‘bad trials', how a study's results can be reworked and polished to distort and exaggerate its findings.
This was a remarkable piece of work, spread over twelve d** from all the major manufacturers, with no stand-out bad guy. It very clearly exposed a broken system: in reality we have thirty-eight positive trials and thirty-six negative ones; in the academic literature we have forty-eight positive trials and three negative ones. Take a moment to flip back and forth between those in your mind: ‘thirty-eight positive trials, thirty-six negative'; or ‘forty-eight positive trials and only three negative'.
If we were talking about one single study, from one single group of researchers, who decided to delete half their results because they didn't give the overall picture they wanted, then we would quite correctly call that act ‘research misconduct'. Yet somehow when exactly the same phenomenon occurs, but with whole studies going missing, by the hands of hundreds and thousands of individuals, spread around the world, in both the public and private sector, we accept it as a normal part of life. 13 It pa**es by, under the watchful eyes of regulators and professional bodies who do nothing, as routine, despite the undeniable impact it has on patients.
Even more strange is this: we've known about the problem of negative studies going missing for almost as long as people have been doing serious science.
This was first formally documented by a psychologist called Theodore Sterling in 1959. 14 He went through every paper published in the four big psychology journals of the time, and found that 286 out of 294 reported a statistically significant result. This, he explained, was plainly fishy: it couldn't possibly be a fair representation of every study that had been conducted, because if we believed that, we'd have to believe that almost every theory ever tested by a psychologist in an experiment had turned out to be correct. If psychologists really were so great at predicting results, there'd hardly be any point in bothering to run experiments at all. In 1995, at the end of his career, the same researcher came back to the same question, half a lifetime later, and found that almost nothing had changed. 15
Sterling was the first to put these ideas into a formal academic context, but the basic truth had been recognised for many centuries. Francis Bacon explained in 1620 that we often mislead ourselves by only remembering the times something worked, and forgetting those when it didn't. 16 Fowler in 1786 listed the cases he'd seen treated with arsenic, and pointed out that he could have glossed over the failures, as others might be tempted to do, but had included them. 17 To do otherwise, he explained, would have been misleading.
Yet it was only three decades ago that people started to realise that missing trials posed a serious problem for medicine. In 1980 Elina Hemminki found that almost half the trials conducted in the mid-1970s in Finland and Sweden had been left unpublished. 18 Then, in 1986, an American researcher called Robert Simes decided to investigate the trials on a new treatment for ovarian cancer. This was an important study, because it looked at a life-or-d**h question. Combination chemotherapy for this kind of cancer has very tough side effects, and knowing this, many researchers had hoped it might be better to give a single ‘alkylating agent' drug first, before moving on to full chemotherapy. Simes looked at all the trials published on this question in the academic literature, read by doctors and academics. From this, giving a single drug first looked like a great idea: women with advanced ovarian cancer (which is not a good diagnosis to have) who were on the alkylating agent alone were significantly more likely to survive longer.
Then Simes had a smart idea. He knew that sometimes trials can go unpublished, and he had heard that papers with less ‘exciting' results are the most likely to go missing. To prove that this has happened, though, is a tricky business: you need to find a fair, representative sample of all the trials that have been conducted, and then compare their results with the smaller pool of trials that have been published, to see if there are any embarra**ing differences. There was no easy way to get this information from the medicines regulator (we will discuss this problem in some detail later), so instead he went to the International Cancer Research Data Bank. This contained a register of interesting trials that were happening in the USA, including most of the ones funded by the government, and many others from around the world. It was by no means a complete list, but it did have one crucial feature: the trials were registered before their results came in, so any list compiled from this source would be, if not complete, at least a representative sample of all the research that had ever been done, and not biased by whether their results were positive or negative.
When Simes compared the results of the published trials against the pre-registered trials, the results were disturbing. Looking at the academic literature – the studies that researchers and journal editors chose to publish – alkylating agents alone looked like a great idea, reducing the rate of d**h from advanced ovarian cancer significantly. But when you looked only at the pre-registered trials – the unbiased, fair sample of all the trials ever conducted – the new treatment was no better than old-fashioned chemotherapy.
Simes immediately recognised – as I hope you will too – that the question of whether one form of cancer treatment is better than another was small fry compared to the depth charge he was about to set off in the medical literature. Everything we thought we knew about whether treatments worked or not was probably distorted, to an extent that might be hard to measure, but that would certainly have a major impact on patient care. We were seeing the positive results, and missing the negative ones. There was one clear thing we should do about this: start a registry of all clinical trials, demand that people register their study before they start, and insist that they publish the results at the end.
That was 1986. Since then, a generation later, we have done very badly. In this book, I promise I won't overwhelm you with data. But at the same time, I don't want any drug company, or government regulator, or professional body, or anyone who doubts this whole story, to have any room to wriggle. So I'll now go through all the evidence on missing trials, as briefly as possible, showing the main approaches that have been used. All of what you are about to read comes from the most current systematic reviews on the subject, so you can be sure that it is a fair and unbiased summary of the results.
One research approach is to get all the trials that a medicines regulator has record of, from the very early ones done for the purposes of getting a licence for a new drug, and then check to see if they all appear in the academic literature. That's the method we saw used in the paper mentioned above, where researchers sought out every paper on twelve antidepressants, and found that a 50/50 split of positive and negative results turned into forty-eight positive papers and just three negative ones. This method has been used extensively in several different areas of medicine:
*Lee and colleagues, for example, looked for all of the 909 trials submitted alongside marketing applications for all ninety new d** that came onto the market from 2001 to 2002: they found that 66 per cent of the trials with significant results were published, compared with only 36 per cent of the rest. 19
*Melander, in 2003, looked for all forty-two trials on five antidepressants that were submitted to the Swedish drug regulator in the process of getting a marketing authorisation: all twenty-one studies with significant results were published; only 81 per cent of those finding no benefit were published. 20
*Rising et al., in 2008, found more of those distorted write-ups that we'll be dissecting later: they looked for all trials on two years' worth of approved d**. In the FDA's summary of the results, once those could be found, there were 164 trials. Those with favourable outcomes were a full four times more likely to be published in academic papers than those with negative outcomes. On top of that, four of the trials with negative outcomes changed, once they appeared in the academic literature, to favour the drug. 21
If you prefer, you can look at conference presentations: a huge amount of research gets presented at conferences, but our current best estimate is that only about half of it ever appears in the academic literature. 22 Studies presented only at conferences are almost impossible to find, or cite, and are especially hard to a**ess, because so little information is available on the specific methods used in the research (often as little as a paragraph). And as you will see shortly, not every trial is a fair test of a treatment. Some can be biased by design, so these details matter.
The most recent systematic review of studies looking at what happens to conference papers was done in 2010, and it found thirty separate studies looking at whether negative conference presentations – in fields as diverse as anaesthetics, cystic fibrosis, oncology, and A&E – disappear before becoming fully-fledged academic papers. 23 Overwhelmingly, unflattering results are much more likely to go missing.
If you're very lucky, you can track down a list of trials whose existence was publicly recorded before they were started, perhaps on a register that was set up to explore that very question. From the pharmaceutical industry, up until very recently, you'd be very lucky to find such a list in the public domain. For publicly-funded research the story is a little different, and here we start to learn a new lesson: although the vast majority of trials are conducted by the industry, with the result that they set the tone for the community, this phenomenon is not limited to the commercial sector.
*By 1997 there were already four studies in a systematic review on this approach. They found that studies with significant results were two and a half times more likely to get published than those without. 24
*A paper from 1998 looked at all trials from two groups of triallists sponsored by the US National Institutes of Health over the preceding ten years, and found, again, that studies with significant results were more likely to be published. 25
*Another looked at drug trials notified to the Finnish National Agency, and found that 47 per cent of the positive results were published, but only 11 per cent of the negative ones. 26
*Another looked at all the trials that had pa**ed through the pharmacy department of an eye hospital since 1963: 93 per cent of the significant results were published, but only 70 per cent of the negative ones. 27
The point being made in this blizzard of data is simple: this is not an under-researched area; the evidence has been with us for a long time, and it is neither contradictory nor ambiguous.
Two French studies in 2005 and 2006 took a new approach: they went to ethics committees, and got lists of all the studies they had approved, and then found out from the investigators whether the trials had produced positive or negative results, before finally tracking down the published academic papers. 28 The first study found that significant results were twice as likely to be published; the second that they were four times as likely. In Britain, two researchers sent a questionnaire to all the lead investigators on 101 projects paid for by NHS R&D: it's not industry research, but it's worth noting anyway. This produced an unusual result: there was no statistically significant difference in the publication rates of positive and negative papers. 29
But it's not enough simply to list studies. Systematically taking all the evidence that we have so far, what do we see overall?
It's not ideal to lump every study of this type together in one giant spreadsheet, to produce a summary figure on publication bias, because they are all very different, in different fields, with different methods. This is a concern in many meta-an*lyses (though it shouldn't be overstated: if there are lots of trials comparing one treatment against placebo, say, and they're all using the same outcome measurement, then you might be fine just lumping them all in together).
But you can reasonably put some of these studies together in groups. The most current systematic review on publication bias, from 2010, from which the examples above are taken, draws together the evidence from various fields. 30 Twelve comparable studies follow up conference presentations, and taken together they find that a study with a significant finding is 1.62 times more likely to be published. For the four studies taking lists of trials from before they started, overall, significant results were 2.4 times more likely to be published. Those are our best estimates of the scale of the problem. They are current, and they are damning.
All of this missing data is not simply an abstract academic matter: in the real world of medicine, published evidence is used to make treatment decisions. This problem goes to the core of everything that doctors do, so it's worth considering in some detail what impact it has on medical practice. Firstly, as we saw in the case of reboxetine, doctors and patients are misled about the effects of the medicines they use, and can end up making decisions that cause avoidable suffering, or even d**h. We might also choose unnecessarily expensive treatments, having been misled into thinking they are more effective than cheaper older d**. This wastes money, ultimately depriving patients of other treatments, since funding for health care is never infinite.
It's also worth being clear that this data is withheld from everyone in medicine, from top to bottom. NICE, for example, is the National Institute for Health and Clinical Excellence, created by the British government to conduct careful, unbiased summaries of all the evidence on new treatments. It is unable either to identify or to access data that has been withheld by researchers or companies on a drug's effectiveness: NICE has no more legal right to that data than you or I do, even though it is making decisions about effectiveness, and cost-effectiveness, on behalf of the NHS, for millions of people. In fact, as we shall see, the MHRA and EMA (the European Medicines Agency) – the regulators that decide which d** can go on the market in the UK – often have access to this information, but do not share it with the public, with doctors, or with NICE. This is an extraordinary and perverse situation.
So, while doctors are kept in the dark, patients are exposed to inferior treatments, ineffective treatments, unnecessary treatments, and unnecessarily expensive treatments that are no better than cheap ones; governments pay for unnecessarily expensive treatments, and mop up the cost of harms created by inadequate or harmful treatment; and individual participants in trials, such as those in the TGN1412 study, are exposed to terrifying, life-threatening ordeals, resulting in lifelong scars, again quite unnecessarily.
At the same time, the whole of the research project in medicine is retarded, as vital negative results are held back from those who could use them. This affects everyone, but it is especially egregious in the world of ‘orphan diseases', medical problems that affect only small numbers of patients, because these corners of medicine are already short of resources, and are neglected by the research departments of most drug companies, since the opportunities for revenue are thinner. People working on orphan diseases will often research existing d** that have been tried and failed in other conditions, but that have theoretical potential for the orphan disease. If the data from earlier work on these d** in other diseases is missing, then the job of researching them for the orphan disease is both harder and more dangerous: perhaps they have already been shown to have benefits or effects that would help accelerate research; perhaps they have already been shown to be actively harmful when used on other diseases, and there are important safety signals that would help protect future research participants from harm. Nobody can tell you.
Finally, and perhaps most shamefully, when we allow unflattering data to go unpublished, we betray the patients who participated in these studies: the people who have given their bodies, and sometimes their lives, in the implicit belief that they are doing something to create new knowledge, that will benefit others in the same position as them in the future. In fact, their belief is not implicit: often it's exactly what we tell them, as researchers, and it is a lie, because the data might be withheld, and we know it.
So whose fault is this?
Why do negative trials disappear?
In a moment we will see more clear cases of drug companies withholding data – in stories where we can identify individuals – sometimes with the a**istance of regulators. When we get to these, I hope your rage might swell. But first, it's worth taking a moment to recognise that publication bias occurs outside commercial drug development, and in completely unrelated fields of academia, where people are motivated only by reputation, and their own personal interests.
In many respects, after all, publication bias is a very human process. If you've done a study and it didn't have an exciting, positive result, then you might wrongly conclude that your experiment isn't very interesting to other researchers. There's also the issue of incentives: academics are often measured, rather unhelpfully, by crude metrics like the numbers of citations for their papers, and the number of ‘high-impact' studies they get into glamorous well-read journals. If negative findings are harder to publish in bigger journals, and less likely to be cited by other academics, then the incentives to work at disseminating them are lower. With a positive finding, meanwhile, you get a sense of discovering something new. Everyone around you is excited, because your results are exceptional.
One clear illustration of this problem came in 2010. A mainstream American psychology researcher called Daryl Bem published a competent academic paper, in a well-respected journal, showing evidence of precognition, the ability to see into the future. fn1 These studies were well-designed, and the findings were statistically significant, but many people weren't very convinced, for the same reasons you aren't: if humans really could see into the future, we'd probably know about it already; and extraordinary claims require extraordinary evidence, rather than one-off findings.
But in fact the study has been replicated, though Bem's positive results have not been. At least two groups of academics have rerun several of Bem's experiments, using the exact same methods, and both found no evidence of precognition. One group submitted their negative results to the Journal of Personality and Social Psychology, – the very same journal that published Bem's paper in 2010 – and that journal rejected their paper out of hand. The editor even came right out and said it: we never publish studies that replicate other work.
Here we see the same problem as in medicine: positive findings are more likely to be published than negative ones. Every now and then, a freak positive result is published showing, for example, that people can see into the future. Who knows how many psychologists have tried, over the years, to find evidence of psychic powers, running elaborate, time-consuming experiments, on dozens of subjects – maybe hundreds – and then found no evidence that such powers exist? Any scientist trying to publish such a ‘So what?' finding would struggle to get a journal to take it seriously, at the best of times. Even with the clear target of Bem's paper on precognition, which was widely covered in serious newspapers across Europe and the USA, the academic journal with a proven recent interest in the question of precognition simply refused to publish a paper with a negative result. Yet replicating these findings was key – Bem himself said so in his paper – so keeping track of the negative replications is vital too.
People working in real labs will tell you that sometimes an experiment can fail to produce a positive result many times before the outcome you're hoping for appears. What does that mean? Sometimes the failures will be the result of legitimate technical problems; but sometimes they will be vitally important statistical context, perhaps even calling the main finding of the research into question. Many research findings, remember, are not absolute black-and-white outcomes, but fragile statistical correlations. Under our current system, most of this contextual information about failure is just brushed under the carpet, and this has huge ramifications for the cost of replicating research, in ways that are not immediately obvious. For example, researchers failing to replicate an initial finding may not know if they've failed because the original result was an overstated fluke, or because they've made some kind of mistake in their methods. In fact, the cost of proving that a finding was wrong is vastly greater than the cost of making it in the first place, because you need to run the experiment many more times to prove the absence of a finding, simply because of the way that the statistics of detecting weak effects work; and you also need to be absolutely certain that you've excluded all technical problems, to avoid getting egg on your face if your replication turns out to have been inadequate. These barriers to refutation may partly explain why it's so easy to get away with publishing findings that ultimately turn out to be wrong. 31
Publication bias is not just a problem in the more abstract corners of psychology research. In 2012 a group of researchers reported in the journal Nature how they tried to replicate fifty-three early laboratory studies of promising targets for cancer treatments: forty-seven of the fifty-three could not be replicated. 32 This study has serious implications for the development of new d** in medicine, because such unreplicable findings are not simply an abstract academic issue: researchers build theories on the back of them, trust that they're valid, and investigate the same idea using other methods. If they are simply being led down the garden path, chasing up fluke errors, then huge amounts of research money and effort are being wasted, and the discovery of new medical treatments is being seriously retarded.
The authors of the study were clear on both the cause of and the solution for this problem. Fluke findings, they explained, are often more likely to be submitted to journals – and more likely to be published – than boring, negative ones. We should give more incentives to academics for publishing negative results; but we should also give them more opportunity.
This means changing the behaviour of academic journals, and here we are faced with a problem. Although they are usually academics themselves, journal editors have their own interests and agendas, and have more in common with everyday journalists and newspaper editors than some of them might wish to admit, as the episode of the precognition experiment above illustrates very clearly. Whether journals like this are a sensible model for communicating research at all is a hotly debated subject in academia, but this is the current situation. Journals are the gatekeepers, they make decisions on what's relevant and interesting for their audience, and they compete for readers.
This can lead them to behave in ways that don't reflect the best interests of science, because an individual journal's desire to provide colourful content might conflict with the collective need to provide a comprehensive picture of the evidence. In newspaper journalism, there is a well-known aphorism: ‘When a dog bites a man, that's not news; but when a man bites a dog …' These judgements on newsworthiness in mainstream media have even been demonstrated quantitatively. One study in 2003, for example, looked at the BBC's health news coverage over several months, and calculated how many people had to die from a given cause for one story to appear. 8,571 people died from smoking for each story about smoking; but there were three stories for every d**h from new variant CJD, or ‘mad cow disease'. 33 Another, in 1992, looked at print-media coverage of drug d**hs, and found that you needed 265 d**hs from paracetamol poisoning for one story about such a d**h to appear in a paper; but every d**h from MDMA received, on average, one piece of news coverage. 34
If similar judgements are influencing the content of academic journals, then we have a problem. But can it really be the case that academic journals are the bottleneck, preventing doctors and academics from having access to unflattering trial results about the safety and effectiveness of the d** they use? This argument is commonly deployed by industry, and researchers too are often keen to blame journals for rejecting negative findings en ma**e. Luckily, this has been the subject of some research; and overall, while journals aren't blameless, it's hard to claim that they are the main source of this serious public-health problem. This is especially so since there are whole academic journals dedicated to publishing clinical trials, with a commitment to publishing negative results written into their constitutions.
But to be kind, for the sake of completeness, and because industry and researchers are so keen to pa** the blame on to academic journals, we can see if what they claim is true.
One survey simply asked the authors of unpublished work if they had ever submitted it for publication. One hundred and twenty-four unpublished results were identified, by following up on every study approved by a group of US ethics committees, and when the researchers contacted the teams behind the unpublished results, it turned out that only six papers had ever actually been submitted and rejected.35 Perhaps, you might say, this was a freak finding. Another approach is to follow up all the papers submitted to one journal, and see if those with negative results are rejected more often. Where this has been tried, the journals seem blameless: 745 man*scripts submitted to the Journal of the American Medical Association (JAMA) were followed up, and there was no difference in acceptance rate for significant and non-significant findings. 36 The same thing has been tried with papers submitted to the BMJ, the Lancet, Annals of Internal Medicine and the Journal of Bone and Joint Surgery. 37 Again and again, no effect was found. Some have argued that this might still represent evidence of editorial bias, if academics know that man*scripts with negative results have to be of higher quality before submission, to get past editors' prejudices. It's also possible that the journals played fair when they knew they were being watched, although turning around an entire publishing operation for one brief performance would be tough.
These studies all involved observing what has happened in normal practice. One last option is to run an experiment, sending identical papers to various journals, but changing the direction of the results at random, to see if that makes any difference to the acceptance rates. This isn't something you'd want to do very often, because it wastes a lot of people's time; but since publication bias matters, it has been regarded as a justifiable intrusion on a few occasions.
In 1990 a researcher called Epstein created a series of fictitious papers, with identical methods and presentation, differing only in whether they reported positive or negative results. He sent them at random to 146 social-work journals: the positive papers were accepted 35 per cent of the time, and the negative ones 26 per cent of the time, a difference that wasn't large enough to be statistically significant. 38
Other studies have tried something similar on a smaller scale, not submitting a paper to a journal, but rather, with the a**istance of the journal, sending spoof academic papers to individual peer reviewers: these people do not make the final decision on publication, but they do give advice to editors, so a window into their behaviour would be useful. These studies have had more mixed results. In one from 1977, sham papers with identical methods but different results were sent to seventy-five reviewers. Some bias was found from reviewers against findings that disagreed with their own views. 39
Another study, from 1994, looked at reviewers' responses to a paper on TENS machines: these are fairly controversial devices sold for pain relief. Thirty-three reviewers with strong views one way or the other were identified, and again it was found that their judgements on the paper were broadly correlated with their pre-existing views, though the study was small. 40 Another paper did the same thing with papers on quack treatments; it found that the direction of findings had no effect on reviewers from mainstream medical journals deciding whether to accept them. 41
One final randomised trial from 2010 tried on a grand scale to see if reviewers really do reject ideas based on their pre-existing beliefs (a good indicator of whether journals are biased by results, when they should be focused simply on whether a study is properly designed and conducted). Fabricated papers were sent to over two hundred reviewers, and they were all identical, except for the results they reported: half of the reviewers got results they would like, half got results they wouldn't. Reviewers were more likely to recommend publication if they received the version of the man*script with results they'd like (97 per cent vs 80 per cent), more likely to detect errors in a man*script whose results they didn't like, and rated the methods more highly in papers whose results they liked. 42
Overall, though, even if there are clearly rough edges in some domains, these results don't suggest that the journals are the main cause of the problem of the disappearance of negative trials. In the experiments isolating the peer reviewers, those individual referees were biased in some studies, but they don't have the last word on publication, and in all the studies which look at what happens to negative papers submitted to journals in the real world, the evidence suggests that they proceed into print without problems. Journals may not be entirely innocent – The Trouble With Medical Journals by Richard Smith, previously editor of the BMJ, is an excellent overview of their faults; 43 and we will see later how they fail to police important and simple flaws in studies. But it would be wrong to lay the blame for publication bias entirely at their door.
In the light of all this, the data on what researchers say about their own behaviour is very revealing. In various surveys they have said that they thought there was no point in submitting negative results, because they would just be rejected by journals: 20 per cent of medical researchers said so in 1998; 44 61 per cent of psychology and education researchers said so in 1991; 45 and so on. 46 If asked why they've failed to send in research for publication, the most common reasons researchers give are negative results, a lack of interest, or a lack of time.
This is the more abstract end of academia – largely away from the immediate world of clinical trials – but it seems that academics are mistaken, at best, about the reasons why negative results go missing. Journals may pose some barriers to publishing negative results, but they are hardly absolute, and much of the problem lies in academics' motivations and perceptions.
More than that, in recent years, the era of open-access academic journals has got going in earnest: there are now several, such as Trials, which are free to access, and have a core editorial policy that they will accept any trial report, regardless of result, and will actively solicit negative findings. Trials registers like clinicaltrials.gov will also post results. With offers like this on the table, it is very hard to believe that anyone would really struggle to disseminate the findings of a trial with a negative result if they wanted to. And yet, despite this, negative results continue to go missing, with vast multinational companies simply withholding results on their d**, even though academics and doctors are desperate to see them.
You might reasonably wonder whether there are people who are supposed to prevent this kind of data from being withheld. The universities where research takes place, for example; or the regulators; or the ‘ethics committees', which are charged with protecting patients who participate in research. Unfortunately, our story is about to take a turn to the dark side. We will see that many of the very people and organisations we would have expected to protect patients from the harm inflicted by missing data have, instead, shirked their responsibilities; and worse than that, we will see that many of them have actively conspired in helping companies to withhold data from patients. We are about to hit some big problems, some bad people, and some simple solutions.
How ethics committees and universities have failed us
By now, you will, I hope, share my view that withholding results from clinical trials is unethical, for the simple reason that hidden data exposes patients to unnecessary and avoidable harm. But the ethical transgressions here go beyond the simple harm inflicted on future patients.
Patients and the public participate in clinical trials at some considerable cost to themselves: they expose themselves to ha**le and intrusion, because clinical trials almost always require that you have more check-ups on your progress, more blood tests, and more examinations; but participants may also expose themselves to more risk, or the chance of receiving an inferior treatment. People do this out of altruism, on the implicit understanding that the results from their experience will contribute to improving our knowledge of what works and what doesn't, and so will help other patients in the future. In fact, this understanding isn't just implicit: in many trials it's explicit, because patients are specifically told when they sign up to participate that the data will be used to inform future decisions. If this isn't true, and the data can be withheld at the whim of a researcher or a company, then the patients have been actively lied to. That is very bad news.
So what are the formal arrangements between patients, researchers and sponsors? In any sensible world, we'd expect universal contracts, making it clear that all researchers are obliged to publish their results, and that industry sponsors – which have a huge interest in positive results – must have no control over the data. But despite everything we know about industry-funded research being systematically biased, this does not happen. In fact, quite the opposite is true: it is entirely normal for researchers and academics conducting industry-funded trials to sign contracts subjecting them to gagging clauses which forbid them to publish, discuss or an*lyse data from the trials they have conducted, without the permission of the funder. This is such a secretive and shameful situation that even trying to document it in public can be a fraught business, as we shall now see.
In 2006 a paper was published in JAMA describing how common it was for researchers doing industry-funded trials to have these kinds of constraints placed on their right to publish the results. 47 The study was conducted by the Nordic Cochrane Centre, and it looked at all the trials given approval to go ahead in Copenhagen and Frederiksberg. (If you're wondering why these two cities were chosen, it was simply a matter of practicality, and the bizarre secrecy that shrouds this world: the researchers applied elsewhere without success, and were specifically refused access to data in the UK. 48) These trials were overwhelmingly sponsored by the pharmaceutical industry (98 per cent), and the rules governing the management of the results tell a story which walks the now-familiar line between frightening and absurd.
For sixteen of the forty-four trials the sponsoring company got to see the data as it accumulated, and in a further sixteen they had the right to stop the trial at any time, for any reason. This means that a company can see if a trial is going against it, and can interfere as it progresses. As we will see later (early stopping, breaking protocols, pp.186, 202), this distorts a trial's results with unnecessary and hidden biases. For example, if you stop a trial early because you have been peeking at the preliminary results, then you can either exaggerate a modest benefit, or bury a worsening negative result. Crucially, the fact that the sponsoring company had this opportunity to introduce bias wasn't mentioned in any of the published academic papers reporting the results of these trials, so nobody reading the literature could possibly know that these studies were subject – by design – to such an important flaw.
Even if the study was allowed to finish, the data could still be suppressed. There were constraints on publication rights in forty of the forty-four trials, and in half of them the contracts specifically stated that the sponsor either owned the data outright (what about the patients, you might say?), or needed to approve the final publication, or both. None of these restrictions was mentioned in any of the published papers, and in fact, none of the protocols or papers said that the sponsor had full access to all the data from the trial, or the final say on whether to publish.
It's worth taking a moment to think about what this means. The results of all these trials were subject to a bias that will significantly distort the academic literature, because trials that show early signs of producing a negative result (or trials that do produce a negative result) can be deleted from the academic record; but nobody reading these trials could possibly have known that this opportunity for censorship existed.
The paper I've just described was published in JAMA, one of the biggest medical journals in the world. Shortly afterwards, a shocking tale of industry interference appeared in the BMJ. 49 Lif, the Danish pharmaceutical industry a**ociation, responded to the paper by announcing in the Journal of the Danish Medical Association that it was ‘both shaken and enraged about the criticism, that could not be recognised'. It demanded an investigation of the scientists, though it failed to say by whom, or of what. Then Lif wrote to the Danish Committee on Scientific Dishonesty, accusing the Cochrane researchers of scientific misconduct. We can't see the letter, but the Cochrane researchers say the allegations were extremely serious – they were accused of deliberately distorting the data – but vague, and without documents or evidence to back them up.
Nonetheless, the investigation went on for a year, because in academia people like to do things properly, and a**ume that all complaints are made in good faith. Peter Gøtzsche, the director of the Cochrane centre, told the BMJ that only Lif's third letter, ten months into this process, made specific allegations that could be investigated by the committee. Two months later the charges were dismissed. The Cochrane researchers had done nothing wrong. But before they were cleared, Lif copied the letters alleging scientific dishonesty to the hospital where four of them worked, and to the management organisation running that hospital, and sent similar letters to the Danish Medical Association, the Ministry of Health, the Ministry of Science, and so on. Gøtzsche and his colleagues said that they felt ‘intimidated and hara**ed' by Lif's behaviour. Lif continued to insist that the researchers were guilty of misconduct even after the investigation was completed. So, researching in this area is not easy: it's hard to get funding, and the industry will make your work feel like chewing on a mouthful of wasps.
Even though the problem has been widely recognised, attempts to fix it have failed. 50 The International Committee of Medical Journal Editors, for example, stood up in 2001, insisting that the lead author of any study it published must sign a document stating that the researchers had full access to the data, and full control over the decision to publish. Researchers at Duke University, North Carolina, then surveyed the contracts between medical schools and industry sponsors, and found that this edict was flouted as a matter of routine. They recommended boilerplate contracts for the relationship between industry and academia. Was this imposed? No. Sponsors continue to control the data.
Half a decade later, a major study in the New England Journal of Medicine investigated whether anything had changed. 51 Administrators at all 122 accredited medical schools in the US were asked about their contracts (to be clear, this wasn't a study of what they did; rather it was a study of what they were willing to say in public). The majority said contract negotiations over the right to publish data were ‘difficult'. A worrying 62 per cent said it was OK for the clinical trial agreement between academics and industry sponsor to be confidential. This is a serious problem, as it means that anyone reading a study cannot know how much interference was available to the sponsor, which is important context for the person reading and interpreting the research. Half of the centres allowed the sponsor to draft the research paper, which is another interesting hidden problem in medicine, as biases and emphases can be quietly introduced (as we shall see in more detail in Chapter 6). Half said it was OK for contracts to forbid researchers sharing data after the research was completed and published, once again hindering research. A quarter said it was acceptable for the sponsor to insert its own statistical an*lyses into the man*script. Asked about disputes, 17 per cent of administrators had seen an argument about who had control of data in the preceding year.
Sometimes, disputes over access to such data can cause serious problems in academic departments, when there is a divergence of views on what is ethical. Aubrey Blumsohn was a senior lecturer at Sheffield University, working on a project funded by Procter & Gamble to research an osteoporosis drug called risedronate. 52 The aim was to an*lyse blood and urine samples from an earlier trial, led by Blumsohn's head of department, Professor Richard Eastell. After signing the contracts, P&G sent over some ‘abstracts', brief summaries of the findings, with Blumsohn's name as first author, and some summary results tables. That's great, said Blumsohn, but I'm the researcher here: I'd like to see the actual raw data and an*lyse it myself. The company declined, saying that this was not their policy. Blumsohn stood his ground, and the papers were left unpublished. Then, however, Blumsohn saw that Eastell had published another paper with P&G, stating that all the researchers had ‘had full access to the data and an*lyses'. He complained, knowing this was not true. Blumsohn was suspended by Sheffield University, which offered him a gagging clause and £145,000, and he was eventually forced out of his job. Eastell, meanwhile, was censured by the General Medical Council, but only after a staggering delay of several years, and he remains in post.
So contracts that permit companies and researchers to withhold or control data are common, and they're bad news. But that's not just because they lead to doctors and patients being misled about what works. They also break another vitally important contract: the agreement between researchers and the patients who participate in their trials.
People participate in trials believing that the results of that research will help to improve the treatment of patients like them in the future. This isn't just speculation: one of the few studies to ask patients why they have participated in a trial found that 90 per cent believed they were making a ‘major' or ‘moderate' contribution to society, and 84 per cent felt proud that they were making this contribution. 53 Patients are not stupid or naïve to believe this, because it is what they are told on the consent forms they sign before participating in trials. But they are mistaken, because the results of trials are frequently left unpublished, and withheld from doctors and patients. These signed consent forms therefore mislead people on two vitally important points. Firstly, they fail to tell the truth: that the person conducting the trial, or the person paying for it, may decide not to publish the results, depending on how they look at the end of the study. And worse than that, they also explicitly state a falsehood: researchers tell patients that they are participating in order to create knowledge that will be used to improve treatment in future, even though the researchers know that in many cases, those results will never be published.
There is only one conclusion that we can draw from this: consent forms routinely lie to patients participating in trials. This is an extraordinary state of affairs, made all the more extraordinary by the huge amounts of red tape that everyone involved in a trial must negotiate, closely monitoring endless arcane pieces of paperwork, and ensuring that patients are fully informed on the minutiae of their treatment. Despite all this regulatory theatre, which hinders good research on routine practice (as we shall see – p.232), we allow these forms to tell patients outright lies about the control of data, and we fail to police one of the most important ethical problems in the whole of medicine. The deceit of these consent forms is, to me, a good illustration of how broken and outdated the regulatory frameworks of medicine have become. But it also, finally, poses a serious problem for the future of research.
We desperately need patients to continue to believe that they are contributing to society, because trial recruitment is in crisis, and it is increasingly hard to persuade patients to participate at all. In one recent study, a third of all trials failed to reach their original recruitment target, and more than half had to be awarded an extension. 54 If word gets out that trials are often more promotional than genuinely scientific, recruitment will become even more difficult. The answer is not to hide this problem, but rather to fix it.
So universities and ethics committees may have failed us, but there is one group of people we might expect to step up, to try to show some leadership on missing trial data. These are the medical and academic professional bodies, the Royal Colleges of General Practice, Surgery and Physicians, the General Medical Council, the British Medical Association, the pharmacists' organisations, the bodies representing each sub-specialty of academia, the respiratory physiologists, the pharmacologists, the Academy of Medical Sciences, and so on.
These organisations have the opportunity to set the tone of academic and clinical medicine, in their codes of conduct, their aspirations, and in some cases their rules, since some have the ability to impose sanctions, and all have the ability to exclude those who fail to meet basic ethical standards. We have established, I hope, beyond any doubt, that non-publication of trials in humans is research misconduct, that it misleads doctors and harms patients around the world. Have these organisations used their powers, stood up and announced, prominently and fiercely, that this must stop, and that they will take action?
One has: the Faculty of Pharmaceutical Medicine, a small organisation of doctors – largely working in the industry – with 1,400 members. And none of the others have bothered.
Not one.
What can be done?
There are several simple solutions to these problems, which fall into three categories. There is no argument against any of the following suggestions that I am aware of. The issue of missing data has been neglected through institutional inertia, and reluctance by senior academics to challenge industry. Their failure to act harms patients every day.
Gagging clauses
If there is nothing to be ashamed of in gagging clauses – if companies, and legislators, and academics, and university contracts departments, all believe that they are acceptable – then everything should be out in the open, prominently flagged up, so that everyone outside those systems can decide if they agree.
1.Until they can be eradicated, where gagging clauses exist, patients should be told, at the time they are invited to participate in a trial, that the company sponsoring it is allowed to hide the results if it doesn't like them. The consent form should also explain clearly that withholding negative results will lead to doctors and patients being misled about the effectiveness of the treatment being trialled, and so exposed to unnecessary harm. Trial participants can then decide for themselves if they think these contracts are acceptable.
2.Wherever the sponsoring company has the contractual right to gag publication, even if it doesn't exercise that right, the fact that a gagging clause existed should be stated clearly: in the academic paper; in the trial protocol; and in the publicly available trial registry entry that goes up before the trial starts. Readers of the trial findings can then decide for themselves if they trust that sponsor and research group to publish all negative findings, and interpret any reported positive findings in that light.
3.All university contracts should follow the same boilerplate format, and forbid gagging clauses. Failing that, all universities should be forced to prominently declare which contracts with gagging clauses they have permitted, and to publish those clauses online for all to see, so that all can be alerted that the institution is producing systematically biased research, and discount any findings from them accordingly.
4.In legislation, gagging clauses should be made illegal, with no possibility of quibbles. If there is a dispute about an*lysis or interpretation between the people running the trial and the people paying for it, it should take place in the published academic literature, or some other public forum. Not in secret.
Professional bodies
1.All professional bodies should take a clear stand on unpublished trial data, declare it clearly as research misconduct, and state that it will be handled like any other form of serious misconduct that misleads doctors and harms patients.That they have not done so is a stain on the reputation of these organisations, and on their senior members.
2.All research staff involved in any trial on humans should be regarded as jointly and severally liable for ensuring that it is published in full within one year of completion.
3.All those responsible for withholding trial data should have their names prominently posted in a single database, so that others can be made aware of the risk of working with them, or allowing them access to patients for research, in future.
Ethics committees
1.No person should be allowed to conduct trials in humans if a research project they are responsible for is currently withholding trial data from publication more than one year after completion. Where any researcher on a project has a previous track record of delayed publication of trial data, the ethics committee should be notified, and this should be taken into account, as with any researcher guilty of research misconduct.
2.No trial should be approved without a firm guarantee of publication within one year of completion.
How regulators and journals have failed us
So far we've established that ethics committees, universities and the professional bodies of medical researchers have all failed to protect patients from publication bias, even though the selective non-publication of unflattering data is a serious issue for medicine. We know that it distorts and undermines every decision that researchers, doctors and patients make, and that it exposes patients to avoidable suffering and d**h. This is not seriously disputed, so you might reasonably imagine that governments, regulators and medical journals must all have tried to address it.
They have tried, and failed. Worse than simply failing, they have repeatedly provided what we might regard as ‘fake fixes': we have seen small changes in regulations and practices, announced with great fanfare, but then either ignored or bypa**ed. This has given false rea**urance to doctors, academics and the public that the problem has been fixed. What follows is the story of these fake fixes.
Registers
The earliest and simplest solution proposed was to open registers of trials: if people are compelled to publish their protocol, in full, before they start work, then we at least have the opportunity to go back and check to see if they've published the trials that they've conducted. This is very useful, for a number of reasons. A trial protocol describes in great technical detail everything that researchers will do in a trial: how many patients they'll recruit, where they'll come from, how they'll be divided up, what treatment each group will get, and what outcome will be measured to establish if the treatment was successful. Because of this, it can be used to check whether a trial was published, but also whether its methods were distorted along the way, in a manner that would allow the results to be exaggerated (as described in Chapter 4).
The first major paper to call for a registry of clinical trial protocols was published in 1986, 55 and it was followed by a flood. In 1990 Iain Chalmers (we can call him Sir Iain Chalmers if you like fn2) published a cla**ic paper called ‘Underreporting Research is Scientific Misconduct', 57 and he has traced the chequered history of trials registers in the UK. 58 In 1992, as the Cochrane Collaboration began to gather influence, representatives of the Association of the British Pharmaceutical Industry (ABPI) asked to meet Chalmers. 59 After explaining the work of Cochrane, and the vital importance of summarising all the trial results on a particular drug, he explained very clearly to them how biased under-reporting of results harms patients.
The industry's representatives were moved, and soon they took action. Mike Wallace, the chief executive of Schering and a member of that ABPI delegation, agreed with Chalmers that withholding data was ethically and scientifically indefensible, and said that he planned to do something concrete to prevent it, if only to protect the industry from having the issue forced upon it in less welcome terms. Wallace stepped out of line from his colleagues, and committed to registering every trial conducted by his company with Cochrane. This was not a popular move, and he was reprimanded by colleagues, in particular those from other companies.
But then GlaxoWellcome followed suit, and in 1998 its chief executive, Richard Sykes, wrote an editorial in the BMJ called ‘Being a modern pharmaceutical company involves making information available on clinical trial programmes'. 60 ‘Programmes' was the crucial word, because as we've seen, and as we shall see in greater detail later, you can only make sense of individual findings if you a**ess them in the context of all the work that has been done on a drug.
GlaxoWellcome set up a clinical trials registry, and Elizabeth Wager, the head of the company's medical writers group, pulled together a group from across the industry to develop ethical guidelines for presenting research. The ABPI, seeing individual companies take the lead, saw the writing on the wall: it decided to commend GlaxoWellcome's policy to the whole industry, and launched this initiative at a press conference where Chalmers – a strong critic – sat on the same side of the table as the industry. AstraZeneca, Aventis, MSD, Novartis, Roche, Schering Healthcare and Wyeth began registering some of their trials – only the ones involving UK patients, and retrospectively – but there was movement at last.
At the same time, there was movement in America. The 1997 FDA Modernization Act created clinicaltrials.gov, a register run by the US government National Institutes of Health. This legislation required that trials should be registered, but only if they related to an application to put a new drug on the market, and even then, only if it was for a serious or life-threatening disease. The register opened in 1998, and the website clinicaltrials.gov went online in 2000. The entry criteria were widened in 2004.
But soon it all began to fall apart. GlaxoWellcome merged with SmithKline Beecham to become GlaxoSmithKline (GSK), and initially the new logo appeared on the old trials register. Iain Chalmers wrote to Jean-Paul Garnier, the chief executive of the new company, to thank him for maintaining this valuable transparency: but no reply ever came. The registry website was closed, and the contents were lost (though GSK was later forced to open a new register, as part of a settlement with the US government over the harm caused by its withholding of data on new drug trials just a couple of years later). Elizabeth Wager, the author of the Good Publication Practice guidelines for drug companies, was out of a job, as her writing department at GSK was closed. Her guidelines were ignored. 61
From the moment that these registries were first suggested, and then opened, it was implicitly a**umed that the shame of producing this public record, and then not publishing your study, would be enough to ensure that people would do the right thing. But the first problem for the US register, which could have been used universally, was that people simply chose not to use it. The regulations required only a very narrow range of trials to be posted, and nobody else was in a hurry to post their trials if they didn't have to.
In 2004 the International Committee of Medical Journal Editors (ICMJE) – a collection of editors from the most influential journals in the world – published a policy statement, announcing that none of them would publish any clinical trials after 2005, unless they had been properly registered before they began. 62 They did this, essentially, to force the hand of the industry and researchers: if a trial has a positive result, then people desperately want to publish it in the most prestigious journal they can find. Although they had no legal force, the journal editors did have the thing that companies and researchers wanted most: the chance of a major journal publication. By insisting on pre-registration, they were doing what they could to force researchers and industry sponsors to register all trials. Everyone rejoiced: the problem had been fixed.
If you think it seems odd – and perhaps unrealistic – that fixing this crucial flaw in the information architecture of a $700 billion industry should be left to an informal gathering of a few academic editors, with no legislative power, then you'd be right. Although everybody began to talk as if publication bias was a thing of the past, in reality it was continuing just as before, because the journal editors simply ignored their own threats and promises. Later (pp.247, 307) we will see the phenomenal financial inducements on offer to editors for publishing positive industry papers, which can extend to millions of dollars in reprint and advertising revenue. But first we should look at what they actually did after their solemn promise in 2005.
In 2008 a group of researchers went through every single trial published in the top ten medical journals, every one of which was a member of the ICMJE, after the deadline for pre-registration. Out of 323 trials published during 2008 in these high-impact academic journals, only half were adequately registered (before the trial, with the main outcome measure properly specified), and trial registration was entirely lacking for over a quarter. 63 The ICMJE editors had simply failed to keep their word.
Meanwhile, in Europe, there were some very bizarre developments. With enormous fanfare, the European Medicines Agency created a registry of trials called EudraCT. EU legislation requires all trials to be posted here if they involve any patients in Europe, and many companies will tell you that they've met their responsibilities for transparency by doing so. But the contents of this EU register have been kept entirely secret. I can tell you that it contains around 30,000 trials, since that figure is in the public domain, but that is literally all I know, and all anyone can know. Despite EU legislation requiring that the public should be given access to the contents of this register, it remains closed. This creates an almost laughable paradox: the EU clinical trials register is a transparency tool, held entirely in secret. Since March 2011, after heavy media criticism (from me at any rate), a subset of trials has slowly been made public through a website called EudraPharm. As of summer 2012, although the agency now claims that its register is accessible to all, at least 10,000 trials are still missing, and the search engine doesn't work properly.64 It's absolutely one of the strangest things I've ever seen, and nobody other than the EU even regards this peculiar exercise as a trials register: I certainly don't, I doubt you do, and both the ICMJE and the World Health Organization have explicitly stated that EudraCT is not a meaningful register.
But new work was being done in the US, and it seemed sensible. In 2007 the FDA Amendment Act was pa**ed. This is much tighter: it requires registration of all trials of any drug or device, at any stage of development other than ‘first-in-man' tests, if they have any site in the US, or involve any kind of application to bring a new drug onto the market. It also imposes a startling new requirement: all results of all trials must be posted to clinicaltrials.gov, in abbreviated summary tables, within one year of completion, for any trial on any marketed drug that completes after 2007.
Once again, to great fanfare, everyone believed that the problem had been fixed. But it hasn't been, for two very important reasons.
Firstly, unfortunately, despite the undoubted goodwill, requiring the publication of all trials starting from ‘now' does absolutely nothing for medicine today. There is no imaginable clinic, anywhere in the world, at which medicine is practised only on the basis of trials that completed within the past three years, using only d** that came to market since 2008. In fact, quite the opposite is true: the vast majority of d** currently in use came to market over the past ten, twenty or thirty years, and one of the great challenges for the pharmaceutical industry today is to create d** that are anything like as innovative as those that were introduced in what has come to be known as the ‘golden era' of pharmaceutical research, when all the most widely used d**, for all the most common diseases, were developed. Perhaps they were the ‘low-lying fruit', plucked from the research tree, but in any case, these are the tablets we use.
And crucially, it is for these d** – the ones we actually use – that we need the evidence: from trials completed in 2005 or 1995. These are the d** that we are prescribing completely blind, misled by a biased sample of trials, selectively published, with the unflattering data buried in secure underground data archives somewhere in the hills (I am told) of Cheshire.
But there is a second, more disturbing reason why these regulations should be taken with a pinch of salt: they have been widely ignored. A study published in January 2012 looked at the first slice of trials subject to mandatory reporting, and found that only one in five had met its obligation to post results. 65 Perhaps this is not surprising: the fine for non-compliance is $10,000 a day, which sounds spectacular, until you realise that it's only $3.5 million a year, which is chickenfeed for a drug bringing in $4 billion a year. And what's more, no such fine has ever been levied, throughout the entire history of the legislation.
So that, in total, is why I regard the ICMJE, the FDA and the EU's claims to have addressed this problem as ‘fake fixes'. In fact, they have done worse than fail: they have given false rea**urance that the problem has been fixed, false rea**urance that it has gone away, and they have led us to take our eyes off the ball. For half a decade now, people in medicine and academia have talked about publication bias as if it was yesterday's problem, discovered in the 1990s and early 2000s, and swiftly fixed.
But the problem of missing data has not gone away, and soon we will see exactly how shameless some companies and regulators can be, in the very present day.
What can be done?
The ICMJE should keep its promises, the EU should be less risible, and the FDA Amendment Act 2007 should be muscularly enforced. But there are many more disappointments to come, so I will leave my action plan on missing data for later in this chapter.
Blood from a stone: trying to get data from regulators
So far we've established that doctors and patients have been failed by a range of different people and organisations, all of whom we might have expected to step up and fix the problem of missing data, since it harms patients in large numbers around the world. We have seen that governments take no action against those who fail to publish their results, despite the public pretence to the contrary; and that they take no action against those who fail to register their trials. We have seen that medical journal editors continue to publish unregistered trials, despite the public pretence that they have taken a stand. We have seen that ethics committees fail to insist on universal publication, despite their stated aim of protecting patients. And we have seen that professional bodies fail to take action against what is obviously research misconduct, despite evidence showing that the problem of missing data is of epidemic proportions. 66
While the published academic record is hopelessly distorted, you might hope that there is one final route which patients and doctors could use to get access to the results of clinical trials: the regulators, which receive large amounts of data from drug companies during the approval process, must surely have obligations to protect patients' safety? But this, sadly, is just one more example of how we are failed by the very bodies that are supposed to be protecting us.
In this section, we will see three key failures. Firstly, the regulators may not have the information in the first place. Secondly, the way in which they ‘share' summary trial information with doctors and patients is broken and shabby. And finally, if you try to get all of the information that a drug company has provided – the long-form documents, where the bodies are often buried – then regulators present bizarre barriers, blocking and obfuscating for several years at a time, even on d** that turn out to be ineffective and harmful. Nothing of what I am about to tell you is in any sense rea**uring.
One: Information is withheld from regulators
Paroxetine is a commonly used antidepressant, from the cla** of d** known as ‘selective serotonin reuptake inhibitors', or SSRIs. You will hear more about this cla** of d** later in this book, but here we will use paroxetine to show how companies have exploited our longstanding permissiveness about missing trials, and found loopholes in our inadequate regulations on trial disclosure. We will see that GSK withheld data about whether paroxetine works as an antidepressant, and even withheld data about its harmful side effects, but most importantly, we will see that what it did was all entirely legal.
To understand why, we first need to go through a quirk of the licensing process. Drugs do not simply come onto the market for use in all medical conditions: for any specific use of any drug, in any specific disease, you need a separate marketing authorisation. So, a drug might be licensed to treat ovarian cancer, for example, but not breast cancer. That doesn't mean the drug doesn't work in breast cancer. There might well be some evidence that it's great for treating that disease too, but maybe the company hasn't gone to the trouble and expense of getting a formal marketing authorisation for that specific use. Doctors can still go ahead and prescribe it for breast cancer, if they want, because the drug is available for prescription, and there are boxes of it sitting in pharmacies waiting to go out (even though, strictly speaking, it's only got marketing approval for use in ovarian cancer). In this situation the doctor will be prescribing the drug legally, but ‘off-label'.
This is fairly common, as getting a marketing authorisation for a specific use can be time-consuming and expensive. If doctors know that there's a drug which has been shown in good-quality trials to help treat a disease, it would be perverse and unhelpful of them not to prescribe it, just because the company hasn't applied for a formal licence to market it for that specific use. I'll discuss the ins and outs of all this in more detail later. But for now, what you need to know is that the use of a drug in children is treated as a separate marketing authorisation from its use in adults.
This makes sense in many cases, because children can respond to d** in very different ways to adults, so the risks and benefits might be very different, and research needs to be done in children separately. But this licensing quirk also brings some disadvantages. Getting a licence for a specific use is an arduous business, requiring lots of paperwork, and some specific studies. Often this will be so expensive that companies will not bother to get a licence specifically to market a drug for use in children, because that market is usually much smaller.
But once a drug is available in a country for one specific thing, as we have seen, it can then be prescribed for absolutely anything. So it is not unusual for a drug to be licensed for use in adults, but then prescribed for children on the back of a hunch; or a judgement that it should at least do no harm; or studies that suggest benefit in children, but that would probably be insufficient to get through the specific formal process of getting marketing authorisation for use in kids; or even good studies, but in a disease where the market is so small that the company can't be bothered to get a marketing approval for use in children.
Regulators have recognised that there is a serious problem with d** being used in children ‘off-label', without adequate research, so recently they have started to offer incentives for companies to conduct the research, and formally seek these licences. The incentives are patent extensions, and these can be lucrative. All d** slip into the public domain about a decade after coming onto the market, and become like paracetamol, which anyone can make very cheaply. If a company is given a six-month extension on a drug, for all uses, then it can make a lot more money from that medicine. This seems a good example of regulators being pragmatic, and thinking creatively about what carrots they can offer. Licensed use in children will probably not itself make a company much extra money, since doctors are prescribing the drug for children already, even without a licence or good evidence, simply because there are no other options. Meanwhile, six months of extra patent life for a blockbuster drug will be very lucrative, if its adult market is large enough.
There's a lot of debate about whether the drug companies have played fair with these offers. For example, since the FDA started offering this deal, about a hundred d** have been given paediatric licences, but many of them were for diseases that aren't very common in children, like stomach ulcers, or arthritis. There have been far fewer applications for less lucrative products that could be used in children, such as more modern medicines called ‘large-molecule biologics'. But there it is.
When GSK applied for a marketing authorisation in children for paroxetine, an extraordinary situation came to light, triggering the longest investigation in the history of UK d** regulation. This investigation was published in 2008, and examined whether GSK should face criminal charges. 67 It turned out that what the company had done – withholding important data about safety and effectiveness that doctors and patients clearly needed to see – was plainly unethical, and put children around the world at risk; but our laws are so weak that GSK could not be charged with any crime.
Between 1994 and 2002 GSK conducted nine trials of paroxetine in children. 68 The first two failed to show any benefit, but the company made no attempt to inform anyone of this by changing the ‘drug label' that is sent to all doctors and patients. In fact, after these trials were completed, an internal company management document stated: ‘it would be commercially unacceptable to include a statement that efficacy had not been demonstrated, as this would undermine the profile of paroxetine'. In the year after this secret internal memo, 32,000 prescriptions were issued to children for paroxetine in the UK alone: so, while the company knew the drug didn't work in children, it was in no hurry to tell doctors that, despite knowing that large numbers of children were taking it. More trials were conducted over the coming years – nine in total – and none showed that the drug was effective at treating depression in children.
It gets much worse than that. These children weren't simply receiving a drug that the company knew to be ineffective for them: they were also being exposed to side effects. This should be self-evident, since any effective treatment will have some side effects, and doctors factor this in, alongside the benefits (which in this case were non-existent). But nobody knew how bad these side effects were, because the company didn't tell doctors, or patients, or even the regulator about the worrying safety data from its trials. This was because of a loophole (on which more in a couple of pages' time): you only had to tell the regulator about side effects reported in studies looking at the specific uses for which the drug has a marketing authorisation. Because the use of paroxetine in children was ‘off-label', GSK had no legal obligation at the time to tell anyone about what it had found.
People had worried for a long time that paroxetine might increase the risk of suicide, though that is quite a difficult side effect to detect in an antidepressant, because people with depression are at a much higher risk of suicide than the general population anyway, as a result of their depression. There are also some grounds to believe that as patients first come out of their depression, and leave behind the sluggish lack of motivation that often accompanies profound misery, there may be a period during which they are more capable of k**ing themselves, just because of the depression slowly lifting.
Furthermore, suicide is a mercifully rare event, which means you need a lot of people on a drug to detect an increased risk. Also, suicide is not always recorded accurately on d**h certificates, because coroners and doctors are reluctant to give a verdict that many would regard as shameful, so the signal you are trying to detect in the data – suicide – is going to be corrupted. Suicidal thoughts or behaviours that don't result in d**h are more common than suicide itself, so they should be easier to detect, but they too are hard to pick up in routinely collected data, because they're often not presented to doctors, and where they are, they can be coded in health records in all sorts of different ways, if they appear at all. Because of all these difficulties, you would want to have every scrap of data you could possibly cobble together on the question of whether these d** cause suicidal thoughts or behaviour in children; and you would want a lot of experienced people, with a wide range of sk**s, all looking at the data and discussing it.
In February 2003, GSK spontaneously sent the MHRA a package of information on the risk of suicide on paroxetine, containing some an*lyses done in 2002 from adverse-event data in trials the company had held, going back a decade. This an*lysis showed that there was no increased risk of suicide. But it was misleading: although it was unclear at the time, data from trials in children had been mixed in with data from trials in adults, which had vastly greater numbers of participants. As a result, any sign of increased suicide risk among children on paroxetine had been completely diluted away.
Later in 2003, GSK had a meeting with the MHRA to discuss another issue involving paroxetine. At the end of this meeting, the GSK representatives gave out a briefing document, explaining that the company was planning to apply later that year for a specific marketing authorisation to use paroxetine in children. They mentioned, while handing out the document, that the MHRA might wish to bear in mind a safety concern the company had noted: an increased risk of suicide among children with depression who received paroxetine, compared with those on dummy placebo pills.
This was vitally important side-effect data, being presented, after an astonishing delay, casually, through an entirely inappropriate and unofficial channel. GSK knew that the drug was being prescribed in children, and it knew that there were safety concerns in children, but it had chosen not to reveal that information. When it did share the data, it didn't flag it up as a clear danger in the current use of the drug, requiring urgent attention from the relevant department in the regulator; instead it presented it as part of an informal briefing about a future application. Although the data was given to completely the wrong team, the MHRA staff present at this meeting had the wit to spot that this was an important new problem. A flurry of activity followed: an*lyses were done, and within one month a letter was sent to all doctors advising them not to prescribe paroxetine to patients under the age of eighteen.
How is it possible that our systems for getting data from companies are so poor that they can simply withhold vitally important information showing that a drug is not only ineffective, but actively dangerous? There are two sets of problems here: firstly, access for regulators; and secondly, access for doctors.
There is no doubt that the regulations contain ridiculous loopholes, and it's dismal to see how GSK cheerfully exploited them. As I've mentioned, the company had no legal duty to give over the information, because prescription of the drug in children was outside of paroxetine's formally licensed uses – even though GSK knew this was widespread. In fact, of the nine studies the company conducted, only one had its results reported to the MHRA, because that was the only one conducted in the UK.
After this episode, the MHRA and the EU changed some of their regulations, though not adequately, and created an obligation for companies to hand over safety data for uses of a drug outside its marketing authorisation, closing the paroxetine loophole.
This whole incident illustrates a key problem, and it is one that recurs throughout this section of the book: you need all of the data in order to see what's happening with a drug's benefits, and risks. Some of the trials that GSK conducted were published in part, but that is obviously not enough: we already know that if we see only a biased sample of the data, we are misled. But we also need all the data for the more simple reason that we need lots of data: safety signals are often weak, subtle and difficult to detect. Suicidal thoughts and plans are rare in children – even those with depression, even those on paroxetine – so all the data from a large number of participants needed to be combined before the signal was detectable in the noise. In the case of paroxetine, the dangers only became apparent when the adverse events from all of the trials were pooled and an*lysed together.
That leads us to the second obvious flaw in the current system: the results of these trials – the safety data and the effectiveness data – are given in secret to the regulator, which then sits and quietly makes a decision. This is a huge problem, because you need many eyes on these difficult problems. I don't think that the people who work in the MHRA are bad, or incompetent: I know a lot of them, and they are smart, good people. But we shouldn't trust them to an*lyse this data alone, in the same way that we shouldn't trust any single organisation to an*lyse data alone, with nobody looking over its shoulder, checking the working, providing competition, offering helpful criticism, speeding it up, and so on.
This is even worse than academics failing to share their primary research data, because at least in an academic paper you get a lot of detail about what was done, and how. The output of a regulator is often simply a crude, brief summary: almost a ‘yes' or ‘no' about side effects. This is the opposite of science, which is only reliable because everyone shows their working, explains how they know that something is effective or safe, shares their methods and their results, and allows others to decide if they agree with the way they processed and an*lysed the data.
Yet for the safety and efficacy of d**, one of the most important of all an*lyses done by science, we turn our back on this process completely: we allow it to happen behind closed doors, because drug companies have decided that they want to share their trial results discreetly with the regulators. So the most important job in evidence-based medicine, and a perfect example of a problem that benefits from many eyes and minds, is carried out alone and in secret.
This perverse and unhealthy secrecy extends way beyond regulators. NICE, the UK's National Institute for Health and Clinical Excellence, is charged with making recommendations about which treatments are most cost-effective, and which work best. When it does this, it's in the same boat as you or me: it has absolutely no statutory right to see data on the safety or effectiveness of a drug, if a company doesn't want to release it, even though the regulators have all of that data. For ‘single technology appraisals', on one treatment, they ask the company to make available to them the information the company thinks is relevant. For ‘guidelines' on treatment in a whole area of medicine, they are more vulnerable to what is published in journals. As a result, even NICE can end up working on distorted, edited, biased samples of the data.
Sometimes NICE is able to access some extra unpublished data from the drug companies: this is information that doctors and patients aren't allowed to see, despite the fact that they are the people making decisions about whether to prescribe the d**, or are actually taking them. But when NICE does get information in this way, it can come with strict conditions on confidentiality, leading to some very bizarre documents being published. On the opposite page, for example, is the NICE document discussing whether it's a good idea to have Lucentis, an extremely expensive drug, costing well over £1,000 per treatment, that is injected into the eye for a condition called acute macular degeneration.
As you can see, the NICE document on whether this treatment is a good idea is censored. Not only is the data on the effectiveness of the treatment blanked out by thick black rectangles, in case any doctor or patient should see it; absurdly, even the names of some trials are missing, preventing the reader from even knowing of their existence, or cross referencing information about them. Most disturbing of all, as you can see in the last bullet point, the data on adverse events is also censored: I'm reproducing the whole page here because I worry that it would otherwise be almost too bizarre for you to believe. This level of censorship isn't an everyday phenomenon; but it illustrates the absurdity of what medicine has come to accept, in professional documents that most people wouldn't bother to read. 69
Why shouldn't we all – doctors, patients and NICE – have access to the information on trials that regulators see? This is something I asked both Kent Woods from the MHRA, and Hans Georg Eichler, Medical Director of the European Medicines Agency, in 2010. Both, separately, gave me the same answer: people outside the agencies cannot be trusted with this information, because they might misinterpret it, either deliberately or through incompetence. Both, separately – though I guess they must chat at parties – raised the MMR vaccine scare, as the cla**ic example of how the media can contrive a national panic on the basis of no good evidence, creating dangerous public-health problems along the way. What if they released raw safety data, and people who don't know how to an*lyse it properly found imaginary patterns, and created scares that put patients off taking life-saving medication?
I accept that this is a risk, but I also believe their priorities are wrong: I think that the advantages of many eyes working on these vitally important problems are enormous, and the possibility of a few irrational scaremongers is no excuse for hiding data. Drug companies and regulators also both say that you can already get all the information you need from regulators' websites, in summary form.
We shall now see that this is untrue.
Two: Regulators make it hard to access the data they do have
When exposed to criticism, drug companies often become indignant, and declare that they already share enough data for doctors and patients to be informed. ‘We give everything to the regulator,' they say, ‘and you can get it from them.' Similarly, regulators insist that all you need to do is look on their website, and you will easily find all the data you need. In reality, there is a messy game, in which doctors and academics trying to find all the data on a drug are sent around the houses, scrabbling for information that is both hard to find and fatally flawed.
Firstly, as we've already seen, regulators don't have all the trials, and they don't share all the ones that they do. Summary documents are available on the early trials used to get a drug onto the market in the first place, but only for the specific licensed uses of the drug. Even where the regulator has been given safety data for off-label uses (following the paroxetine case above) the information from these trials still isn't made publicly available through the regulator: it simply sits quietly in the regulator's files.
For example: duloxetine is another drug in fairly widespread use, which is usually given as an antidepressant. During a trial on its use for a completely different purpose – treating incontinence – there were apparently several suicides. 70 This is important and interesting information, and the FDA holds the relevant data: it conducted a review on this issue, and came to a view on whether the risk was significant. But you cannot see any of that on the FDA website, because duloxetine never got a licence for use in treating incontinence. 71 The trial data was only used by the FDA to inform its internal ruminations. This is an everyday situation.
But even when you are allowed to see trial results held by regulators, getting this information from their public websites is supremely tricky. The search functions on the FDA website are essentially broken, while the content is haphazard and badly organised, with lots missing, and too little information to enable you to work out if a trial was prone to bias by design. Once again – partly, here, through casual thoughtlessness and incompetence – it is impossible to get access to the basic information that we need. Drug companies and regulators deny this: they say that if you search their websites, everything is there. So let's walk, briefly, through the process in all its infuriating glory. The case I will use was published three years ago in JAMA as a useful illustration of how broken the FDA site has become: 72 replicating it today, in 2012, nothing has changed.
So: let's say we want to find the results from all the trials the FDA has, on a drug called pregabalin, in which the drug is used to treat pain for diabetics whose nerves have been affected by their disease (a condition called ‘diabetic peripheral neuropathy'). You want the FDA review on this specific use, which is the PDF document containing all the trials in one big bundle. But if you search for ‘pregabalin review', say, on the FDA website, you get over a hundred documents: none of them is clearly named, and not one of them is the FDA review document on pregabalin. If you type in the FDA application number – the unique identifier for the FDA document you're looking for – the FDA website comes up with nothing at all.
If you're lucky, or wise, you'll get dropped at the Drugs@FDA page: typing ‘pregabalin' there brings up three ‘FDA applications'. Why three? Because there are three different documents, each on a different condition that pregabalin can be used to treat. The FDA site doesn't tell you which condition each of these three documents is for, so you have to use trial and error to try to find out. That's not as easy as it sounds. I have the correct document for pregabalin and diabetic peripheral neuropathy right here in front of me: it's almost four hundred pages long, but it doesn't tell you that it's about diabetic peripheral neuropathy until you get to here. There's no executive summary at the beginning – in fact, there's no title page, no contents page, no hint of what the document is even about, and it skips randomly from one sub-document to another, all scanned and bundled up in the same gigantic file.
If you're a nerd, you might think: these files are electronic; they're PDFs, a type of file specifically designed to make sharing electronic documents convenient. Any nerd will know that if you want to find something in an electronic document, it's easy: you just use the ‘find' command: type in, say, ‘peripheral neuropathy', and your computer will find the phrase straight off. But no: unlike almost any other serious government document in the world, the PDFs from the FDA are a series of photographs of pages of text, rather than the text itself. This means you cannot search for a phrase. Instead, you have to go through it, searching for that phrase, laboriously, by eye.
I could go on. I will. There's some kind of ‘table of contents' on the seventeenth page, but it gets the page numbers wrong. I've finished now. There is simply no reason for this obfuscation and chaos. These problems aren't caused by technical issues specific to trials, and they would hardly cost any money at all to fix. This is plainly, simply, unhelpful, and the best we can hope is that it's driven by thoughtlessness.
That's a tragedy, because if you can unearth this document, and decode it, you will find that it is full of terrifying gems: perfect examples of situations in which a drug company has used dodgy statistical methods to design and an*lyse a study, in such a way that it is predestined – from the outset – to exaggerate the benefits of the drug.
For example, in the five trials on pregabalin and pain, lots of people dropped out during the study period. This is common in medical trials, as you will shortly see, and it often happens because people have found a drug to be unhelpful, or have had bad side effects. During these trials you're measuring pain at regular intervals. But if some people drop out, you're left with an important question: what kind of pain score should you use for them in your results? We know, after all, that people dropping out are more likely to have done badly on the drug.
Pfizer decided to use a method called ‘Last Observation Carried Forward', which means what you'd expect: you take the last measurement of pain severity while the patients were on the drug, from just before they dropped out, and then paste that in for all the remaining pain measures that they missed, after they stopped coming to follow-up appointments.
The FDA disapproved of this: it pointed out, quite correctly, that Pfizer's strategy would make the drug look better than it really is. For a fairer picture, we have to a**ume that the drop-outs stopped taking the drug because of side effects, so their pain score should reflect the reality, which is that they would never get any benefit from the drug in normal use. The correct level of pain to record for them is, therefore, their pain at the beginning of the study, before they had any kind of treatment (if you're interested, this is called ‘Baseline Observation Carried Forward'). The an*lysis was duly redone, properly, and a more modest, more accurate view of the benefits of the drug was produced. In this case, it turns out that using the ‘last observation' method overestimated the improvement in pain by about a quarter.
Here's the catch. Four out of five of these trials were then published in the peer-reviewed academic literature, the place where doctors look for evidence on whether a drug works or not (one trial wasn't published at all). Every single one of the published an*lyses used ‘Last Observation Carried Forward', the dodgy method, the one that exaggerates the benefits of the drug. Not one of them acknowledges that ‘last observation' is a technique that overstates these benefits.
You can see why it is important that we have access to all the information we can possibly get on every drug trial: not only are some whole trials withheld from us, but there are often hidden flaws in the methods used. The devil is in the detail, and there are many dodgy trials, as we shall soon see, with flaws that may not be clear even in the academic papers, let alone in the thin and uninformative summaries from regulators. Furthermore, as we shall also see very shortly, there are often worrying discrepancies between the regulators' summary documents and what actually happened in the trial.
This is why we need to get hold of a more detailed document on each trial: something called the Clinical Study Report (CSR). These are long pieces of work, sometimes thousands of pages, but they are complete enough for the reader to reconstruct exactly what happened to all the participants; and they will let you find out where the bodies are buried. Drug companies give this study report to the regulator – though still only for formally licensed uses of the drug – so both have a copy, and both should be happy to hand it over.
We will now see what happens when you ask them.
Three: Regulators withhold study reports that they do have
In 2007, researchers from the Nordic Cochrane Centre were working on a systematic review for two widely used diet d**, orlistat and rimonabant. A systematic review, as you know, is the gold-standard summary of the evidence on whether a treatment is effective. These are life-saving, because they give us the best possible understanding of the true effects of a treatment, including its side effects. But doing this requires access to all of the evidence: if some is missing, especially if unflattering data is deliberately harder to obtain, we will be left with a distorted picture.
The researchers knew that the trial data they were able to find in the published academic literature was probably incomplete, because negative trials are routinely left unpublished. But they also knew that the European Medicines Agency (EMA) would have much of this information, since the manufacturers of d** are obliged to give the study reports to the regulator when trying to get them onto the market. Since regulators are supposed to act in the interests of patients, they applied to the EMA for the protocols and the study reports. That was in June 2007.
In August, the EMA responded: it had decided not to give out the study reports for these trials, and was invoking the section of its rules which allows it to protect the commercial interests and intellectual property of drug companies. The researchers replied immediately, almost by return of post: there is nothing in the study reports that will undermine the protection of someone's commercial interests, they explained. But if there was, could the EMA please explain why it felt the commercial interests of the drug companies should override the welfare of patients?
We should pause for just one moment, and think about what the EMA is doing here. It is the regulator that approves and monitors d** for the whole of Europe, with the aim of protecting the public. Doctors and patients can only make meaningful decisions about treatments if they have access to all the data. The EMA has that data, but has decided that the drug companies' interests are more important. Having spoken to a lot of people in regulation, I can offer one small insight into what on earth they might be thinking. Regulators, in my experience, are preoccupied with the idea that they see all the data, and use it to make the decision about whether a drug should go on the market, and that this is enough: doctors and patients don't need to see the data, because the regulator has done all that work.
This misunderstands a crucial difference between the decisions made by regulators and the decisions made by doctors. Contrary to what some regulators seem to think, a drug is not either ‘good' and therefore on the market, or ‘bad' and therefore off it. A regulator makes a decision about whether it's in the interests of the population as a whole that the drug should be available for use, at all, ever – even if only in some very obscure circumstance, infrequently and cautiously. This bar is set pretty low, as we shall see, and lots of d** that are on the market (in fact, the overwhelming majority) are hardly ever used.
A doctor needs to use the same information as that available to the regulator in order to make a very different decision: is this the right drug for the patient in front of me right now? The simple fact that a drug is approved for prescription doesn't mean it's particularly good, or the best. In fact, there are complex decisions to be made in each clinical situation about which drug is best. Maybe the patient has failed to get better on one drug, so you want to try another, from a different cla** of d**; maybe the patient has mild kidney failure, so you don't want to use the most popular drug, as that causes very occasional problems in patients with dodgy kidneys; maybe you need a drug that won't interfere with other d** the patient is taking.
These complex considerations are the reason we are OK with having a range of d** on the market: even if some of them are less useful overall, they might be useful in specific circumstances. But we need to be able to see all of the information about them, in order to make these decisions. It is not enough for the regulators to grandly state that they have approved a drug, and therefore we should all feel happy to prescribe it. Doctors and patients need the data just as much as regulators do.
In September 2007 the EMA confirmed to the Cochrane researchers that it wasn't going to share the study reports on orlistat and rimonabant, and explained that it had a policy of never disclosing the data given as part of a marketing authorisation. A serious problem had emerged. These weight-loss d** were being widely prescribed throughout Europe, but doctors and patients were unable to access important information about whether they worked, how bad the side effects were, which was more effective, or any of a whole host of other important questions. Real patients were being exposed to potential harm, in everyday prescribing decisions, through this lack of information that was enforced by the EMA.
The researchers went to the European Ombudsman with two clear allegations. Firstly, the EMA had failed to give sufficient reasons for refusing them access to the data; and secondly, the EMA's brief, dismissive claim that commercial interests must be protected was unjustified, because there was no material of commercial interest in the trial results, other than the data on safety and effectiveness, which doctors and patients obviously need to access. They didn't know it at the time, but this was the beginning of a battle for data that would shame the EMA, and would last more than three years.
It took four months for the EMA to respond, and over the next year it simply reiterated its position: as far as it was concerned, any kind of information the disclosure of which would ‘unreasonably undermine or prejudice the commercial interests of individuals or companies' was commercially confidential. The study reports, it said, might contain information about the commercial plans for the drug. The researchers responded that this was unlikely, but was in any case of marginal importance, as part of a much more important and pressing situation: ‘As a likely consequence of [the] EMA's position, patients would die unnecessarily, and would be treated with inferior and potentially harmful d**.' They regarded the EMA's position as ethically indefensible. More than that, they said, the EMA had a clear conflict of interest: this data could be used to challenge its summary views on the benefits and risks of these treatments. The EMA had failed to explain why doctors and patients having access to study reports and protocols should undermine anyone's reasonable commercial interests, and why these commercial interests were more important than the welfare of patients.
Then, almost two years into this process, the EMA changed tack. Suddenly, it began to argue that study reports contain personal data about individual patients. This argument hadn't been raised by the EMA before, but it's also not true. There may have been some information in some whole sections of the study reports that gave detail on some individual participants' odd presentations, or on possible side effects, but these were all in the same appendix, and could easily be removed.
The EU Ombudsman's conclusions were clear: the EMA had failed in its duty to give an adequate or even coherent explanation of why it was refusing access to this important information. He made a preliminary finding of maladministration. After doing that, he was under no obligation to offer any further opinion on the weak excuses offered by the EMA, but he decided to do so anyway. His report is damning. The EMA had flatly failed to address a serious charge that its withholding information about these trials was against the public interest, and exposed patients to harm. The Ombudsman also described how he had gone through the study reports in detail himself, and had found that they contained neither any commercially confidential information, nor any details about the commercial development of the d**. The EMA's claims that answering the request would have put it under an inappropriate administrative burden were untrue, and it had overestimated the work this would have involved: specifically, he explained, removing any personal data, where it did sometimes appear, would be easy.
The Ombudsman told the EMA to hand over the data, or produce a convincing explanation of why it shouldn't. Amazingly, the EMA, the d** regulator covering the whole of Europe, still refused to hand over the documents. During this delay, people certainly suffered unnecessarily, and some possibly also died, simply for want of information. But the behaviour of the EMA then deteriorated even further, into the outright surreal. Any scrap of the company's thinking about how to run the trial, it argued, which could be intuited from reading the study reports and protocols, was commercially sensitive with respect to its thoughts and plans. This was true, the EMA said, even though the d** were already on the market, and the information was from final clinical trials, at the very end of the commercial drug-development process. The researchers responded that this was perverse: they knew that withheld data is often negative, so if anything, any other company seeing negative data about these d** would be less likely to try to get a competitor to market, if it appeared that the benefits of the d** were more modest than had been thought.
That wasn't the end of it. The EMA also grandly dismissed the idea that lives were at risk, stating that the burden of proof was on the researchers to demonstrate this. To me this is – if you can forgive me – a slightly contemptuous attitude, especially given what happens in the next paragraph. It's plainly true that if doctors and patients are unable to see which is the best treatment, then they will make worse decisions, exposing patients to unnecessary harm. Furthermore, it is obvious that larger numbers of academics making transparent judgements about publicly accessible trial data are a much more sensible way of determining the risks and benefits of an intervention than a brief blanket ‘yes or no' edict and summary from a regulator. This is all true of d** like orlistat and rimonabant, but it's also true of any drug, and we will see many cases where academics spotted problems with d** that regulators had missed.
Then, in 2009, one of the two d**, rimonabant, was taken off the market, on the grounds that it increases the risk of serious psychiatric problems and suicide. This, at the same time that the EMA was arguing that researchers were wrong to claim that withholding information was harming patients.
And then the EMA suddenly claimed that the design of a randomised trial itself is commercially confidential information.
In case it needs reiterating, let me remind you that the first trial appears in the Bible, Daniel 1:12, and although the core ideas have certainly been refined over time, all trials are essentially identical experiments, generic across any field, with the basics of a modern trial sketched out at least half a century ago. There is absolutely no earthly sense in which anyone could realistically claim that the design of a randomised controlled trial is a commercially confidential or patentable piece of intellectual property.
This was now a farce. The researchers opened all barrels. The EMA was in breach of the Declaration of Helsinki, the international code of medical ethics, which states that everyone involved in research has a duty to make trial findings public. The researchers knew that published papers give a flattering subset of the trial data, and so did the EMA. Patients would die if the EMA continued to withhold data. There was nothing of serious commercial value in there. The EMA's brief public summaries of the data were inaccurate. The EMA was complicit in the exploitation of patients for commercial gain.
It was now August 2009, and the researchers had been fighting for more than two years for access to data on two widely prescribed d**, from the very organisation that is supposed to be protecting patients and the public. They weren't alone. The French ‘prescribers' bulletin' Prescrire was trying at the same time to get the EMA's documents on rimonabant. It was sent some unhelpful documents, including the rather remarkable ‘Final Assessment Report', from the Swedish agency that had handled the drug's approval much earlier. You can read this PDF in full online. Or rather, you can't. On the following page you can see exactly what the scientific an*lysis of the drug looked like – the document the EMA sent to one of France's most respected journals for doctors. 73 I think it tells a rather clear story, and to add to the insult, there are sixty pages of this in total.
In the meantime, the Danish Medical Authority had handed over fifty-six study reports to Cochrane (though it still needed more from the EMA); a complaint from the drug company about this had been rejected by the Danish government, which had seen no problem with commercial information (there was none), nor administrative ha**le (it was minimal), nor the idea that the design of a randomised trial is commercial information (which is laughable). This was chaos. The EMA – which you may remember was responsible for EudraCT, the transparency tool that is held in secret – was going out on a very peculiar limb. It seemed that it would do anything it could to withhold this information from doctors and patients. As we shall see, sadly, this level of secrecy is its stock in trade.
But now we come to the end of this particular road for the EMA. It handed the full, final study reports over to the Ombudsman, reminding him that even the table of contents for each was commercial. Once he had had these documents in his hand, his final opinion came swiftly. There was no commercial data here. There was no confidential patient information here. Hand it over to the public, now. The EMA, at glacial pace, agreed to set a deadline for delivering the data to the researchers, doctors and patients who needed it. The Ombudsman's final ruling was published at the end of November 2010. 74 The initial complaint was made in June 2007. That is a full three and a half years of fighting, obstruction and spurious arguments from the EMA, during which one of the d** had to be removed from the market because it was harming patients.
After this precedent had been set, things had to shift at the EMA; it was forced to revise its approach, and study reports were briefly made more widely available under a new policy (this window has just been shut down, in 2013, as you will read in the Afterword). But even this policy of sharing reports did not solve the problem of access to all information about a trial: the EMA often doesn't hold all the modules, of all the reports, for all the uses of all the medicines it has approved. And it only holds records for the most recent few years, which is hopelessly incomplete, since in medicine we use treatments that have come on the market over the past few decades. In fact, this loophole is illustrated by the very first request made by the Cochrane researchers who won this great breakthrough with the Ombudsman. They applied for documents on antidepressants: a good place to start, since these d** have been the focus of some particularly bad behaviour over the years (though we should remember that missing trial data pervades every corner of medicine). What happened next was even more bizarre than the EMA's three-year battle to withhold information on orlistat and rimonabant.
The researchers put in their request to the EMA, but were told that the d** had been approved back in the era when marketing authorisations were given out by individual countries, rather than by the EMA centrally. These local authorisations were then ‘copied' to all other nations. The MHRA, the UK d** regulator, held the information the researchers wanted, so they would have to approach it for a copy. The researchers dutifully wrote to the MHRA, asked for the reports on a drug called fluoxetine, and then waited patiently. Finally the answer came: the MHRA explained that it would be happy to hand over this information, but there was a problem.
The documents had all been shredded. 75
This was in keeping, it explained, with the agency's retention policy, which was that such documents were only kept if they were of particular scientific, historical or political interest, and the files didn't meet those criteria. Let's just take a moment to think through what the criteria must be. The SSRI antidepressant d** have seen many scandals on hidden data, and that should be enough on its own, but if you think back to the beginning of this chapter, one of them – paroxetine – was involved in an unprecedented, four-year-long investigation into whether criminal charges could be brought against GSK. That investigation into paroxetine was the largest investigation that the MHRA has ever conducted into drug safety: in fact, it was the largest investigation the MHRA has ever conducted, of any kind, ever. Quite apart from that, these original study reports contain vitally important data on safety and efficacy. But the MHRA shredded them all the same, feeling that they were not of sufficient scientific, historical or political interest. 76
I can only leave you there.
Where are we so far?
The missing data saga is long and complex, and involves patients around the world being put at risk, and some major players which have let us down to an extraordinary extent. Since we are almost at the end of it, this is a good moment to gather together what we've seen so far.
Trials are frequently conducted, but then left unpublished, and so are unavailable to doctors and patients. Only half of all trials get published, and those with negative results are twice as likely to go missing as those with positive ones. This means that the evidence on which we base decisions in medicine is systematically biased to overstate the benefits of the treatments we give. Since there is no way to compensate for this missing data, we have no way of knowing the true benefits and risks of the d** doctors prescribe.
This is research misconduct on a grand, international scale. The reality of the problem is universally recognised, but nobody has bothered to fix it:
*Ethics committees allow companies and individual researchers with a track record of unpublished data to do more trials on human participants.
*University administrators and ethics committees permit contracts with industry that explicitly say the sponsor can control the data.
*Registers have never been enforced.
*Academic journals continue to publish trials that were unregistered, despite pretending that they won't.
*Regulators have information that would be vital to improve patient care, but are systematically obstructive:
*they have poor systems to share poor summaries of the information that they do have;
*and they are bizarrely obstructive of people asking for more information.
*Drug companies hold trial results that even regulators don't see.
*Governments have never implemented laws compelling companies to publish data.
*Doctors' and academics' professional bodies have done nothing.
What to do about all this?
You will need to wait, because in the next section there are even greater horrors to come.
Trying to get trial data from drug companies: the story of Tamiflu
Governments around the world have spent billions of pounds on stockpiling a drug called Tamiflu. In the UK alone we have spent several hundred million pounds – the total figure is not yet clear – and so far we've bought enough tablets to treat 80 per cent of the population if an outbreak of bird flu occurs. I'm very sorry if you have the flu, because it's horrid being ill; but we have not spent all this money to reduce the duration of your symptoms in the event of a pandemic by a few hours (though Tamiflu does do this, fairly satisfactorily). We have spent this money to reduce the rate of ‘complications': a medical euphemism, meaning pneumonia and d**h.
Lots of people seem to think Tamiflu will do this. The US Department of Health and Human Services said it would save lives and reduce hospital admissions. The European Medicines Agency said it would reduce complications. The Australian d** regulator said so too. Roche's website said it reduces complications by 67 per cent. 77 But what is the evidence that Tamiflu really will reduce complications? Answering questions like this is the bread and bu*ter of the Cochrane Collaboration, which you will remember is the vast and independent non-profit international collaboration of academics producing hundreds of systematic reviews on important questions in medicine every year. In 2009 there was concern about a flu pandemic, and an enormous amount of money was being spent on Tamiflu. Because of this, the UK and Australian governments specifically asked the Cochrane Acute Respiratory Infections Group to update its earlier reviews on the drug.
Cochrane reviews are in a constant review cycle, because evidence changes over time as new trials are published. This should have been a pretty everyday piece of work: the previous review, in 2008, had found some evidence that Tamiflu does indeed reduce the rate of complications. But then a Japanese paediatrician called Keiji Hayashi left a comment that would trigger a revolution in our understanding of how evidence-based medicine should work. This wasn't in a publication, or even a letter: it was a simple online comment, posted underneath the Tamiflu review on the Cochrane website.
You've summarised the data from all the trials, he explained, but your positive conclusion is really driven by data from just one of the papers you cite, an industry-funded meta-an*lysis led by an author called Kaiser. This, ‘the Kaiser paper', summarises the findings of ten earlier trials, but from these ten trials, only two have ever been published in the scientific literature. For the remaining eight, your only information comes from the brief summary in this secondary, industry source. That's not reliable enough.
In case it's not immediately obvious, this is science at its best. The Cochrane review is readily accessible online; it explains transparently the methods by which it looked for trials, and then an*lysed them, so any informed reader can pull the review apart, and understand where the conclusions came from. Cochrane provides an easy way for readers to raise criticisms. And, crucially, these criticisms did not fall on deaf ears. Tom Jefferson is an editor at the Cochrane Acute Respiratory Infections Group, and the lead author on the 2008 review. He realised immediately that he had made a mistake in blindly trusting the Kaiser data. He said so, without any defensiveness, and then set about getting the information in a straight, workpersonlike fashion. This began a three-year battle, which is still not resolved, but which has thrown stark light on the need for all researchers to have access to clinical study reports on trials wherever possible.
First, the Cochrane researchers wrote to the authors of the Kaiser paper, asking for more information. In reply, they were told that this team no longer had the files, and that they should contact Roche, the manufacturer of Tamiflu. So naturally they wrote to Roche and asked for the data.
This is where the problems began. Roche said it would hand over some data, but the Cochrane reviewers would need to sign a confidentiality agreement. This was an impossibility for any serious scientist: it would prevent them from conducting a systematic review with any reasonable degree of openness and transparency. More than this, the proposed contract also raised serious ethical issues, in that it would have required the Cochrane team to actively withhold information from the reader: it included a clause saying that on signing it, the reviewers would never be allowed to discuss the terms of this secrecy agreement; and more than that, they would be forbidden from ever publicly acknowledging that it even existed. Roche was demanding a secret contract, with secret terms, requiring secrecy about trial data, in a discussion about the safety and efficacy of a drug that has been taken by hundreds of thousands of people around the world. Jefferson asked for clarification, and never received a reply.
Then, in October 2009, the company changed tack: they would like to hand the data over, they explained, but another meta-an*lysis was being conducted elsewhere. Roche had given them the study reports, so Cochrane couldn't have them. This was a simple non-sequitur: there is no reason why many groups shouldn't all work on the same question. In fact, quite the opposite: replication is the cornerstone of good science. Roche's excuse made no sense. Jefferson asked for clarification, but never received a reply.
One week later, unannounced, Roche sent seven short documents, each around a dozen pages long. These contained excerpts of internal company documents on each of the clinical trials in the Kaiser meta-an*lysis. This was a start, but it didn't contain anything like enough information for Cochrane to a**ess the benefits, or the rate of adverse events, or fully to understand exactly what methods were used in the trials.
At the same time, it was rapidly becoming clear that there were odd inconsistencies in the information on this drug. Firstly, there was considerable disagreement at the level of the broad conclusions drawn by different organisations. The FDA said there were no benefits on complications, while the Centers for Disease Control and Prevention (in charge of public health in the USA – some wear nice naval uniforms in honour of their history on the docks) said it did reduce complications. The Japanese regulator made no claim for complications, but the EMA said there was a benefit. In a sensible world, we might think that all these organisations should sing from the same hymn sheet, because all would have access to the same information. Of course, there is also room for occasional, reasonable disagreement, especially where there are close calls: this is precisely why doctors and researchers should have access to all the information about a drug, so that they can make their own judgements.
Meanwhile, reflecting these different judgements, Roche's own websites said completely different things in different jurisdictions, depending on what the local regulator had said. It's naïve, perhaps, to expect consistency from a drug company, but from this and other stories it's clear that industry utterances are driven by the maximum they can get away with in each territory, rather than any consistent review of the evidence.
In any case, now that their interest had been piqued, the Cochrane researchers also began to notice that there were odd discrepancies between the frequency of adverse events in different databases. Roche's global safety database held 2,466 neuropsychiatric adverse events, of which 562 were cla**ified as ‘serious'. But the FDA database for the same period held only 1,805 adverse events in total. The rules vary on what needs to be notified to whom, and where, but even allowing for that, this was odd.
In any case, since Roche was denying them access to the information needed to conduct a proper review, the Cochrane team concluded that they would have to exclude all the unpublished Kaiser data from their an*lysis, because the details could not be verified in the normal way. People cannot make treatment and purchasing decisions on the basis of trials if the full methods and results aren't clear: the devil is often in the detail, as we shall see in Chapter 4, on ‘bad trials', so we cannot blindly trust that every study is a fair test of the treatment.
This is particularly important with Tamiflu, because there are good reasons to think that these trials were not ideal, and that published accounts were incomplete, to say the least. On closer examination, for example, the patients participating were clearly unusual, to the extent that the results may not be very relevant to normal everyday flu patients. In the published accounts, patients in the trials are described as typical flu patients, suffering from normal flu symptoms like cough, fatigue, and so on. We don't do blood tests on people with flu in routine practice, but when these tests are done – for surveillance purposes – then even during peak flu season only about one in three people with ‘flu' will actually be infected with the influenza virus, and most of the year only one in eight will really have it. (The rest are sick from something else, maybe just a common cold virus.)
Two thirds of the trial participants summarised in the Kaiser paper tested positive for flu. This is bizarrely high, and means that the benefits of the drug will be overstated, because it is being tested on perfect patients, the very ones most likely to get better from a drug that selectively attacks the flu virus. In normal practice, which is where the results of these trials will be applied, doctors will be giving the drug to real patients who are diagnosed with ‘flu-like illness', which is all you can realistically do in a clinic. Among these real patients, many will not actually have the influenza virus. This means that in the real world, the benefits of Tamiflu on flu will be diluted, and many more people will be exposed to the drug who don't actually have flu virus in their systems. This, in turn, means that the side effects are likely to creep up in significance, in comparison with any benefits. That is why we strive to ensure that all trials are conducted in normal, everyday, realistic patients: if they are not, their findings may not be relevant to the real world.
So the Cochrane review was published without the Kaiser data in December 2009, alongside some explanatory material about why the Kaiser results had been excluded, and a small flurry of activity followed. Roche put the short excerpts it had sent over online, and committed to make full study reports available (it still hasn't done so).
What Roche posted was incomplete, but it began a journey for the Cochrane academics of learning a great deal more about the real information that is collected on a trial, and how that can differ from what is given to doctors and patients in the form of brief, published academic papers. At the core of every trial is the raw data: every single record of blood pressure of every patient, the doctors' notes describing any unusual symptoms, investigators' notes, and so on. A published academic paper is a short description of the study, usually following a set format: an introductory background; a description of the methods; a summary of the important results; and then finally a discussion, covering the strengths and weaknesses of the design, and the implications of the results for clinical practice.
A clinical study report, or CSR, is the intermediate document that stands between these two, and can be very long, sometimes thousands of pages. 78 Anybody working in the pharmaceutical industry is very familiar with these documents, but doctors and academics have rarely heard of them. They contain much more detail on things like the precise plan for an*lysing the data statistically, detailed descriptions of adverse events, and so on.
These documents are split into different sections, or ‘modules'. Roche has shared only ‘module 1', for only seven of the ten study reports Cochrane has requested. These modules are missing vitally important information, including the an*lysis plan, the randomisation details, the study protocol (and the list of deviations from that), and so on. But even these incomplete modules were enough to raise concerns about the universal practice of trusting academic papers to give a complete story about what happened to the patients in a trial.
For example, looking at the two papers out of ten in the Kaiser review which were published, one says: ‘There were no drug-related serious adverse events,' and the other doesn't mention adverse events. But in the ‘module 1' documents on these same two studies, there are ten serious adverse events listed, of which three are cla**ified as being possibly related to Tamiflu. 79
Another published paper describes itself as a trial comparing Tamiflu against placebo. A placebo is an inert tablet, containing no active ingredient, that is visually indistinguishable from the pill containing the real medicine. But the CSR for this trial shows that the real medicine was in a grey and yellow capsule, whereas the placebos were grey and ivory. The ‘placebo' tablets also contained something called dehydrocholic acid, a chemical which encourages the gall bladder to empty. 80 Nobody has any clear idea of why, and it's not even mentioned in the academic paper; but it seems that this was not actually an inert, dummy pill placebo.
Simply making a list of all the trials conducted on a subject is vitally important if we want to avoid seeing only a biased summary of the research done on a subject; but in the case of Tamiflu even this proved to be almost impossible. For example, Roche Shanghai informed the Cochrane group of one large trial (ML16369), but Roche Basel seemed not to know of its existence. But by setting out all the trials side by side, the researchers were able to identify peculiar discrepancies: for example, the largest ‘phase 3' trial – one of the large trials that are done to get a drug onto the market – was never published, and is rarely mentioned in regulatory documents. fn3
There were other odd discrepancies. Why, for example, was one trial on Tamiflu published in 2010, ten years after it was completed? 82 Why did some trials report completely different authors, depending on where they were being discussed? 83 And so on.
The chase continued. In December 2009 Roche had promised: ‘full study reports will also be made available on a pa**word-protected site within the coming days to physicians and scientists undertaking legitimate an*lyses'. This never happened. Then an odd game began. In June 2010 Roche said: Oh, we're sorry, we thought you had what you wanted. In July it announced that it was worried about patient confidentiality (you may remember this from the EMA saga). This was an odd move: for most of the important parts of these documents, privacy is no issue at all. The full trial protocol, and the an*lysis plan, are both completed before any single patient is ever touched. Roche has never explained why patient privacy prevents it from releasing the study reports. It simply continued to withhold them.
Then in August 2010 it began to make some even more bizarre demands, betraying a disturbing belief that companies are perfectly entitled to control access to information that is needed by doctors and patients around the world to make safe decisions. Firstly, it insisted on seeing the Cochrane reviewers' full an*lysis plan. Fine, they said, and posted the whole protocol online. Doing so is completely standard practice at Cochrane, as it should be for any transparent organisation, and allows people to suggest important changes before you begin. There were few surprises, since all Cochrane reports follow a pretty strict manual anyway. Roche continued to withhold its study reports (including, ironically, its own protocols, the very thing it demanded Cochrane should publish, and that Cochrane had published, happily).
By now Roche had been refusing to publish the study reports for a year. Suddenly, the company began to raise odd personal concerns. It claimed that some Cochrane researchers had made untrue statements about the drug, and about the company, but refused to say who, or what, or where. ‘Certain members of Cochrane Group involved with the review of the neuraminidase inhibitors,' it announced, ‘are unlikely to approach the review with the independence that is both necessary and justified.' This is an astonishing state of affairs, where a company feels it should be allowed to prevent individual researchers access to data that should be available to all; but still Roche refused to hand over the study reports.
Then it complained that the Cochrane reviewers had begun to copy journalists in on their emails when responding to Roche staff. I was one of the people copied in on these interactions, and I believe that this was exactly the correct thing to do. Roche's excuses had become perverse, and the company had failed to keep its promise to share all study reports. It's clear that the modest pressure exerted by researchers in academic journals alone was having little impact on Roche's refusal to release the data, and this is an important matter of public health, both for the individual case of this Tamiflu data, and for the broader issue of companies and regulators harming patients by withholding information.
Then things became even more perverse. In January 2011 Roche announced that the Cochrane researchers had already been given all the data they need. This was simply untrue. In February it insisted that all the studies requested were published (meaning academic papers, now shown to be misleading on Tamiflu). Then it declared that it would hand over nothing more, saying: ‘You have all the detail you need to undertake a review.' But this still wasn't true: it was still withholding the material it had publicly promised to hand over ‘within a few days' in December 2009, a year and a half earlier.
At the same time, the company was raising the broken arguments we have already seen: it's the job of regulators to make these decisions about benefit and risk, it said, not academics. Now, this claim fails on two important fronts. Firstly, as with many other d**, we now know that not even the regulators had seen all the data. In January 2012 Roche claimed that it ‘has made full clinical study data available to health authorities around the world for their review as part of the licensing process'. But the EMA never received this information for at least fifteen trials. This was because the EMA had never requested it.
And that brings us on to our final important realisation: regulators are not infallible. They make outright mistakes, and they make decisions which are open to judgement, and should be subject to second-guessing and checking by many eyes around the world. In the next chapter we will see more examples of how regulators can fail, behind closed doors, but here we will look at one story that illustrates the benefit of ‘many eyes' perfectly.
Rosiglitazone is a new kind of diabetes drug, and lots of researchers and patients had high hopes that it would be safe and effective. 84 Diabetes is common, and more people develop the disease every year. Sufferers have poor control of their blood sugar, and diabetes d**, alongside dietary changes, are supposed to fix this. Although it's nice to see your blood sugar being controlled nicely in the numbers from lab tests and machines at home, we don't control these figures for their own sake: we try to control blood sugar because we hope that this will help reduce the chances of real-world outcomes, like heart attack and d**h, both of which occur at a higher rate in people with diabetes.
Rosiglitazone was first marketed in 1999, and from the outset it was a magnet for disappointing behaviour. In that first year, Dr John Buse from the University of North Carolina discussed an increased risk of heart problems at a pair of academic meetings. The drug's manufacturer, GSK, made direct contact in an attempt to silence him, then moved on to his head of department. Buse felt pressured to sign various legal documents. To cut a long story short, after wading through documents for several months, in 2007 the US Senate Committee on Finance released a report describing the treatment of Dr Buse as ‘intimidation'.
But we are more concerned with the safety and efficacy data. In 2003 the Uppsala Drug Monitoring Group of the World Health Organization contacted GSK about an unusually large number of spontaneous reports a**ociating rosiglitazone with heart problems. GSK conducted two internal meta-an*lyses of its own data on this, in 2005 and 2006. These showed that the risk was real, but although both GSK and the FDA had these results, neither made any public statement about them, and they were not published until 2008.
During this delay, vast numbers of patients were exposed to the drug, but doctors and patients only learned about this serious problem in 2007, when cardiologist Professor Steve Nissen and colleagues published a landmark meta-an*lysis. This showed a 43 per cent increase in the risk of heart problems in patients on rosiglitazone. Since people with diabetes are already at increased risk of heart problems, and the whole point of treating diabetes is to reduce this risk, that finding was big potatoes. His findings were confirmed in later work, and in 2010 the drug was either taken off the market or restricted, all around the world.
Now, my argument is not that this drug should have been banned sooner, because as perverse as it sounds, doctors do often need inferior d** for use as a last resort. For example, a patient may develop idiosyncratic side effects on the most effective pills, and be unable to take them any longer. Once this has happened, it may be worth trying a less effective drug, if it is at least better than nothing.
The concern is that these discussions happened with the data locked behind closed doors, visible only to regulators. In fact, Nissen's an*lysis could only be done at all because of a very unusual court judgement. In 2004, when GSK was caught out withholding data showing evidence of serious side effects from paroxetine in children, the UK conducted an unprecedented four-year-long investigation, as we saw earlier. But in the US, the same bad behaviour resulted in a court case over allegations of fraud, the settlement of which, alongside a significant payout, required GSK to commit to posting clinical trial results on a public website.
Professor Nissen used the rosiglitazone data, when it became available, found worrying signs of harm, and published this to doctors, which is something that the regulators had never done, despite having the information years earlier. (Though before doctors got to read it, Nissen by chance caught GSK discussing a copy of his unpublished paper, which it had obtained improperly. 85)
If this information had all been freely available from the start, regulators might have felt a little more anxious about their decisions, but crucially, doctors and patients could have disagreed with them, and made informed choices. This is why we need wider access to full CSRs, and all trial reports, for all medicines, and this is why it is perverse that Roche should be able even to contemplate deciding which favoured researchers should be allowed to read the documents on Tamiflu.
Astonishingly, a paper published in April 2012 by regulators from the UK and Europe suggests that they might agree to more data sharing, to a limited extent, within limits, for some studies, with caveats, at the appropriate juncture, and in the fullness of time. 86 Before feeling any sense of enthusiasm, we should remember that this is a cautious utterance, wrung out after the dismal fights I have already described; that it has not been implemented; that it must be set against a background of broken promises from all players across the whole field of missing data; and that in any case, regulators do not have all the trial data anyway. But it is an interesting start.
Their two main objections – if we accept their goodwill at face value – are interesting, because they lead us to the final problem in the way we tolerate harm to patients from missing trial data. Firstly, they raise the concern that some academics and journalists might use study reports to conduct histrionic or poorly conducted reviews of the data: to this, again, I say, ‘Let them,' because these foolish an*lyses should be conducted, and then rubbished, in public.
When UK hospital mortality statistics first became easily accessible to the public, doctors were terrified that they would be unfairly judged: the crude figures can be misinterpreted, after all, because one hospital may have worse figures simply because it is a centre of excellence, and takes in more challenging patients than its neighbours; and there is random variation to be expected in mortality rates anyway, so some hospitals might look unusually good, or bad, simply through the play of chance. Initially, to an extent, these fears were realised: there were a few shrill, unfair stories, and people overinterpreted the results. Now, for the most part, things have settled down, and many lay people are quite able to recognise that crude an*lyses of such figures are misleading. For drug data, where there is so much danger from withheld information, and so many academics desperate to conduct meaningful an*lyses, and so many other academics happy to criticise them, releasing the data is the only healthy option.
But secondly, the EMA raises the spectre of patient confidentiality, and hidden in this concern is one final prize.
So far I have been talking about access to trial reports, summaries of patients' outcomes in trials. There is no good reason to believe that this poses any threat to patient confidentiality, and where there are specific narratives that might make a patient identifiable – a lengthy medical description of one person's idiosyncratic adverse event in a trial, perhaps – these can easily be removed, since they appear in a separate part of the document. These CSRs should undoubtedly, without question, be publicly available documents, and this should be enforced retrospectively, going back decades, to the dawn of trials.
But all trials are ultimately run on individual patients, and the results of those individual patients are all stored and used for the summary an*lysis at the end of the study. While I would never suggest that these should be posted up on a public website – it would be easy for patients to be identifiable, from many small features of their histories – it is surprising that patient-level data is almost never shared with academics.
Sharing data of individual patients' outcomes in clinical trials, rather than just the final summary result, has several significant advantages. Firstly, it's a safeguard against dubious an*lytic practices. In the VIGOR trial on the paink**er Vioxx, for example, a bizarre reporting decision was made. 87 The aim of the study was to compare Vioxx against an older, cheaper paink**er, to see if it was any less likely to cause stomach problems (this was the hope for Vioxx), and also if it caused more heart attacks (this was the fear). But the date cut-off for measuring heart attacks was much earlier than that for measuring stomach problems. This had the result of making the risks look less significant, relative to the benefits, but it was not declared clearly in the paper, resulting in a giant scandal when it was eventually noticed. If the raw data on patients was shared, games like these would be far easier to spot, and people might be less likely to play them in the first place.
Occasionally – with vanishing rarity – researchers are able to obtain raw data, and re-an*lyse studies that have already been conducted and published. Daniel Coyne, Professor of Medicine at Washington University, was lucky enough to get the data on a key trial for epoetin, a drug given to patients on kidney dialysis, after a four-year-long fight. 88 The original academic publication on this study, ten years earlier, had switched the primary outcomes described in the protocol (we will see later how this exaggerates the benefits of treatments), and changed the main statistical an*lysis strategy (again, a huge source of bias). Coyne was able to an*lyse the study as the researchers had initially stated they were planning to in their protocol; and when he did, he found that they had dramatically overstated the benefits of the drug. It was a peculiar outcome, as he himself acknowledges: ‘As strange as it seems, I am now the sole author of the publication on the predefined primary and secondary results of the largest outcomes trial of epoetin in dialysis patients, and I didn't even participate in the trial.' There is room, in my view, for a small army of people doing the very same thing, rean*lysing all the trials that were incorrectly an*lysed, in ways that deviated misleadingly from their original protocols.
Data sharing would also confer other benefits. It allows people to conduct more exploratory an*lyses of data, and to better investigate – for example – whether a drug is a**ociated with a particular unexpected side effect. It would also allow cautious ‘subgroup an*lyses', to see if a drug is particularly useful, or particularly useless, in particular types of patients.
The biggest immediate benefit from data sharing is that combining individual patient data into a meta-an*lysis gives more accurate results than working with the crude summary results at the end of a paper. Let's imagine that one paper reports survival at three years as the main outcome for a cancer drug, and another reports survival at seven years. To combine these two in a meta-an*lysis, you'd have a problem. But if you were doing the meta-an*lysis with access to individual patient data, with treatment details and d**h dates for all of them, you could do a clean combined calculation for three-year survival.
Darby S, McGale P, Correa C, Taylor C, Arriagada R, Clarke M, Cutter D, Davies C, Ewertz M, Godwin J, Gray R, Pierce L, Whelan T, Wang Y, Peto R.Albain K, Anderson S, Arriagada R, Barlow W, Bergh J, Bliss J, Buyse M, Cameron D, Carrasco E, Clarke M, Correa C, Coates A, Collins R, Costantino J, Cutter D, Cuzick J, Darby S, Davidson N, Davies C, Davies K, Delmestri A, Di Leo A, Dowsett M, Elphinstone P, Evans V, Ewertz M, Gelber R, Gettins L, Geyer C, Goldhirsch A, Godwin J, Gray R, Gregory C, Hayes D, Hill C, Ingle J, Jakesz R, James S, Kaufmann M, Kerr A, MacKinnon E, McGale P, McHugh T, Norton L, Ohashi Y, Paik S, Pan HC, Perez E, Peto R, Piccart M, Pierce L, Pritchard K, Pruneri G, Raina V, Ravdin P, Robertson J, Rutgers E, Shao YF, Swain S, Taylor C, Valagussa P, Viale G, Whelan T, Winer E, Wang Y, Wood W, Abe O, Abe R, Enomoto K, Kikuchi K, Koyama H, Masuda H, Nomura Y, Ohashi Y, Sakai K, Sugimachi K, Toi M, Tominaga T, Uchino J, Yoshida M, Haybittle JL, Leonard CF, Calais G, Geraud P, Collett V, Davies C, Delmestri A, Sayer J, Harvey VJ, Holdaway IM, Kay RG, Mason BH, Forbes JF, Wilcken N, Bartsch R, Dubsky P, Fesl C, Fohler H, Gnant M, Greil R, Jakesz R, Lang A, Luschin-Ebengreuth G, Marth C, Mlineritsch B, Samon** H, Singer CF, Steger GG, Stöger H, Canney P, Yosef HM, Focan C, Peek U, Oates GD, Powell J, Durand M, Mauriac L, Di Leo A, Dolci S, Larsimont D, Nogaret JM, Philippson C, Piccart MJ, Masood MB, Parker D, Price JJ, Lindsay MA, Mackey J, Martin M, Hupperets PS, Bates T, Blamey RW, Chetty U, Ellis IO, Mallon E, Morgan DA, Patnick J, Pinder S, Olivotto I, Ragaz J, Berry D, Broadwater G, Cirrincione C, Muss H, Norton L, Weiss RB, Abu-Zahra HT, Portnoj SM, Bowden S, Brookes C, Dunn J, Fernando I, Lee M, Poole C, Rea D, Spooner D, Barrett-Lee PJ, Mansel RE, Monypenny IJ, Gordon NH, Davis HL, Cuzick J, Lehingue Y, Romestaing P, Dubois JB, Delozier T, Griffon B, Mace Lesec'h J, Rambert P, Mustacchi G, Petruzelka, Pribylova O, Owen JR, Harbeck N, Jänicke F, Meisner C, Schmitt M, Thomssen C, Meier P, Shan Y, Shao YF, Wang X, Zhao DB, Chen ZM, Pan HC, Howell A, Swindell R, Burrett JA, Clarke M, Collins R, Correa C, Cutter D, Darby S, Davies C, Davies K, Delmestri A, Elphinstone P, Evans V, Gettins L, Godwin J, Gray R, Gregory C, Hermans D, Hicks C, James S, Kerr A, MacKinnon E, Lay M, McGale P, McHugh T, Sayer J, Taylor C, Wang Y, Albano J, de Oliveira CF, Gervásio H, Gordilho J, Johansen H, Mouridsen HT, Gelman RS, Harris JR, Hayes D, Henderson C, Shapiro CL, Winer E, Christiansen P, Ejlertsen B, Ewertz M, Jensen MB, Møller S, Mouridsen HT, Carstensen B, Palshof T, Trampisch HJ, Dalesio O, de Vries EG, Rodenhuis S, van Tinteren H, Comis RL, Davidson NE, Gray R, Robert N, Sledge G, Solin LJ, Sparano JA, Tormey DC, Wood W, Cameron D, Chetty U, Dixon JM, Forrest P, Jack W, Kunkler I, Rossbach J, Klijn JG, Treurniet-Donker AD, van Putten WL, Rotmensz N, Veronesi U, Viale G, Bartelink H, Bijker N, Bogaerts J, Cardoso F, Cufer T, Julien JP, Rutgers E, van de Velde CJ, Cunningham MP, Huovinen R, Joensuu H, Costa A, Tinterri C, Bonadonna G, Gianni L, Valagussa P, Goldstein LJ, Bonneterre J, Fargeot P, Fumoleau P, Kerbrat P, Luporsi E, Namer M, Eiermann W, Hilfrich J, Jonat W, Kaufmann M, Kreienberg R, Schumacher M, Bastert G, Rauschecker H, Sauer R, Sauerbrei W, Schauer A, Schumacher M, Blohmer JU, Costa SD, Eidtmann H, Gerber G, Jackisch C, Loibl S, von Minckwitz G, de Schryver A, Vakaet L, Belfiglio M, Nicolucci A, Pellegrini F, Pirozzoli MC, Sacco M, Valentini M, McArdle CS, Smith DC, Stallard S, Dent DM, Gudgeon CA, Hacking A, Murray E, Panieri E, Werner ID, Carrasco E, Martin M, Segui MA, Galligioni E, Lopez M, Erazo A, Medina JY, Horiguchi J, Takei H, Fentiman IS, Hayward JL, Rubens RD, Skilton D, Scheurlen H, Kaufmann M, Sohn HC, Untch M, Dafni U, Markopoulos C, Dafni D, Fountzilas G, Mavroudis D, Klefstrom P, Saarto T, Gallen M, Margreiter R, de Lafontan B, Mihura J, Roché H, Asselain B, Salmon RJ, Vilcoq JR, Arriagada R, Bourgier C, Hill C, Koscielny S, Laplanche A, Lê MG, Spielmann M, A'Hern R, Bliss J, Ellis P, Kilburn L, Yarnold JR, Benraadt J, Kooi M, van de Velde AO, van Dongen JA, Vermorken JB, Castiglione M, Coates A, Colleoni M, Collins J, Forbes J, Gelber RD, Goldhirsch A, Lindtner J, Price KN, Regan MM, Rudenstam CM, Senn HJ, Thuerlimann B, Bliss JM, Chilvers CE, Coombes RC, Hall E, Marty M, Buyse M, Possinger K, Schmid P, Untch M, Wallwiener D, Foster L, George WD, Stewart HJ, Stroner P, Borovik R, Hayat H, Inbar MJ, Robinson E, Bruzzi P, Del Mastro L, Pronzato P, Sertoli MR, Venturini M, Camerini T, De Palo G, Di Mauro MG, Formelli F, Valagussa P, Amadori D, Martoni A, Pannuti F, Camisa R, Cocconi G, Colozza A, Pa**alacqua R, Aogi K, Takashima S, Abe O, Ikeda T, Inokuchi K, Kikuchi K, Sawa K, Sonoo H, Korzeniowski S, Skolyszewski J, Ogawa M, Yamash**a J, Bastiaannet E, van de Velde CJ, van de Water W, van Nes JG, Christiaens R, Neven P, Paridaens R, Van den Bogaert W, Braun S, Janni W, Martin P, Romain S, Janauer M, Seifert M, Sevelda P, Zielinski CC, Hakes T, Hudis CA, Norton L, Wittes R, Giokas G, Kondylis D, Lissaios B, de la Huerta R, Sainz MG, Altemus R, Camphausen K, Cowan K, Danforth D, Lichter A, Lippman M, O'Shaughnessy J, Pierce LJ, Steinberg S, Venzon D, Zuj**ski JA, D'Amico C, Lioce M, Paradiso A, Chapman JA, Gelmon K, Goss PE, Levine MN, Meyer R, Parulekar W, Pater JL, Pritchard KI, Shepherd LE, Tu D, Whelan T, Nomura Y, Ohno S, Anderson A, Ba** G, Brown A, Bryant J, Costantino J, Dignam J, Fisher B, Geyer C, Mamounas EP, Paik S, Redmond C, Swain S, Wickerham L, Wolmark N, Baum M, Jackson IM, Palmer MK, Perez E, Ingle JN, Suman VJ, Bengtsson NO, Emdin S, Jonsson H, Del Mastro L, Venturini M, Lythgoe JP, Swindell R, Kissin M, Erikstein B, Hannisdal E, Jacobsen AB, Varhaug JE, Erikstein B, Gundersen S, Hauer-Jensen M, Høst H, Jacobsen AB, Nissen-Meyer R, Blamey RW, Mitchell AK, Morgan DA, Robertson JF, Ueo H, Di Palma M, Mathé G, Misset JL, Levine M, Pritchard KI, Whelan T, Morimoto K, Sawa K, Takatsuka Y, Crossley E, Harris A, Talbot D, Taylor M, Martin AL, Roché H, Cocconi G, di Blasio B, Ivanov V, Paltuev R, Semiglazov V, Brockschmidt J, Cooper MR, Falkson CI, A'Hern R, Ashley S, Dowsett M, Makris A, Powles TJ, Smith IE, Yarnold JR, Gazet JC, Browne L, Graham P, Corcoran N, Deshpande N, di Martino L, Douglas P, Hacking A, Høst H, Lindtner A, Notter G, Bryant AJ, Ewing GH, Firth LA, Krushen-Kosloski JL, Nissen-Meyer R, Anderson H, k**ander F, Malmström P, Rydén L, Arnesson LG, Carstensen J, Dufmats M, Fohlin H, Nordenskjöld B, Söderberg M, Carpenter JT, Murray N, Royle GT, Simmonds PD, Albain K, Barlow W, Crowley J, Hayes D, Gralow J, Green S, Hortobagyi G, Livingston R, Martino S, Osborne CK, Adolfsson J, Bergh J, Bondesson T, Celebioglu F, Dahlberg K, Fornander T, Fredriksson I, Frisell J, Göransson E, Iiristo M, Johansson U, Lenner E, Löfgren L, Nikolaidis P, Perbeck L, Rotstein S, Sandelin K, Skoog L, Svane G, af Trampe E, Wadström C, Castiglione M, Goldhirsch A, Maibach R, Senn HJ, Thürlimann B, Hakama M, Holli K, Isola J, Rouhento K, Saaristo R, Brenner H, Hercbergs A, Martin AL, Roché H, Yoshimoto M, Paterson AH, Pritchard KI, Fyles A, Meakin JW, Panzarella T, Pritchard KI, Bahi J, Reid M, Spittle M, Bishop H, Bundred NJ, Cuzick J, Ellis IO, Fentiman IS, Forbes JF, Forsyth S, George WD, Pinder SE, Sestak I, Deutsch GP, Gray R, Kwong DL, Pai VR, Peto R, Senanayake F, Boccardo F, Rubagotti A, Baum M, Forsyth S, Hackshaw A, Houghton J, Ledermann J, Monson K, Tobias JS, Carlomagno C, De Laurentiis M, De Placido S, Williams L, Hayes D, Pierce LJ, Broglio K, Buzdar AU, Love RR, Ahlgren J, Garmo H, Holmberg L, Liljegren G, Lindman H, Wärnberg F, Asmar L, Jones SE, Gluz O, Harbeck N, Liedtke C, Nitz U, Litton A, Wallgren A, Karlsson P, Linderholm BK, Chlebowski RT, Caffier H.
This is exactly the kind of work being done in the area of breast cancer research, where a small number of charismatic and forceful scientists just happen to have driven a pioneering culture of easier collaboration. The summaries they are publishing represent real collaboration, between vast numbers of people, and on vast numbers of patients, producing highly reliable guidance for doctors and patients.
The process sheds a stark light on the reality of data collaboration on such a large scale. Here, for example, is the author list on an academic paper from the Lancet in November 2011: it's reporting an immense, definitive, and incredibly useful metaan*lysis of breast cancer treatment outcomes, using individual patient data pooled from seventeen different trials. The author list is printed in four-point font size (though I suspect that might go wrong in the e-book edition …) because there are seven hundred individual researchers named in it. I typed each of them in by hand for you.
This is what medicine should look like. An honest list of all the people involved, free access to information, and all the data pooled together, giving the most accurate information we can manage, to inform real decisions, and so prevent avoidable suffering and d**h.
We are a very, very long way away from there.
What can be done?
We urgently need to improve access to trial data. In addition to the previous suggestions, there are small changes that would vastly improve access to information, and so improve patient care.
1.The results of all trials conducted on humans must be made available within one year of completion, in summary table form if academic journal publication has not occurred. This requires the creation of a body that is charged with publicly auditing whether or not trials have withheld results at twelve months; and primary legislation that is enforced, as a matter of urgency, internationally, with stiff penalties for transgression. In my view these penalties should include fines, but also prison terms for those who are found to be responsible for withholding trial data, as patients are harmed by this process.
2.All systematic reviews – such as Cochrane reviews – that draw together trial results on any clinical question should also include a section on the trials which they know have been conducted, but whose results are being withheld. This should state: which completed trials have not reported results; how many patients' worth of information there is in each unreported trial; the names of the organisations and the individuals who are withholding the data; the efforts the reviewers have made to get the information from them. This is trivial extra work, as review teams already attempt to access this kind of data. Documenting this will draw attention to the problem, and make it easier for doctors and the public to see who is responsible for harming patient care in each area of medicine.
3.All clinical study reports should also be made publicly available, for all the trials that have ever been conducted on humans. This will be cheap, as the only costs are in finding one paper copy, scanning it and placing it online, perhaps with a check to remove confidential patient information. There is a vast mountain of highly relevant data on d** that is currently being withheld, distorting what we know about treatments in widespread current use. Many of these documents will be sitting in the dry paper archives of drug companies and regulators. We need legislation compelling the industry to hand them over. Our failure to fix this is costing lives.
4.We need to work on new methods for academics to extract summary information from these documents, as they are more detailed than published academic papers. The Cochrane group working on Tamiflu have made great progress here, learning as they go, and this field will need manuals.
5.We should work towards all triallists having an obligation to share patient-level data wherever possible, with convenient online data warehouses, 89 and streamlined systems whereby legitimate senior researchers can make requests for access, in order to conduct pooled an*lyses and double-check the results reported in published trials.
None of this is difficult, or impossible. Some of it is technical, for which I apologise. The field of missing data is a tragic and strange one. We have tolerated the emergence of a culture in medicine where information is routinely withheld, and we have blinded ourselves to the unnecessary suffering and d**h that follows from this. The people we should have been able to trust to handle all this behind the scenes – the regulators, the politicians, the senior academics, the patient organisations, the professional bodies, the universities, the ethics committees – have almost all failed us. And so I have had to inflict the details on you, in the hope that you can bring some pressure to bear yourself.
If you have any ideas about how we can fix this, and how we can force access to trial data – politically or technically – please write them up, post them online, and tell me where to find them.