When US federal regulators started giving certain drugs "breakthrough" status in 2012, pharmaceutical company interest exploded.
The goal was to speed up the approval of drugs for serious diseases when there was preliminary evidence the new drugs were better than existing treatment options.
Giving these drugs "breakthrough" status comes with a side benefit: an evocative name.
The word "breakthrough" suggests scientific triumphs and miracle cures to many people, including physicians. Companies and media reports often tout the "breakthrough" designation for experimental drugs that have not yet been proved effective.
A new study published in the Journal of the American Medical Association digs deeply into the studies behind the flood of "breakthrough" therapies that have been approved by the Food and Drug Administration over the past six years and finds many of the trials that supported approval lacked the strongest kind of medical evidence.
"These drugs are being accelerated through development, and as this study shows, they are being tested on the basis of far less rigorous studies - studies that have substantial limitations to them in terms of their scientific rigor," said Aaron Kesselheim, an associate professor of medicine at Harvard Medical School.
Kesselheim, who was not involved in the study, thinks the name of the "breakthrough" designation should be changed.
Typically, researchers conducting clinical trials take steps, such as randomly assigning patients to groups that receive the drug or a placebo, but no randomized trial was used for 40 percent of the approved drugs, the Yale School of Medicine study found.
Joseph Ross, associate professor of medicine at Yale, said the study shows that efforts to speed up drug approval have been successful but come with a trade-off: uncertainty.
Because the trials are shorter and smaller, the long-term risks and benefits - and whether the same results will be seen in a broad population of patients - are not fully understood.
"If we are going to be making this trade-off to allow novel drugs to come to market on the basis of evidence that is generally accompanied by greater uncertainty, we must be committed as a clinical and scientific community to ensuring that high-quality, rigorous post-market trials are conducted within a reasonable period," Ross said.
Janet Woodcock, director of the Center for Drug Evaluation and Research at the FDA, said in an interview the study's authors failed to take into account the strength of the evidence and looked too narrowly at the academic question of how the trials were designed. She said drugs with dramatic treatment effects or drugs that can be targeted to those who are likely to respond may not need the same type of trial design to demonstrate they work.
"This is probably the trajectory of drug development, as the science advances so much," Woodcock said. "What has been traditional drug development is changing, and I know that's causing discomfort, but we're very confident the drugs out there are making tremendous" benefit for patients.
A Journal of Clinical Oncology study done this year evaluated cancer drugs that had received breakthrough therapy designation and found that although they were approved faster, there was no evidence the medicines were safer, more effective or more novel than drugs approved without the designation.
Kesselheim published a study two years ago that posed a hypothetical scenario to physicians, asking whether they would prescribe an FDA-designated breakthrough therapy or a drug with early promising results that had not yet been shown to improve survival.
It was a trick question, because breakthrough therapy means there is "preliminary clinical evidence" a drug might be better than existing therapies, according to the technical definition. Physicians overwhelmingly chose the one called a breakthrough.