|
You could say that large clinical trials have an image problem. With multiple authors, multiple centers, and research that goes on for years, large trials are seen as the not-so-sexy cousin to the short and sweet retrospective analysis.
Speaking to the politics of the issue is Dr. Paul Tornetta, Professor and Vice Chairman in the Department of Orthopaedic Surgery at Boston University School of Medicine, “We have a real issue in orthopedics with the difference between surgeons who participate in substantial, long-term trials and those who take a shortcut towards recognition in the field. Let’s say you are a junior-level surgeon in an academic center and your boss wants you to publish; you want a promotion. In such a situation, the easiest thing to do is a retrospective analysis of your cases, with a report on the outcomes, complication rates, and surgical techniques. This can be done in approximately 10 hours and you can get residents to help. Boom…you are the first author on a peer-reviewed publication.”
“Contrast that with the SPRINT trial,” says Dr. Tornetta, a lead investigator on SPRINT. “There were 24 principal investigators at 24 sites. In the end, surgeons who provided patients and donated long hours to this group effort only have their names at the bottom or in the index. Chairs of some orthopedic departments are contributing to the discrepancy between types of studies as they encourage junior faculty not to do large-scale collaborative efforts. And journal editors often fail to recognize the importance of these collaborations as they have rules limiting the number of authors who can be listed on a publication. After years of hard work, that seems patently unfair.”
Giving an example of the complexity involved in multicenter trials, Dr. Michael McKee, an orthopedic surgeon at St. Michael’s Hospital in Toronto, notes, “We belong to the Canadian Orthopaedic Trauma Society, a loose network of 15 trauma centers that collaborate on randomized trials in multiple centers. The process is that someone in the group has an idea for a trial, brings it to the rest of the group, and we all discuss its merits. Around the table we have people with multiple areas of expertise, including statistical training, epidemiological training, expertise in fracture care, etc. The individual promoting the idea finalizes the protocol and the approval process begins. Eventually four or five centers pull through, but this all takes a significant investment of time and money. Just to get to the starting line on the trials usually takes from one to one-and-a-half years.”
But you get much more reliable data than in a smaller, short-term study. Explains Dr. McKee, “The problem with, say, a retrospective analysis is that you usually get one person’s viewpoint, which is often biased. And because it is a retrospective view, follow-up and outcome measures are incomplete. Let’s say you are studying elbow pain. There is no real starting point and you end up with, for example, a score of 85, but you really don’t know what that means. Also troublesome is that the procedure may have been done using multiple protocols. One person operates on a joint and there is some immediate motion; other patients have been immobilized for four weeks, but you don’t know that. In a retrospective review all the outcomes get jumbled together and nothing is standardized. Taking the SPRINT study as a contrast, everything was standardized so that the only difference between the two groups was reamed versus unreamed nails. Another issue with retrospective reviews is that there is never enough to prove much. You have the beta error in which there is a failure to demonstrate a true difference in the experiment because there are too few patients to prove a statistical difference. If, for example, the range of motion in group A that was treated with a certain device improves by a 14 degree better range motion than in group B and the p-value is .2, that is not a significant difference. That study may have had only 30 patients and was thus underpowered.”
As is true in many areas of life, he who holds the bullhorn gets the attention. Dr. McKee: “There is so much information disseminated at conferences and not all of it is reliable. In the past, audiences were naïve regarding epidemiology and study design, but certain researchers have done a great job of educating the orthopedic community about what these statistics really mean. As far as the credibility of presentations, the way to get ‘bad’ people off the podium is to put ‘good’ people on the podium. Conferences should be full of information on randomized, professionally run prospective trials, as opposed to presentations where someone says, ‘We did 35 of these new xyz procedures and they are the greatest thing ever.’ But who ends up on the podium depends on who the program committee members are and their levels of sophistication.”
Although a champion of large trials, Dr. McKee is also a realist. “Prospective studies are extremely difficult to do and not everything lends itself to randomized trials. They are unique and complex and hard to organize. All of the doctors must agree on the protocol and the expenses involved are at least $100,000…and that is when people have the infrastructure in place.”
Speaking to how actionable certain data can be is Dr. Kurt Spindler, Professor and Vice-Chairman of the Department of Orthopaedics and Rehabilitation at Vanderbilt University Medical School. “If you undertake a retrospective review it just doesn’t give you a solid level of confidence by which to make clinical decisions. Multicenter trials are so well organized and sophisticated that the information resulting from them actually changes the way surgeons practice. It is a major problem that publication committees don’t recognize this and that editors won’t allow all of the authors on a paper to have their names listed. And yet with a smaller, less powerful study, the author is getting credit on his or her CV and in the literature for being first author. This also gives him or her more weight to vie for a promotion.”
And because most orthopedic surgeons are not researchers, they often don’t comprehend the importance of the type of study with which they are being presented. Dr. Spindler: “The majority of orthopedic surgeons don’t understand how to use levels of evidence and quality of study design in making decisions. There is a fundamental lack of education on these issues. We need to reach the point of, when a presenter gets on the podium and says, ‘This is my study and what we found,’ that someone in the audience asks about the level of evidence. Too often it was just a case series or, even worse, someone’s opinion. If the speaker doesn’t reveal the level of evidence then the audience tends to think what the presenter says is gospel and that all studies have the same level of rigor.”
Musing about the wider effects of this issue, Dr. Spindler notes, “Unless orthopedics develops good prospective multicenter trials then the field will fall further behind disciplines that do this well, such as cardiology and rheumatology. They have good networks of clinical outcomes trials and lots of NIH funding. A future of few multicenter trials will mean that our treatments will be more experienced-based and that the discipline will not prosper. There will be less reimbursement for physicians and less remuneration for companies because solid evidence for techniques, surgeries, and devices will be absent.”
He concludes, “What with all the talk about conflicts of interest, there has been a great push in the field to become more transparent. Here is another opportunity to introduce the light of truth. The ultimate question is, ‘Can a young field of medicine be a truly great discipline?’ We can’t do that without looking at outcomes, doing large prospective multicenter trials, and having the funding to do so. Tens of millions of dollars are spent on trials in cardiology and cancer research and the result is that you have very reliable, actionable information. These fields have grown up.”
|