The Foundation for Chiropractic Education and Research

Response to Meta-Analysis Published By Assendelft Et Al

By Anthony L. Rosner, Ph.D., LL.D. (Hon.)
June 25, 2003

The recent meta-analysis by Assendelft et al., published in the June 3, 2003 edition of the Annals of Internal Medicine [1] is a troubling example of how clinical data is to be interpreted and presented for public consumption and policy development. In addition to raising numerous issues as to the clinical applicability of meta-analyses, it may even belie its basic premises as will be pointed out below. In reviewing this document, one must remain vigilant as to how the rowdy qualities of human bias, subjectivity, and disagreement extend well into the rarefied atmospheres of randomized clinical trials, meta-analyses, and actual clinical guidelines.

The overall conclusion of this report—that there is no evidence that spinal manipulation therapy is superior to either standard treatments for patients with acute or chronic low back pain—can be interpreted in the same breath to indicate that, in terms of the pain or disability outcomes scales evaluated, it is neither inferior. Before one analyzes the methodological issues of the report itself, one is entirely justified to ask whether the treatments are truly equivalent.


1.   Comparative Side Effects and Relative Safety:

For spinal manipulation, the occurrence of major complications (regardless of the region of the spine manipulated) has generally been shown to be less than one per million. [2–5] Even transient, minor side-effects have been estimated to occur at 1 per 120,000 cervical manipulations. [6] These figures pale when compared to an extensive body of literature describing as many as 220,000 deaths and other complications produced in the United States each year by medications in general [7–14] or the 10,000–20,000 fatalities and multiple organ systems adversely affected by NSAIDs. [15–23] Even what has been regarded as the more relatively benign COX-2 inhibitors [24–27] and acetaminophen medications [28] have been described to generate serious GI, cardiovascular, and hepatic problems at rates orders of magnitude greater than side-effects attributed to spinal manipulation. The overall picture comparing spinal manipulation to the commonly used treatment alternatives of either direct analgesic ingestion or visits to the general practitioner (80% resulting in analgesic use by the authors’ own citation [1, 29] ) should be one of relative clarity to the patient: In one instance there is an option with a low rate of lasting side-effects and in the other a treatment regimen with severe and sometimes fatal complications that are inexplicably deemed to be "acceptable." [30]


2.   Mix of Clinical Judgment with Data from the Literature:

The authors strongly imply that this study is intended to be more rigorous than the systematic reviews and meta-analyses that preceded it. Yet their admission to the effect that the comparison of spinal manipulative therapy with each different treatment alternative for each outcome for each back pain stratum "was not possible because the data were sparse" raises one’s suspicion that this particular review may not have been as "systematic" as first presumed. These fears are confirmed in the very next sentence which informs the reader that the clinical judgment of effectiveness benchmarks from members of the Cochrane Editorial Board was used to fill in the gaps of experimental data, undermining the very process championed in this study. Indeed, the noted epidemiologist David Sackett applauds the use of clinical expertise as well as experimental outcomes data to build a truly effective evidence base for optimum patient care, [31] a sentiment echoed elsewhere as well. [32] But this undercuts the very process that the authors suggest that they are undertaking in pursuit of the most definitive experimental data available. In other words, how much adulteration of this "systematic review" has taken place?


3.   Inadmissible Criterion of Quality:

One of the criteria for methodologic quality of RCTs by the Cochrane Group—the blinding of the care provider (V3)—is impossible in the administration of manual therapy, particularly high-velocity spinal manipulation. Accordingly, its inclusion by the authors as a determinant of inclusion or rejection of RCTs is without justification. As many as 11 studies have erroneously reported double-blinding in the chiropractic experimental literature; the nonfeasibility of blinding the practitioner in numerous modalities of alternative medicine has been extensively discussed elsewhere and needs to be duly noted. [33]


4.   Guideline Rationale:

As part of their rationale for embarking upon this investigation, Assendelft et al. bemoan the disparity of recommendations for spinal manipulative therapy from different countries, citing in particular the dissension expressed in the guidelines from Australia, Israel, and The Netherlands. What the authors do not disclose is the preponderance of support for spinal manipulation expressed in 8 out of a total of 11 such guidelines, with perhaps an additional half thrown in for The Netherlands (which found sufficient justification for treating acute but not chronic back pain by spinal manipulation, [34] an oddity since van Tulder [who is Dutch] decisively supported the chronic over the acute evidence in a recent systematic review). [35] Furthermore, in the comparison of guidelines cited by the authors, there was concordance among all 11 nations (United States, United Kingdom, The Netherlands, Israel, New Zealand, Finland, Australia, Switzerland, Germany, Denmark, and Sweden) in six aspects of healthcare: (i) diagnostic triage, history taking, and physical examination; (ii) their conclusion that radiographs were not useful for managing nonspecific low back pain; (iii) their recognition of the importance of psychosocial factors; (iv) their discouragement of bed rest; (v) certain stipulations regarding the prescription of medications; and (vi) their concluding that the vast majority of low back pain cases should be managed in a primary care setting. Other areas besides spinal manipulation in which differences arose were (i) exercise therapy, (ii) muscle relaxants, and (iii) patient information.

The reason this census of nations regarding guideline and medical practices has been taken is to point out that the reported concordances and discordances do not appear to correlate with the amount, design, and quality of randomized clinical trials or systematic literature reviews that have been published. Rather, there appear to be human and cultural values at work here that I would maintain have not necessarily been eliminated in the study currently under discussion. This leads directly to our next point of critique.


5.   Meta-Analyses Themselves Are Subject to Bias and Omissions:

Regarding their clinical relevance, the very basis of meta-analyses (including the report of Assendelft et al.) has to be closely scrutinized. One report has gone so far as to compare meta-analyses to statistical alchemy, due to their intrinsic nature:

"...the removal and destruction of the scientific requirements that have been so carefully developed and established during the 19th and 20th centuries. In the mixtures formed for most statistical meta-analyses, we lose or eliminate the elemental scientific requirements for reproducibility and precision, for suitable extrapolation, and even sometimes for fair comparison." [36]

Specifically, Feinstein raises the following deficiencies of meta-analyses, most having to do with the sloughing of important clinical information:

i.   Disparate groups of patients of varying homogeneity across different studies are thrown into one analysis, often called a "mixed salad";

ii.   The weighting of studies of different quality may be inaccurate or absent altogether;

iii.   One needs to know about the real-world effects in the presentation and treatment of patients; in particular

(a) the severity of the illness,
(b) co-morbidities,
(c) pertinent co-therapies, and
(d) clinically relevant and meaningful outcomes;

iv.   Inconsistent statistical techniques pertaining to increments, effect size, correlation coefficients, and relative risk and odds ratios;

v.   Omission of the reference denominator; and

vi.   The fact that the odds ratio inflates the true value of the relative risk under certain conditions.

In any event, the numbers of patients needed to treat must be reported in order to observe a true difference in treatment groups, a practice often overlooked in meta-analyses. [36] To make matters worse, a recent report involving four medical areas (cardiovascular disease, infectious disease, pediatrics, and surgery) indicates that individual quality measures were not reliably associated with the strength of the treatment effect in 276 RCTs analyzed in 26 meta-analyses. [37]

The fact that arbitrariness and bias can not only creep into but actually dominate in meta-analyses is both convincingly and dramatically demonstrated in a recent publication in the Journal of the American Medical Association. In their efforts to compare two different preparations of heparin for their respective abilities to prevent post-operative thrombosis, Juni and his colleagues have revealed that diametrically opposing results can be obtained in different meta-analyses, depending upon which of 25 scales is used to distinguish between high- and low-quality RCTs. The root of the problem is evident from the variability of weights given to three prominent features of RCTs (randomization, blinding, and withdrawals) by the 25 studies which have compared the two therapeutic agents. In one investigation, for example, a third of the total weighting of the quality of the trial is afforded to both randomization and blinding, whereas in another, none of the quality scoring is derived from these two features. Widely skewed intermediate values for the three aspects of RCTs under discussion are apparent from the 23 other scales presented. The astute reader will immediately suspect that sharply conflicting conclusions might be drawn from these different studies—and these fears are amply borne out by the forest plot presented in the study. Here each of the meta-analyses listed resolve the studies they have reviewed into high- and low-quality strata, based upon each of their scoring systems. It can be seen that ten of the studies selected show a statistically superior effect of one heparin preparation over the other, but only for the low-quality studies. Seven other studies reveal precisely the opposite effect, in which the high- but not the low-quality studies display a statistically significant superiority of low-molecular weight heparin. Depending upon which scale one uses, therefore, one can either demonstrate or refute the clinical superiority of one clinical treatment over the other. In this manner, therefore, all the rigor and labor-intensive elements of the RCT and its interpretation by the meta-analysis are simply reduced to the subjective and undoubtedly capricious human element of value judgment through the arbitrary assignment of numbers in the weighting of experimental quality. [38] Reduced to lay terms often used to describe the limits of computer capabilities, one might summarize this undertaking as an apt demonstration of the principle, "Garbage in, garbage out."


6.   Contradictions in Design:

There appear to be contradictions in the design in the authors’ comparison of spinal manipulative therapy to 7 other treatment therapies (sham, conventional general practitioner, analgesics, physical therapy, exercises, back school, or collection of therapies judged to be ineffective or even harmful [traction, corset, bed rest, home care, topical gel, no treatment, diathermy, or minimal massage]). Specifically:

a.   Conventional general practitioner and analgesic use is considered to be synonymous, based upon a single reference which suggests that 80% of visits to the general practitioner result in a prescription for using an analgesic. Why then, should analgesic use then be presented in the report as a discrete intervention?

b.   Physical therapy is stated to include exercise in amounts up to 100%. Again, why should exercise then be presented elsewhere as a separate intervention?

7. Contradictions in Evaluating Statistical and Clinical Significance:

One especially troubling situation arises with the authors’ interpretation of the forest plots comparing spinal manipulative and sham therapies. In one instance (Figure 3), spinal manipulative therapy is shown to have "clinically important" short-term improvements in pain and disability; however, these differences are deemed to have "failed to reach a conventional level of statistical significance." In comparing spinal manipulative therapy to the group of treatments deemed ineffective, however, we now find a statistically significant advantage for the former intervention. It is perplexing indeed to then find the authors stating that "the clinical significance of this finding is questionable (italics mine)." [1] In the very simplest of terms, one cannot have it both ways. It would almost seem as if there were a deliberate effort to minimize a treatment effect of potential interpret pertaining to spinal manipulation.


8.   Data are Not Shown in Areas of Interest:

Given the aforementioned arbitrary characteristics of meta-analyses and perhaps of the authors’ presentation as well, one has every reason to wish for the opportunity to examine the data which support the authors’ contention that "our sensitivity analyses supported the robustness of our results with respect to the type of manipulative therapy, profession of the manipulator, and the quality of the studies included." [1] However, none pertaining to these critical areas were presented in the body of the paper. The issue is particularly important with regard to the skill and training of the manipulator, who at times has been misrepresented in the scientific literature. [39, 40] It is questionable how effectively the authors were able to draw comparisons of different chiropractic techniques, as they overlooked the most recent and arguably comprehensive attempts to do so from both the points of view of clinical effectiveness [41] and a literature review. [42]


9.   Clinical as Opposed to Fastidious Treatments:

Some treatments (traction, diathermy, minimal massage) have been deemed by the authors to lack sufficient evidence for their effectiveness as stand-alone applications, and as such have been rejected from consideration in this investigation. What is not clear, however, is whether they are effective in a synergistic manner as ancillary treatments and whether they have been excluded as potentially helpful adjuncts to manual therapy. This was alluded to in Feinstein’s discussion of meta-analyses presented above (critique #5). [36]


10.   Lack of Long-Term Followup:

Followups for back pain outcome assessments are limited in this study to 6 months. However, numerous studies cite recurrences of low back pain for up to 1 year. [43–45] This not only makes the definition of an episode problematical, [46] but it demands that follow-up times for at least a year be observed in order to assess a more durable and perhaps economical treatment effect. Indeed, the longevity of treatment effects of spinal manipulation in managing back pain for 12 months [47–49] to 3 years [50] have been amply demonstrated. In comparison to medications for the treatment of headaches, it has been shown to be markedly superior. [51, 52] As in several aforementioned areas of this study, this particular aspect for comparing treatments might be expected to diminish the actual capacity of spinal manipulation to display its full benefits.


CONCLUDING REMARKS

From a variety of perspectives, this meta-analyses appears to be both flawed and to have either obscured or overlooked the maximal clinical benefits that might be expected to have been conferred upon patients by spinal manipulation, particularly as performed by a chiropractor. The patient response to intervention is far more complex than the dimensions offered by the authors in their discussion. Tonelli points out, for example, that there will always be a region in which discrete differences between individuals cannot be made explicit and quantified, called an epistemological zone. [53] This degree of sophistication is best summarized by Horwitz, who points out that to assume that the entire range of clinical treatment in any modality has been successfully captured by the precision of existing analytical methods in the scientific literature "would be like saying that a medical librarian who has access to systematic reviews, meta-analyses, Medline, and practice guidelines provides the same quality of healthcare as an experienced physician." [54] Hopefully, these shortcomings in the current meta-analyses can be appreciated by the public and addressed more meaningfully in future research.

1  Assendelft WJJ, Morton SC, Yu EI, Suttorp MJ, Shekelle PG.
Spinal manipulative therapy for low back pain: A meta-analysis of effectiveness relative to other therapies.
Annals of Internal Medicine 2003; 138: 871-881.

2  Haldeman S, Rubinstein SM.
Cauda equina syndrome in patients undergoing manipulation of the lumbar spine.
Spine 1992; 17(12): 1469-1473.

3  Terrett AGL, Kleynhans AM.
Complications from manipulations of the low back.
Chiropractic Journal of Australia 1992; 22(4): 129-140.

4  Hurwitz EL, Aker PO, Adams AH, Meeker WC, Shekelle PG.
Manipulation and Mobilization of the Cervical Spine:
A Systematic Review of the Literature
Spine (Phila Pa 1976) 1996 (Aug 1); 21 (15): 1746–1760

5  Haldeman S, Carey P, Townsend M, Papadopoulos C.
Arterial Dissections Following Cervical Manipulation: The Chiropractic Experience
Canadian Medical Association Journal 2001 ( Oct 2);   165 (7):  905–906

6  Klougart N, Leboeuf-Yde C, Rasmussen LR.
Safety in Chiropractic Practice Part II: Treatment to the Upper Neck
and the Rate of Cerebrovascular Incidents
J Manipulative Physiol Ther 1996 (Nov);   19 (9):   563–569

7  Lazarou J, Pomeranz B, Corey P.
Incidence of adverse drug reactions in hospitalized patients.
Journal of the American Medical Association 1998; 279(15): 1200-1205.

8  Phillips D, Christenfeld N, Glynn L.
Increase in US medication-error deaths between 1983 and 1993.
Lancet 351: 643-644.

9  Schuster M, McGlynn E. Brook R.
How good is the quality of health care in the United States?
Milbank Quarterly 1998; 76: 517-563.

10  Weingart SN, Wilson RM, Gibberd RW, Harrison B.
Epidemiology and medical error.
British Medical Journal 2000; 320: 774-777.

11  Starfield B.
Is US health really the best in the world?
Journal of the American Medical Association 2000; 284(4): 483-485.

12  Report from the Inspector General of the Department of Health and Human Services;
Lauran Neergard, Associated Press, December 15, 1999.

13  Gandhi TK, Weingart SN, Borus J, Seger AC, Peterson J, Burdick E, Seger DL, Shu K, Federico F
Adverse drug events in ambulatory care.
New England Journal of Medicine 2003; 348(16): 1556-1564.

14  Gurwitz JH, Field TS, Harrold LR, Rothschild J, Debellis K, Seger AC, Cadoret C, Fish LS, Garber L
Incidence and preventability of adverse drug events among older persons in the ambulatory setting.
Journal of the American Medical Association 2003; 289(9): 1107-1116.

15  Wolfe MM, Lichtenstein DR, Singh G.
Gastrointestinal Toxicity of Nonsteroidal Anti-inflammatory Drugs
New England Journal of Medicine 1999 (Jun 17); 340 (24): 1888–99

16  Ament PW, Childers RS
Prophylaxis and treatment of NSAID-induced gastropathy.
American Family Physician 1997; 55(4): 1323-1326, 1331-1332.

17  Simon LS. Osteoarthritis: An overview.
Clinical Cornerstone 1999; 2(2): 26-34.

18  Fries JF.
Assessing and understanding patient risk.
Scandinavian Journal of Rheumatology 1992; 92(Suppl): 21-24.

19  Armstrong CP, Blower AL.
Non-steroidal anti-inflammatory drugs and life-threatening complications of peptic ulceration.
Gut 1987; 28: 527-532.

20  Gabriel SE, Jaakkimainen L, Bombardier C.
Risk for serious gastrointestinal complications related to the use of nonsteroidal anti-inflammatory drugs: A meta-analysis.
Annals of Internal Medicine 1991; 115: 787-796.

21  Carson JL, Willett LR.
Toxicity of nonsteroidal anti-inflammatory drugs: An overview of the epidemiological evidence.
Drugs 1993; 46(Suppl 1): 243-248.

22  Carson JL, Strom BL, Soper KA, West SL, Morse SL.
The association of nonsteroidal anti-inflammatory drugs with upper gastrointestinal tract bleeding.
Archives of Internal Medicine 1987; 147: 85-88.

23  Page J, Henry D.
Consumption of NSAIDS and the development of congestive heart failure in elderly patients.
Archives of Internal Medicine 2000; 160: 777-784.

24  Bombardier C, Laine L, Reicin A, Shapiro D, Burgos-Vargas R, Davis B, Day R, Ferraz MB, Hawkey CJ.
Comparison of upper gastrointestinal toxicity of Rofecoxib and naproxen in patients with rheumatoid arthritis. VIGOR Study Group.
New England Journal of Medicine 2000; 343: 1520-1528.

25  Important drug safety information: Vioxx (Dear Healthcare Professional Letter).
Point-Claire, Dorval (QC): Merck Frosst Canada; 2002 April 15. Available:
www.hc-sc.gc.ca/hpb-dgps/therapeut/zfiles/english/advisory/industry/vioxx_e.html
(accessed 2002 May 30).

26  Important drug safety information: Celebrex (Dear Healthcare Professional Letter).
Mississauga (ON): Pharmacia Canada Inc.; 2002 May 13. Available:
www.hc-sc.gc.ca/hpb-dgps/therapeut/zfiles/english/industry/celebrex_e.html
(accessed 2002 May 30).

27  Konstam MA, Weir MR, Reicin A, Shapiro D, Sperling RS, Barr E, Gertz BJ.
Cardiovascular thrombotic events in controlled clinical trials of rofecoxib.
Circulation 2001; 104: 2280-2288.

28  Whitcomb DC, Block GD.
Association of acetaminophen hepatoxicity with fasting and ethanol use.
Journal of the American Medical Association 1994; 272(23): 1845-1850.

29  Cherkin DW, Dyer S, Browne W, Townsend J. Frank AO.
Medication use for low back pain in primary care.
Spine 1998; 23: 607-14.

30  Rome PL.
Perspectives: An Overview of Comparative Considerations of Cerebrovascular Accidents
Chiropractic Journal of Australia 1999; 29 (3): 87—102

31  Sackett DL.
Editorial: Evidence-Based Medicine.
Spine 1998; 23(10): 1085-1086.

32  Jonas W.
The Evidence House: How to Build an Inclusive
Base for Complementary Medicine
Western Journal of Medicine 2001 2001 (Aug); 175 (2): 79–80

33  Caspi O, Millen C, Sechrest L.
Integrity and research: Introducing the concept of dual blindness: How blind are double-blind clinical trials in alternative medicine?
Alternative Therapies in Health and Medicine 2000; 6(6): 493-498.

34  Koes BW, van Tulder MW, Ostelo R, Kim Burton A, Waddell G.
Clinical Guidelines for the Management of Low Back Pain
in Primary Care: An International Comparison
Spine (Phila Pa 1976) 2001 (Nov 15); 26 (22): 2504–2513

35  van Tulder MW, Koes BW, Bouter LM.
Conservative treatment of acute and chronic nonspecific low back pain: A systematic review of randomized controlled trials of the most common interventions.
Spine 22(18): 2128-2156, 1997.

36  Feinstein AR.
Meta-analysis: Statistical alchemy for the 21st century.
Journal of Clinical Epidemiology 1995; 48(1): 71-79.

37  Balk EM, Bonis PAL, Moskowitz H, Schmid CH, Ioannidis JPA, Wang C, Lau J.
Correlation of quality measures with estimates of treatment effect in meta-analyses of randomized controlled trials.
Journal of the American Medical Association 2002; 287(22): 2973-2982.

38  Juni P, Witsch A, Bloch R, Egger M.
The hazards of scoring the quality of clinical trials for meta-analysis.
Journal of the American Medical Association 1999; 282(11): 1054-1060.

39  Terrett AGJ.
Current concepts in vertebrobasilar complications following spinal manipulation.
West Des Moines, IA: NCMIC Group Inc., 2001.

40  Terrett AGJ:
Misuse of the Literature by Medical Authors in Discussing Spinal Manipulative Therapy Injury
J Manipulative Physiol Ther 1995 (May); 18 (4): 203–210

41  Gatterman MI, Cooperstein R, Lantz C, Perle SM, Schneider MJ.
Rating specific chiropractic technique procedures for common low back conditions.
Journal of Manipulative and Physiological Therapeutics 2001; 24(7): 449-456.

42  Cooperstein RL, Perle SM, Gatterman MI, Lantz C, Schneider MJ.
Chiropractic technique procedures for specific low back conditions: Characterizing the literature.
Journal of Manipulative and Physiological Therapeutics 2001; 24(6): 407-424.

43  Croft PR, Macfarlane GJ, Papageorgiou AC, Thomas E, Silman AJ.
Outcome of low back pain in general practice:
A prospective study
BMJ 1998 (May 2); 316 (7141): 1356–1359

44  McGorry RW, Webster BS, Snook SH, Hsiang SM.
The relation between pain intensity, disability, and the episodic nature of chronic and recurrent low back pain.
Spine 2000; 25(7): 834-841.

45  Hestbaek L, Leboeuf-Yde C, Engberg M, Lauritzen T, Bruun stat NH.
LBP: What is the long-term course?
European Spine Journal 2003; 12(2): 149-165.

46  Smith M, Stano M.
Costs and Recurrences of Chiropractic and Medical Episodes of Low-back Care
Journal of Manipulative and Physiological Therapeutics 1997 (Jan); 20 (1): 5–12

47  Koes BW, Bouter LM, van Mameren H, et al.
Randomised Clinical Trial of Manipulative Therapy and Physiotherapy
for Persistent Back and Neck Complaints: Results of
One Year Follow Up
British Medical Journal 1992 (Mar 7); 304 (6827): 601–605

48  Skargren EI, Oberg BE, Carlsson PG, Gade M.
Cost and effectiveness analysis of chiropractic and physiotherapy treatment for low back and neck pain.
Spine 1997; 22: 2167-2177.

49  Meade TW, Dyer S, Browne W, et al:
Randomised Comparison of Chiropractic and Hospital Outpatient
Management for Low Back Pain: Results from Extended Follow up
British Medical Journal 1995 (Aug 5); 311 (7001): 349–351

50  Meade TW, Dyer S, Browne W, et al:
Randomised Comparison of Chiropractic and Hospital Outpatient Management for Low Back Pain:
Results from Extended Follow up
British Medical Journal 1995 (Aug 5); 311 (7001): 349–351

51  Boline, PD, Kassak, K, Bronfort, G, Nelson, C, and Anderson, AV.
Spinal Manipulation vs. Amitriptyline for the Treatment of Chronic Tension-type Headaches:
A Randomized Clinical Trial
J Manipulative Physiol Ther 1995 (Mar); 18 (3): 148–154

52  Nelson Nelson CF, Bronfort G, Evans R, Boline P, Goldsmith C, Anderson AV:
The Efficacy of Spinal Manipulation, Amitriptyline and the Combination of Both Therapies
for the Prophylaxis of Migraine Headache
J Manipulative Physiol Ther 1998 (Oct);   21 (8):   511–519

53  Tonelli MR.
The philosophical limits of evidence-based medicine.
Academic Medicine 1998; 73(12): 1234-1240.

54  Horwitz RI.
The dark side of evidence-based medicine.
Cleveland Clinic Journal of Medicine 1996; 63: 320-323.