Is Chiropractic Evidence Based?
A Pilot Study

This section is compiled by Frank M. Painter, D.C.
Send all comments or additions to:    Frankp@chiro.org

FROM:   J Manipulative Physiol Ther 2003 (Jan); 26 (1): 47 ~ FULL TEXT

Adrian B. Wenban, BAppSc, MMedSc

Private practice of Chiropractic,
Barcelona, Spain.

---

OBJECTIVE:   To calculate the proportion of care delivered in a chiropractic practice supported by good–quality clinical trials.

DESIGN:   Retrospective survey.

METHODS:   Data were collected from patient files relating to 180 consecutive patient visits in a suburban chiropractic practice in northern Spain. Each patient's presenting complaint was paired with the chiropractor's chosen primary intervention. Based on a literature review (Medline, Mantis, and nonautomated searches of local medical libraries), each presenting complaint–primary intervention pairing was categorized according to the level of supporting evidence as follows: Category I, intervention based on good quality clinical trial evidence; Category II, intervention based on poor–quality or no clinical trial evidence. To distinguish between good– and poor–quality clinical trials, studies were critically appraised and assigned quality scores.

RESULTS:   Of the 180 cases surveyed, 123 (68.3%) (95% CI, 61.5%–75.1%) were based on clinical trials of good methodologic quality (Category I). Only 57 (31.7%) (95% CI, 24.9%–38.5%) of the cases were based on poor–quality or no clinical trial evidence (Category II).

CONCLUSION:   When patients were used as the denominator, the majority of cases in a chiropractic practice were cared for with interventions based on evidence from good–quality, randomized clinical trials. When compared to the many other studies of similar design that have evaluated the extent to which different medical specialties are evidence based, chiropractic practice was found to have the highest proportion of care (68.3%) supported by good–quality experimental evidence.

---

From the Full–Text Article:

Introduction

A large number of recent editorials [1–11] in major medical journals related to so–called complementary and alternative medicine (CAM) contain 2 common themes of relevance to this manuscript:

(1) The editorials all consider chiropractic to be a CAM;

(2) the editorials contend that CAMs, and therefore chiropractic, by definition, are not evidence based (see Table 1).

Opposing the contention that chiropractic is not evidence based is a trend noted in a recently published review article. [12] The author of that review concluded that “the increase in the number of reports of clinical trials (published in mainstream medical journals across the last 30 years) indicates an increasing level of original research activity in complementary medicine (and chiropractic) and suggests a trend toward an evidence–based approach in this discipline.” [12]

Recent trends in health care include the growing popularity of CAM, [13] editorial comments in medical journals contending that CAM is not evidence based (Table 1), and mainstream medicine’s growing preoccupation with practicing in an evidence–based model. These observations led 1 group of researchers to investigate the question:

“Why do people attend complementary practitioners?”

They concluded,

“The advances of scientific medicine and expert evidence–based advice were not always relevant to patients.” [14]

Discussions and debates regarding the utility and limitations of the evidence–based model of health care, and its relevance and implications for chiropractic patients and chiropractors, have been few and far between in the chiropractic peer–review literature. These limited discussions have largely taken the form of admonitions from within [15–18] and outside [7, 19, 20] the chiropractic profession for chiropractors to take up the challenge of making clinical practice more evidence based. One author, [11] who considered chiropractic part of CAM, stated,

“It is therefore timely to ask whether CAM is, at all, evidence based.”

That same author also wrote,

“Recent studies suggest that around 80% of all acute orthodox interventions used in a hospital or general practice setting are based on evidence.”

The author then poses the question,

“How solid is the evidence base in CAM?”

It is noteworthy that the 80% referred to by that author was derived from studies [21, 22] that included evidence in the form of randomized clinical trials (RCTs) and non–experimental studies.

To date, very little effort has been directed toward quantifying the extent to which the day–to–day care delivered in chiropractic practice is based on evidence. One attempt has been made to quantify the extent to which chiropractic care for low back pain (LBP) is congruent with recent US practice guidelines. [23] That study reported on the appropriate use of what is termed spinal manipulative therapy (SMT). The authors found that the rate of appropriate use of SMT for low back pain by chiropractors was 46%. [23] That study was structured such that it used similar appropriateness criteria to the guidelines published in 1994 by the Agency for Health Care Policy and Research. [24]

This retrospective analysis used a research methodology based on previous studies [21, 22, 25–27] that endeavored to determine the extent to which different specialties of medicine could be deemed evidence based. I have applied a similar method in a chiropractic practice with the aims of determining

(1) the proportion of care delivered (in the form of primary interventions) in a chiropractic practice, which is based on evidence from good–quality RCTs;

(2) whether chiropractic practice can be evaluated with methods as rigorous as those used to evaluate specialties of medicine; and

(3) how the proportion of care delivered, and supported by good–quality RCTs, compares between chiropractic and specialties of medicine.

For those authors who have previously applied this research method, [21, 22, 26–36] the stimulus in asking what proportion of medical care is evidence based came from previous observations, which estimated that only approximately 10% to 20% of all procedures currently used in medical practice have been shown to be efficacious by controlled trials. [20, 37] However, the studies on which those 10% to 20% estimates were based “used clinical maneuvers rather than patients as the denominator for their rates, [so that] treatments rarely used received the same weight as common ones.” [21] A recent evaluation of health technologies using that method concluded that only 21% were based on evidence. [38]

I believe, as do other investigators, [21, 22, 25–36] that it is more meaningful to make an estimate of the proportion of patients in common clinical situations who receive interventions based on evidence. In this way, the commonly used interventions, for which there is likely to be more evidence, are weighted more heavily than less commonly used interventions. A more clinically meaningful estimate of the proportion of care delivered that is evidence based thus becomes possible.

---

Methods

Together with the chiropractor who delivered the care related to this survey, I reviewed the case notes of 180 consecutive patients seen over the course of 5 working days. I randomly selected the block of 180 different, but consecutive, patient visits from the files of a chiropractic practice in a suburban area in northeastern Spain. The care was delivered June 7–11, 1999. The chiropractor was a graduate of an American college, had 6 years of clinical experience, and only a very basic understanding of the concepts espoused by the proponents of evidence–based health care. At the time of rendering the care that is the subject of this study, the chiropractor involved was not aware that her case notes would subsequently be reviewed for the purposes of this study. Therefore, the chiropractor involved was blinded to the intent of the study, thus removing any practitioner bias.

The patient’s chief complaint was defined as the condition most responsible for the patient seeking care at the chiropractic practice. The primary intervention was defined as the type of care, clinical maneuver, or advice that was applied by the chiropractor involved and considered to be most important to that particular case. Both the chief complaint and the primary intervention were decided through subsequent discussions between the author and chiropractor given that both reviewed the case notes. The chiropractor was not made privy to my intent with regard to discussing the patient’s case notes. Moreover, confidentiality was preserved by omitting the personal demographics from the patient’s file during the discussion periods.

Literature searches were carried out in 2 computerized bibliographic databases:

Medline [Index Medicus] searches 1966–1998 (key words: exercise, back school, chiropractic, osteopathy, randomized clinical trial, evaluation studies), and

Mantis (formerly Chirolars).

Nonautomated searches of local medical libraries were also undertaken. The goal of the literature search was not to identify all relevant RCTs. Instead, as was the case with studies that examined the extent to which a number of medical specialties could be deemed evidence based, the goal was to locate at least 1 relevant RCT published in a peer–reviewed journal that supported the care delivered. In contrast to the past studies of similar design, an attempt was made to elevate the rigor of this study by distinguishing between good – and poor quality RCTs through a process of critical appraisal and the subsequent assignment of quality scores. RCTs located and found to be relevant to this survey were critically appraised with the use of a modified version of the scoring system of Koes et al. [39] Each RCT was scored in relation to the criteria listed in Appendix I. Each criterion was given a weight, and the maximum score was set at 100 points for each study. Each presenting complaint – primary intervention pair, from here forward in this manuscript referred to as “the care delivered”, was then categorized according to the level of supporting evidence as follows:

Category I:   Interventions whose value has been established in 1 or more good quality clinical trials. (Note: For the purposes of this study, a clinical trial was deemed to be of good quality if, through critical appraisal, it was awarded a quality score greater than, or equal to, 50 out of a possible 100.)

Category II:   Interventions with poor quality or no located, supportive experimental evidence. (Note: For the purposes of this study, an experimental study was deemed to be “unconvincing” or of “poor quality” if, through critical appraisal, it was awarded a quality score of less than 50 out of 100.) Category II studies that were both relevant and supportive of interventions but did not have a quality score of 50 or greater were excluded from the study.

Clinical trials that were located and found to be supportive and relevant to this survey were critically appraised, awarded a quality score, and deemed to qualify for Category I evidence (a clinical trial with a quality score greater than or equal to 50), or Category II evidence (clinical trial with a quality score of less than 50). Subsequent to determining the proportion of care supported by Category I evidence and delivered by the chiropractor involved, the 95% confidence intervals (CIs) were calculated in the hope of establishing the range of values within which one can be 95% sure that the population value lies.

---

Results

Table 1 Table 2 Table 3 Table 4 Table 5 Table 6

Table 1 lists quotes from a number of editorials and highlights their common theme concerning the supposed lack of evidence for chiropractic and CAM. There are a number of similarities between the presenting complaints of the patient population in this study and patients presenting complaints in 2 large, population–based studies recently conducted in North America that related to chiropractors and their patients [40, 41] (see Table 2).

Nineteen relevant and supportive RCTs were located. After critical appraisal and assignment of quality scores, 14 of the RCTs were found to have a quality score of 50 or more and therefore met the required criteria for inclusion in Category I (see Table 3).

The processes of critical appraisal and quality score assignment found that 5 of the 19 supportive clinical trials were not of good methodologic quality. Those 5 trials [42–46] did not qualify for inclusion in the evidence base for this study and were relegated to Category II status (see Table 4).

Of 180 patients presenting at a chiropractic center, 124 (68.3%; 95% CI, 61.5%–75.1%) received primary interventions in the presence of chief complaints that were supported by good–quality clinical trials. [39, 47–65] The remaining 57 patients (31.7%; 95% CI, 24.9%–38.5%) were deemed to be based on poor–quality [42–46] or no RCT evidence (Category II) (see Table 5).

Once the proportion of interventions supported by good–quality RCTs in chiropractic practice were determined, a comparison with the findings from similar studies performed on medical specialties was made. The proportion of delivered care supported by RCT evidence was less than 50% in the majority of medical specialties examined to date. Of 14 studies that have applied this methodology, only 5 have found the discipline they examined to have at least 50% of the care delivered supported by RCT evidence. Based on the results of this study, chiropractic practice can be included with 3 medical specialties — 53% inpatient general medicine, [21] 50% and 64.8% internal medicine, [30, 36] and 65% acute general psychiatry [25] — as disciplines with at least 50% of the care delivered supported by RCT evidence. Furthermore, when compared to medical specialties that have been similarly evaluated, [21, 22, 25–36] the results of this study imply that chiropractic practice may provide the highest proportion of care (68.3%) supported by good–quality experimental evidence (see Table 6).

---

Discussion

The retrospective nature of this survey helped to prevent the occurrence of observation bias; that is, the chiropractor who rendered the care was not aware that her case notes would be reviewed for the purposes of this study, thus blinding her to the intent of the study and protecting the integrity of the case notes as they pertained to the quality of intervention.

However, a number of problems arise out of basing this analysis solely on the primary presenting condition and the primary intervention applied. This study is based on a very simplified model of clinical practice, whereas, in reality, clinical practice is a complex encounter. Often, patients have more than just 1 presenting complaint (comorbidity), and chiropractors often resort to using more than 1 intervention with the same patient. Moreover, many questions yet to be asked of the clinical encounter are not amenable to the RCT format, such as “What do practicing chiropractors consider valid and useful sources of evidence for clinical practice?” and “Do chiropractors deem the methods taught under the banner of evidence–based practice (EBP) relevant and useful in answering those questions that arise out of clinical encounters with their patients?”

One author has criticized other studies similar in design to this study, by stating, “In measuring what is most readily measurable, they reduce the multidimensional doctor–patient encounter to a bald dichotomy and may therefore distort rather than summarize the doctor's overall performance.” [66] This criticism must certainly be considered in this study and chiropractic in general, where the clinical encounter is multidimensional, and where many practitioners see themselves delivering care, as opposed to treatment, from a wellness perspective, which in the health/disease continuum is always viewed relative to the pursuit of an individual's optimum potential.

It has been suggested that “at the very least, future attempts to answer the question ‘How evidence based is my practice?’ should include some measure of how competing clinical questions were prioritized for each case and how the evidence obtained was specified to reflect the needs and choices of the individual patient.” [67] Although this is a legitimate concern, no attempt was made in this study to measure or assess such qualitative information. Regardless, this study lends further weight to the belief that chiropractic can be evaluated with methods as rigorous, or more so, as those used to evaluate specialties of medicine.

Probably the weakest point is this study's methodology, a weakness that exists in all the similar, previous studies that have examined the extent to which medical specialties are “evidence based” simply because 1 supportive RCT is located in the literature. These studies are further weakened by their failure to search for and take into account nonsupportive RCTs. These flawed benchmarks were set by the authors who designed and carried out the first of these types of studies21 and have been perpetuated by a number of subsequent authors, [22, 25–36] myself included. The use of integrative studies, including systematic reviews, are recommended as a critical step in improving the quality of this type of study in the future.

The proponents of EBP have stated that when seeking answers to questions about the effectiveness of therapy, “we should avoid the non–experimental approaches, since these routinely lead to false–positive conclusions about efficacy.” [68] Therefore, in keeping with that statement, I made an effort to exclude RCTs of poor methodologic quality (quality score <50), and did not accept, or search for, convincing nonexperimental studies as evidence of intervention effectiveness. In contrast, similar previous studies that asked how evidence based are a number of medical specialties, 1 of which was coauthored by a leading proponent of EBP, [21] did not distinguish between good– and poor–quality RCTs, and accepted nonexperimental studies as proof that an intervention, and ultimately a medical specialty, was evidence based. It appears contradictory to me that the proponents of EBP promote 1 very high standard when teaching EBP (ie, in questions about therapy, we should avoid nonexperimental approaches) but then lower the bar when it comes to assessing the extent to which their own medical specialties are evidence based. The study by Gill et al, [22] assessing the proportion of interventions in general medical practice that were evidence based, is a case in point. That study's inclusion criteria for what counts as evidence in support of an intervention's effectiveness were set such that they included nonexperimental studies. The study authored by Nordin–Johansson et al [36] went even further when, in studying the extent to which internal medicine was evidence based, they allowed the inclusion of the consensus opinions of national expert panels to count toward calling internal medicine evidence based.

As a result of the very loose criteria they employed, Gill et al [22] were able to claim in the conclusion to their study that 81% of interventions used in general practice are evidence based. However, had the authors of that study set their inclusion criteria in accordance with the industry standard as promulgated by the proponents of EBP, thereby only allowing RCTs to count as evidence of an intervention's effectiveness, the proportion of interventions used in general practice that could be claimed to be evidence based would have dropped to only 31%. Very probably, had the authors of that article taken the time to appraise critically the supportive RCTs they located, and to exclude RCTs of poor quality, the proportion of care provided in general practice may have been less than 31%. Interestingly, chiropractic, which has come under ongoing criticism for its lack of research, fares comparatively well (see Table 6). Of the care provided by the chiropractor in this study, 68.3% was deemed evidence based when examined with a similar, if not more stringent, methodology than that used in assessing the extent to which a number of medical specialties are evidence based.

The Cochrane collaboration model [69] of compiling systematic reviews of evidence argues that, when they exist, most weight should be given to carefully controlled trials. However, this approach is inclined to provide answers to questions that are easily addressed with existing research methodologies. It does not necessarily address all the needs posed by clinical practice. For example, in chiropractic, to date, much of the research has been directed toward treating individuals who are not well. In this study, which involves chiropractic patients, 27.2% of patients received care for indications that were not supported by RCTs of good methodologic quality. Those conditions may provide researchers of a biomedical inclination with fruitful areas for future research into the effectiveness of chiropractic care. If, however, the purpose of chiropractic is to optimize health, as suggested by the Association of Chiropractic Colleges paradigm, [70] then an equally appropriate research focus might be to explore why and how chiropractic care can best support well individuals in maintaining a disease–free state and attaining optimal well–being.

One author, [71] after examining the medical and chiropractic literature in relation to the role of the chiropractor within the broader health care system, has suggested that studies [72–74] performed to date “reflect a much broader scope of practice for chiropractic than is suggested by the epidemiology of the patient complaints.” In light of such preliminary findings, it may be that the chiropractic profession and public would benefit from further exploration of the role of chiropractic care from a wellness perspective. Even the chiropractor, from whom the data for this study was collected, contends that the care delivered was not directed specifically toward the resolution of a patient's chief complaint or symptoms. Instead, the chiropractor's stated intention was “to optimize the individual's inherent healing capacity.” Preliminary studies do support the observation that some quality–of–life measures do improve in patients who receive ongoing chiropractic care, [75, 76] but whether such improvements accrue from an optimized healing capacity due to periodic spinal adjustment awaits further research.

Because this study involved performing a thorough but far from exhaustive literature search, only a more extensive investigation of the literature will reveal whether further studies exist that support the care delivered by the chiropractor in this study. Furthermore, since the literature review for this study was performed, a number of published RCTs [77, 78] support interventions that formed part of this study and may therefore result in a greater proportion of chiropractic practice being deemed evidence based should this type of study be repeated.

Eleven of the 14 previous studies of this type were structured such that they considered positive, convincing, nonexperimental studies worthy of qualifying an intervention and discipline as “evidence based.” In contrast to that approach, I attempted to raise the standard on what qualified for “evidence–based” status, as did Geddes et al, [25] who examined to what extent psychiatric care was evidence based, and Michaud et al, [30] when they similarly examined internal medicine. This was achieved by acknowledging interventions to be “evidence based” only if supported by at least 1 relevant randomized clinical trial. Additionally, this study went 1 step beyond all similar, previous studies by critically appraising those RCTs that were found and excluding those that did not achieve a quality score of 50 points or more. Despite this more rigorous approach, it should be remembered that individual studies are rarely ever conclusive and, as mentioned earlier in this discussion, the use of integrative studies, including systematic reviews, are recommended as a critical step in improving the quality of this type of study in the future. [79] Other researchers [27] that previously used a design similar to this study did incorporate systematic reviews into Category I evidence but were able to locate only 2 reviews of relevance.

Further use of this research method in examining the extent to which a given area of health care is evidence based may require a more thorough evaluation of the methodologic quality of the RCTs used in support of the care delivered. In this study, quality scores were used in an attempt to improve the quality of permissible evidence. However, the validity of such quality assessment scales has recently been criticized, with one author stating, “Perhaps the most insidious form of subjectivity masquerading as objectivity . . . is ‘quality scoring’” and “I wholeheartedly condemn quality scores because they conflate objective study properties, such as study design, with subjective and often arbitrary quality weighting schemes.” [80] Furthermore, a recent meta–analysis of studies using different quality assessment scales concluded that the use of summary scores are, at best, problematic. [81] A number of the other studies that have used this research design have been criticized as follows: “Apart from anything else, they were undertaken in specialized units and looked at the practice of world experts in Evidence Based Medicine; hence, the figures arrived at can hardly be generalized beyond their immediate setting.” [66] This study does not suffer from such restrictive shortcomings, because it was carried out in a typical chiropractic practice, and the practitioner involved had only a basic grounding in EBP. However, generalization of these findings to other chiropractic practices must await, at a minimum, supportive findings from larger similar surveys. I am presently conducting further studies using an evolved version of this pilot study methodology to examine the extent to which chiropractic practice might be considered evidence based in a number of different countries.

In chiropractic, as has been observed in CAM, [82] there is a strong emphasis on tailoring care to the individual patient. This creates problems for the incorporation of EBP into chiropractic, because the actions of the proponents of EBP, despite what they may claim, reveal that research designs other than the RCT format are considered to be of little value in questions concerning the efficacy of an intervention. [83] The paradox of the clinical trial is that it may be able to assess whether an intervention works under artificially specified controlled conditions, but in no way can it assess who will benefit from a given intervention. Therefore, although accurate decision making in a clinical setting may benefit from evidence derived from RCTs, the successful application of that evidence to the individual patient before us requires evidence from both qualitative and quantitative research. [84]

The importance of research paradigms, other than the quantitative, have previously been discussed in the chiropractic literature. [85, 86] I concur with those authors, in that chiropractic needs to make a balanced investment in quantitative, qualitative, and emergent research paradigms if it is to optimize the health of its science, art, and philosophy.

Read the FULL TEXT Article now.

Return to ChiroZINE ARTICLES

Return to EVIDENCE–BASED PRACTICE

Return to CHIROPRACTIC RESEARCH ARTICLES

Since 2–06–2003