Select Committee on Science and Technology Sixth Report


CHAPTER 4: EVIDENCE

4.1 There are several types of evidence that it is desirable to obtain before a therapy is advocated:

  • Evidence that the therapy is efficacious above and beyond the placebo effect (see paras 3.19 - 3.34);

  • Evidence that the therapy is safe;

  • Evidence that the therapy is cost-effective;

  • Evidence concerning the mechanism of action of the therapy.

  • Methods available for obtaining such evidence will be discussed in Chapter 7 : Research.

Evidence for Efficacy?

4.2 The conclusions from research into the efficacy of the various CAMs are outside the remit of this report. However, it is necessary to understand the general state of the CAM evidence base, in order to consider what type of evidence needs to be collected and to understand why CAM's claims often cause controversy.

4.3 CAM has been criticised by some witnesses for not having scientific evidence to back its claims. The Academy of Medical Sciences (P 1) told us that they are concerned that many CAM practitioners do not take a 'scientific' approach to treatment. They suggest that whereas conventional medicine makes efforts to conduct rigorous research, and changes its clinical practice when new information is discovered, CAM practitioners are more likely to stick to their belief systems despite any negative evidence that may emerge. This is one reason the Academy of Medical Sciences uses to explain why CAM lacks an adequate evidence base to convince them, as conventional scientists, of its claims. This is a controversial statement, and as discussed in para 2.19, Sir Iain Chalmers, Director of the UK Cochrane Centre, suggested that conventional medicine is biased against CAM and conventional medical practitioners and scientists are likely to require lower standards of proof for conventional medical treatments then they do for CAM (P 225). Nevertheless, as we concluded in para 2.7, we are satisfied that there is at present no credible evidence base to support the value of any of the therapies that we list in our Group 3.

4.4 The Department of Health summed up their opinion of the evidence base for CAM by saying that "Evidence for CAM in the form of research has been criticised as being inadequate, and there is some justification in this claim" (P 101). This is a controversial area, as the definition of an adequate evidence base varied across our witnesses. Some CAM practitioners have claimed that a history of safe and apparently successful traditional use is enough evidence to justify advocating the use of their particular therapy. However, most of our witnesses with a conventional medical or scientific background have asserted that, in order for CAM therapies to be more widely accepted, it is important that they have a critical mass of scientifically-controlled evidence to support their claims; and that at the moment most of CAM lacks such evidence. Many of our conventional medical witnesses have suggested that, since much of conventional medicine is required to undergo rigorous trials to justify its use, no less should be expected of CAM. But even this view is controversial as some of the CAM advocates we have heard from have suggested that much of conventional medicine lacks a rigorously tested evidence base, and that to require one of CAM is to operate a double standard. In fact, the Department of Health followed their statement that there was some justification behind the claim that the CAM evidence base was inadequate, by acknowledging that the same could be said for some conventional medicine. It is our view that most modern conventional therapies are backed by scientific evidence. It is in the case of some of the older and traditional treatments surviving from the past (such as cold 'cures' and 'tonics') where evidence is, like the evidence for much of CAM, lacking.

4.5 There are two notable weaknesses of the evidence base for CAM that at present exists. One is that in most of these areas little research is being done, and the second is that the few studies which have been completed are given disproportionate weight. It is worth considering this second feature in some detail. Some CAMs have embarked upon research in order to build up an evidence base. All the therapies that we have included in Group 1 either have done, or are working on, rigorous trials to test their claims. However, one or two studies with positive results in support of their claims for efficacy are not enough. It must be remembered that with a statistical significance of p<.05[29] (the commonly accepted level of significance), one in twenty studies of any procedure will show a possible significant effect; hence a few positive results with small effects are not yet enough to prove a therapy's efficacy, nor to justify its wider provision.

4.6 Another problem with the way existing studies are used is that many old studies are recycled again and again through reviews and meta-analyses[30]. There are some doubts about the usefulness and validity of the results of meta-analysis. Professor Peter Lachmann, on behalf of the Academy of Medical Sciences, told us: "Meta-analysis is a highly contentious but very important issue and is subject to all sorts of problems, of which comparable design, selective publication of positive against negative results and various other problems are well-known. Not all meta-analyses should be afforded the same weight" (Q 1411). This problem highlights the need for more original work, involving well-designed clinical trials, to be done on CAM disciplines.

4.7 The importance of evidence of efficacy is less clear than the importance of evidence of safety. Many witnesses have suggested that if a person feels that a therapy is helpful to them and can be shown not to be harming them, then it is not necessary for there to be statistically valid research supporting its claims. But the question then arises as to whether such a treatment should be made available at public expense. The role of patient satisfaction in evaluating therapies will be considered in paragraphs 4.24 - 4.27.

4.8 One argument that has been repeated to us is that the existence of evidence which supports a therapy's claims is of secondary importance, provided that patients are aware of whether there is any evidence or not. Consumer bodies such as Patient Concern (P 166) believe that treating a patient with a therapy that lacks evidence of efficacy is not wrong if the patient is happy with the treatment, as long as he or she knows that there is no definitive proof of efficacy and has not been led to believe that the treatment will definitely work. They call for strong measures to be taken against practitioners who mislead patients with false claims of evidence of efficacy.

4.9 Another issue to consider in this area is how much evidence there is to support the claims of other healthcare interventions so as to consider the position of CAM in context. The Medicines Control Agency require evidence of efficacy (and of quality and safety) before licensing any new pharmaceutical product. However, the British Dental Association (P 35) gave evidence suggesting that much of clinical dental practice has a weak evidence base. We have also heard evidence concerning several commonly used conventional medical treatments that have a long history of use but little research evidence to support such use. Examples include the use of electro-convulsive therapy for the treatment of depression and cervical and uterine curettage for treating dysfunctional uterine bleeding.

4.10 However, the Institute of Biology (P 125), suggest that if health practitioners are to be held liable for their services to their patients then the medicines they prescribe must be proven to be efficacious.

4.11 There are complications in this area beyond simply evaluating the importance of evidence of efficacy. Many submissions from CAM representatives, as well as the submission from the NHS Confederation (P 144), claim that, for some forms of CAM in some situations, there is already evidence of efficacy. Therefore they suggest that the lack of mainstream acceptance and the slow NHS uptake must be due to other factors. The NHS Confederation claims CAM has suffered from 'unscientific prejudice' from the scientific orthodoxy; however most conventional medicine submissions deny this, and reiterate the argument that a few positive studies should not be given too much weight and do not constitute a critical mass of evidence (para 4.5). In particular, positive trials of homeopathic treatment in allergic disorders have not yet convinced many conventional practitioners. Specifically, trials at the Glasgow Homeopathic Hospital, demonstrating benefit in the treatment of asthma and allergic rhinitis with homeopathic remedies, are thought by some independent observers to need larger and longer trials for confirmation of the perceived effects.

4.12 Beyond these general points the diversity of CAM therapies is such that our comments must be related to the three groups of disciplines that we have listed in Box 1.

4.13 Of the therapies in Group 1 we were made aware of good evidence of the efficacy of osteopathy and chiropractic[31]. Indeed, they appear to be somewhat more effective than the manipulative techniques employed by conventional physiotherapists. There is also scientific evidence of the efficacy of acupuncture, notably for pain relief and the treatment of nausea[32]. The evidence for the efficacy of herbal medicine is mixed. Many herbs have established activities while others do not; among those which are active many are claimed to have numerous other actions for which evidence is lacking. Many powerful drugs used in conventional medicine are of herbal origin, such as morphine derived from the poppy, or digoxin from the foxglove. Problems sometimes arise when mixtures of herbs are used. Even when these are of proven efficacy it may be difficult to identify the active ingredient or ingredients and some preparations may be difficult to standardise and control. In the case of homeopathy, although it is covered by a separate Act of Parliament, we were not able to find any totally convincing evidence of its efficacy. Nevertheless, we accept that there is anecdotal evidence of benefit from homeopathic remedies in animals, where presumably a placebo effect is less significant. Much more research is needed.

4.14 Of the therapies in Group 2 there are many claims of efficacy, usually for a limited range of ailments. We have not examined each in detail. We see many of these complementary therapies as inducing relaxation and a sense of well-being so as presumably to stimulate the immune response, as in the placebo effect. Many are greatly appreciated for the comfort they provide to terminally ill patients.

4.15 We find no convincing evidence of efficacy for any of the remedies in Groups 3a or 3b, but we did not carry out a detailed examination.

4.16 Evidence for the efficacy of the treatment itself is not the only important factor. The Royal Society of Edinburgh (P 212) makes the point that evidence of the validity of diagnostic procedures is as important as evidence supporting efficacy of a treatment. Diagnostic procedures must be reliable and reproducible and more attention must be paid to whether CAM diagnostic procedures as well as CAM therapies, have been scientifically validated. We agree that this is an issue that should always be kept in mind when doing research in this area.

4.17 More research is needed on the efficacy of most CAMs. In the case of therapies which possess research evidence, but whose practitioners believe that conventional scientific views are standing in the way of their acceptance, it would be constructive if a body such as the National Institute for Clinical Excellence (NICE) could evaluate such evidence as exists. (NICE did point out that topics they enquire into are determined by the Department of Health and are selected against a framework of the State's priorities for the NHS (Q 1839). However, they did acknowledge that, in their view, such subjects may be suitable for appraisals). It would also help if such bodies made sure that on their evaluation committees were doctors and scientists who were aware of CAM's intricacies, philosophy and research (see Chapter 7).

4.18 In our opinion any discipline whose practitioners make specific claims for being able to treat specific conditions should have evidence of being able to do this above and beyond the placebo effect. This is especially true for therapies which aim to be available on the NHS and aim to operate as an alternative to conventional medicine, specifically therapies in Group 1. The therapies in our Groups 3a and 3b also aim to operate as an alternative to conventional medicine, and have sparse, or non-existent, evidence bases. Those therapies in our Group 2 which aim to operate as an adjunct to conventional medicine and mainly make claims in the area of relaxation and stress management are in lesser need of proof of treatment-specific effects but should control their claims according to the evidence available to them.

Evidence for Safety

4.19 Evidence that a therapy has few, if any, significant adverse effects and will not cause avoidable harm must be considered important in all medicine, including CAM. However, there are two potential complications which confound this seemingly simple statement:

  • What level of safety should be demanded?

  • What type of evidence of safety is acceptable?

4.20 In determining what level of safety should be sought, the risk/benefit ratio of the therapy in question must be considered. If the potential benefits of a therapy are likely to be very significant, or even life-saving, then the level of risk a patient may be willing to take with the therapy is likely to be higher than the level of risk they are willing to accept for the benefit of temporary symptom relief or the cure of a minor complaint. Another consideration is whether the risks a therapy may possess are inherent or can be minimised through proper regulation of its practitioners. For example we received some evidence about the risks of acupuncture causing pneumothorax due to a needle being inserted into the pleural cavity; however if practitioners are properly trained and well-regulated this risk is minimised. In determining what evidence of safety is acceptable it is important to consider what weight should be given to a history of safe traditional use. Within CAM such evidence is common and is often given reasonable weight by CAM advocates and to a certain extent by policy-makers. For example there are exemptions from licensing in the Medicines Acts for natural remedies of traditional use, and a third category of medicines, which will include traditional-use herbal medicines, is being examined by the European Union (see Chapter 5).

4.21 There is no doubt that many CAM therapies are very safe, as compared to many new powerful conventional remedies. This is often used as an argument for approving the increasing use of CAM, but it must be remembered that the use of a "safe" CAM remedy to treat a serious or potentially lethal disease, so that the use of conventional preparations with proven efficacy is denied, is of course a real danger.

4.22 Several submissions we received suggest that minimum standards of safety need to be defined and widely disseminated in order to protect the public. The British Holistic Medical Association have suggested that such work should be carried out by NICE and the Commission for Health Improvement (CHI) who should then issue national guidelines.

4.23 The evidence that we received from almost all the different therapies indicated that at the point of diagnosis, if the practitioners thought that their treatment would not work, they would refer their patients to a conventional medical practitioner. We were encouraged by this sentiment, even though it was not universal.

PATIENT SATISFACTION AS EVIDENCE FOR EFFICACY

4.24 We have heard many conflicting opinions on the idea that high levels of patient satisfaction could be used as evidence for a therapy's efficacy. It has been argued by some that such satisfaction is very important. The International Federation of Reflexologists (P 129) suggest that evaluation of patient satisfaction is particularly important in CAM because much of CAM emphasises patients' participation in the therapy and evaluation of its effects. Many other witnesses have asserted that although patient satisfaction has its place it is not sufficient to justify accepting that a therapy works so that objective rather than subjective evidence is needed. The Academy of Medical Sciences explained why this may be: "It needs to be emphasised that patient satisfaction is not in itself a sufficient estimate of clinical benefit. While it is very important that patients be satisfied with the efforts made on their behalf, it is at least equally important that they should obtain objective benefit. The two do not always go together. For example, patients with peripheral vascular disease, if they go to a practitioner who allows them to continue smoking will show a high patient satisfaction although their outcome will be poor. In contrast, if they are made to stop smoking they are likely to be dissatisfied but their outcome will be much better" (p 286).

4.25 NICE, who have been charged with the responsibility of evaluating the evidence for different NHS treatments over the coming years, also express concern about the validity of anecdotal evidence such as patient satisfaction. Professor Sir Michael Rawlins, Chairman of NICE, told us: "Anecdote, by and large, is not a very reliable method for determining efficacy and 2000 years of medicine demonstrate the fragility of anecdote as a basis for practising medicine" (Q 1833).

4.26 One point that most of our witnesses have agreed upon is that patient experience is important enough to warrant patients being involved in the appraisal of therapies. NICE have made moves towards incorporating patients' views into their appraisals. Mr Andrew Dillon, Chief Executive of NICE, told us: "In the process we have established we invited nationally-based patient advocate groups to make submissions into our individual appraisals. So we have a written statement of their assessment of, as far as they understand it, the patient's perspective of the disease, and if it is an intervention which is currently in use in the NHS, their understanding of the patient's experience of using that intervention in the management of their illness. We also invite patient advocates to join the appraisal committee meetings themselves" (Q 1843).

4.27 In conclusion, patient satisfaction has its place as part of the evidence base for CAM but its position is complicated, as Sir Michael Rawlins, explained: "The difficulty, of course, is that very often the anecdotal evidence relates to conditions where there is fluctuation in the clinical course and people who start an intervention at a time when there is a natural resolution of the disease, very understandably, are likely to attribute cause and effect when it may not be. But, on the other hand, there are some anecdotes that are quite clearly important." Therefore, ideally studies should include patient satisfaction as one of a number of measures in evaluating a treatment, but it alone cannot be taken as a proof or otherwise of a treatment's efficacy or as evidence to justify provision.

Evidence About Mechanisms of Action

4.28 The position of therapies without a scientifically plausible mechanism of action (e.g. healing and homeopathy) needs to be considered. If there is no scientifically plausible mechanism through which a treatment may work in the human body, can the use of such a therapy be justified? Should such therapies be considered a product of the placebo effect enhanced by "tender loving care" or should consideration be given to the possibility that they may have explanations not yet understood by modern science? (The role of the placebo effect is discussed in paras 3.19-3.34.)

4.29 Many of our witnesses have argued that if there is evidence for efficacy then it is not necessary to understand exactly how the effect is achieved, and we agree. This is, indeed, the position with several conventional therapies. Professor Sir Michael Rawlins explained that in NICE's search for clinical excellence, it is evidence of efficacy and not the mechanism of action that is prioritised: "I do not mind and I do not think the Institute minds whether it understands how a treatment works or not. I do not understand how many treatments do work, and this is after 35 years as a pharmacologist, but what we do like is good evidence that they do whatever they claim to do" (Q 1833).

4.30 However, despite these arguments, the opposite view is that if a therapy has no plausible mechanism of action then spending research money on it and providing patients with access to it is likely to be a waste of resources. It is worth considering this argument in more detail, by asking two distinct questions:

  • Should mechanisms of action be plausible before research into the efficacy of a therapy is funded?
  • Should mechanisms of action be understood before access to a therapy is provided?

4.31 It is of course true that many treatments have been used for a long time without understanding their mechanisms of action and only now are possible explanations for how they work coming to light. Professor Lesley Rees, a Trustee of FIM, used acupuncture as a case in point. Acupuncture is traditionally said to work through affecting energy meridians that according to Traditional Chinese medicine circulate around each person. This explanation is not congruent with current scientific thought and if an understanding of mechanisms of action were considered of paramount importance doctors should have shunned acupuncture years ago. Now, however, other possible mechanisms, which are more amenable to modern scientific thought (e.g. concerning the effect on the central nervous system and the stimulation of endorphin receptors), are being discovered and evidence for acupuncture's efficacy is growing. As Professor Lesley Rees summed up: "…acupuncture has been used for thousands of years, yet there was no real information about how it might work and I think it would be fair to say that it would have been terrible if the benefits of acupuncture had not been appreciated and used over all the years because we did not have any real understanding of perhaps some of the mechanisms about how they work" (Q 77).

4.32 In terms of research Professor Tom Meade from the Royal Society told us that "…the distinction between the effect and the explanation for the effect is central, and you do not need to believe in the explanation in order to believe in the effect" (Q 181). Therefore, "...it would be perfectly possible for a funding body to allow a bit of research to go forward even though the theoretical backdrop is totally irrelevant to whether the treatment works or not. I think probably what we will see now, increasingly, is applications for funds which simply say 'There is good reason to think there is an effect here and we want to study that. We are going to use these methods which are well attested.' And if everyone agrees that then, if this works out, it will reduce the ambiguity of the effectiveness of this particular treatment" (Q 181).

4.33 A reason for funding efficacy studies of therapies without a plausible mechanism of action is that research into that area can help elucidate routes through which mechanisms of action might work. Professor Meade explained: "…I think it is possible, in some circumstances, that the result of the trial - in other words that something is effective - will actually then give clues as to studying the mechanisms. Equally, if it is not effective then it is beginning to exclude possible explanations as well" (Q 181).

4.34 However, there is an alternative view, articulated by Professor Lewis Wolpert, a fellow of the Academy of Medical Sciences, that: "It is not just efficacy that you should be thinking about. Medicine aims to base itself upon science. Let me tell you what I mean. If you have therapy which you can in no plausible way relate to the behaviour of cells…I personally could not support research in that field" (Q 1404). Based on the limited amount of research funding available in the medical sciences, he suggests that research into therapies such as homeopathy should not be funded: "A liquid which contains no active molecule, which no chemist could plausibly give an account of, is not an area where I would want to invest money. I am sorry: one cannot give up all of chemistry just because one believes homeopathy works" (Q 1404 and 1406). Professor Patrick Bateson, giving evidence on behalf of the Royal Society, summed up this argument by saying the role of mechanisms of action comes into importance because "... the critical thing here is going to be whether there is enough evidence to justify us spending more time and trouble testing the efficacy and safety of treatment" (Q 175).

4.35 The mechanism through which homeopathy may work on the body is a specific case in point, about which we have heard much. Samuel Hahnemann at the turn of the 19th century put forward the "law of similars", claiming that any disease can be treated successfully with minute amounts of a drug which in larger doses gives rise to the same symptoms. Therefore, homeopathy is based on the idea of treating 'like with like' by administering hugely diluted versions of basically dangerous substances, such that a dose given to a patient may not contain even a single molecule of the active principle. Many conventional doctors and scientists cannot accept that infinitesimal dilutions can have any effect on the body.

4.36 The arguments about homeopathy illustrate the weight given to understanding mechanisms of action. The Department of Health explained their position on homeopathy which clearly shows they prioritise safety before anything else and give less weight to issues of scientific plausibility: "In relation to homeopathic medicines, we very much agree that there is uncertainty, or limited evidence, about the specific mechanism whereby homeopathy works. The starting point is that homeopathic medicines as such are very much at the safe end of the spectrum; they are very dilute. Often these substances do not have a clearly measurable effect on the body, which is why the simplified homeopathic registration scheme introduced in 1994 concentrates specifically on safety and quality and not efficacy. We have taken a fairly pragmatic approach: if homeopathy does not harm then it is less important to have an in-depth understanding of its mechanism for effectiveness" (Q 34).

4.37 In terms of research funding for therapies without a scientifically plausible mechanism of action, it seems that opinion within the world of conventional medicine is very divided. However, we recommend that if a therapy whose mechanism of action is unclear does gain sufficient evidence to support its efficacy, then the NHS and the medical profession should ensure that the public have access to it and its potential benefits.

4.38 The question of NHS provision for therapies such as homeopathy was answered by the Department of Health by prioritising safety together with consumer choice. On the other hand, as evidence from the Academy of Medical Sciences suggests, the only reason for using therapies such as homeopathy is as a vehicle for the placebo effect to work safely (see paras 3.19 - 3.34). Professor Peter Lachmann told us: "Other effects of homeopathy apropos the placebo effect have already been mentioned and I personally am entirely happy with the idea that homeopathy is a good way of administering a placebo because it is free from harm. I am well aware of the fact that in conventional medicine placebo effects are sometimes produced by the administration of drugs. That is less harmless because all drugs have some side-effects. If drugs are given not for a good purpose but just given to make the patients feel that something is being done for them, then I would entirely agree that a homeopathic preparation, which would produce the same placebo effect without possible harmful side-effects, is to be preferred" (Q 1410).

4.39 The intricate arguments concerning the use of the placebo effect as a therapy were discussed in chapter 3; this does not contradict the argument that safety is a priority, and as long as a therapy is safe, use of any benefits it may bring to patients is justifiable without necessarily understanding its mechanisms. In an era when the Government are hoping that NHS treatments will live up to a standard of evidence set by NICE, we welcome the fact that, as the quotation from Professor Sir Michael Rawlins shows, NICE are willing to accept that a therapy can be efficacious and worth considering even when its mechanisms of action are unclear (see para 4.29).

4.40 It is our opinion that as long as the treatments are known to carry no, or few, adverse effects, it would be against the principle of clinical freedom[33] to prevent patients from having access to therapies which fulfil these criteria and have never been restricted. This is especially the case if the patients believe that such therapies help them and the only argument against them is that an adequate evidence base, derived from controlled trials, does not exist. It is also our opinion that mechanisms of action are of secondary importance to efficacy, a view shared by NICE (Q 1833). We also believe that the principle of clinical freedom should allow therapy with any credible evidence of efficacy the opportunity of validation by further research and the possibility of NHS provision. Any medicine with credible, accepted evidence for efficacy should be available, whatever the controversy over its underlying mechanisms.


29   A significance level of p<.05 means that there is a probability of less than 5% that the results a trial has produced could occur by chance. Back

30   Meta-analysis is the combination of data from several studies to produce a single estimate. From the statistical point of view, meta-analysis is a straightforward application of multi-factorial methods. If there are several studies of the same thing with each giving an estimate of an effect, the meta-analysis provides a common estimate representative of all the work. Back

31   See: Vincent, C. & Furnham, A. (1997) (Op.cit.), 'The quality of medical information and the evaluation of acupuncture, osteopathy and chiropractic'. Back

32   British Medical Association. The evidence base of acupuncture in: Acupuncture: efficacy, safety practice. Harwood Academic Publishers, London (2000); pp 7 - 37. Back

33   By "the principle of clinical freedom" we mean the ability of a medical practitioner to exercise freedom of choice in preventing, diagnosing and treating disease within the limits of his or her clinical competence, having regard solely to the welfare and well-being of the individual, and casting all other considerations aside. Back