Informatics Skills:
Weakness in the Foundation of Research

This section is compiled by Frank M. Painter, D.C.
Send all comments or additions to:    Frankp@chiro.org

FROM:   Proceedings of the 2002 International Conference
on Spinal Manipulation (OCT)

Rupert, R.L., Makasha Colonvega, B.S.

---

INTRODUCTION:

One of the challenges facing today’s health care professionals is the difficulty in keeping current despite the proliferation of medical literature. This need is compounded with the increasing advocacy for evidence-based medicine. Current estimates suggest that clinicians would need to read 19 articles per day, every day of the year to keep abreast of relevant clinical developments. [1 ] Other than consulting colleagues in the field or experts, health care professionals can peruse literature reviews for a concise, qualitative or quantitative meta-analysis of pertinent information. However, expert bias and errors in the literature review process has been shown to be a reason for caution. [2] It is often impossible to separate fact from opinion or to decipher the authors’ methods for selecting material. [3]

Part of the problem lies in poor informatic skills which effects the critical process of information gathering and synthesis. In essence, the failure of authors to describe the procedures and methods they used to prepare the literature review makes it impossible for the work to either be verified or reproduced. Previous research suggests that researchers fail to completely describe the databases used to gather relevant information and/or rely upon a single database for the synthesis of current knowledge. One study concluded that, “no single database is always capable of retrieving all the relevant or important citations.” [4]

Failure to identify database sources is also accompanied by an inadequate description of the literature search semantics. Researchers not only omit a description of MeSH terms, subheadings, check tags, and keywords employed in the search, but do not include a description of the semantic logic utilized, the Boolean expression. Semantic relationships used with the search terms can profoundly effect medical bibliographical retrieval. [5] Researchers also introduce bias into the process of information synthesis by exclusion of references because of language. [6] There is a tendency to restrict the literature search to a single database like Medline which already has a documented language bias and further then further restrict the Medline articles to only those written in English. Hence, a large pool of possible pertinent information from other countries may be ignored.

This analysis of the literature will investigate a cross section of biomedical peer-reviewed scientific journals for the quality of the methods used by the authors in their review of the literature. It will explore to what extent the author(s) have provided the key elements in the methods section that are required for verification or replication of the work. Specifically, the information to be tracked includes the following: was the database(s) that were searched identified, the MeSH terms and or keywords identified, the syntax of the Boolean arguments described, the inclusion years/time of the search provided? The study will also look at the assumption that the quality of the literature review process might be influenced by the number of authors contributing to the work. This investigation will explore the degree to which problems exist within both the medical and chiropractic professions. This paper will follow-up with a recommended set of guidelines to improve the quality of future research.

---

METHODS:

A convenience sample of ten biomedical peer reviewed journals was utilized, a heterogeneous compilation of medical and chiropractic journals was selected for analysis as follows: The Journal of Family Practice (medical), The Clinical Journal of Pain (medical), Annals of Internal Medicine (medical), Spine (medical), Annals of Allergy, Asthma, & Immunology (medical), Topics in Clinical Chiropractic (chiropractic), Journal of Canadian Chiropractic Association (chiropractic), Journal of Neuromusculoskeletal System (chiropractic), Journal of Sports Chiropractic & Rehabilitation (chiropractic), and Journal of Manipulative and Physiological Therapeutics (chiropractic). All articles that purported to be qualitative and/or quantitative literature reviews published in the year 2000 were extracted from these journals. Each review was critiqued for the primary information required to make the review reproducible.

Criteria for replication were defined as

(1) specification of the database(s) source(s),
(2) search terms utilized (i.e. MeSH headings, subheadings, check tags, and keywords),
(3) Boolean arguments (i.e. logical operation of search words with operators, “and”, “or”, “not”, etc.),
(4) timeframe of review content, and
(5) language(s) of original articles.

The criteria involving search terms and Boolean arguments were intended to demonstrate the limited amount of information used as the basis for research. In addition, the criterion for the language(s) of the original articles was intended to show the exclusion of information in languages other than English. Inclusion of the number of authors’ criterion was utilized in order to investigate a possible relationship between the number of authors contributing to the literature review and the degree of reproducibility or quality of the research.

---

RESULTS:

From the ten biomedical journals, seventy-three reviews of the literature were identified. Table I represents a systematic review of the combined statistics for the ten biomedical journals. Table II represents the combined statistics for the replication criteria i.e. the key information components needed from the author(s) in order to replicate the study. These percentages reflect the ratio of the total number of articles demonstrating the specific criteria versus the sample total. In order to investigate the extent of reproducibility obtained by our sample of journal articles, percentages were calculated for the inclusion of replication variables in Table III. Table III provided a continuum of articles demonstrating a high degree of reproducibility to those with a lower degree of reproducibility based upon all six variables (reporting databases, keywords, MeSH terms, Boolean expression, content timeframe, and original article languages).

---

CONCLUSIONS:

From the ten biomedical peer reviewed journals chosen for this analysis, and the total of seventy-three (73) reviews of the literature, only three (3) articles had all the six criteria for replication and verification of the literature search. Forty percent gave none of the necessary criteria at all. These omissions and errors can ultimately effect clinical decisions upon which readers, health care professionals and other experts, are challenged to make on a daily basis. These findings suggest that despite the need for an accurate blueprint of how the research is conducted the research community in general is seriously deficient in describing their methods when it comes to literature reviews. Those reviews that did provide a description of their methods, unfortunately demonstrate that they are not familiar with the variety or proper search strategies for biomedical databases. A serious effort by both the medical and chiropractic research communities must be made to become more familiar with informatics to provide a solid foundation for their research.

Return to OPINION DEPARTMENT

Return to EVIDENCE–BASED PRACTICE

Since 11-27-2004