Annals of Surgical Oncology Cite Track
HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS
This Article
Right arrow Abstract Freely available
Right arrow Full Text (PDF)
Right arrow Alert me when this article is cited
Right arrow Alert me if a correction is posted
Services
Right arrow Similar articles in this journal
Right arrow Similar articles in PubMed
Right arrow Alert me to new issues of the journal
Right arrow Download to citation manager
Citing Articles
Right arrow Citing Articles via Google Scholar
Google Scholar
Right arrow Articles by Brooks, A. D.
Right arrow Articles by Jaques, D. P.
Right arrow Search for Related Content
PubMed
Right arrow PubMed Citation
Right arrow Articles by Brooks, A. D.
Right arrow Articles by Jaques, D. P.
Related Collections
Right arrow Education
Annals of Surgical Oncology 8:638-643 (2001)
© 2001 Society of Surgical Oncology

ORIGINAL ARTICLES

The Search for Level I Evidence in Solid-Tumor Oncology

Ari D. Brooks, MD, Howard M. Karpoff, MD, Isabel Sulimanoff, MLS, Daniel Coit, MD, Murray F. Brennan, MD and David P. Jaques, MD

From the Department of Surgery (ADB, HMK, DC, MFB, DPJ) and Medical Library (IS), Memorial Sloan-Kettering Cancer Center, New York, New York.

Correspondence: Address correspondence and reprint requests to: David P. Jaques, MD, Vice Chairman, Department of Surgery, Memorial Sloan-Kettering Cancer Center, 1275 York Ave., New York, NY 10021; Fax: 212-717-3645; E-mail: jaquesd{at}mskcc.org


   ABSTRACT
TOP
 ABSTRACT
INTRODUCTION
 METHODS
 RESULTS
 DISCUSSION
 REFERENCES
 
Background: We have developed a method to identify, filter, review, and distribute the published level I evidence for solid tumor oncology.

Methods: A standardized MEDLINE search identified prospective randomized controlled trials (PRCTs) in solid tumor oncology. Only PRCTs with therapeutic end points were included. All references were reviewed by a surgical oncology fellow in consultation with experts in the field. The full citations were imported into a comprehensive database. Data on statistical methods according to the Consolidated Standard of Reporting Trials statement were tabulated along with reviewer’s comments. A designation of Ia was given to articles that were well designed and significant contributions to their field. The database powers a dynamic, easily searchable Web site on our intranet and is available in personal digital assistant (PDA) format.

Results: By using standard search criteria, only .03% of the 11 million articles listed in MEDLINE are PRCTs concerning therapy for solid organ malignancies. Approximately 14% of reviewed articles were given a designation of Ia. Having comprehensive data readily available with intranet access or PDAs during conferences enhances their educational value and specificity.

Conclusions: We have developed an exciting tool that uses a highly trained filter to screen and record the medical data available to the clinician. This information has been made available and portable by using the Internet and PDAs.

Key Words: Evidence-based medicine • Oncology • Database searching • Internet • Personal digital assistant


   INTRODUCTION
TOP
 ABSTRACT
 INTRODUCTION
METHODS
 RESULTS
 DISCUSSION
 REFERENCES
 
An essential element of clinical practice is an understanding of the rationale for our management decisions. Mastery of the evidence justifying our management in a given field of medicine should be one of the goals of postgraduate training programs. Furthermore, a prerequisite for the design of clinical trials should be a complete awareness of the relevant available evidence to avoid unnecessary duplication of efforts and allow clarification of research goals.

The National Library of Medicine maintains a bibliographic database (MEDLINE) to aid clinicians and researchers in their search for published evidence about specific topics. This computerized system has made comprehensive searches of the literature simple and accessible. The system is not perfect, however, and several authors have demonstrated that MEDLINE is not able to identify all of the randomized trials in a given field with complete accuracy.1–13 Some investigators have resorted to hand-searching journals for comprehensive identification of randomized trials.2,13 Indeed, there is now an international collaborative effort to create a register of all randomized trials, known as the Cochrane Collaboration and library.14 In addition, the National Library of Medicine has expanded its available search terms for level I evidence and continues to update its records retrospectively.

For those performing systematic reviews, a comprehensive listing of level I evidence is essential to avoid missing relevant articles. For others, it may be more important to know the strengths and limitations of the most significant trials to guide our management and research decisions. As Etzioni15 pointed out in 1972, as an increasing amount of information is made available because of additional publications and more powerful computers, there is a need for quality filters to aid the user in identifying the important information.

In 1996, epidemiologists, editors, and statisticians from the Standards of Reporting Trials group and the Asilomar Working Group met and drafted a set of recommendations for standardizing the reporting of randomized, controlled trials.16 This checklist of 21 items, along with a flowchart listing numbers of patients eligible, randomized, and treated, is called the Consolidated Standard of Reporting Trials (CONSORT) statement and is intended to improve the reporting of randomized trials. By using this information, a researcher may assess and compare the quality of randomized trials.

The acceptance of the CONSORT statement by most major journals is an effort to encourage authors to publish the essential statistical information for a thorough evaluation of these trials.16 In addition, the Cochrane collaboration has formed disease-specific study groups that evaluate the available evidence and make treatment recommendations on the basis of meta-analysis.14

As these international efforts continue for the cataloging, registration, and evaluation of prospective randomized controlled trials (PRCTs), there is a real need for a comprehensive, easily searchable and accessible database of level I evidence in oncology. In our institution, a working knowledge of the level I evidence in oncology is essential for purposes of education, identification of research questions, and development of guidelines for management. We have developed a method for the collection, review, and access to the level I evidence in oncology.


   METHODS
TOP
 ABSTRACT
 INTRODUCTION
 METHODS
RESULTS
 DISCUSSION
 REFERENCES
 
A list of human solid tumors was compiled (Table 1). A standard search strategy was used to identify PRCTs for each tumor type, which consisted of the MeSH neoplasm term corresponding to the organ being searched, limited to the subheadings "/prevention & control," "/drug therapy," "/radiotherapy," "/surgery," or "/therapy." This search was combined with publication type, "randomized control trial.pt," for the years 1991 to 2001. Alternatively, the search was combined with the text word search term "randomi$.tw" for the years 1966 to 1991. The results of the combined searches were limited to "human" studies and "English" language.


View this table:
[in this window]
[in a new window]
  		TABLE 1. Organ sites in the evidence-based oncology database
 
The resulting list of references, including abstracts, was divided by tumor type and distributed to individual surgical oncology fellows in the Memorial Sloan-Kettering Cancer Center training program. The abstracts were reviewed for inclusion in the study meeting the selection criteria of a randomized trial with a therapeutic end point. Articles with basic science end points, methodology, preliminary reports with no end point data, and retrospective analysis of end points not initially intended in the randomized trials were not included. All selected complete references were obtained from the library.

The selected references were electronically entered into a bibliographic software database (Reference ManagerTM, ISI Researchsoft, Berkeley, CA) and arranged by tumor type. A second, multi-user database was designed in FileMaker ProTM (FileMaker Inc., Santa Clara, CA) to capture information about the references obtained during the article review process. In addition to bibliographic information and the abstract imported directly from Reference Manager, the FileMaker Pro database (Fig. 1) contains information about the authors and their institution or study group. Components of the CONSORT requirements, such as hypothesis, outcome measures, randomization method, blinding, and so on, were included as well. A group of fields was included to further index the database for simple searching. The study was classified first by disease stage: local (primary tumor), regional (nodal), or metastatic. Second, treatment modality was classified: surgery, chemotherapy, radiation therapy, or immunotherapy/other. Combining two or more stages and modalities allows a search of trials of chemotherapy for regional disease or surgery for metastatic disease. Additional modifiers recorded were adjuvant therapy, neoadjuvant therapy, curative intent, and palliative intent. The database contains a summary of the strengths, weaknesses, and conclusions authored by the surgical oncology fellows and edited by the appropriate staff surgeons and physicians.



View larger version (30K):
[in this window]
[in a new window]
  		FIG. 1. Data fields collected in the Evidence-Based Oncology Database.

 
The surgical oncology fellows read all of the selected journal articles and completed the fields in the database. Articles found on review of the complete reference to be ineligible according to the previously listed criteria were deleted from the database. The fellows were asked to comment on the statistical merits of the article. More specifically, the power of the study (probability of type II error), the strengths and weaknesses of the design and completion of the study, and the overall relevance of the question and results to clinical practice were analyzed and described.

A ranking system was devised to aid in identifying the strongest articles in each field. This system divides all publications on randomized trials into three levels. The highest rank (Ia) was reserved for publications that described a well-designed, adequately powered study of a relevant question in the therapy of solid organ malignancy. These articles have influenced, or should influence, the way we practice medicine. Some publications in this group had an enormous impact on therapy and are included despite being statistically flawed, whereas others, although no longer relevant, represent an important step in the evolution of our treatment approach. Most articles fell into the Ib classification; these included well-conducted trials that did not significantly alter our treatment approach or underpowered studies that serve to offer more evidence to support other, stronger studies. Finally, the lowest rank within this group is the Ic classification, reserved for publications that do not report enough information to adequately judge the results, poorly designed and executed studies, and studies of questionable clinical relevance.

At the completion of the review process, the database was printed and reviewed by attending physicians with expertise in the fields of surgical oncology, medical oncology, and radiation oncology. The review process was focused establishing consensus on the level Ia articles for each field. Additional articles identified by review of the references or known to the experts were added at this time. The written reviews of the Ia articles were edited to reflect the effect of the findings in addition to the strengths and weaknesses of the articles.

The review process culminated in a series of presentations to the surgical oncology fellows and attending physicians during which the state of level I evidence within each tumor type was described. A summary of each Ia article and its effect was presented as well. Constructive criticisms were incorporated into the final versions of the reviews.

The database is kept current by an automated search of MEDLINE for new prospective randomized trials, which are reviewed weekly within the Department of Surgery for inclusion in the updated database.

Access to the data is provided in two forms. First, the FileMaker Pro database is used to run an internal (intranet) Web site providing easy access to the data to anyone in the institution. The Web site is database-driven with LassoTM Dynamic Markup Language (LassoTM Web Data Engine 3.5, Blue World Communications, Inc. Bellevue, WA) for access to the latest information. The multiuser database is hosted on a Windows NTTM (Microsoft Inc., Redmond, WA) network. The Web site is run on a Microsoft server running on an NT platform.

The second method of access to the data is portable. A version of the database suitable for use on the PalmTM (PalmTM, Inc., Santa Clara, CA) hand-held personal digital assistant (PDA) was developed by using FMSyncTM (FMSyncTM Software, Cupertino, CA) database synchronization software (on a Macintosh G3TM, Apple Computer, Inc., Cupertino, CA) to create a copy of the database for use in the J-FileTM (Land-J Technologies, Orlando, FL) database program for the Palm. This program provides a fully searchable form of the database, including the reference data with abstract and MESH terms, the additional stage and modality search fields, and the review section.


   RESULTS
TOP
 ABSTRACT
 INTRODUCTION
 METHODS
 RESULTS
DISCUSSION
 REFERENCES
 
The standard MEDLINE search identified 1967 articles in all listed organ sites (Table 1) since 1975. After review, 1206 articles were found to fit the inclusion criteria of randomized trials with clinical end points. This translates into 61% precision for the standard MEDLINE search. Evaluation by experts in the field identified 32 articles missed by the MEDLINE search (97% sensitivity). Critical review of the studies assigned a rank of Ia to 11% of reviewed articles and 7% of all articles found with the MEDLINE search. The breakdown for each completed tumor type is listed in Table 1.

Reference manager databases containing all included level I publications were created for each organ site. These databases were made available to the fellows for use in preparation of manuscripts regarding evidence-based management of their assigned tumor sites.

The complete database has been placed on the institutional intranet for access during conferences or for research purposes. Individual copies for the Palm PDA have been distributed to the fellows and attendings and are used daily for teaching rounds and to help answer questions by using available evidence at conferences augmenting the educational experience at our institution.


   DISCUSSION
TOP
 ABSTRACT
 INTRODUCTION
 METHODS
 RESULTS
 DISCUSSION
REFERENCES
 
The concept of evidence-based medicine evolved from a desire to grade treatment recommendations from proven efficacy, through questionable efficacy, to experimental. These recommendations are based on assigning a rank to the various types of evidence in the literature.17 Level I evidence, the highest rank, includes PRCTs and meta-analyses. With varying interpretations, levels II and below run from prospective controlled clinical trials through large observational studies and finally to expert opinion and case reports. Any clinician or researcher wishing to identify the best evidence for a given therapy, either to guide management or plan a study, needs to assemble all the level I evidence for that particular question.

Most clinicians and clinical researchers begin their search at a bibliographical database. The National Library of Medicine’s database, MEDLINE, contains more than 11 million references,18 which are searchable by title and abstract text words or by MESH terms. Significant shortcomings of the MEDLINE database lie in the facts that some journals are not even indexed in MEDLINE and that some coding is incomplete for prospective randomized trials. A number of authors have commented on the difficulty in identifying all randomized trials in MEDLINE, reporting a sensitivity1–13 of 30% to 89%. To improve the sensitivity of the search, a less specific search strategy must be used that decreases the precision of the search to the range of 10% to 30%. The MEDLINE trial type search term for prospective randomized trial was added beginning in 199119 and has improved precision. All of these reports have confirmed that regardless of the search strategy, a significant amount of time must be spent to identify the best evidence.

In 1972, Archie Cochrane20 pointed out that the provision of health care is not always based on the best available evidence. He realized that if health care providers had access to the most up-to-date reviews of the best evidence, they would provide superior health care. In addition, access to a comprehensive list of randomized trials and systematic reviews would identify research questions and avoid duplication. A project named in honor of Dr. Cochrane began in 1992 at the United Kingdom National Health Service (NHS) and was later joined by the New York Academy of Science in 1993 to form the Cochrane Collaboration. Among the many goals of the Cochrane Collaboration was the formation of a comprehensive database of randomized trials to assist researchers in the performance of systematic reviews as well as to organize the systematic reviews and recommendations on the basis of the results of randomized trials. The work of this admirable organization is available to subscribing libraries in the form of a series of searchable database modules. In addition, they have updated the coding of more than 30,000 references in MEDLINE to aid in the search for level I evidence.14 The Cochrane library is accessible to most medical center libraries by subscription. The database includes a comprehensive registry of randomized trials without a quality rating, and separately, a series of systematic reviews of specific topics. Although these reviews are very helpful, they do not cover all aspects of therapy for oncology at present.

Once the studies are identified, the clinician or researcher must still ascertain whether the methods are sound and the conclusions valid and whether they are applicable to practice. This is not always a straightforward task. In 1996, the CONSORT statement was published in response to a growing need for standards in the reporting of randomized trials.16 The elements of the statement are a checklist for the submitting author and a diagram of the study design for publication that allow the reader to easily gain the essential information on study design. A by-product of the CONSORT statement is a heightened awareness on the part of the study designer to take these essential elements into account from inception. The expectation is that the quality of the reports and the studies themselves will improve.

There are many published checklists or scales to rate the quality of randomized trials.21 The most frequently cited measures within these scales are the number of patients, the use of blinding, and the randomization methodology. Several investigators have evaluated the quality of the randomized trials in dermatology, otolaryngology, and stroke trials.22–24 They found that the overall quality of the reports, and possibly the studies themselves, were poor. By using one of these scales to weigh the results of randomized trials within meta-analyses, Moher et al.25 found that poor trial quality can be associated with a difference in effect. Therefore the results must be evaluated in the context of the quality of a specific trial.

Several institutions have established evidence-based medicine centers to review and distribute quality information and recommendations. Evidence Based Medicine Reviews is a joint venture between the American College of Physicians and the British Medical Journal which provides a search of the American College of Physicians journal club and the Journal of Evidence Based Medicine. This subscription service encompasses only reviews and studies from these two sources, so that the topics covered vary widely. The University of York hosts the NHS Center for Reviews and Dissemination (http://www.york.ac.uk/inst/crd), which includes the Database of Reviews of Effectiveness maintained by the NHS (in the UK). This database contains systematic reviews; not all areas of medicine are covered. Oxford university (http://www.evidence-basedmedicine.com) is the host for the NHS Center for Evidence-Based Medicine, and their database contains 355 critically appraised topics, mostly related to acute care. The British Medical Journal publishing group has an on-line resource (http://www.clinicalevidence.org) and a paper resource26 called "Clinical Evidence." This resource includes concise evidence summaries spanning a wide range of clinical problems. The main shortcomings of these resources are that they are not comprehensive and that none is oncology specific. The Society of Surgical Oncology has just published27 and launched an online form of their annotated bibliography (http://www.surgonc.org). This resource is not meant as a comprehensive review, nor is it limited to level I evidence, but it does highlight the important publications that support the management of solid tumors.

To provide a filtered database of PRCTs, a method was developed to aid the busy clinician or researcher in identifying the best evidence in oncology. The resulting database is 0.0001% of the current size of MEDLINE. This database is easily searchable and cataloged by tumor and treatment characteristics. Objective methodologic data are cataloged for assessment of the quality of the trials. In addition, a quality filter is included as well as a review edited by an expert in the field to put the study in context. A system is in place for the periodic contemporaneous updating of the database. The database is easily accessible via the intranet at our institution and by using a PDA. For this project, it is evident that unpublished reports as well as publications in non–MEDLINE-indexed journals will be excluded by our search, but we believe that the MEDLINE search in combination with the expert opinions will avoid excluding any significant trials.

The use of improved search terms, article ranking, and continued careful editing ensure that the highest quality publications will be identified reliably and presented with the essential details to allow critical interpretation by the user of the Memorial Sloan-Kettering Cancer Center Evidence Based Oncology database.


   Footnotes
 
Presented in part at the 54th Annual Cancer Symposium of the Society of Surgical Oncology, Washington, DC, March 17, 2001.

Received for publication March 16, 2001. Accepted for publication June 15, 2001.


   REFERENCES
TOP
 ABSTRACT
 INTRODUCTION
 METHODS
 RESULTS
 DISCUSSION
 REFERENCES
 

  1. Gotzsche PC, Lange B. Comparison of search strategies for recalling double-blind trials from MEDLINE. Dan Med Bull 1991; 38: 476–8.[Medline]
  2. Jadad AR, McQuay HJ. A high-yield strategy to identify randomized controlled trials for systematic reviews. Online J Curr Clin Trials 1993;Doc No. 33 p.
  3. Silagy C. Developing a register of randomised controlled trials in primary care. BMJ 1993; 306: 897–900.
  4. Dickersin K, Scherer R, Lefebvre C. Identifying relevant studies for systematic reviews. BMJ 1994; 309: 1286–91.[Abstract/Free Full Text]
  5. Adams CE, Power A, Frederick K, Lefebvre C. An investigation of the adequacy of MEDLINE searches for randomized controlled trials (RCTs) of the effects of mental health care. Psychol Med 1994; 24: 741–8.[Medline]
  6. Solomon MJ, Laxamana A, Devore L, McLeod RS. Randomized controlled trials in surgery. Surgery 1994; 115: 707–12.[Medline]
  7. Marson AG, Chadwick DW. How easy are randomized controlled trials in epilepsy to find on Medline? The sensitivity and precision of two Medline searches. Epilepsia 1996; 37: 377–80.[CrossRef][Medline]
  8. Balas EA, Stockham MG, Mitchell JA, Sievert ME, Ewigman BG, Boren SA. In search of controlled evidence for health care quality improvement. J Med Syst 1997; 21: 21–32.[CrossRef][Medline]
  9. Bender JS, Halpern SH, Thangaroopan M, Jadad AR, Ohlsson A. Quality and retrieval of obstetrical anaesthesia randomized controlled trials. Can J Anaesth 1997; 44: 14–8.[Medline]
  10. Nwosu CR, Khan KS, Chien PF. A two-term MEDLINE search strategy for identifying randomized trials in obstetrics and gynecology. Obstet Gynecol 1998; 91: 618–22.[CrossRef][Medline]
  11. Croft AM, Vassallo DJ, Rowe M. Handsearching the Journal of the Royal Army Medical Corps for trials. J R Army Med Corps 1999; 145: 86–8.[Medline]
  12. Langham J, Thompson E, Rowan K. Identification of randomized controlled trials from the emergency medicine literature: comparison of hand searching versus MEDLINE searching. Ann Emerg Med 1999; 34: 25–34.[CrossRef][Medline]
  13. Suarez-Almazor ME, Belseck E, Homik J, Dorgan M, Ramos-Remus C. Identifying clinical trials in the medical literature with electronic databases: MEDLINE alone is not enough. Control Clin Trials 2000; 21: 476–87.[CrossRef][Medline]
  14. Jadad AR, Haynes RB. The Cochrane Collaboration—advances and challenges in improving evidence-based decision making. Med Decis Making 1998; 18: 2–9.[Abstract/Free Full Text]
  15. Etzioni A. The need for quality filters in information systems. Science 1971; 171: 133.[Free Full Text]
  16. Begg C, Cho M, Eastwood S, et al. Improving the quality of reporting of randomized controlled trials. The CONSORT statement. JAMA 1996; 276: 637–9.[Abstract/Free Full Text]
  17. Cook DJ, Guyatt GH, Laupacis A, Sackett DL. Rules of evidence and clinical recommendations on the use of antithrombotic agents [published erratum appears in Chest 1994;105:647]. Chest 1992; 102 (Suppl 4): S305–S311.[Free Full Text]
  18. Search MEDLINE. PubMed and Internet Grateful Med, www.nlm.nih.gov/pubs/factsheets/medline.html. National Library of Medicine, 2000.
  19. Wilczynski NL, Walker CJ, McKibbon KA, Haynes RB. Assessment of methodologic search filters in MEDLINE. Proc Annu Symp Comput Appl Med Care 1993 601–5.
  20. Cochrane AL. Effectiveness and Efficiency. Random Reflections on Health Services. London: Nuffield Provincial Hospitals Trust, 1999.
  21. Moher D, Jadad AR, Nichol G, Penman M, Tugwell P, Walsh S. Assessing the quality of randomized controlled trials: an annotated bibliography of scales and checklists. Control Clin Trials 1995; 16: 62–73.[Medline]
  22. Adetugbo K, Williams H. How well are randomized controlled trials reported in the dermatology literature? Arch Dermatol 2000; 136: 381–5.[Abstract/Free Full Text]
  23. Ah-See KW, Molony NC. A qualitative assessment of randomized controlled trials in otolaryngology. J Laryngol Otol 1998; 112: 460–3.[Medline]
  24. Bath FJ, Owen VE, Bath PM. Quality of full and final publications reporting acute stroke trials: a systematic review. Stroke 1998; 29: 2203–10.[Abstract/Free Full Text]
  25. Moher D, Pham B, Jones A, et al. Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses? Lancet 1998; 352: 609–13.[CrossRef][Medline]
  26. Clinical Evidence. London: BMJ Publishing Group, 2000.
  27. Pisters P, Edge S. Surgical Oncology. Yesterday, Today and Tomorrow. Society for Surgical Oncology, 2000.




This Article
Right arrow Abstract Freely available
Right arrow Full Text (PDF)
Right arrow Alert me when this article is cited
Right arrow Alert me if a correction is posted
Services
Right arrow Similar articles in this journal
Right arrow Similar articles in PubMed
Right arrow Alert me to new issues of the journal
Right arrow Download to citation manager
Citing Articles
Right arrow Citing Articles via Google Scholar
Google Scholar
Right arrow Articles by Brooks, A. D.
Right arrow Articles by Jaques, D. P.
Right arrow Search for Related Content
PubMed
Right arrow PubMed Citation
Right arrow Articles by Brooks, A. D.
Right arrow Articles by Jaques, D. P.
Related Collections
Right arrow Education

HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS