Annals of Surgical Oncology Sign the Guestbook
HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS
10.1245/s10434-006-9060-6
Annals of Surgical Oncology 14:417-423 (2007)
© 2007 Society of Surgical Oncology
This Article
Right arrow Abstract Freely available
Right arrow Full Text (PDF)
Right arrow Alert me when this article is cited
Right arrow Alert me if a correction is posted
Services
Right arrow Similar articles in this journal
Right arrow Similar articles in PubMed
Right arrow Alert me to new issues of the journal
Right arrow Download to citation manager
Citing Articles
Right arrow Citing Articles via HighWire
Right arrow Citing Articles via Google Scholar
Google Scholar
Right arrow Articles by Körner, H.
Right arrow Articles by Söreide, J. A.
Right arrow Search for Related Content
PubMed
Right arrow PubMed Citation
Right arrow Articles by Körner, H.
Right arrow Articles by Söreide, J. A.

Original Article

Diagnostic Accuracy of Serum-Carcinoembryonic Antigen in Recurrent Colorectal Cancer: A Receiver Operating Characteristic Curve Analysis

Hartwig Körner, MD, PhD1,2, Kjetil Söreide, MD1, Pål Johan Stokkeland, MD1 and Jon Arne Söreide, MD, PhD, FACS1,2

1 Department of Surgery, Stavanger University Hospital, P.O. Box 8100, 4068, Stavanger, Norway
2 Institute of Surgical Sciences, University of Bergen, Bergen, Norway

Correspondence: Address correspondence and reprint requests to: Hartwig Körner, MD, PhD; E-mail: koerner{at}online.no


   ABSTRACT
TOP
 ABSTRACT
INTRODUCTION
 PATIENTS AND METHODS
 RESULTS
 DISCUSSION
 REFERENCES
 
Background: Serial measurements of carcinoembryonic antigen (CEA) are frequently used in the follow-up after colorectal cancer (CRC), but its usefulness remains debatable. Choosing the appropriate cut-off point is crucial to the diagnostic accuracy (DA) of continuous test variables. Receiver operating characteristic curve (ROC) analysis is the appropriate statistical method for this purpose, but has not been applied in previous studies.

Methods: One hundred ninety-four consecutive patients surgically treated with curative intent for CRC between July 1996 and June 1999 had systematic follow-up for five years. Follow-up included imaging, coloscopy and serial CEA measurements. Complete data including CEA measurements were available from 153 patients. ROC analysis of CEA was done with regard to detection of recurrent disease.

Results: Depending on the chosen cut-off value of CEA, DA varied widely within the normal range (CEA ≤10 U/ml). CEA >4 U/ml provided the highest sensitivity (0.78) and specificity (0.91), compared to a sensitivity and specificity at the upper normal range (CEA = 10 U/ml) of 0.51 and 0.99, respectively. Thirty-three patients (24%) developed recurrence. Among 11 (5%) asymptomatic patients diagnosed by elevated CEA levels, only two patients (1.5%) were amenable to secondary curative surgery. A threefold increase of CEA in an individual patient had the same DA as the best cut-off value (>4 U/ml).

Conclusions: Diagnostic accuracy of CEA in follow-up after curative surgery for CRC is influenced by the chosen cut-off value. A threefold increase of CEA may indicate recurrent disease. The value of serial measurement of CEA was limited.

Key Words: Colorectal cancer • Follow-up • CEA • Recurrence • ROC analysis • Diagnostic accuracy

Abbreviations: CI, confidence interval • CEA, carcinoembryonic antigen • CRC, colorectal cancer • DA, diagnostic accuracy • PPV, positive predictive value • NPV, negative predictive value • LR, likelihood ratio • ROC, receiver operator characteristic analysis • AUC, area under the curve


   INTRODUCTION
TOP
 ABSTRACT
 INTRODUCTION
PATIENTS AND METHODS
 RESULTS
 DISCUSSION
 REFERENCES
 
Colorectal cancer (CRC) represents a formidable health burden, with approximately one million new cases diagnosed annually in the Western world.1 Surgery remains the mainstay of therapy, but recurrence after curative resection of colorectal cancer occurs in 30–40% of patients.2–5 Accordingly, various follow-up strategies have been applied after curative surgery for CRC in order to detect recurrent disease, but they have resulted in debate concerning their efficacy and compliance as well as their cost to implement.5,6 Surgical indications for metastatic CRC are expanding, with a demonstrated survival benefit even for patients with extensive disease.7,8 Over the past three decades, carcinoembryonic antigen (CEA) has been widely used as a tumor marker to detect asymptomatic or early recurrences amenable to curative surgery. Also, CEA has been studied extensively in order to evaluate its place in clinical practice.9–17 Use of this tumor marker has been shown to be cost-effective, but its possible survival benefit remains controversial.14,15,18

CEA is a diagnostic test with a continuous spectrum of test results. Therefore, diagnostic properties expressed by sensitivity, specificity, positive predictive value (PPV) and negative predictive value (NPV) or likelihood ratio (LR) depend on the chosen cut-off value used to differentiate between normal and pathologic values.

Receiver operating characteristic curve (ROC) analysis is an appropriate statistical method for assessing the diagnostic accuracy (DA) of a test with a continuous spectrum of results. The area under the curve (AUC) is a measure of the overall DA of the test, and the cut-off value providing the highest sensitivity and specificity is calculated.19–21

While most studies have analyzed CEA at a single cut-off level (e.g., the upper limit of the normal range or at a particular specificity level),12,18,22 ROC analysis of CEA has not been used with regard to follow-up after colorectal cancer according to the literature.23–26 The purpose of this study was to analyze the DA of CEA by means of ROC analysis in patients undergoing systematic surveillance for recurrent disease after curative surgery for CRC in a large, unselected, population-based cohort.


   PATIENTS AND METHODS
TOP
 ABSTRACT
 INTRODUCTION
 PATIENTS AND METHODS
RESULTS
 DISCUSSION
 REFERENCES
 
Between July 1996 and June 1999, 314 consecutive patients from a catchment area of 285,000 inhabitants had curative surgery for adenocarcinoma of the colon and rectum.5 Our institution is the only one providing surgery for colorectal cancer for this population. Accordingly, our patient cohort is to be considered to be population-based.

Preoperative serum-CEA was analyzed routinely. Patients <75 years of age (n = 194, 62%), had a systematic postoperative follow-up schedule according to the guidelines of the Norwegian GastroIntestinal Cancer Group (NGICG).27 The follow-up schedule included serial CEA measurements—in addition to imaging studies of the liver, lungs and colonoscopy—until the end of the follow-up period (five years) or until a diagnosis of recurrence was made, Table 1Go.5,27 Complete CEA values for analysis of DA were available in 153 patients (79%). This study cohort was similar to all the 194 patients with regard to age, gender and tumor characteristics.


View this table:
[in this window]
[in a new window]

  		TABLE 1. Follow-up schedule after curative surgery for colorectal cancer according to the recommendations of the Norwegian Gastro-Intestinal Cancer Group (NGICG)
 
Definitions
Recurrence was defined as either a locoregional recurrence (tumor tissue at the primary site of resection, either intra- or extraluminal) and/or distant spread and/or a metachronous CRC. Disease-free interval was defined as the time interval from primary treatment until the diagnosis of recurrence, or until end of follow-up. Patients with a disease-free interval of five years or more were considered as cured of CRC. Evaluation of CEA during follow-up was set as either the CEA value associated with the diagnosis of recurrence, or in the event of no evidence of disease at completion of follow-up.

The reference standard for the accuracy of CEA measurement was either the diagnosis of recurrence established by biopsy and/or imaging studies (positive target patient) or a disease-free interval of 60 months or more without proof of recurrence (negative target patient; end of follow-up).

Sensitivity was defined as the fraction of all diseased patients with a positive test (true positives). Specificity was defined as the fraction of all healthy patients with a negative test (true negatives). Likelihood ratio for a positive test (LR+) was defined as the ratio of the fractions of the true positives and false positives (sensitivity/1–specificity). Likelihood ratio for a negative test (LR–) was defined as the fraction of the false negatives and true negatives (1–sensitivity/specificity).28,29

Pre-test probability was defined as the probability that a patient suffers from recurrence, i.e., prevalence of recurrent disease after curative resection. Post-test probability was defined as the probability of recurrent disease in a patient with a certain test result of CEA. Thus, the LR+ is a measurement of the increase in probability of disease given a certain test result. LR+ >10 or LR– <0.1 were considered to be excellent test results (LR+ >10: a positive test is ten times more likely to occur in a patient with recurrence; LR– <0.1: a false-negative test result in less than one out of ten patients with a negative test).28

Laboratory Analysis of CEA
CEA was analyzed with the immunoassay kit from Abbot Diagnostic Division, Abbot Park, IL, USA. The upper limit of normal values was <10 U/ml.

Statistical Analysis
Categorical data were analyzed by frequency tables and the chi-square test. Continuous variables were tested for normality by Kolmogorov–Smirnov test. CEA data were found to have non-normal distributions (Kolmogorov–Smirnov test P < 0.001) for both raw data and after log transformation. Central tendency was expressed by median with a 95% con-fidence interval (CI), and variables were compared by Mann–Whitney U-test.

For the analysis of diagnostic accuracy, the CEA value at evaluation (i.e., CEA value at last follow-up or at the diagnosis of recurrence) was compared with first postoperative CEA value of the follow-up schedule. We adopted this approach from our clinical practice to follow serial CEA measurements as a marker of potential recurrence, by comparing the most recent measurement with the immediate postoperative value. Increased CEA may indicate both locoregional recurrence and distant spread, but might also occur when a metachronous CRC develops. Consequently, CEA was analyzed with regard to all events. Data were stratified with regard to Dukes classification. The ratio of the CEA value at evaluation to the postoperative CEA (hereafter referred to as the "CEA ratio") was calculated to assess the increase in CEA, which indicates recurrence of CRC.

We performed ROC analysis of the CEA value at the time of evaluation, and of the CEA ratio values at evaluation. The area under the ROC curve (AUC) with 95% CI was calculated to express the overall diagnostic accuracy (DA) of the test. Cut-off values with the highest sensitivity and specificity, PPV, NPV, as well as positive and negative likelihood ratio (LR+ and LR–) were calculated. Time-dependent events were analyzed by Kaplan–Meier statistics and compared by log-rank test.

Statistical analysis was performed with MedCalc Statistical Software v. 8.1.1 (Mariakerke, Belgium).


   RESULTS
TOP
 ABSTRACT
 INTRODUCTION
 PATIENTS AND METHODS
 RESULTS
DISCUSSION
 REFERENCES
 
Diagnostic Accuracy of CEA
The tumor characteristics are given in Table 2Go. Thirty-seven patients (24%) developed recurrence, Table 3Go. Five-year overall survival in patients without recurrence was 81%. Median survival in patients with recurrence was 40 months (P < 0.0001). Preoperative and postoperative CEA values, and the values at evaluation are shown in Table 4Go and Fig. 1Go. Distribution of CEA by Dukes stages showed preoperatively increased values only in stages B and C. All patients with increased preoperative CEA values (CEA >10 U/ml; n = 21, 16%) had postoperative CEA measurements within the reference range (≤10 U/ml). The first postoperative CEA measurement was done within six months after surgery.5 The distribution pattern pre- and postoperatively as well as at evaluation was similar with regard to recurrence (Kruskal–Wallis test; P = 0.68), but some of the patients with Dukes A had recurrence associated with increased CEA measurements, while they had normal CEA values at the time of primary diagnosis. Eighteen (49%) of 37 patients with recurrence had CEA values within the normal range (≤10 U/ml).


View this table:
[in this window]
[in a new window]

  		TABLE 2. Tumor characteristics of 153 patients treated with curative surgery for colorectal cancer
 

View this table:
[in this window]
[in a new window]

  		TABLE 3. Recurrences in 37 (24%) of 153 patients treated with curative surgery for colorectal cancer. Six patients (2%) developed both locoregional recurrence and distant spread
 

View this table:
[in this window]
[in a new window]

  		TABLE 4. Analysis of carcinoembryonic antigen in 153 patients undergoing curative surgery for colorectal cancer. Preoperative and postoperative values are given, as well as those for evaluation (either at end of follow-up or at diagnosis of recurrence)
 

Figure 1
View larger version (11K):
[in this window]
[in a new window]

  		FIG. 1. Distribution plot of CEA values preoperatively, postoperatively and at evaluation according to Dukes stadium. CEA values are given in a logarithmic scale.

 
Results from the ROC analysis are shown in Table 5Go. CEA values and CEA ratio at evaluation had similar AUC values (0.88 and 0.91 respectively). This indicates an overall DA of 88% and 91% in patients with increased CEA values at recurrence (Fig. 2Go). A cut-off value of >4 U/ml provided the highest sensitivity and specificity. Sensitivity of CEA in the upper normal range was 0.51, as compared to 0.73 when the cut-off value (CEA of 4 U/ml) defined by ROC analysis was used. This difference resulted in an increased fraction of false negative test results, by 22% (Fig. 3Go).


View this table:
[in this window]
[in a new window]

  		TABLE 5. Receiver operating characteristics curve analysis of CEA and the increase in CEA (ratio of CEA at evaluation to CEA postoperatively) with regard to the diagnosis of recurrence after curative surgery for colorectal cancer. Values are given with 95% CI
 

Figure 2
View larger version (18K):
[in this window]
[in a new window]

  		FIG. 2. Comparison of receiver operating characteristics (ROC) curves for CEA and the CEA ratio (ratio of CEA at evaluation to postoperative CEA). The areas are nearly identical, indicating similar diagnostic accuracies. For further explanation, see Table 5Go.

 

Figure 3
View larger version (29K):
[in this window]
[in a new window]

  		FIG. 3. Sensitivity and specificity with 95% confidence intervals (CI) of the observed test spectrum of CEA. The upper cut-off value of the reference range is marked with a black line (10 U/ml). CEA values are given on a logarithmic scale. Sensitivity decreases from nearly 100% at the lowest value to 51% at the upper normal border.

 
The results for LRs are shown in Table 5Go. The LR+ increased from lower to higher values within the normal range. Thus, the diagnostic abilities of the test changed from low in the range of CEA ≤4 U/ml to excellent in the upper half of the normal range (CEA 5–10 U/ml). A three-fold increase of CEA was associated with a 20-fold increase in the probability of recurrence as compared to the pre-test probability. LR– performed best in the lower end of the normal range.

ROC analysis was also done for both variables with regard to Dukes stages. The best cut-off value for the CEA measurement changed within the normal range of 10 U/ml (>3 U/ml in Dukes A, >8 U/ml in Dukes B, and >4 U/ml in Dukes C).

The cut-off value for the ratio of CEA at evaluation and postoperatively was the same in all Dukes stages (ratio of 3). The results from ROC analysis with regard to AUC, sensitivity, specificity, likelihood ratios, and predictive values were similar within the Dukes stage.

Clinical Outcome
Seventeen of the 37 patients (49%; 11% of all patients) had an asymptomatic recurrence. Fifteen patients (40%; 10% of all patients) had curative surgery for recurrent disease (Table 6Go). Eleven asymptomatic patients (30%; 7% of all patients) were diagnosed solely due to increased CEA level. Only two of these patients (5%; 1.3% of all patients) were amenable to curative resection of recurrence.


View this table:
[in this window]
[in a new window]

  		TABLE 6. Surgical procedures performed with curative intent for recurrences of colorectal cancer
 

   DISCUSSION
TOP
 ABSTRACT
 INTRODUCTION
 PATIENTS AND METHODS
 RESULTS
 DISCUSSION
REFERENCES
 
According to our knowledge, this is the first report on the application of ROC curve analysis in order to evaluate the DA of serial CEA monitoring in the follow-up after curative surgery for colorectal cancer. Our results show that the cut-off point with the highest sensitivity and specificity (>4 U/ml) was lower than the upper normal limit of the test (10 U/ml). Sensitivity was low (0.73), and this may reflect the fact that only 19 of 37 (51%) of the patients with recurrence had increased CEA measurements. The sensitivity decreased to 0.51 when the upper normal limit was applied (Table 5Go and Fig. 3Go). Because positive and negative results of a diagnostic test with a continuous spectrum usually overlap to some degree, the choice of the cut-off value determines the fractions of possible false positives and false negatives.19 Consequently, the choice of cut-off value is of crucial importance to the DA of the test. In our study, 22% more patients with recurrence were falsely classified as healthy when the upper normal range was used, compared to the cut-off value revealed by ROC analysis. To our knowledge, DA of CEA has been analyzed hitherto only with regard to single cut-off values.10,11,14,30

ROC analysis of the ratio CEA revealed that a more than threefold increase was as accurate as the absolute CEA value for detection of recurrence. This observation shows that the absolute value of CEA is not sufficient for a correct interpretation of the test result, but has to be interpreted within the context of previous tests. Increase of CEA within the normal range (e.g., from 3 to 9 U/ml) was associated with recurrence in some patients, which strengthens the observation. In the various Dukes stages, where the best cut-off values were different within the normal range, the ratio CEA remained unchanged. A twofold increase of CEA was used in a study protocol for recurrent colorectal cancer, without an explained statistical rationale.10 As far as we know, the factor of increase of CEA has not been addressed extensively in the literature. However, half of the patients with recurrent disease had no CEA increase, and therefore no benefit from serial CEA measurement.

The results for PPV and NPV have to be interpreted with caution, because they depend directly on the prevalence of the disease.28 The figures in our study are based on a recurrence rate of 24% in an unselected population-based group of patients, and our results should apply well to a general Western population. In contrast, the use of LR avoids this problem.28 Our study showed LR+ figures of 9 for CEA >4 U/ml and 20 for the ratio of CEA, which indicate a nine- and 20-fold increase in the probability of disease with a positive test result, as compared to the pre-test probability. In contrast, the LR– was only 0.27 and 0.24, respectively, which translates into one out of four patients with recurrence being falsely classified as healthy. Thus, a negative test result was not able to rule out recurrence, as a negative LR lower than 0.10 is generally considered to be a clinically useful test.21,31 Truly, this reflects the fact that many patients will have normal CEA values in spite of recurrent disease.

The CEA values are not corrected for smoking habits of the patients. It is known that CEA values can be falsely elevated in smokers.32 However, CEA measurements were taken serially in the individual patients preoperatively and during follow-up, and the individual patients are regarded to be their own controls. The event of a cancer disease might change smoking habits in some patients, but we do not think that this could influence our results. We are not aware of any study reporting different reference limits for smokers and nonsmokers. It is important to note that results from our analysis apply to the particular immunoassay used during the study period at our institution.

The ideal situation for evaluating a diagnostic test would be to have a reference standard which reflects the truth, and that the results from this particular test would not influence the reference standard.29 In the present study, recurrence was confirmed by biopsy, endoscopy and/or by imaging. In addition, a long follow-up time of at least five years ensured that the patients could be classified clearly at evaluation as either healthy or diseased. It might be possible that CEA changes prompted further examinations and thus led to a work-up bias and a falsely high DA. However, as CEA was used with other diagnostic tools according to the follow-up schedule (Table 1Go), we believe a work-up bias is less likely. A study where clinicians are blinded for CEA results and possible clues for recurrent disease would be ethically controversial and difficult to perform. We think therefore that our data from a large population-based patient group are highly valid.

CEA is one of the most frequently used tests in follow-up after CRC, and a large number of studies on it have been published since the 1970s.10–14,30,33 Several authors conclude that serial measurement of CEA is of limited value in the follow-up of colorectal cancer,11,30 while others found CEA measurements to be crucial in the management of these patients, even when they reported that only between 2% and 3% could benefit from a curative resection of recurrence based on CEA measurement.15,18 In our study, recurrence was diagnosed in eleven asymptomatic patients out of 153 (7%) by CEA monitoring, and only two of them (1.3%) could undergo curative surgery.

In conclusion, serial measurement of CEA during follow-up after curative surgery was of limited value. The choice of the cut-off value is of great importance when interpreting the test results. Many patients will never have increased CEA levels, even when incurable recurrence occurs. These aspects are, in our opinion, of great importance for the clinical use of CEA and for future studies of CEA relating to the follow-up of patients with colorectal cancer.


   ACKNOWLEDGMENTS
 
This study was financially supported by the Centre for Clinical Research, Armauer Hansens House, Haukeland University Hospital, Bergen. The authors wish to thank the Clinical Cancer Research Office, Haukeland University Hospital, Bergen, for valuable help with the database.


   FOOTNOTES
 
The study was financially supported by the Centre for Clinical Research, Armauer Hansens House, Haukeland University Hospital, Bergen.

Received for publication January 9, 2006. Accepted for publication March 11, 2006.


   REFERENCES
TOP
 ABSTRACT
 INTRODUCTION
 PATIENTS AND METHODS
 RESULTS
 DISCUSSION
 REFERENCES
 

  1. Weitz J, Koch M, Debus J, et al. Colorectal cancer. Lancet 2005; 365:153–65.[CrossRef][Medline]
  2. Biggs CG, Ballantyne GH. Sensitivity versus cost effectiveness in postoperative follow-up for colorectal cancer. Curr Opin Gen Surg 1994; 94–102.
  3. Bruinvels DJ, Stiggelbout AM, Kievit J, et al. Follow-up of patients with colorectal cancer. A meta-analysis. Ann Surg 1994; 219:174–82.[Medline]
  4. Detry R. Follow-up after curative surgery for colorectal cancer. Acta Gastroenterol Belg 2001; 64:268–71.[Medline]
  5. Körner H, Söreide K, Stokkeland PJ, Söreide JA. Systematic follow-up after curative surgery for colorectal cancer in Norway: A population-based audit of effectiveness, costs, and compliance. J Gastrointest Surg 2005; 9:320–8.[CrossRef][Medline]
  6. Audisio RA, Setti-Carraro P, Segala M, et al. Follow-up in colorectal cancer patients: a cost–benefit analysis. Ann Surg Oncol 1996; 3:349–57.[Abstract]
  7. Fong Y, Cohen AM, Fortner JG, et al. Liver resection for colorectal metastases. J Clin Oncol 1997; 15:938–46.[Abstract/Free Full Text]
  8. Weber SM, Jarnagin WR, DeMatteo RP, et al. Survival after resection of multiple hepatic colorectal metastases. Ann Surg Oncol 2000; 7:643–50.[Abstract]
  9. Wanebo JH, Stearns M, Schwartz MK. Use of CEA as an indicator of early recurrence and as a guide to a selected second-look procedure in patients with colorectal cancer. Ann Surg 1978; 188:481–93.[Medline]
  10. Wood CB, Ratcliffe JG, Burt RW, et al. The clinical significance of the pattern of elevated serum carcinoembryonic antigen (CEA) levels in recurrent colorectal cancer. Br J Surg 1980; 67:46–8.[Medline]
  11. Persijn JP, Hart AA. Prognostic significance of CEA in colorectal cancer: a statistical study. J Clin Chem Clin Biochem 1981; 19:1117–23.[Medline]
  12. Armitage NC, Davidson A, Tsikos D, Wood CB. A study of the reliability of carcinoembryonic antigen blood levels in following the course of colorectal cancer. Clin Oncol 1984; 10:141–7.[CrossRef][Medline]
  13. Northover J. The use of prognostic markers in surgery for colorectal cancer. Eur J Cancer 1995; 31A:1207–9.
  14. Duffy MJ. Carcinoembryonic antigen as a marker for colorectal cancer: is it clinically useful?. Clin Chem 2001; 47:624–30.[Abstract/Free Full Text]
  15. Wichmann MW, Müller C, Lau-Werner U, et al. The role of carcinoembryonic antigen for the detection of recurrent disease following curative resection of large-bowel cancer. Langenbecks Arch Surg 2000; 385:271–5.[CrossRef][Medline]
  16. Chau I, Allen MJ, Cunningham D, et al. The value of routine serum carcinoembryonic antigen measurement and computed tomography in the surveillance of patients after adjuvant chemotherapy for colorectal cancer. J Clin Oncol 2004; 22:1420–9.[Abstract/Free Full Text]
  17. Kievit J. Follow-up of patients with colorectal cancer: numbers needed to test and treat. Eur J Cancer 2002; 38:986–99.[CrossRef][Medline]
  18. Wichmann MW, Lau-Werner U, Müller C, et al. Carcinoembryonic antigen for the detection of recurrent disease following curative resection of colorectal cancer. Anticancer Res 2000; 20:4953–5.[Medline]
  19. Robertson EA, Zweig MH. Use of receiver operating characteristic curves to evaluate the clinical performance of analytical systems. Clin Chem 1981; 27:1569–74.[Abstract/Free Full Text]
  20. Zweig MH. Evaluation of the clinical accuracy of laboratory tests. Arch Pathol Lab Med 1988; 112:383–6.[Medline]
  21. Zweig MH, Campbell G. Receiver-operating characteristic (ROC) plots: a fundamental evaluation tool in clinical medicine. Clin Chem 1993; 39:561–77.[Abstract/Free Full Text]
  22. Carriquiry LA, Pineyro A. Should carcinoembryonic antigen be used in the management of patients with colorectal cancer?. Dis Colon Rectum 1999; 42:921–9.[CrossRef][Medline]
  23. Stajic S, Novakovic R, Kostic N, et al. The importance of monitoring colorectal adenomas by tumor markers (ROC analysis). Med Pregl 1993; 46:69–71.
  24. Vallejo J, Torres-Avisbal M, Contreras P, et al. CEA, CA 19.9 and CA 195 in patients with colorectal carcinoma. ROC analysis. Rev Esp Med Nucl 1999; 18:281–6.[Medline]
  25. Pasanen PA, Eskelinen M, Partanen K, et al. Receiver operating characteristic (ROC) curve analysis of the tumour markers CEA, CA 50 and CA 242 in pancreatic cancer; results from a prospective study. Br J Cancer 1993; 67:852–5.[Medline]
  26. Silver HK, Archibald BL, Ragaz J, Coldman AJ. Relative operating characteristic analysis and group modeling for tumor markers: comparison of CA 15.3, carcinoembryonic antigen, and mucin-like carcinoma-associated antigen in breast carcinoma. Cancer Res 1991; 51:1904–9.[Abstract/Free Full Text]
  27. Norwegian Gastro-Intestinal Cancer Group (NGICG) Kolorektalcancer og Analcancer. En Veiledning for Leger Oslo: NGICG; 1999.
  28. Altman DG. Diagnostic tests. In: Practical Statistics for Medical Research. London: Chapman and Hall, 1991:411–416.
  29. Bhandari M, Guyatt GH. How to appraise a diagnostic test. World J Surg 2005; 29:561–6.[CrossRef][Medline]
  30. Moertel CG, Fleming TR, Macdonald JS, et al. An evaluation of the carcinoembryonic antigen (CEA) test for monitoring patients with resected colon cancer. JAMA 1993; 270:943–7.[Abstract/Free Full Text]
  31. Simel DL, Samsa GP, Matchar DB. Likelihood ratios with confidence: sample size estimation for diagnostic test studies. J Clin Epidemiol 1991; 44:763–70.[CrossRef][Medline]
  32. Fukuda I, Yamakado M, Kiyose H. Influence of smoking on serum carcinoembryonic antigen levels in subjects who underwent multiphasic health testing and services. J Med Syst 1998; 22:89–93.[CrossRef][Medline]
  33. Wanebo HJ. Cancer trends: the role of CEA in managing colorectal cancer. VA Med 1983; 110:103–8.[Medline]



This article has been cited by other articles:


Home page
 J. Clin. Pathol.Home page
K. Soreide
Receiver-operating characteristic curve analysis in diagnostic, prognostic and predictive biomarker research
J. Clin. Pathol., January 1, 2009; 62(1): 1 - 5.
[Full Text] [PDF]

This Article
Right arrow Abstract Freely available
Right arrow Full Text (PDF)
Right arrow Alert me when this article is cited
Right arrow Alert me if a correction is posted
Services
Right arrow Similar articles in this journal
Right arrow Similar articles in PubMed
Right arrow Alert me to new issues of the journal
Right arrow Download to citation manager
Citing Articles
Right arrow Citing Articles via HighWire
Right arrow Citing Articles via Google Scholar
Google Scholar
Right arrow Articles by Körner, H.
Right arrow Articles by Söreide, J. A.
Right arrow Search for Related Content
PubMed
Right arrow PubMed Citation
Right arrow Articles by Körner, H.
Right arrow Articles by Söreide, J. A.

HOME HELP FEEDBACK SUBSCRIPTIONS ARCHIVE SEARCH TABLE OF CONTENTS