Proteomic Profiling in Ovarian Cancer: Is it Plausible?
Abstract & Commentary
By Robert L. Coleman, MD, Associate Professor, University of Texas; M.D. Anderson Cancer Center, Houston TX. Dr. Coleman is on the speaker’s bureau for GlaxoSmithKline, Bristol Myers-Squibb, and Ortho Biotech.
Synopsis: The reproducibility of the proteomic profiling approach has yet to be established.
Source: Baggerly KA, et al. Signal in noise: evaluating reported reproducibility of serum proteomic tests for ovarian cancer. J Natl Cancer Inst. 2005;97:307-309.
Reproducibility of complex proteomic signatures in detecting early stage ovarian cancer has proven to be a significant challenge in the field of biomarker discovery. Although early reports of proteomic profiling demonstrated near 100% sensitivity and specificity in discriminating ovarian cancer from non-cancer, the ability to reproduce these results across datasets has been difficult. Baggerly and colleagues examined 2 proteomic spectral datasets available in the public domain with the intent of testing the published classification method and assessing reproducibility. They then calculated the probability that classification could occur by chance alone. Using statistical methodology similar to the original reports then followed by random allocation of spectral clusters, Baggerly et al report that the predictive spectral peaks were less accurate (less than 80%) than originally reported. Applying a separate methodology (jack-knife approach) to test the original peaks to a second dataset produced high accuracy (98.4%). However, this level of accuracy was met or exceeded in 6% of the simulations using completely random values. On the basis of these 2 findings, Baggerly et al concluded that proteomic signature profiling to accurately and reproducibly discriminate cancer from non-cancer was not plausible and may be the effect of procedural bias.
Commentary
News of accurate discrimination of cancer patients from unaffected patients via evaluation of blood samples was a welcomed and widely publicized advance in the field of ovarian cancer screening research. Although difficult to understand, the technology of developing and interpreting proteomic signatures from banked blood of both cancer patients, patients with non-cancerous lesions, and normal controls, appeared to offer renewed hope in the ability to accurately classify patients with disease, particularly those with early stage disease where survival is the best. While the data being produced were patterns (spectral peaks) recognizable only by sophisticated statistical algorithms sensitivity, specificity and accuracy were very high and far better than our best (and current) biomarker methodology. However, almost as soon as the data were reported, news of the inability to accurately reproduce the data permeated the clinical community who were already baited with the promise of a commercial product awaiting approval. The current report, on the surface, seems to dampen that enthusiasm even more where the results of 2 independent data sets were found to not be reproducible and could be explained by chance—that is by overfitting. Overfitting errors are encountered when a multivariate model is used to fit a very large number of possible predictors. Indeed, Baggerly et al demonstrated that simply choosing random spectral peak values for a prediction model met or exceeding the predictive accuracy of the historical model in 6% of their samplings. This would imply that the plausibility for prediction based on the public domain datasets was low and confounded.
The Baggerly article is followed by 2 commentaries in the Journal of the National Cancer Institute that help to put this apparent conundrum into perspective.1 We find from the original authors of one of the sentinel articles that the datasets placed into the public domain were generated from experimental settings where one or more parameters were purposefully altered to study the effects on output spectra. In this situation, the datasets would likely not be reproducible as the baseline between sets is intentionally and artificially adjusted. In addition, it was raised that issues of reproducibility are premature as the technology is a moving target and continuously updated. However, as is discussed in the second commentary, reproducibility is the lifeblood of moving this complex promising field (termed “-omics”) forward. It is further complicated by the dichotomy of discovery; on the one hand, you have a test based on pattern recognition alone, where the results are based on unknown and un-described proteins, and on the other, you have biomarkers development, which would take on the tedious process of describing these proteins, showing their relevance (biological impact) to ovarian cancer and then develop a measurable antibody in the patient’s sera. Calls for a new trial have been made, where reproducibility is the mission, in a single data set with careful attention to statistical methodology.2,3
What is clear from the 10 pages of commentary that accompanies the 3-page brief communication is that this field holds tremendous promise in promoting our ability to identify “at-risk” patients, but must be embarked upon carefully. In addition, free interchange and discussion through publicly disseminated data will help perfect this important task.
References
1. Ransohoff DF. Lessons from controversy: ovarian cancer screening and serum proteomics. J Natl Cancer Inst. 2005;97:315-319.
2. Zhang Z, et al. Three biomarkers identified from serum proteomic analysis for the detection of early stage ovarian cancer. Cancer Res. 2004;64:5882-5890.
3. Zhu W, et al. Detection of cancer-specific markers amid massive mass spectral data. Proc Natl Acad Sci USA. 2003;100:14666-14671.
The reproducibility of the proteomic profiling approach has yet to be established.You have reached your article limit for the month. Subscribe now to access this article plus other member-only content.
- Award-winning Medical Content
- Latest Advances & Development in Medicine
- Unbiased Content