The notion of reproducibility is fundamental to good science.
If a finding cannot be reproduced by independent research groups, its relevance is very limited, regardless of its validity. It is therefore imperative that scientists describe their experiments in sufficient detail so that they can be reproduced, challenged and built upon.
However, due to recent technological advances in the biological and computational sciences, experimental protocols, data analysis and interpretation have become increasingly complex. This has made reproducing research findings more challenging.
As the number of high-profile cancer studies that cannot be reproduced continues to grow (see eLife Reproducibility Project), some researchers have suggested that the biomedical sciences are experiencing a “reproducibility crisis”.
To address this important issue, the US Food and Drug Administration (FDA) led the MicroArray and SEquencing Quality Control (MAQC/SEQC) projects to assess the reproducibility of the new high-throughput biotechnologies used in cancer research for diagnosis and prognosis. Recognizing that new technologies and analytical approaches are being developed at a rapid pace, members of the MAQC/SEQC consortium decided to form a new international society, the Massive Analysis and Quality Control (MAQC) Society, whose mission is to promote the best research practices for enhanced reproducibility.
As described in a letter published in Nature Biotechnology, the initial focus of the new society will be on data analysis. “While it is obvious that assays used in biomedical studies must be robust, the reproducibility of computational analyses is a relatively new concept that is yet to be fully adopted by the scientific community,” says Dr. Benjamin Haibe-Kains, Scientist at the Princess Margaret (PM) Cancer Centre and co-founder of the MAQC Society.
Given the complexity and diversity of the computational analyses used in biomedical research, the society will select examples of fully reproducible studies, which will be collectively analyzed in order to draft practical guidelines for future studies. “This is an ambitious initiative that will only be possible with the support of the society members who will directly benefit from the resulting guidelines,” adds Dr. Haibe-Kains.
To foster collaborations and the active involvement of the community, Carl Virtanen, Director and Research Lead of UHN Digital and head of the Bioinformatics and HPC Core, created the MAQC Society website. “I followed with great interest the previous efforts led by the FDA to assess the reproducibility of cutting-edge technologies used in cancer research and was involved in similar initiatives back in the era of microarrays. Because the Bioinformatics Core also runs clinical software pipelines for genetic testing at UHN which are, by law, mandated to be fully reproducible, I was eager to get involved and help build a web-resource for the new MAQC Society,” says Mr. Virtanen. “Data analysis is a critical part of all cancer studies, and we must ensure that the studies’ findings are fully reproducible. This initiative is vital and will help to ensure that the massive investments in cancer research deliver benefits to patients and the health system in the long-term.”
This initiative was supported by The Princess Margaret Cancer Foundation.
The international MAQC Society launches to enhance reproducibility of high-throughput technologies. Shi L , Kusko R, Wolfinger RD, Haibe-Kains B , Fischer M, Sansone SA, Mason CE, Furlanello C , Jones WD, Ning B, Tong W. Nature Biotechnology, Volume 35, Number 12, December 2017.