An incentive-based approach for improving data reproducibility

An experience in an Australian university echos the statements of widespread cultural problems. While reputation damage may lead some scientists to put emphasis on reproducibility, these are rare. Many senior researchers who already have established track records and are in the twilight years of their career simply do not care and will not investigate and/or correct/retract published studies if they suspect them to be flawed. Many scientists do not want to invest the time or resources into exploring possible problems with assumptions behind the data and it returns to their own moral code to publish or not. Unfortunately, there appears to be little accountability and recourse for others to take when they suspect reproducibility problems. Australian universities mainly deal with these issues through self-governance and there is well-documented issues with obtaining access to public funding to reproduce studies. While an overhaul to reporting reproducibility issues with penalties for repeat offenders could go a long way to improving the problem there are alternatives.

It is interesting to note that the cost estimates of reproducing studies of $500 000 to $2 000 000 align closely with basic science funding from the Australian NHRMC towards individual 3-5 year research grants in biomedical research. Many early-mid career post-doctoral researchers do not have the track record to regularly obtain similar funding and many studies have documented the high rates of attrition of early career researchers. Based on these high rates of non-reproducibility it clear that track record is an extremely poor indicator of study quality or the ability to generate reproducible research.

Unfortunately, many academic researches who want to drive cultural change and improve research reproducibility levels lack the financial resources or track record to implement these changes. In water cooler conversations a colleague proposed to fund the researcher not the project. Pharmaceutical companies should listen to this suggestion and consider moving first to fund researchers in an academic environment who have a proven ability to generate reproducible research. This might be done by funding data reproduction studies or relevant new areas of research. The ability to complete the project and funding level should be gauged on research quality alone, which puts all researchers on an even footing and means that quality research is valued. Highly prized fellowships awarded for quality research would drive cultural change as it provides quality researchers who will scrutinize their data, ensure that results are reproducible and possibly take longer to publish the financial stability to complete with more abundant poorer-quality research. This might be implemented by companies advertising for research proposals in a relevant field, however, if commercial interests precluded advertising in relevant fields it could be more creative such as holding competitions to problem solve how to test if research is reproducible followed by commercial in-confidence collaborations. While detractors will no doubt raise concerns of bias with commercially funded research, strong high-quality reproducible results should be able to withstand all scrutiny.

Until someone with the financial resources and influence to promote change moves first the status quo will remain the same. Direct action from significant stakeholders could rapidly drive the revolution in research culture that is necessary to fix research reproducibility issues.