Submitted on 14 July 2023
RE: Reproducibility and research integrity report
Dear Science, Innovation and Technology Committee
As co-chairs and members of the UK Research Integrity Committee, we welcomed the publication of your recent report that concludes the inquiry into reproducibility and research integrity. Your committee’s focus on reproducibility, openness and transparency provides vital direction to the UK’s thriving research sector. Our remit, current work and forward plans align with many of your conclusions and recommendations.
On 6 July 2023 we published our first annual statement on research integrity in the UK, which we include with this letter. Our 2023 statement provides an overview of research integrity framed by the Concordat to Support Research Integrity’s five principles, as well as research misconduct. Our statement draws on recent evidence and our new analyses undertaken over the past 12 months. We suggest actions that we will carry out alongside suggested actions for the wider research sector, including organisations, disciplinary groups, and leaders.
We agree with your committee’s view that describing the sum of reproducibility challenges as a ‘crisis’ risks detracts from the successes of UK research. We also suggest that the term ‘crisis’ neither reflects the situation in many fields nor motivates positive engagement with integrity practices that underpin reproducibility, research excellence and trust in research. Our work to promote research integrity underpins, supports, and embeds reproducibility and encompasses the diverse conceptualisations and terminology preferences of reproducibility held by the numerous disciplines and fields working in the UK.
Several of your inquiry’s conclusions and recommendations relate to the UK Committee on Research Integrity. Our responses are as follows:
Recommendation 3 (Paragraph 43). “We ask that UK CORI commit to produce an annual state of the nation report on research integrity, including reproducibility, issues. These reports should contain a breakdown of the scale of these issues across different disciplines. They should also contain action plans for addressing identified issues.”
Our 2023 annual statement describes the current state of research integrity in the UK, with integrity understood through the principles set out in the UK Concordat to Support Research Integrity. In this, aspects of integrity—including openness, transparency, rigour, honesty and accountability— all support reproducibility. Together, these principles are the foundation for confidence and trust in research. Our statement includes actions that we are taking and suggests actions for the wider research sector.
Our work to listen to the needs and concerns of the research sector indicates that disciplines want to learn more from one another. However, identification of the scale of issues in particular disciplines may neither be possible nor beneficial and may harm openness about integrity. Work on integrity and reproducibility should encourage open dialogue about mistakes, and the whole research sector needs to recognise that correction of errors in the research record demonstrates honesty. Much of the UK’s cutting-edge research is catalysed in cross-disciplinary partnerships and with international groups, including in diverse disciplines and regulatory environments. A focus on individual disciplines may obscure innovations that take place across disciplines either in the UK or in international collaborations.
We recommend that it would be productive to encourage disciplines to learn from one another’s practice in integrity, reproducibility and research quality. As a way forward, we suggest that there is a need to develop open discussions about how rigour is constituted in diverse disciplines, and that there is a need to evaluate the impact of initiatives that may bolster rigour, for instance registered reports. We will continue to address new developments related to research integrity and will produce a statement annually.
Recommendation 3 cont. (Paragraph 43). “UK CORI should also publish an action plan detailing how it will carry out each of the steps set out in its strategy, including timelines and what resources will be committed for the activities it plans to carry out.”
In keeping with our commitment to transparency we published our 2023 workplan alongside our strategic plan, and our forward plan for 2023-2025. The forward plan contains information about activities and their timeframes. In our statement we encourage the whole sector to take responsibilities and actions in relation to research integrity, including reproducibility. This includes the need for a clear articulation of rigour, for openness and for visible lines of accountability.
Our 2023 statement identifies areas in which more evidence needs to be assembled, these include information about the effectiveness of research integrity training and information about the integrity of research conducted outside higher education institutions. Our statement also highlights the need for greater consistency in the way in which some information relating to integrity is collected, notably in higher education institutions’ annual statements about research integrity. This year, the committee is working with the sector to explore indicators of research integrity, mindful of the importance of bureaucracy reduction and responsible metrics.
Recommendation 4 (Paragraph 44). “UK CORI should make sure reproducibility challenges are given due attention and not overlooked in deference to other pressing research integrity issues. UK CORI should develop a sub-committee to focus on reproducibility challenges in research. This sub-committee should establish the relative weight of reproducibility within research integrity concerns, and UK CORI should use this evidence to plan its prioritisation.”
It is vital that attention is given to the challenges relating to reproducibility. Reproducibility is achieved when research is carried out with rigour and adheres to principles of openness and transparency, accountability, and honesty. We welcome the inclusion of an Arts and Humanities perspective in your report, which highlighted that the concept of reproducibility ‘is not applicable to the research methods employed’ and that transparency is used to discuss rigour in these fields. This aligns with views that we have heard from a number of fields, and AI is another field in which reproducibility might require reframing. It is important that debates about reproducibility draw on the expertise of all disciplines. Rather than establish a sub-committee on reproducibility we include a focus on reproducibility in all our activities so that it is embedded across our work. In 2023 our focus on reproducibility will include work on priority areas, including artificial intelligence, indicators of integrity, and research misconduct.
Recommendation 5 (Paragraph 45). “The UK Committee on Research Integrity (UK CORI) should specifically: investigate the impact of deploying AI—and other increasingly complex software—on reproducibility in research; consider whether specialised action is needed; and set out these findings in its annual reports.”
We are pleased that your committee also recognises the importance of AI in research. There is increased public and research awareness of AI because of the availability of generative AI platforms. AI research is large and growing, is used in research processes such as publication, is used to develop research findings and is a research area in its own right. We have convened a group of experts to focus on research integrity and AI. To date this group includes representatives from the Ada Lovelace Institute, the Royal Academy of Engineering, and the Alan Turing Institute. Our 2023 statement includes discussion of some of the concerns and opportunities afforded by AI. We expect to report further on AI, research integrity and reproducibility in 2023-2024.
Further recommendations
In addition, there were several other conclusions and recommendations on which we offer views.
Recommendations 7 and 8 (Paragraphs 60 and 61). Both recommendations relate to research misconduct. Recommendation 7: “The UK Government should lead on a co-produced framework with the UK Reproducibility Network, UKRIO and UK CORI, which sets out the roles and expectations for key actors when cases of misconduct are identified.”
Recommendation 8: “The UK Government should assess the benefits that an additional body, set up to investigate malpractice, could bring to the UK’s research integrity governance architecture.”
Clear roles and expectations about research misconduct are vital to UK research, including in assessment of potential misconduct and actions when misconduct is identified. We work closely with the UK Research Integrity Office (UKRIO) and the UK Reproducibility Network (UKRN) and look forward to continuing to do so.
We lead a programme of work to understand the current landscape and recommend next steps for addressing research misconduct. The programme benefits from an advisory group that includes representatives of UKRIO, the Concordat Signatories Group, industry and government.
In parallel, we are looking at how investigative bodies work in other countries. We note that establishment of such a body in the UK would require that it was set up with appropriate legal and regulatory powers in place. It would be important that any such body is based on robust evidence about likely impact on integrity and reproducibility. However, the remit of our committee is promotion of research integrity.
Recommendation 9 (Paragraph 68). “The research community, including research institutions and publishers, should work alongside individuals to create an environment where research integrity and reproducibility are championed.”
This recommendation aligns with recent focus on research culture and the importance of a system-wide approach to integrity and reproducibility. Reproducibility is enhanced when all parts of the research sector work together and when culture supports honesty. This includes within and between research-performing organisations, publishers, academies and learned societies.
Everyone at all levels should take responsibility for reproducibility, and it is vital that research environments support openness in relation to correction of mistakes. We support the FAIR data principles and the Concordat on Open Research Data and we note that training in matters such as research methodology are established for early career researchers, particularly in doctoral training. The ongoing shift towards greater use of data availability statements, open access of data and publications, and moves relating to openness such as authorship credit, are all positive and beneficial. In 2022-2023 we inputted into the work of the Future Research Assessment Framework. We welcome its recent publication and its considerable focus on open research and research culture.
Recommendation 28 (Paragraph 143). “Publishers and funders should work together to offer a ‘registered report partnership model.’ Not only will this benefit researchers who will receive feedback on their research plans and guaranteed publication, but greater transparency in research methodologies will be achieved, which should have a positive impact on research reproducibility.”
The registered report partnership model has been adopted in some parts of the research system. We agree that this approach may have potential to benefit reproducibility through enhancement of transparency. Registered reports may also offer potential to improve rigour in research, which also enhances reproducibility. More work is now needed to evaluate their impact on rigour and reproducibility.
We hope that you find our response comprehensive and interesting. We look forward to further discussions with you about research reproducibility and integrity.
Yours sincerely
UK Committee on Research Integrity
You can access a pdf of the response in this link: UKCORI response