The ADD blog provides insight into the ADD project’s research across six university partners. Meet our researchers from Aalborg University, Aarhus University, Copenhagen Business School, Roskilde University, the University of Copenhagen, and the University of Southern Denmark. Read about their projects, activities, ideas, and thoughts—and gain a new perspective on the controversies and dilemmas we face in the digital age, along with ideas on how to strengthen digital democracy.
By David Budtz Pedersen
Recently, the European Commission presented its proposal for the use of artificial intelligence in scientific institutions at a seminar in Brussels. Several of the participants expressed very high hopes for the many tasks in science that AI can help solve. As in other areas of the public and private sector, AI is expected to facilitate more efficient workflows in the research ecosystem. But the European research community needs to consider carefully the consequences that AI may have for scientific integrity and ethics.
In various instances, the use of generative AI in scientific projects is harmless. Like with other digital solutions, AI is finding a natural application in research. This applies to AI assistants that help to translate texts, take meeting notes, suggest grammatical improvements, suggest layout for publications, or find scientific references. But even at this stage, challenges to integrity can arise. For example, evaluations of large language models show that they sometimes invent references to scientific studies that do not actually exist, and that recounts of complex scientific theories need rigorous fact-checking. To avoid hallucinations, human oversight is becoming increasingly relevant. For even rudimentary AI use, scientists need to be aware of risks and shortcomings – and confirm their commitment to test for reliability.
But challenges become even more persistent with more pervasive use cases. Several major data companies, including large international publishers, increasingly profit from selling metadata from research. They see a great potential in using AI more extensively and train proprietary models on peer reviewed research. Because large publishers own enormous amounts of scientific data, they are able to deploy AI models to perform a number of more fundamental research tasks.
Examples include using AI to write original texts, perform statistical analysis, conduct simulations, propose new hypotheses, design experiments, develop new research questions, define work processes, and assess and evaluate research proposals, articles and results. In a not-too-distant theoretical future, it is possible that AI will perform or facilitative almost all parts of the research process, from hypothesis to data collection and analysis to publication and assessment.
In this scenario, AI-based research will pose a number of epistemic challenges to the integrity and ethics of science. For the past 150 years, science has been the gold standard of knowledge in liberal democracy. Scientific institutions, journals, and academies have rigorously controlled scientific outputs for validity: they have discarded poor methods, rejected flawed empirical claims, and zealously used human judgment to separate science from non-science.
Philosophers have recognized a strict demarcation between scientific and non-scientific knowledge cannot be maintained in its most fundamental sense. Knowledge comes in many forms and is distributed across a spectrum of validity claims. The transition from hypothesis to result and authorized knowledge is dynamic, consisting of learning processes and feedback loops. But as Karl Popper once stated, the best criterion for science is openness to falsification and the ability to achieve provisional consensus.
In other words, results produced by researchers must be verifiable, accepted, and recognized among peers. This is the very premise of the »social contract« between science and society. Science is represented by specialized institutions that are tasked with the responsibility to conduct quality-assurance and certify that expert knowledge lives up to the highest standards of transparency and reproducibility. While most lay citizens are unable to verify individual scientific knowledge claims, they rely on a specialized division of labor. They are required to trust the ability of scientific institutions to self-correct and to reward and incentivize research at the highest level of collective excellence.
The social contract between science and society relies on the ability of researchers to understand and reconstruct how particular scientific claims are produced and verified. In turn, this requires open sharing of results and transparency about methodology, data collection, and how conclusions are reached and presented. If this process is made opaque by a mediating layer of AI products, it will erode the quality and trust in science. It might sound appealing if all the hard work of setting up experiments, conducting analysis, writing publications, submitting grant applications, and performing peer review can be automated by artificial intelligence. But this should be treated with great caution and regulated in by institutional codes of conduct.
Unlike AI models trained on high-quality scientific data, such as AlphaFold, the output of most language models cannot be verified or reproduced. GenAI is based on statistical models. Changing even the smallest input will change the output in ways that are inaccessible to humans. Add to this the fact that algorithms are kept secret and accessible only through company paywalls. Public researchers don’t have access to verify the programs they use themselves.
So far, the publishing giants have defined the rules of the game and the direction of digital infrastructures in the research system. Commercial solutions can appear better and cheaper than local alternatives and experiments. This creates serious vulnerabilities in science and society. At a larger scale, relying on AI models for research can pose treats to security, democracy, and trust.
For these reasons, it is crucial to regulate the digital infrastructure of science. Introducing and experimenting with AI models require oversight from universities and research foundations. Additionally, research data needs to be open and independent of commercial publishers. By retaining ownership of data and actively enforcing intellectual property rights, universities can retain the rights to research. Open scholarship in its veracious forms can help accelerate the push towards more responsible use of AI in research. In addition, university managers and funders need to create the right incentives and start disrupting the past and current model of publishing, which leads to data loss and transfer of property to private companies. Relying on ethics and responsibility, and incorporation CoARA and DORA principles are starting points.
If European research institutions maintain ownership and control over data, AI models can also be trained on public research. These models can be verified and fact-checked; their algorithms are not held secret but can be published on equal terms with other research contributions. In this way, European research can help advance the digital sovereignty that policymakers worldwide are increasingly focused on.
David Budtz Pedersen is Professor of Science Communication at Department of Communication and Psychology, Aalborg University (Denmark) and Director of the Knowledge Broker Unit of ADD Project. His research focuses on the management, communication and impact of science and technology. He is a regular adviser to the European Commission, the Danish Ministry of Science, and a Member of the Norwegian Research Council. He takes a special interest in open, responsible and collaborative research focused on creating positive impact in society. He is Chair of the EU High-Level Presidency Conference on Reforming Research Assessment, 3-4 December 2025.