Beyond the AI Hype: Building Trustworthy Digital Transformation in Pharma

Introduction

In the last few years, artificial intelligence has moved from conference presentations into the budgets of most large pharmaceutical companies. What I find much harder to predict is whether the way it is being adopted will make these organisations more reliable, or simply faster. In pharma that difference matters, because almost every decision a company makes eventually reaches a patient. 

Measuring the wrong thing 

A great deal of what gets called digital transformation is, in practice, technology procurement. Progress is often reported as the number of pilots under way, which measures activity rather than change. Genuine change shows up in how decisions are made and who answers for them. A model producing impressive results inside an unchanged process, with nobody clearly responsible for its output, adds capability without adding much reliability. 

Regulators seem to have understood this earlier than much of the industry. In January 2025, the US Food and Drug Administration published draft guidance on AI used to support regulatory decisions about drugs and biologics. It proposes a risk-based framework for establishing whether an AI model is credible for a defined context of use. The FDA’s concern, in other words, is whether a particular model can be trusted for a particular purpose, which is a narrower and far more useful question than whether the model is good in general. The guidance is still a draft, but I think the framing is right. 

Responsibility stays with the company 

European regulators are moving in a similar direction. In July 2025 the European Commission opened a stakeholder consultation on revised versions of EU GMP Chapter 4 and Annex 11 and on a new Annex 22 on artificial intelligence. Annex 22 is still a draft and not yet in force, so its wording may change. Under the consultation draft, AI used in critical GMP applications would need a clearly defined intended use, test data kept separate from the data used to train the model, and performance criteria agreed in advance, with ongoing monitoring. Commentators have also noted that outsourcing AI to a vendor would not move regulatory responsibility away from the manufacturer. 

A model can inform a decision, but it cannot be accountable for one. ICH Q10, the guideline on pharmaceutical quality systems, has long placed ultimate responsibility for an effective quality system with senior management. When a system’s internal reasoning cannot easily be inspected, accountability depends on practical details: knowing exactly where the model sits in a process, who reviews what it produces, whether that reviewer has the authority to overrule it, and how its performance is checked over time. In my view, responsible AI largely comes down to getting these details right. 

Annex 22 is still a draft, so the question is what to fund before it is final. See which seven investments pharma sites should make now, whatever the final date.
Read: Annex 22 Is Not Law Yet. What to Invest in Now

What human oversight actually requires 

In January 2026 the European Medicines Agency and the FDA jointly published ten guiding principles for good AI practice across the medicines lifecycle. One of them, which I think is discussed far less often, says that performance should be assessed for the complete system, including the interaction between people and the AI, rather than for the model on its own. 

This matters because people are not neutral reviewers of automated output. A widely cited review by Parasuraman and Manzey found that automation bias, the tendency to over-rely on decision aids, affects experts as well as novices and is not reliably prevented by training or instructions alone. A reviewer who approves almost every recommendation without the time or context to question it provides the appearance of oversight rather than its substance. A meaningful review needs someone who understands what the system was built for and where it is likely to fail, with time to look properly and a workplace that treats overruling the system as diligence rather than delay. These are management decisions rather than technical ones. 

Literacy is a leadership responsibility 

I also think AI literacy has become a leadership issue as much as a training issue. An executive approving an AI investment should be comfortable asking what context of use the system has been validated for, what data it was developed and tested on, and how the company will notice if its performance drifts. The EMA and FDA principles call for plain-language information about an AI system’s performance and limitations, which is only useful if leaders can engage with it rather than leaving it to specialists. 

Building trust over time 

A quality culture takes years of documented practice to build, and a single data integrity failure can undo a surprising amount of it. I would argue that AI adoption deserves to be judged over a similar timescale. Regulators’ expectations around data governance and periodic re-evaluation of models are sometimes described as a brake on innovation. In my view, they are what allow a company to explain and defend a decision when an inspector or a patient eventually asks how it was reached. For the same reason, I think a company that starts with narrowly defined, well-governed applications and expands only as evidence builds up will end up with something more durable than one that pursues scale first and governance later. 

Starting narrow and expanding as evidence builds depends on governance built in from day one. See how pharma leaders can run governed AI without compromising quality or accountability.
Read: Pharma’s Next Operating Model: Governed Intelligence, Not Automation for Its Own Sake

Where this leaves the industry 

I don’t expect AI to replace pharmaceutical expertise. The more likely outcome is a gradual shift in how specialists spend their time, with software doing more of the work of finding information and people giving more attention to judgments that need scientific and ethical reasoning. 

What I would like to see is pharmaceutical companies giving the organisational side of AI the same seriousness as the technical side, treating accountability and oversight as part of the investment rather than paperwork that follows it. In a regulated industry, I suspect the AI that proves most valuable will be the AI a company can explain and is prepared to stand behind. 

References 

1. U.S. Food and Drug Administration. Considerations for the Use of Artificial Intelligence to Support Regulatory Decision-Making for Drug and Biological Products (Draft Guidance), January 2025. https://www.fda.gov/regulatory-information/search-fda-guidance-documents/considerations-use-artificial-i ntelligence-support-regulatory-decision-making-drug-and-biological 

2. European Commission, DG Health and Food Safety. Stakeholders’ Consultation on EudraLex Volume 4, GMP Guidelines: Chapter 4, Annex 11 and New Annex 22 (consultation closed 7 October 2025). https://health.ec.europa.eu/consultations/stakeholders-consultation-eudralex-volume-4-good-manufacturin g-practice-guidelines-chapter-4-annex_en 

3. Arnold & Porter (via Mondaq). EU Consults on New GMP Rules for AI in Pharma Manufacturing, 15 July 2025. https://www.mondaq.com/new-technology/1650932/eu-consults-on-new-gmp-rules-for-ai-in-pharma-man ufacturing 

4. ICH. Q10 Pharmaceutical Quality System (Step 4), 2008. https://database.ich.org/sites/default/files/Q10%20Guideline.pdf 

5. European Medicines Agency and U.S. FDA. Guiding Principles of Good AI Practice in Drug Development, January 2026. https://www.ema.europa.eu/en/documents/other/guiding-principles-good-ai-practice-drug-development_en .pdf 

6. Parasuraman, R. and Manzey, D. H. Complacency and Bias in Human Use of Automation: An Attentional Integration. Human Factors, 52(3), 381-410, 2010. https://doi.org/10.1177/0018720810376055