Bot now?

The ability to explain AI’s reasoning cannot be a proxy for safety in governance frameworks

Qinecsa’s Beena Wood sets out other critical provisions that need to be designed into intelligently automated PV systems, and adapted by use case and associated risk profile, for an AI-based PV system to be deemed trustworthy and ‘safe’ – both at the outset and over time.

While large swathes of the life sciences industry have enthusiastically bought into what next generations of AI technology appear to offer them, they now need to double down on what will make AI-optimised systems safe and compliant. That’s if they want to avoid exposing themselves to avoidable risk in a pharmacovigilance context.

Looking beyond explainability (an AI-enabled system’s ability to evidence its reasoning) is an important part of this. Justifying how a conclusion was reached does not prevent error or ensuing harm.

Recognising this, regulators have made clear that they will not tolerate errors arising from vigilance failures linked to AI. AI governance in a PV setting now needs to be multilayered, adapted to the given use case and reviewed and honed over time.

MHRA in the UK, the US FDA and the EU’s EMA all now refer to the need for governance and trust to be risk-based and proportionate, focused around outcomes and context of use. Meanwhile validation should not be a one-time activity, given how quickly needs and capabilities can change.

PV technology vendors are already on the case, but not so much pharma companies’ procurement teams. That is, they are often looking for reassurance in the wrong places. Automation bias, where organisations or functions deploy AI and blindly trust its output, is problematic, certainly, and regulators are toughening their stance on such over-reliance.

In April 2026, the FDA issued its first-ever warning letter to explicitly cite over-reliance on AI as a violation, after a company (Purolea) used AI to generate compliance documents without human review and then attributed a missed regulatory requirement to the AI not having flagged it.

To avoid such scenarios, pharma companies need to be reassured not only that a system can explain its reasoning, but also that an AI-supported system will fail ‘safely’ if something goes wrong (such as by triggering alerts and prompting human checks).

In the future, inspectors will want to see operational evidence of what happens in practice in an organisation – that is, whether an AI system has followed the designated SOPs, not just that expectations have been documented.

Recommended selection criteria

Going forwards, a vetting framework for pharma companies procuring AI systems for PV should ideally seek to determine that the data feeding the system is verified and trustworthy, while the system reasons only in the context relevant to that specific decision.

It should also confirm that the AI system operates within the organisation’s own rules, and governance for this use case has been shaped around its specific risk.

Oversight should be calibrated to that particular risk and ‘designed in’ ahead of time. Assurance should hold up continuously across the whole life cycle, and the organisation should be able to produce operational evidence that the system has followed its SOPs if an inspector asks.

The way that the risk profile and controls will need to vary according to what the use case touches and where it comes in the PV life cycle, meanwhile, is usefully illustrated by the relative needs of case intake versus signal detection and evidence building.

In the former scenario, the primary risk is source and context (does the incoming data accurately reflect a single reported event, correctly triaged for seriousness and causality?).

When it comes to signal detection and evidence building, on the other hand, data is being pulled together across multiple sources and contexts – e.g. in building a disproportionality trend or cumulative evidence across cases, potentially for onward review by a safety review committee.

The stakes here are structurally different. A single case processed incorrectly is a serious error affecting one case. Incomplete scoping or provenance of the underlying data could mean missing an entire signal, with farther-reaching implications.

An additional, related risk concerns what is feeding the decision to escalate or close a case: i.e. whether the statistics, trends and supporting cases shown to a reviewer are being reliably calculated from real data, or derived from an AI-generated summary (this may be plausible yet inaccurate).

Staying alert

Governance needs to be adapted to the identified risk profile rather than be applied uniformly, including considerations around how much an AI-enhanced system is allowed to do without human review.

For the most consequential decisions, it may be that AI judgement needs to be removed altogether – either by requiring human sign-off before anything proceeds (so that the AI capability is never permitted to act on its own initiative), or by handling the decision using a fixed rule.

A further factor to look out for is ‘automation complacency’. Take the scenario of a reviewer required to oversee 100 cases a day that have been processed by AI. After a while, the responsible individual is unlikely to be actively reviewing them, but rather rubber-stamping them. Companies will need to mitigate for that, if only by restating personal accountability, or monitoring for throughput of human checks that is unfeasibly quick.

Continuous monitoring capabilities should ideally include detection and flagging when a normally active data source has gone quiet, and of possible contradictions between sources – for instance, a patient reporting no concomitant medication against an HCP report vs. contradictory lab data (which could have a bearing on the causality assessment).

Evidence generation should be built into the way that an AI-based system operates day to day, if PV leads are to have real confidence in the reliability and trustworthiness of their systems.

Layering in all of the appropriate governance measures, and reviewing and adapting them over time, has to be the way to go now.

Only with a comprehensive and adaptable approach to governance will fears around AI ‘going rogue’ be calmed and the full spectrum of AI benefits be elicited.

Beena Wood is Chief Product Officer at Qinecsa