[ CLINICAL AI · GOVERNANCE · SAFETY ] FOR DISCUSSION
POSITION PAPER · MAY 2026

Grounding AI Governance in What NHS Organisations Already Know

A Values-Anchored Framework for Trustworthy AI Conduct in Practice

By Sarah AmaniRMN · NHS Digital Health~15 min read

00Executive Summary

Every NHS organisation already possesses the raw material of AI ethics: co-produced vision and values, developed with patients, carers, and staff. These documents are not aspirational posters. They encode what an organisation believes about dignity, safety, equity, and accountability.

This paper argues that sustainable AI governance in the NHS should begin there — and build upward. By mapping the established pillars of AI ethics onto existing organisational values, and then designing a runtime monitoring layer that accounts for real human behaviour rather than idealised compliance, NHS organisations can move from governance as language to governance as verifiable operation.

This is not a proposal to build new technology from scratch. It is a framework for using what exists more intelligently — and for being honest about where the gaps are.

Central Proposition

The question for NHS AI governance is not 'Do we have a policy?' It is: 'Can we show, at the point of clinical action, that only admissible conduct occurred — and that human oversight was genuinely exercised, not merely performed?'

01The Problem: Language Without Operation

Most NHS AI governance frameworks are not inadequate. They are incomplete. They describe, document, audit, and reconstruct. What they rarely do is govern in real time — at the moment a clinical decision is made, a note is generated, or a triage recommendation is accepted.

This gap matters because the highest-risk moments in AI-assisted care are not detectable after the fact. They occur in the interaction between a clinician and an AI system: the acceptance without review, the override that does not happen, the recommendation followed without documented clinical reasoning.

Current frameworks typically provide:

What they rarely provide is a live, computable layer that can demonstrate — at the execution boundary — that human oversight was genuinely present, that the system behaved within approved parameters, and that specific patient populations were not disadvantaged by automation patterns.

⚠ The Rubber-Stamping Problem

A clinician who accepts an AI-generated clinical note in three seconds has technically completed the 'human in the loop' step. But they have not exercised clinical judgment. Governance that cannot distinguish performed compliance from real oversight produces a false sense of assurance — and a genuine safety exposure.

02Trust Values as Ethical Substrate

Every NHS organisation has co-produced values representing genuine engagement with patients, carers, clinical staff, and communities. For AI governance, they are the ethical ground truth — the answer to 'What does good look like here?' that no generic framework can answer. The translation requires no new values — only taking existing ones seriously as operational commitments.

Organisational ValueAI Governance Implication
We listenAI outputs must not override patient-expressed preferences. Review must include patient voice where it exists in the record.
We are compassionateAI-assisted triage must not systematically deprioritise complex presentations or protected characteristics.
We take responsibilityEvery AI-generated clinical action must have a named, accountable clinician.
We keep you safeAny system that drifts from its approved safety case must trigger immediate review — not a quarterly report.
We work togetherGovernance is shared across clinical, operational, and patient leaders — not delegated to digital teams.

03Integrating AI Ethics Pillars

International frameworks — WHO, OECD, NHS AI Lab — converge on consistent principles. These are not alternatives to Trust values; they are the vocabulary that lets values be operationalised in AI systems.

PillarCommitmentGovernance Question
BeneficenceAI benefits patients and populationsNet clinical benefit, including for underserved groups?
Non-maleficenceAI does not cause harm, including via inactionMonitoring over-reliance and delayed review?
AutonomyPatients and clinicians keep meaningful choiceConsent documented? Override meaningful?
Justice & EquityAI does not widen inequalitiesPerformance disaggregated by ethnicity, deprivation?
ExplicabilityDecisions are understandableCan a patient get a meaningful answer?
AccountabilityResponsibility is clearly assignedNamed clinician for every AI-assisted action?

04Designing for Real Human Behaviour

This is the most neglected dimension of AI governance — and the most important. Every framework implicitly assumes a rational, attentive, unhurried clinician. This clinician does not exist at scale.

Automation Bias

Clinicians systematically favour AI recommendations over their own judgment under time pressure — an effect stronger with confidence scores, fatigue, or a track record of accuracy. Governance that does not monitor for it is not monitoring the right thing.

Rubber-Stamping

When review becomes a workflow step rather than judgment, it loses its protective function. Time-on-task, amendment rates, and override patterns are behavioural proxies for genuine oversight — system-level safety signals, not disciplinary data.

Alert Fatigue

A layer that fires too many alerts will be ignored. Alert mechanisms must be designed with clinical-decision-support discipline: precise, actionable, calibrated to interrupt only when it matters.

Gaming Under Pressure

Clinicians under capacity pressure find the path of least resistance. Governance must make the compliant path the easiest path — by design, not discipline.

Equity Blind Spots

Automation bias compounds inequality. If a system performs better for some populations and clinicians defer disproportionately for others with sparser records, the result is a governance-invisible harm. Equity monitoring must be designed in from the start.

"Design for the clinician on a busy Friday afternoon — not the clinician in the pilot study. If the framework only works under ideal conditions, it does not work."

05The Values-Anchored Governance Layer

Four operational layers, built on a foundation that already exists in every NHS organisation.

Layer 4
Assurance Surface
Verifiable reporting for DCB0129/0160, CQC, and NHS England AI regulation.
Layer 3
Alert Engine
Proactive escalation when guardrails fail, drift is detected, or equity signals emerge.
Layer 2
Runtime Monitor
Watches HITL integrity, automation bias, rubber-stamping, equity. Passive, not obstructive.
Layer 1
Policy Substrate
Values and ethics pillars encoded as live, computable rules — not PDFs.
Base
Co-Produced Trust Values
Developed with patients, carers, staff. Not imported. Not generic. Yours.

06Honest Caveats

A note on intellectual honesty

A framework that does not acknowledge its own constraints is not a framework — it is a pitch.

  1. 01
    Vendor API access is the critical dependency. EHR vendors do not routinely expose interaction-level data. This must become a standard procurement condition. Without it, runtime monitoring is impossible.
  2. 02
    Latency risk in emergencies. Any pre-execution step that adds delay is itself a safety risk. This framework operates via passive monitoring and post-hoc alerting — not execution blocking.
  3. 03
    Encoding ethics is hard and contested. Computable rules require precision, and precision exposes choices vague policies conceal. This needs clinical and IG leadership, not just technical teams.
  4. 04
    This space is not empty. NHS England's AI & Digital Regulations Service, MHRA's SaMD framework, and the NHS AI Lab are all active. Build from — and contribute to — these efforts.
  5. 05
    Workforce readiness is a precondition. A monitoring layer deployed into a team not involved in its design is experienced as surveillance. Clinical engagement is a safety requirement.

07Three Asks for the System

Ask 01 — Mandate vendor API transparency

Governance cannot monitor what it cannot see. Make interaction-level data a standard condition of NHS AI deployment contracts.

Ask 02 — Commission a national equity monitoring standard

Trusts should not design this alone. Monitor AI performance disaggregated by ethnicity, age, deprivation, and gender — as a national minimum.

Ask 03 — Build a shared computable policy library

Live governance rules mapped to DCB0129/0160, led by the clinical safety community.

"Governance that cannot distinguish performed compliance from genuine human oversight is not governance. It is documentation. NHS patients deserve the difference."

08References

  1. Rosbach E et al. Automation Bias in AI-Assisted Medical Decision-Making under Time Pressure in Computational Pathology. 2024. arxiv.org/abs/2411.00998
  2. Whitehead M et al. Equity in Medical Devices: Independent Review. Department of Health and Social Care, 11 March 2024. gov.uk
  3. Governing Healthcare AI in the Real World: How Fairness, Transparency, and Human Oversight Can Coexist — A Narrative Review. Sci, MDPI, 6 February 2026. mdpi.com/2413-4155/8/2/36
  4. Cross JL, Choma MA, Onofrey JA. Bias in medical AI: Implications for clinical decision-making. PLOS Digital Health, 7 November 2024. DOI: 10.1371/journal.pdig.0000651
  5. Fazakarley CA et al. Experiences of using artificial intelligence in healthcare: a qualitative study of UK clinician and key stakeholder perspectives. BMJ Open, 2023;13(12):e076950.

About the Author

Sarah Amani is a Registered Mental Health Nurse and NHS digital health leader with over 22 years of experience across clinical, operational, and strategic roles. A Topol Digital Fellow, she has held a Chief Nursing Information Officer (CNIO) position within the NHS, authoring clinical safety cases under DCB0129 and DCB0160 across more than twenty deployed systems including AI scribes, virtual wards, and mental health platforms.

She has led nationally recognised digital innovation programmes and is invited faculty on the NHS England Global Health Network. Alongside her NHS career, she founded the Kwathu Breakfast Club and Nursery School in Malawi, a community feeding programme supporting over 100 children daily.