I'm watching a wave of health AI products launch with impressive demos and no clinical oversight framework. Chatbots dispensing health advice. Symptom checkers making triage decisions. Wellness platforms offering "AI-powered health insights." The technology is advancing faster than the guardrails. That gap is going to hurt people.
What You Need to Know
- Health AI products are launching at unprecedented speed, but many lack clinical governance frameworks that ensure safe and appropriate outputs
- The risk isn't that AI gets every answer wrong - it's that it gets most answers right, building false confidence before failing on the cases that matter most
- Effective clinical guardrails include scope boundaries, confidence thresholds, escalation pathways, clinician review loops, and adverse event monitoring
- Organisations deploying health AI without clinical governance are creating liability exposure and, more importantly, patient safety risk
The Confidence Problem
Here's what makes health AI uniquely dangerous compared to AI in other domains. A chatbot that gives bad restaurant recommendations wastes your evening. A health AI that gives bad clinical guidance can cause real harm.
And the failure mode is insidious. Health AI products are typically right most of the time. The symptom checker correctly identifies a common cold 95% of the time. The wellness chatbot gives reasonable general health advice for straightforward questions. This high baseline accuracy builds trust.
67%
of consumers say they would trust AI-generated health advice if the platform seemed reliable, even without verification from a health professional
Source: Accenture Digital Health Consumer Survey, 2024
The danger lives in the other 5%. The cases where the symptom checker misses the early signs of something serious because the presentation is atypical. The wellness chatbot that gives advice that's safe for most people but dangerous for someone with an undisclosed condition. The AI that's never wrong until it's wrong in a way that matters.
In health, the edge cases aren't annoying exceptions. They're the patients who get hurt.
What Guardrails Actually Look Like
Clinical guardrails aren't about limiting AI capability. They're about ensuring AI operates safely within defined boundaries. Here's what a robust framework includes.
Scope Boundaries
Every health AI product needs a clearly defined scope of what it can and cannot do. A symptom information tool is not a diagnostic tool. A wellness chatbot is not a clinical advisor. A risk assessment platform is not a substitute for clinical judgement.
These boundaries need to be explicit in the product design, the user interface, and the user communication. When the AI encounters a query outside its scope, it should decline gracefully and direct the user to appropriate clinical resources.
Confidence Thresholds
AI outputs should carry confidence indicators, and the system should behave differently at different confidence levels. High-confidence, well-supported responses can be delivered directly. Low-confidence or ambiguous responses should be flagged, softened, or withheld.
This isn't the same as slapping a disclaimer on everything. It's engineering the system to know its own limitations and act accordingly.
Escalation Pathways
When the AI reaches the boundary of its competence, there must be a clear pathway to human clinical review. Not a phone number at the bottom of a page. An integrated, low-friction escalation that connects the user to a qualified health professional with context about the interaction.
The measure of a good health AI isn't how many questions it answers. It's how well it handles the questions it shouldn't answer.
Jay Harrison
Health Technology Advisory
Clinician Review Loops
AI outputs in health should be subject to ongoing clinician review. Not every output individually, but systematic sampling that catches patterns of error, identifies edge cases the system handles poorly, and feeds back into model improvement.
This requires clinical expertise within the organisation, not just engineering capability. A team of engineers reviewing AI health outputs without clinical training will miss the errors that matter.
Adverse Event Monitoring
Health products have adverse event reporting obligations. AI health products should have equivalent mechanisms. When an AI output contributes to a negative health outcome, the organisation needs to know, investigate, and learn from it.
25%
of AI health startups surveyed had no formal process for monitoring or reporting adverse events related to AI-generated health advice
Source: Rock Health, Digital Health Funding Report, H1 2024
The absence of adverse event monitoring in many health AI products isn't a gap. It's a failure of basic clinical governance.
The Regulatory Landscape
Regulators are catching up, but they're behind the deployment curve.
In New Zealand, health AI sits in an evolving regulatory space. Medsafe regulates medical devices, and some AI tools may fall under that framework. The Privacy Act governs health data use. But the specific application of these frameworks to AI-powered health products is still being clarified.
Internationally, the EU's AI Act classifies health AI as high-risk, requiring conformity assessments, human oversight, and transparency obligations. The US FDA has been approving AI medical devices under existing frameworks but is developing AI-specific guidance.
Organisations deploying health AI in New Zealand should be building to the highest international standard, not waiting for local regulation to catch up. When regulation arrives, being ahead of it is a competitive advantage. Being behind it is a compliance crisis.
Building It Right
The organisations that will lead in health AI are the ones building clinical governance from day one, not the ones with the flashiest demo.
Hire clinical alongside engineering. Your health AI team needs clinicians who understand both the clinical domain and the AI capability. Not as advisors consulted quarterly, but as team members shaping the product daily.
Define scope before capability. Decide what your AI should and shouldn't do before you build it. The temptation to let capability define scope - "the model can do this, so let's let it" - is how you end up with ungoverned health advice.
Test with clinical scenarios. Not just accuracy benchmarks. Realistic clinical scenarios including edge cases, atypical presentations, and ambiguous situations. Test what the AI does when it should say "I don't know."
Build the escalation first. Before you build the AI response engine, build the pathway to human clinical review. The escalation pathway is more important than the AI output, because it catches the failures that matter.
Monitor continuously. Clinical governance isn't a launch checklist. It's an ongoing programme. Regular review of AI outputs, systematic adverse event monitoring, and continuous feedback from clinical oversight.
The health AI space is moving fast. Moving fast without guardrails in healthcare isn't bold. It's irresponsible.
- Does clinical governance slow down health AI development?
- It changes what you prioritise, not how fast you move. Building governance from the start is faster than retrofitting it after a safety incident. The organisations I've seen move fastest in health AI are the ones with clinical expertise embedded in the team from day one.
- Who should be responsible for clinical governance in a health AI company?
- A clinician with both clinical expertise and product understanding. Not the CTO who read a textbook, and not an external advisor who reviews quarterly. Someone with clinical authority who's in the room when product decisions are made.