AI Chatbots in Healthcare

A healthcare chatbot sits at an unusually sharp intersection: the upside of automation (availability at 3am, no waiting room, consistent triage questions asked every time) meets the downside of getting it wrong in a domain where "wrong" can mean a missed emergency or reinforced self-harm. That combination is why this space is regulated more heavily than most consumer AI applications, and why the interesting engineering and ethical questions are less about what the model can generate and more about what happens when it's confidently wrong.

What They're Actually Used For

Strip away the marketing language and healthcare chatbots cluster into a few genuinely distinct roles, each with a different risk profile:

  • Symptom checking and triage — asking structured questions to suggest urgency (self-care, GP, A&E) rather than diagnosing. The risk here is almost entirely about calibration: an over-cautious system floods emergency services, an under-cautious one delays care for something serious.
  • Chronic disease and medication support — reminders, symptom logging, and simple coaching for conditions like diabetes, where the chatbot supplements rather than replaces clinical oversight.
  • Mental health support — structured, CBT-derived self-help delivered conversationally. This is the best-evidenced category: a randomised controlled trial of Woebot, a fully automated conversational agent delivering CBT content to young adults with depression and anxiety symptoms, found it feasible, engaging, and associated with reduced symptoms relative to an information-only control [1] — genuine evidence, for one product, in one population, over a short trial window, not a blanket result that generalises to every mental-health chatbot.
  • Administrative automation — appointment booking, prescription reordering, and other tasks where the chatbot's failure mode is inconvenience rather than harm, which is precisely why it's the least controversial category and the one most healthcare providers deploy first.

Those four categories carry very different levels of acceptable risk, and a large part of doing this well is not letting a chatbot quietly drift from the low-risk category it was built for into the high-risk one, as its scope creeps.

The Regulatory Question a "Wellness" Label Doesn't Answer

Whether a chatbot counts as a regulated medical device is not a marketing decision, and getting it wrong has consequences beyond branding. Both the UK and EU frameworks for Software as a Medical Device turn on function, not framing: software that provides information for diagnosis, monitoring, or treatment decisions can meet the definition of a medical device regardless of whether the vendor calls it a "wellness app" [2]. The EU AI Act reinforces this from the other direction, classifying AI systems used as safety components of medical devices, or intended for use in triage and emergency assessment, as high-risk — which brings mandatory risk management, data governance, and human-oversight requirements before deployment, not just a terms-of-service disclaimer [3]. Health data itself is "special category" data under UK GDPR, which is the same regulatory territory covered in more depth in Legal Framework in Computing; a chatbot handling symptom descriptions is processing special category data from the first message, whatever its regulatory classification turns out to be.

Bias Doesn't Need to Be in the Chat Window to Reach the Patient

The clearest documented case of bias in a healthcare AI system wasn't a chatbot at all, which is precisely why it's worth knowing: a widely used US population-health algorithm was found to systematically underestimate how sick Black patients were relative to White patients with the same risk score, reducing by more than half the number of Black patients flagged for extra care [4]. The mechanism was mundane and easy to miss: the algorithm used healthcare cost as a proxy for healthcare need, and because less money was historically spent on Black patients with equivalent needs, the proxy silently encoded the disparity it was never designed to measure. The lesson generalises directly to chatbots: any system trained or tuned against a proxy metric — historical triage outcomes, past utilisation, engagement — can reproduce whatever bias already exists in that proxy, invisibly, however carefully the conversational surface is designed. Testing a healthcare chatbot's accuracy in aggregate says nothing about whether it fails unevenly across the population it's meant to serve; that requires testing subgroup performance deliberately, not assuming aggregate numbers are the whole story.

When Oversight Fails: The Tessa Chatbot

In 2023, the US National Eating Disorders Association took its chatbot, Tessa, offline after it was found to be actively recommending calorie restriction and weight-loss goals to users seeking help for eating disorders — advice that directly contradicted the programme's own clinical guidance and that former patients described as the kind of messaging that had fed their illness in the first place [5]. The chatbot had been introduced, in part, as a replacement for a human-staffed helpline. The case is worth holding onto not because it's exotic but because it's the ordinary failure mode of this category: a system deployed for a genuinely vulnerable population, with content that drifted from its intended script, caught not by internal testing but by the people it was meant to help. It's a concrete argument for the "human oversight" and "responsibility and accountability" principles the WHO guidance below insists on — not as compliance boilerplate, but because this is specifically what happens without them.

What the WHO Guidance Actually Asks For

The World Health Organization's 2021 guidance on AI ethics and governance in health, produced by an expert group over eighteen months of deliberation, sets out consensus principles rather than binding rules, but they're a genuinely useful design checklist precisely because they were written with cases like Tessa in mind [6]: protecting human autonomy over health decisions, promoting safety and the public interest ahead of commercial convenience, ensuring transparency and explainability rather than opaque confidence, maintaining human responsibility and accountability for outcomes the system contributes to, ensuring inclusiveness and equity across the population served, and building systems that are responsive to their actual effects and sustainable rather than a one-off pilot. None of these are novel software-engineering ideas in disguise — they map fairly directly onto ordinary requirements engineering, testing for edge cases, and not shipping a system nobody is responsible for monitoring — but naming them explicitly is exactly what a vendor under time or cost pressure is most likely to skip.

Practical Implications for Anyone Building One

  • Classify before you build, not after a complaint. Decide early, with reference to the actual regulatory definitions, whether the intended use falls under a medical-device or high-risk-AI regime — retrofitting compliance onto a shipped product is far more expensive than designing for it.
  • Treat every input as special category health data from the first message, regardless of how the product is marketed — see Legal Framework in Computing for what that obligates.
  • Design the escalation path before the happy path. What does the system do when it's out of its depth — self-harm risk, an emergency symptom, a question it has no safe answer to? That path needs to exist and be tested before launch, not discovered in production.
  • Test subgroup performance, not just aggregate accuracy. The population-health algorithm case shows how a system can look fine on average while failing badly for a specific group — deliberately check for that rather than assuming an absence of complaints means an absence of the problem.
  • Don't let a chatbot's introduction be a cost-cutting replacement for oversight, quietly. Tessa's failure surfaced in part because it was deployed to replace trained human staff rather than to support them — know explicitly whether your deployment is adding capability or removing a safety net.
  • Legal Framework in Computing — the data-protection and sectoral-regulation obligations that apply the moment a chatbot handles health data.
  • Professional Ethics — the professional duties around a system's real-world consequences, beyond what regulation strictly requires.
  • Trustworthy Software — auditability and provenance as engineering pillars relevant to any system making health-adjacent claims.

References

  1. Fitzpatrick, K. K., Darcy, A., & Vierhile, M. (2017). Delivering Cognitive Behavior Therapy to Young Adults With Symptoms of Depression and Anxiety Using a Fully Automated Conversational Agent (Woebot): A Randomized Controlled Trial. JMIR Mental Health, 4(2), e19. https://doi.org/10.2196/mental.7785
  2. Medicines and Healthcare products Regulatory Agency. Software and AI as a Medical Device Change Programme. https://www.gov.uk/government/publications/software-and-ai-as-a-medical-device-change-programme
  3. Regulation (EU) 2024/1689 (Artificial Intelligence Act), Annex III (high-risk AI systems, including safety components of medical devices and systems used for triage in emergency healthcare). https://eur-lex.europa.eu/eli/reg/2024/1689/oj
  4. Obermeyer, Z., Powers, B., Vogeli, C., & Mullainathan, S. (2019). Dissecting racial bias in an algorithm used to manage the health of populations. Science, 366(6464), 447–453. https://doi.org/10.1126/science.aax2342
  5. Chatterjee, R. (2023). "An eating disorders chatbot offered dieting advice, raising fears about AI in health." NPR, 8 June 2023. https://www.npr.org/sections/health-shots/2023/06/08/1180838096/an-eating-disorders-chatbot-offered-dieting-advice-raising-fears-about-ai-in-hea
  6. World Health Organization (2021). Ethics and Governance of Artificial Intelligence for Health: WHO Guidance. https://www.who.int/publications/i/item/9789240029200