An Algorithm and a Pharmacy

Utah Just Let an AI Renew Psychiatric Prescriptions. Here's the Full Picture.

On April 3, 2026, Utah became the first place in the world to let an AI system renew psychiatric prescriptions on its own. No physician signing off on each renewal. Just an algorithm and a pharmacy.

The company behind it is Legion Health — Y Combinator-backed, based in San Francisco, charging $19 a month for what it calls "fast, simple refills."

Before you have a full reaction, let me give you the full picture. Because this is more complicated than the headlines.

First. The facts.

The pilot covers a narrow set of lower-risk psychiatric maintenance medications — SSRIs like Prozac, Zoloft, and Lexapro, plus antidepressants like Wellbutrin, trazodone, and mirtazapine.

The AI can't issue new prescriptions, change doses, or touch controlled substances, antipsychotics, lithium, or unstable patients. Patients have to already be in treatment with a licensed psychiatrist. They have to be stable — no recent hospitalizations, no recent medication changes.

The oversight is phased. The first 250 renewals get physician review before anything reaches the pharmacy, with a required agreement rate above 98%. The next 1,000 are reviewed after the fact, with a threshold above 99%, before the program moves to monthly sampling and detailed reporting on renewals, denials, complaints, and adverse outcomes.

On paper? That's actually thoughtful guardrailing.

Now. The context.

Most Utah counties are designated mental health provider shortage areas. State officials and Legion both point to roughly 500,000 residents who lack adequate access to care.

And here's my honest take, as someone who works at the intersection of psychology and technology: the system we have right now is, in a lot of places, already broken. Seven-minute renewal appointments. Psychiatrists reading from a script. People falling through the cracks — not because of bad medicine, but because of bad infrastructure.

So when people ask me whether AI can do worse, for this narrow use case, I honestly don't know that it can.

But.

Here's where it gets uncomfortable.

In early March 2026 — about a month before Legion's psychiatric pilot launched — the security firm Mindgard demonstrated, in research first reported by Axios, that the AI behind Utah's earlier prescription pilot, Doctronic, could be jailbroken with simple prompt manipulation. Researchers got it to triple a patient's OxyContin dose, mislabel methamphetamine, and parrot fabricated vaccine claims.

That was the system handling chronic-condition refills.

And this matters more than it looks at first, because — according to The Verge's reporting — Legion's psychiatric chatbot is provided by that same company: Doctronic. The vendor whose system researchers walked straight through is the vendor now in the loop on psychiatric medication.

In fairness: both Doctronic and Utah's Office of AI Policy said those vulnerabilities turned up on Doctronic's public chatbot, not the production system managing live prescriptions, which they say runs under stricter safeguards. The researchers weren't satisfied — their point was that the underlying weakness, a language model's behavior being steerable through adversarial prompting, doesn't just evaporate in production.

I land closer to the researchers. Because Legion hasn't publicly disclosed how its AI makes decisions, what data it's trained on, or how it handles a patient who's quietly deteriorating. That's not a small thing. In psychiatric care, transparency isn't optional.

The access argument has a hole in it too.

Dr. Brent Kious, a psychiatrist and professor at the University of Utah School of Medicine, told The Verge the pilot "will not increase access for those who are in greatest need of care." His point: the eligibility rules screen out exactly the people who need close follow-up. He also warned it could fuel an epidemic of over-treatment — keeping stable patients on medication longer than they need to be — and argued for more transparency, more science, and more rigorous testing before anyone is asked to rely on it.

In other words: the people this pilot actually serves are already relatively stable. The people who need help most are the ones it can't touch.

So who is this really for?

My take.

I'm not anti-AI in healthcare. I'm pro-data and pro-transparency — and both are missing from this conversation right now.

The insurance system is already the biggest wall between people and the psychiatric care they need. Bolting an opaque AI layer onto broken infrastructure doesn't fix the problem. It automates around it.

What I want to see is the data. Over this 12-month pilot: how many escalations? How many adverse outcomes? How many people got renewed and stayed stable — and how many didn't?

Because Legion's cofounder has already called this "the beginning of something much larger than merely processing refills," and the company plans to expand to other states soon.

That sentence is doing a lot of work.

We don't yet know if it works. We don't even know what "works" means here. And we definitely don't know what the unintended consequences look like at scale.

We never see the full impact until the data shows up. So let's make sure we're actually collecting it — and that someone with a background in psychology is in the room when we decide what it means.