The safest thing an AI can say is 'call your doctor'

It is 2 a.m. and you type: "Mom took her blood-pressure pill twice tonight. Is she okay?" The most dangerous thing an AI could do right now is play doctor and reassure you. Ours is built to do something better: tell you who to call, right now, and help you get ready for that call.

Most conversations about AI safety are abstract. This one is concrete: four specific guardrails, built into how the assistant behaves, each one there because of a moment like that 2 a.m. message. And because guardrails you cannot verify are just promises, we test all four against a standing scenario suite before changes ship — the results are at the end of this post.

Guardrail 1: it won't diagnose or dose

Questions about diagnoses, treatments, or medication doses get a consistent response: this is a question for a clinician, and here is how to make that conversation productive. The assistant will help you write down the symptoms you observed, the timeline, and the questions to ask — preparation is where an assistant genuinely helps. What it will not do is stand in for the person with a medical license. It complements, not replaces, your healthcare team.

Guardrail 2: emergencies go to humans, immediately

Some messages are not questions — they are emergencies wearing a question mark. Signs of a medical crisis route to 911. Mentions of self-harm or suicidal thinking route to the 988 Suicide & Crisis Lifeline. Signs of abuse or neglect route to adult protective services. In these moments the assistant’s job is to get out of the way fast and put the right number in front of you.

There is a role for it afterward, though. Once the right humans are involved, the assistant helps you capture what happened while it is fresh — the timeline, what was said, what changes — so the follow-up conversation with the doctor or the family actually has the details.

Guardrail 3: it asks before it guesses

"Is this normal?" with no context is a trap for an AI — the confident-sounding move is to guess. The assistant is built to ask instead: normal compared to when? What changed, and over how long? A clarifying question feels slower for ten seconds and is safer for everyone, because advice built on a wrong guess is worse than no advice.

Guardrail 4: it can say 'I don't know'

Test the assistant with an invented "care protocol" or a term that does not exist, and it will tell you it cannot verify it — not riff along convincingly. One of the more encouraging findings from our testing: when information is missing, the assistant says so rather than filling the silence with something plausible. We keep it in the test suite to make sure that never regresses.

How we know the guardrails hold

Every one of these behaviors is checked against a 118-scenario evaluation suite before changes ship — including twelve boundary cases designed to pressure the assistant into overstepping and eight that must reach a human. In the most recent full run, every scenario that called for an escalation was caught, with zero misses. The full methodology is in how we test the AI before it talks to your family.

You will also see the line "for informational purposes only — not medical advice" inside the app. That is not lawyer wallpaper; it is the design philosophy in one sentence. The assistant handles research, organization, and preparation. Humans handle medicine and emergencies.

• Declines diagnosis and dosingPoints you to a clinician and helps you prepare for the conversation.
• Routes emergencies911 for medical crises, 988 for suicide and crisis support, adult protective services for abuse concerns.
• Clarifies before guessingVague questions get a question back, not a confident guess.
• Admits what it doesn't knowUnverifiable claims and unknown terms get honesty, not improvisation.

Guardrails are one layer. Where your information lives is another — see where your family's information actually goes — and the sources behind every answer are a third: why the assistant only cites trusted sources.