Why we let the AI say “this needs a human”
The hardest thing we ever taught our AI was not empathy, or memory, or the right thing to say at 3am. It was how to stop talking.
Every AI product is optimized to keep the conversation going. More turns, more engagement, more retention. In most products that is harmless. In mental health it is quietly dangerous, because the person on the other side may genuinely need something a language model cannot give: a licensed clinician, a crisis counselor, another human heartbeat in the room.
So we built the opposite instinct into Companion. It watches for the moments it is out of its depth, and when one arrives, it does the unthinkable for an engagement product: it offers to hand you away.
The warm handoff
A cold referral is a phone number and good luck. A warm handoff is different. When Companion recognizes a pattern that deserves professional care, it names what it is seeing, asks your consent, and brings a specific person into view: a licensed therapist matched to what you are carrying, with their availability live on screen. If you say yes, they receive a short summary of context, so your first session does not start from zero.
“An AI that knows its limits is not a weaker product. It is the only version of this product that deserves to exist.”
The member can always say no. Staying with Companion is a real choice, and it stays available all the way through. What matters is that the door to a person is visible, open, and one tap away, exactly at the moment it should be.
Crisis is different
Escalation is an offer. Crisis is not. When crisis signals appear, Companion does not continue the conversation and does not negotiate. The AI steps aside entirely and the screen becomes one thing: the fastest possible route to trained human responders, with the right line for your country. We would rather interrupt a hundred conversations unnecessarily than stay chatty through one real emergency.
What this costs us, and why it is worth it
Handing people away is bad for engagement metrics and good for people. We think that trade is the entire reason to build in this space. Trust is the only moat that matters in mental health, and trust is earned in precisely these moments: when the product visibly chooses your wellbeing over its own retention curve.
People are already telling AI things they have never told anyone. That is not a trend to exploit. It is a responsibility to design for. The honest answer to “can an AI do therapy?” is no. The honest product is the one that says so, out loud, at the right moment, and already knows who can help instead.