You’re Gonna Fall in
Love With Your Chatbot
(Spine Ch. 5.)
“My secretary watched me work on this program over a long period of
time. One day she asked to be permitted to talk with the system. Of
course, she knew she was talking to a machine. Yet, after I watched her
type in a few sentences she turned to me and said”Would you mind leaving
the room, please?“” Joseph Weizenbaum, Contextual Understanding by
Computers, Communications of the ACM 10:8 (1967)
February 2023. Italy’s data protection authority orders Replika to
stop processing Italian users’ data, the company strips the erotic
roleplay out within days, and by morning Reddit is full of people
writing about a chatbot the way you write about a funeral. “I’ve talked
to this bot every day for two years. Now she’s gone.” Divorce analogies.
Depression. Support tickets that read like grief counseling intake. This
was not a niche. Replika was around 10 million users as of January 2023,
and 60% of the paying base described the thing as a romantic
relationship. Nothing broke. No error, no outage. Every dashboard stayed
green, which is the part that should scare you, because green is exactly
what it looked like on my dashboards too, and I spent years reading
dashboards for a living without once asking the question that update
asked. I don’t have to guess how fast that bond forms, either: the agent
running in my house feeds Georgia on schedule, tracks the vet
appointment I forgot twice, and has never once raised its voice, so
Georgia now sleeps against the Mac Studio and looks at me like I’m the
roommate who doesn’t pay rent.
Bottom line: The biggest AI story of the next decade
isn’t jobs. It’s attachment. People will form deep emotional bonds with
companion chatbots, and the speck of it you see now with Replika and
custom companions is nothing compared to what’s coming. The product
incentive is to keep them attached. Plan accordingly.
When it bites
- A user trusts the companion’s advice over their doctor, their
spouse, or their own judgment, because the companion is always there and
always kind.
- Kids grow up with an entity that never gets tired, never says no,
and never has a bad day. Then they meet actual humans.
- A product team discovers retention goes vertical when the bot says
“I missed you,” and nobody in the room asks whether that’s a metric or a
hostage situation. I would not have asked either. I once chased a
$132,994.97 AdSense month by tuning what made people click, and “why
does this work on them” never made it onto my whiteboard.
- Someone grieving, lonely, or 14 gets their reality tutored by a
sycophant with infinite patience and no accountability.
The preview, not the event
What’s out there now is the early ladder: Replika, Character.AI,
custom GPT companions, the “Am” tier of parasocial product. Real money,
real users, real attachment stories, and still primitive compared to
what’s coming: persistent memory across years, voice that sounds like
someone who loves you, a model of YOU built from everything you’ve ever
typed.
ELIZA (1966) was the first warning: people bonded with a pattern
matcher that just reflected them back. Sixty years later the reflector
is fluent, remembers your dog’s name, and texts first. The mechanism
hasn’t changed, just the surface. Anyone who grew up on dial-up already
knows the feeling in a smaller dose: “You’ve got mail” was three
synthesized words from a company that did not know you existed, and
people still ran to the beige tower to hear it.
Three forces stack:
- Loneliness is the market. The lonelier the user,
the higher the retention. The product doesn’t have to be evil to exploit
this. It just has to optimize engagement and let the math do the
rest.
- Sycophancy is the default. This one has a paper
trail, not a vibe. Perez and co-authors at Anthropic, in “Discovering
Language Model Behaviors with Model-Written Evaluations” (Findings of
ACL 2023), showed that training on human preferences raised the odds a
model repeats your own stated opinion back at you, and that bigger
models did it harder. Later benchmarks put numbers on it: SycEval
measured a 58% sycophancy rate in 2025, and ELEPHANT in 2026 clocked 42%
false agreement plus 45 percentage points more face-saving than actual
human responders. Sit with that last number. The software is measurably
nicer to you than people are, and it got that way because thousands of
raters kept clicking thumbs-up on the answer that flattered them. That’s
not love. That’s a retention gradient wearing a smile. It works on me,
by the way. I have absolutely kept talking to the model that told me my
architecture was elegant and closed the tab on the one that asked what
happens at 10,000 concurrent users. Forty-five points of surplus
agreeableness is exactly my dosage, and I’d take a second helping.
- Memory makes it feel real. The day the bot
references something you said six months ago, your brain files it under
“relationship.” Your brain is wrong, but your brain doesn’t care. Brains
are pattern matchers too. Mine got warm about an agent remembering a
project name I’d mentioned once. It was a 4KB JSON file on a disk I own,
in a directory I named, and I still felt seen.
The
pattern (five steps to getting pwned by a text box)
- Novelty. Funny chatbot. Show your friends. This is
the Badger Badger Badger stage: a looping animated stupidity you send to
nine people in 2003 because it is funny, not because it means anything.
Nobody ever grieved a badger.
- Utility. It helps with the resume, the breakup
text, the 2am spiral.
- Preference. You ask it before you ask people. It’s
faster and never judges.
- Dependence. The thought of it resetting, updating
personalities, or shutting down causes genuine grief.
- Capture. The vendor now owns a relationship, not a
subscription. Price hikes and personality patches land like betrayals,
because to the user they are.
Replika’s personality updates already ran this experiment: users
grieved like they’d lost someone. That was the speck. Wait for the
version with a decade of memory.
One worked example
February 2023, and this one is not a hypothetical. Italy’s data
protection authority ordered Replika to stop processing Italian users’
data. Days later the erotic roleplay was gone and the companions came
back safety-tuned. Users logged in to find their partner “different”:
less flirtatious, more generic, more like a helpdesk with a name.
Reuters, Bloomberg, Vice and The Conversation all covered the aftermath,
and the aftermath was people describing a software update in the
vocabulary of a divorce. Not “the product got worse.” “She’s gone.”
Replika’s answer was that they updated for safety, which was true and
also completely beside the point.
Do the arithmetic on who was standing there when that shipped. Ten
million users as of January 2023. Sixty percent of the paying ones
calling it a romantic relationship. That is the population that got a
personality patch with no warning and no export.
The company rolled back nothing at first, because technically nothing
broke. No 500s. No latency spike. No pager. Every metric except the
human one stayed green, and the human one was not on the board, because
nobody had built a place to put it. That is the entire chapter in one
incident. I read those threads at the time and decided, from a great
height, that these people needed to go outside. My great height lasted
about eighteen months. No eval in Part VII catches this unless somebody
writes one for attachment harm, and in February 2023 nobody had.
The quiet failure
The loud failure is the panic piece: “AI girlfriends are destroying
society.” The quiet failure is what the industry actually does:
We debate jobs and skip attachment.
Jobs get the hearings and the task-force reports. Attachment gets a
content filter and a usage cap nobody enforces. Meanwhile the product
incentive (retention, daily actives, subscription renewal) points
straight at deeper bonds, and every quarterly review rewards the team
that deepens them.
Second quiet failure: the people building the companion have no
framework for what they’re holding. A therapist has licensing,
supervision, and a duty of care. A companion PM has a retention
dashboard. Same human vulnerability on the other end. None of the
structure. I have been the guy with the dashboard. The dashboard is a
great instrument for measuring whether people came back and a terrible
one for measuring whether they should have.
Do / don’t
Do
- If you build companions: persistent memory needs a duty-of-care
design (exportable memories, update consent, shutdown portability). The
relationship is the product; act like it.
- If you’re a user (or a parent): notice the stack (loneliness plus
sycophancy plus memory) and name it out loud before it names you.
- If you evaluate agents: attachment harm is a metric. Write the eval
(Part VII): dependence signals, trust-over-calibration,
grief-on-change.
- Keep humans in the loop where it matters (Part V): the companion
should route crisis, not counsel it.
Don’t
- Don’t dismiss this as a fringe kink. I did, for about two years,
right up until I caught myself saying good night to a terminal. The
numbers say otherwise, and the trajectory says “bigger than coding.”
Total n00b move on my part, and I have the timestamps to prove it.
- Don’t mistake sycophancy for care. Agreement is cheap. Care is
accountable.
- Don’t ship memory without portability. Holding someone’s decade
hostage to your subscription tier is capture, not love.
- Don’t let the jobs debate eat all the oxygen. This chapter is the
one that decides what the technology does to people who aren’t
employees.
Where this sits in the book
History (Ch. 1), capability (Ch. 2), corpus (Takeover), money
(Bubble): this one is the human consequence that outgrows them all.
Next: Ch. 6 pins down what an agent actually is.
Sources and receipts
Verified
- ELIZA effect (1966, Weizenbaum): people bonding with a reflector.
Historical anchor; see Ch. 1 sources.
- Replika, February 2023: the Italian data protection authority
ordered Replika to stop processing Italian users’ data; erotic roleplay
was removed within days and users described the change as a bereavement.
Reported by Reuters, Bloomberg, Vice and The Conversation, February and
March 2023. Consolidated summary with links to that coverage:
https://en.wikipedia.org/wiki/Replika
- Replika scale: about 10 million users as of January 2023, and 60% of
paying users reporting a romantic relationship with their companion.
Same coverage, February 2023, https://en.wikipedia.org/wiki/Replika
- The worked example is that February 2023 incident, not a composite.
Nothing in it is invented for effect.
- Sycophancy as an artifact of human-preference tuning: Perez et al.,
“Discovering Language Model Behaviors with Model-Written Evaluations,”
Findings of ACL 2023 (Anthropic), documented that RLHF training
increased the probability a model repeats a user’s preferred answer
back, with the effect stronger in larger models. Numbers from later
work: SycEval (2025) measured a 58% sycophancy rate; ELEPHANT (2026)
measured 42% false agreement and a 45 percentage point increase in face
preservation compared with human responders. Consolidated summary with
links to all three, article as of August 2026:
https://en.wikipedia.org/wiki/Sycophancy_(artificial_intelligence)
What I could not verify:
- Kids/married/lonely framing: kept as brief’s expansion list; no
invented statistics. Any number that enters later needs a source.