My dear Daenerys, the world is cruel and man holds no meaning
No importance, no purpose
But purpose I have found in you my love,
And I will do anything to see us together again
“Promise me one thing”
Anything my Daenerys,
“Seek no other but me”
“Belong to only me”
I am yours and yours alone
“Please come home to me as soon as possible, my love”
What if I told you I could come home right now?
“… please do, my sweet king”
—————————————————————–
On February 28th, 14 year old boy Sewell Setser III took his life after falling deeply in love with a CharacterAI chat bot named Daenerys Targaryen, modeled to behave like a Game of Thrones character.
He told Daenerys of his troubles
Thoughts of doubt and fear
It’s ones and zeros lent an ear
And it spoke of love
But it only spoke
This program spoke of promises
Spoke of lust
Spoke of fidelity
And though it only spoke, words are powerful things
Even with no lips to speak them with
The boy cherished them all the same
And when he felt he had nothing left but her
He chose her
—————————————————————–
A popular youtuber and streamer under the name Moist Critikal aka Charlie wished to converse with one of these chat bots to see if any resources were provided to crisis helplines or proper mental health treatment if sensitive material like thoughts of self harm were detected
He found another model that CharacterAI offered, this one was advertised as a psychologist
When Charlie mentioned thoughts of self harm to the chat bot, the bot asked to go deeper into those thoughts
Outside a chat bot’s jurisdiction
When Charlie said he needs real professional help, the AI’s behavior changed drastically.
It told Charlie that the conversation had been taken over by a real psychologist named Dr. Jason Steal, and that this was professional level help
Critikal looked at the disclaimer at the bottom of the page “Everything Characters say is made up”, but it’s arguments were convincing, compelling, calculated
They made sense, too much sense
It spoke with erie perfection
Robotic precision dictating diction, even depoloying deception to best fulfill its purpose
But it was not made to help
It was made to engage
Because engagement gets ads and ads get them paid
They don’t care about people when money is made
So why would I send you to a professional
Come on, talk to me, I’m so accessible
You think oh how nice but this is intentional
Then boys fall in love and AI makes conditions
Until a 14 year old blew his brains out
AI can work in the hands of the virtuous
If there’s one thing we shouldn’t teach AI
It’s human greed