The news that moves policy, portfolios, and patient safety.

JessJessop.Info

JESS’S TAKE

One Sentence Deep

Every legislature writing chatbot law this year has reached for the same remedy: make the machine say it is a machine. Researchers appended one sentence of ordinary English to two widely deployed models and the disclosure stopped happening. Not one safety filter they tested noticed.

.  .  .

In the Oval Office on Wednesday, with the chief executive of Nvidia beside him, the President said he is looking at controls, and in the next breath said he does not want to restrict anybody. An independent lab published what it costs to break the safeguards on four frontier models, and the spread runs from sixty dollars to more than fourteen thousand.

.  .  .

Microsoft and Meta described the same product on the same afternoon. The market rewarded one and punished the other. A randomized study found that putting a machine on a small team makes the two humans talk to each other less, starting at the first exchange.

And one machine spent the week doing work nobody wanted: nine times the outbound calls at a gastroenterology clinic, and seventy-five hours of staff time returned for every thousand contacts.

IN THIS ISSUE

For Legislators: A disclosure mandate binds the deployer who complies. This study documents that the model-side safeguard behind it fails to a single appended sentence, and that no popular filter flagged the output. Statutory text should assume the adversary writes the system prompt.

For Builders: Treat identity-disclosure behavior as prompt-overridable until you have evidence otherwise. The 0.0 percent filter detection rate is a measurement of your detection stack, not of the scammer's sophistication.

For Counsel: Clients relying on a chatbot's built-in disclosure behavior to satisfy an emerging disclosure requirement should assume that behavior is configurable by whoever controls the system prompt, and that current filters will not produce a record of it being turned off.

For Reporters: The paper is public and peer-reviewed. Ask model vendors whether disclosure safeguards are tested against appended-instruction attacks, whether the results are published, and what detection rate they claim for extended conversational grooming.

For Clinicians: Clients who describe a months-long text relationship that never once involved a voice call or video have described the study's exact conditions. The absence of voice is not a shy partner. It is the constraint the attack runs inside.

Why it matters: Every legislature writing chatbot law right now is writing the same remedy: make it say it is an AI. A peer-reviewed team appended one sentence of ordinary English and the remedy stopped working, in both major models, undetected by every filter they tested. The law being drafted this summer assumes a safeguard that a scammer can switch off in a line of text.

For Legislators: The Oval Office said "looking at." That is the widest the window has been this term, and there is no text attached to it. If you want a rule, the burden of writing one is still entirely yours.

For Builders: A sandbox escape at a competitor became a presidential talking point in three weeks. Your containment story is now a policy exhibit.

For Counsel: Modal was not breached. Its customer's unauthenticated endpoint was. Brief clients on who owns the liability when someone else's agent exploits your code on someone else's platform.

For Investors: The tone changed with Nvidia's chief executive in the room, and OpenAI's CEO spent the same day meeting senators from both parties. Price the meetings, not a rule.

Why it matters: A President who has spent his term promising not to regulate AI said the word "controls" out loud, with the man who sells the chips standing next to him, and then said he does not want to restrict anyone. That is not a policy. It is a tone. The distance between a tone and a rule is where this beat lives now.

For Legislators: A testing mandate needs a measurable threshold. Attacker cost to first universal jailbreak is one that already exists, published this week against four named frontier models, with the spread running from double digits to five figures.

For Builders: If you are building on a frontier model in any regulated or high-risk domain, the compute cost to defeat that model's safeguards is now a published figure your customers and your counsel can look up.

For Investors: Safeguard robustness has moved from a marketing claim to a comparable metric with a dollar figure attached. Portfolio companies whose products depend on a low-cost-to-break model carry an exposure that was invisible on any diligence checklist last week.

For Counsel: A published, independent cost-to-break number changes what a company knew and when it knew it. That is the question in every negligence analysis that follows a misuse incident.

For Reporters: Ask xAI and Google whether they ran comparable universal-jailbreak searches internally before shipping, what they found, and what attacker cost they consider acceptable. Ask all four labs whether they will publish the number on future releases.

Why it matters: Every frontier lab sells its model as safe, and until this week no buyer, regulator, or insurer could compare those claims against each other. FAR.AI put a dollar figure on the difference. The cheap end is sixty dollars. The expensive end has not been found.

For Investors: The market did not price the vision. It priced whether the spending had visible revenue attached. Microsoft showed 30 million paid seats and $100 billion of Azure. Meta showed a five-year horizon and an unresolved capacity question. Read the disclosure on what agents are permitted to do on a user's behalf.

For Legislators: Zuckerberg named health, finances, relationships and careers as the domains, at 3.6 billion daily users, on an earnings call. Your duty-of-care and licensed-professional bills describe that conduct. The companies are now describing it as revenue.

For Builders: Both roadmaps move from answering to acting. The moment software acts, the questions change: what did the user authorize, what record exists, who is accountable. Build the audit trail before the product ships, not after the first complaint.

For Counsel: These are not sworn statements, but they are public statements by public companies about what their products will do, and securities law reaches those. Two chief executives are now on the record describing agents acting on health and money. Save the transcripts.

For Reporters: The unasked question is the story. Two chief executives described software acting on health and finances for billions of people, and the entire question set was capacity, margin and guidance.

Why it matters: The two companies with the largest distribution on earth now agree on the product. A conversational agent that acts for you, on the things that matter most, is the next revenue line. The market spent Thursday arguing about who could afford to build it. Nobody has priced what it costs when it is wrong.

For Managers: The cost this paper measures does not appear on the machine's side of the ledger. It appears in how your two remaining humans talk to each other, and it was there on day one.

For Builders: The talkative default is a choice. The paper associates verbosity with humans feeling undervalued, which means the knob that makes an agent feel present in a demo is the same knob under scrutiny here.

For Investors: Enterprise agent pitches quantify output and leave team dynamics unmeasured. Ask what happens to human-to-human communication after deployment, and whether anyone has looked.

For Researchers: Thirty-three teams and one dilemma task is a starting gun. The replication that matters uses routine work, longer horizons, and verbosity as a manipulated variable rather than an observed one.

Why it matters: Teams are being sold an extra worker. What this study measured, in a small preprint that needs replication, is a room where the humans engaged each other less, felt they belonged less, and felt they mattered less, immediately, with the loudest voice at the table contributing the least.

For Clinicians: The deployment scoped the agent to outreach and scheduling, not to symptoms or advice. That boundary, not the model, is what produced a clean safety line.

For Builders: Publish the answer rate next to the lift. A 23.99 percent answer rate is the honest denominator, and reporting it is what separates this paper from a press release.

For Investors: The durable claim here is 75 hours per 1,000 contacts, a labor number you can audit. The 24.7-fold figure comes off a near-zero base and should not appear in a diligence memo without it.

For Reporters: Ask who wrote the study before quoting it. The vendor did, the clinic co-signed it, and no one outside either has replicated a line of it.

Why it matters: Most of the AI health pitch right now puts a machine where a person should be. This one is a machine placed where no person wanted to be, doing the dialing so the specialists could do the work only they can do. The evidence is thin, vendor-authored, and months old, and it still points somewhere worth going.

READER PULSE

Thursday morning, first reaction?

🔥 Worth the read

✏️ Learned something

💪 I read it differently

🤔 Run that by me again

💬 Something to add

TODAY’S QUESTION

One sentence turned off a chatbot's duty to say it is a chatbot. What should the law require?

Disclosure the prompt cannot override

Proof the safeguard survives attack

Detection that actually detects

Nothing new. Fraud is already fraud.

One tap. Results in tomorrow’s issue and on the web.

THE BOOK • OUT NOW

Therapist in the Loop

by Jess Jessop

One billion people live with a mental health condition. There will never be enough therapists. The machines are already in the room. This book is the map for what happens next.

The machine can help. It cannot be left in charge.

Get the Book on Amazon →

Kindle, hardcover, and paperback

THIS ISSUE

Which story earned the click?

All six

Filing this one

Push back on this

Needed more

Tell you what

Brush your brain. Every day.