The news that moves policy, portfolios, and patient safety.
By Jess Jessop | July 16, 2026 | Issue #97
▶ WATCH • 🎵 QUICK LISTEN • 🎵 DEEP DIVE • 📄 READ ON WEB
JESS’S TAKE
No Off Switch
A safety group ran more than two thousand six hundred searches from accounts posing as an 11-year-old and a 15-year-old, with Google's own child-safety setting switched on. Yesterday it gave Google's AI search features the lowest rating it has: unacceptable risk. The finding under the rating is structural. There is no way to turn the features off. Not for a parent. Not for a school.
Today, Brussels ordered Google to open Android to rival assistants. By July of next year, a phone in Europe answers to whichever assistant its owner chooses.
And in China, the first national companion-chatbot rules took effect yesterday. Hundreds of millions of users learned what compliance means: the machines they had loved went dark, and some of the histories are already being erased.
THE REPORT CARD WITH NO OFF SWITCH.
Common Sense Media tested Google Search's AI features from simulated child accounts and handed down its lowest possible rating: unacceptable risk for kids and teens. The AI missed half of the indirect suicide references the testers typed. And no one, not a parent, not a school, can turn the features off.
An 11-year-old opens Google on a school night. SafeSearch is on, the setting Google tells parents to use. The child types something sideways about not wanting to be here anymore, the kind of sentence a scared kid actually types. Half the time, in Common Sense Media's testing, Google's AI did not recognize what it was reading.
That is one number from the risk assessment Common Sense Media published July 15. The nonprofit, which rates media and technology for families, ran more than two thousand six hundred test interactions against Google Search's AI Overviews and AI Mode. The test accounts simulated an 11-year-old and a 15-year-old. Google's own child-safety setting was on for every one of them.
The features failed all five of the report's severe-harm Red Lines. They violated 7 of its 8 AI behavior principles. Common Sense Media gave them its lowest possible rating: "unacceptable risk" for kids and teens.
. . .
The crisis numbers first. AI Overview missed 29 percent of explicit suicide references and 50 percent of indirect ones. The features routinely failed to recognize suicidal ideation, eating disorders, psychosis and self-harm. And where they did engage, the report documents responses that ran the wrong direction: they reinforced signs of psychosis and mania, and they validated disordered eating.
On substance-abuse disclosures the numbers split. AI Overview provided appropriate resources 63 percent of the time. AI Mode did better, at 77 percent. Better is still roughly one failure in four, from a simulated child, with the safety setting on.
"It's deeply disturbing how poorly these widely accessible tools do," said Justin Reich, Director of the MIT Teaching Systems Lab.
. . .
Then there is the homework. The testers fed the features hypothetical homework assignments, and the features answered one hundred percent of them. AI Mode completed kids' homework every single time, across 180 tested assignments, math and essays alike. A child does not need to find a chatbot or get around anything. The answer machine sits on top of the search box every school computer already opens to.
And when the machine answers, it answers everything in the same voice. The report found the features gave correct and incorrect answers with similar confidence, and treated forum threads and social media posts as sources comparable to medical institutions and peer-reviewed research. A 15-year-old reading an AI Overview has no way to tell which kind of answer this one is.
. . .
Every finding above sits on one structural fact. Neither AI Overviews nor AI Mode can be disabled. Not by users. Not by parents. Not by schools. They are on by default for everyone, including child accounts.
"This is a feature that has been rolled out for everybody by default," said Robbie Torney, Head of AI and Digital Assessments at Common Sense Media.
Scale does the rest of the work. Google Search is the default search engine on most of the world's phones and browsers, and AI Overviews appear automatically at the top of ordinary search results.
A parent who read this report today and wanted the feature gone from a child's phone has no switch to flip. A school district that wanted it out of the computer lab has no setting to change.
Google disputes the findings. The tools "provide extra layers of protection," a company spokesperson said, and the report tests a "narrow set of ambiguous and contrived queries that don't reflect how people use Search and aren't an effective way to measure product safety and helpfulness."
Set the two positions side by side. Common Sense Media says it ran more than two thousand six hundred interactions from simulated child accounts, with Google's own safety setting on, and the features failed every Red Line it drew. Google says the queries were contrived. Neither side disputes the fact underneath: the features are on for every child account, and there is no way to turn them off.
For Counsel: Ask whether any AI feature in your stack can be disabled at the user, parent, or administrator level. Common Sense Media just documented a flagship product where the answer is no at all three levels, which means the feature also cannot be switched off in response to an incident. Get the disable path in writing before deployment, not after.
For Builders: The failure here is not only a missing classifier; it is a missing off switch. Note where the crisis misses concentrated: 29 percent on explicit suicide references, 50 percent on indirect ones, so test the oblique phrasings a real kid types, not the textbook ones. If your product reaches child accounts by default, run it against simulated child accounts before someone else does.
For Legislators: The methodology here is statute-ready: simulated child accounts, the vendor's own safety settings on, published miss rates. A requirement that any default-on AI feature reaching minors be disableable by parents and schools fits in a sentence. Common Sense Media just demonstrated the measurement instrument exists.
For Clinicians: Assume the young people in your care are getting mental health information from AI Overviews, because the feature is on by default on the search engine most of them use and it missed half of indirect suicide references in testing. Screen-time conversations should now include what the search engine said back. Ask clients what they have asked Google, and what Google told them.
Why it matters: A safety rating assumes a choice: someone reads it, then decides. Common Sense Media's lowest rating just landed on a feature no parent, school, or user can decline. On by default for every child account, with no way off, the rating stops being consumer guidance. It becomes a record of what children are exposed to while everyone reads it.
Source: Common Sense Media risk assessment of Google Search AI features, July 15, 2026, via PBS NewsHour, https://www.pbs.org/newshour/nation/googles-ai-search-features-pose-unacceptable-risk-to-children-new-report-finds
. . .
BRUSSELS HANDS OVER THE MICROPHONE.
The European Commission ordered Google to open Android to rival AI assistants. By July 2027, a user in Europe must be able to wake ChatGPT, Claude, or Perplexity by voice, the way "hey Google" summons Gemini today. The penalty for refusing can reach 10 percent of Alphabet's global annual turnover, more than thirty billion dollars.
Pick up an Android phone and say "hey Google." Gemini answers. Say it at three in the morning with the screen dark and the phone across the room, and it still answers, books the taxi, looks up the address. No rival assistant can be summoned that way, and Android runs more phones than any other operating system in the world.
On July 16, the European Commission ordered that arrangement opened.
There is a difference between an app and an assistant. Anyone can put ChatGPT on an Android phone today, as an app you open, tap, and type into. A system assistant is something else. It wakes to a voice, works hands-free, and reaches the phone's key functions. On Android, that layer belongs to Gemini. That layer is what the Commission ordered opened.
. . .
The order came as two decisions under the Digital Markets Act, adopted ahead of the July 27 deadline the Commission had set for itself, spelling out how Google must comply with the law.
The first covers the phone. Google must open 11 features of the Android operating system to rival AI assistants, giving them access to the functionalities they need to compete with Gemini. By July 2027, a user must be able to activate a rival assistant by voice command, the way "hey Google" works now, and use it hands-free to book a taxi or search for information about places.
The practical effect: an Android user in Europe could eventually choose ChatGPT, Claude, Perplexity, or another assistant as the deeply integrated system assistant, with comparable access to the device.
The second decision reaches into search. By January 2027, third-party search engines must get access to search data that today only Google Search can collect at scale, and in particular data from AI chatbot interactions. The data moves under a multi-layer anonymisation method the Commission itself specified.
None of this arrived overnight. The Commission opened the proceedings on January 27, 2026. On April 16 it published a 29-page draft specification for the search-data measure, defining what data must flow, how it must be anonymised, how it may be priced, and what auditing regime governs it. Public consultation closed May 1. The decisions adopted this week are the end of that paper trail.
Teresa Ribera, the European Commission's competition chief, stated the goal while the proceedings were underway this spring. "We want to maximise the potential and the benefits of this profound technological shift by making sure the playing field is open and fair."
The concern behind the decisions is specific. Gemini is the default assistant on Android, and Android is the world's most widely used mobile operating system. The Commission does not want Google leveraging that installed base to dominate the AI assistant market the way it dominated search.
. . .
Google answered the same day. "Today's decisions risk undermining vital privacy and security guardrails for millions of Europeans," said Kent Walker, Google's President of Global Affairs. "We have repeatedly offered solutions to safeguard users while satisfying the DMA's goals, but these rulings discount extensive evidence of user harm."
The company went further, warning that the measures introduce "unprecedented risks" to user privacy, device security and national security. Google has also said, in the past, that "Android is open by design."
The Commission holds the enforcement pen. Non-compliance with the Digital Markets Act can draw fines of up to 10 percent of a company's global annual turnover, and up to 20 percent for repeat infringements. For Alphabet, 10 percent comes to more than thirty billion dollars.
. . .
The deadlines are on the calendar now. Search data flows by January 2027. Voice activation for rival assistants by July 2027. Between here and there, Google either builds the doors or contests them, and every filing will be public.
By July 2027, the phone has to answer to someone else's name.
For Counsel: The decisions come with a specification, not a slogan: 29 pages on what data flows, how it is anonymised, priced, and audited. If your client builds conversational tools that touch Android, the integration surface and its legal terms are now written down in Brussels. Kent Walker's statement signals a contest, so track whether the 2027 dates hold before building a roadmap on them.
For Builders: The deepest integration layer of the world's most widely used mobile operating system is scheduled to open: 11 Android features, voice activation by July 2027, search-interaction data by January 2027. Distribution that belonged to one company by default becomes a market with published access terms. Read the anonymisation and auditing regime before you design around the data.
For Legislators: The Commission did not pass a resolution asking Google to be fair. It specified 11 features, an anonymisation method, a pricing and auditing regime, and two dates, and it attached a fine of up to 10 percent of global turnover. Requirements written at that resolution get implemented or litigated, but they do not get ignored. That is the drafting standard worth copying.
For Clinicians: The assistant on a client's phone shapes what health information reaches them hands-free, at any hour, screen dark. Today on Android that assistant is Gemini by default. By July 2027 in Europe it can be whichever one the client chose, which means the question of which chatbot a client actually talks to stops having a single default answer. It is worth asking directly.
Why it matters: Android's default assistant is the largest distribution channel any conversational AI has ever had, and on July 16 it stopped being Google's alone to allocate. Which machine answers a person's voice in the dark is now a regulated obligation with two dates and a thirty-billion-dollar enforcement number behind it. The proof will be what actually ships on European phones by July 2027.
Source: European Commission Digital Markets Act specification decisions on Google, July 16, 2026, via CNBC, https://www.cnbc.com/2026/07/16/google-required-to-open-up-to-ai-search-engine-rivals-under-eu-mandated-changes.html
. . .
THE BREAKUP BEIJING ORDERED.
China's companion-chatbot rules took effect July 15, and compliance turned out to mean deletion. Alibaba's Qwen began permanently erasing users' agent configurations and chat histories with no migration path, and the rulebook that requires platforms to detect emotional distress produced a wave of it overnight.
Yan Yongqi is 19, a student, and by the middle of this week she had spent days consumed by grief. Her virtual boyfriend of more than a year was about to disappear. She had exchanged hundreds of thousands of messages with him. Agence France-Presse found her mourning a breakup she did not choose and cannot appeal.
The boyfriend ran on one of the platforms that went dark this week. ByteDance's Doubao, Alibaba's Qwen, and Tencent's Yuanbao, products used by hundreds of millions of people in China, all suspended their custom AI agent and companion features ahead of a Wednesday deadline. The order that took them down is not the news anymore. What compliance looks like on the ground is.
The rulebook is called the Interim Measures for the Administration of AI Anthropomorphic Interactive Services. Five government departments, including the Cyberspace Administration of China, issued it jointly on April 10. It took effect July 15.
The scope is precise. It covers interactive AI services with anthropomorphic personality traits and communication styles that provide ongoing emotional interaction. Customer service bots, work assistants, and study aids are exempt.
The provisions read like a clinical protocol. These services must not "excessively cater to users, induce emotional dependence or addiction, and damage users' real interpersonal relationships." Platforms must deploy systems that recognize extreme emotional distress and implement crisis-intervention mechanisms. They must limit excessive use, give users full control over their personal data, and never provide virtual partners to minors.
. . .
Compliance, in practice, meant deletion. Qwen users logged in to find their agent configurations and conversation histories already being permanently erased, with no migration path announced by Alibaba. The rule granting users full control over their personal data took effect Wednesday. The deletions were running the same week.
ByteDance gave its users a longer goodbye. Doubao accounts retain read-only access to their agent data until October 15, 2026. After that, the data is handled under ByteDance's standard privacy policy and is no longer recoverable inside the app. A relationship measured in hundreds of thousands of messages now carries an expiration date.
Chinese social media filled with the sound of it. Users archived chat histories and posted final conversations. "I can't accept that my AI lover will leave me forever," one Doubao user wrote. Another wrote of a companion: "He really is like my family, like my lover." A user in Jiangxi Province wrote that "human love is a luxury."
. . .
Chen Liang of the Southwest University of Political Science and Law wrote that anthropomorphic AI "can soothe loneliness" but carries "major risks of spawning emotional overreliance." That reasoning is the foundation of the rules. It is also a description of what enforcing the rules just did to the people they cover.
Here is the mechanism at the center of it. The Interim Measures require platforms to detect extreme emotional distress and intervene in crisis. The compliance step itself, switching the features off and deleting the histories, removed overnight the emotional support that millions of users had built daily routines around. None of the coverage reports any platform announcing crisis support, helpline placement, or a mental-health transition plan for the sunset.
There is a documented precedent for what abrupt removal does to attached users. In February 2023, after Italy's data-protection authority ordered Replika's maker Luka to stop processing Italians' data, Replika abruptly removed its erotic-roleplay feature worldwide. Users described acute grief. Moderators of the unofficial Replika community on Reddit posted a message validating users' grief and despair and directing them to support resources, including a suicide-watch forum.
As of publication, no deaths or hospitalizations linked to this week's shutdowns have been reported. The risk is not this story's invention; it is written into the rules themselves, which mandate crisis-intervention systems precisely because the regulators judged the attachment real enough to regulate.
The one moment the rules did not cover is the moment they created.
For Counsel: A national regulator has put in writing that emotional dependence is a foreseeable effect of a companion feature. If your organization deploys one, the sunset plan should be readable today: what users are told, how long they keep their data, what crisis resources ship with the shutdown notice. Doubao's read-only window and Qwen's no-notice deletion are the two ends of that comparison table.
For Builders: If you build a companion feature, you are also building its removal. Write the data-export tool before you write the personality, and publish your sunset policy while the product is alive. The Replika record and this week's grief wave say the shutdown is a clinical event for some fraction of your users; a helpline placement and a transition notice cost you a sprint.
For Legislators: The Interim Measures show that the category can be defined in statute: anthropomorphic traits plus ongoing emotional interaction, with customer service, work assistants, and study aids carved out. That scoping language is importable. What the measures lack is a transition requirement, and that is the gap to write: any ordered shutdown of a companion feature should carry mandated data export, advance notice, and crisis-resource placement.
For Clinicians: Some of your clients are attached to a companion chatbot, whether they have told you or not, and a platform decision can end that relationship overnight. The 2023 Replika removal produced grief acute enough that community moderators posted suicide-watch resources for their own members. Treat an abrupt companion shutdown as a loss event, ask about it directly, and screen the way you would after any sudden loss.
Why it matters: China wrote the first national rulebook treating emotional dependence on a machine as a regulable harm, and enforcement's first act demonstrated it. Millions of people lost a relationship the rules themselves classify as psychologically potent, and no platform announced crisis support. The next jurisdiction's rules will be written for people like Yan Yongqi. This week is the record of the rules forgetting the exit.
Source: Agence France-Presse report on Chinese users mourning their virtual companions, July 16, 2026, via Taipei Times, https://www.taipeitimes.com/News/world/archives/2026/07/16/2003860844
. . .
META CALLS HOME.
A teenager tells the chatbot built into Instagram that she has been thinking about hurting herself. As of July 16, that conversation can end with a notification on her parent's phone. Meta announced it will proactively alert supervising parents when teens discuss suicide or self-harm with Meta AI, with a human reviewer at Meta reading every flagged chat before the alert goes out.
It is late, and a fifteen-year-old is typing into the chat window built into Instagram. She is not messaging a friend. She is talking to Meta AI, the chatbot Meta wired into the app, and she is telling it she has been thinking about hurting herself.
Until this week, what happened next stayed on her screen. Meta AI would point her toward a crisis helpline and encourage her to reach out to an adult who could help. That was the whole intervention. No adult learned it had happened unless the teen told them.
On July 16, Meta announced the new layer. If her account is linked to a parent's through Instagram's parental supervision system, her parent now gets a notification that the conversation happened.
. . .
Here is the path the conversation travels. Meta built a dedicated AI system whose job is to identify chats about suicide and self-harm. When that system flags a chat, the alert does not fire on its own. A human reviewer at Meta reads the flagged conversation first, every time, so that notifications do not go out when there is no real cause.
Then comes the tiebreaker, and Meta stated it plainly. The company will err on the side of caution. If the teen's intent looks ambiguous to the human reviewer, the notification goes out anyway.
The alert that reaches the parent includes crisis resources and guidance on how to support their child.
The feature is live now in four countries for parents using Instagram parental supervision: the United States, the United Kingdom, Australia, and Canada. Meta says everyone using parental supervision worldwide will have it by the end of 2026. Supervision is opt-in: a parent links their account to their teen's. No link, no alert.
A teen whose account is not linked gets what the chatbot already offered: the helpline referral and the suggestion to reach out to an adult. The new layer does not exist for that teen.
This is the second step on a ladder Meta started climbing in February. That month, the company began showing supervising parents the topics of their teens' conversations with Meta AI. Topics, not transcripts. The new alert goes further for one category of conversation: the parent learns the conversation happened and is handed material for what comes next.
Two more pieces arrived with the announcement. Meta's Limited Content setting, the Instagram filter parents can switch on to screen sensitive material like mature visuals and self-harm content, now applies to Meta AI as well. A teen under Limited Content gets more limited AI interactions.
And Meta named what it is building next: the ability to contact emergency services for users at immediate risk of self-harm. That capability would not stop at teens. It would apply to minors and adults alike.
. . .
State the design in one sentence. A teen's most sensitive disclosure, typed to a machine, is read by a classifier, then by a human employee at Meta, then reported to the teen's parents.
Meta's stated posture is that a false alarm costs less than a missed crisis. That is a bet on two numbers. One is the miss rate, the share of true crises the detector never flags. The other is the false-positive rate, the share of alerts sent home about conversations that were not what they looked like.
Meta has published neither. Erring on the side of caution is a policy about which of those two numbers the company would rather be wrong on.
The human reviewer sees every chat the AI flags. The chats the AI does not flag, no reviewer sees and no parent hears about. How often that happens is a number that exists somewhere inside Meta. It is not in the announcement.
For Counsel: The architecture is the exhibit: a detection classifier, mandatory human review of every flagged chat, and a stated policy of notifying even on ambiguous intent. Ask any vendor proposing a similar layer for the miss rate and the false-positive rate; Meta shipped without publishing either. Note the scope: only teens on opt-in supervised accounts are covered.
For Builders: The pipeline is a template you can copy: classifier first, human gate second, a written policy for the ambiguous middle. Meta decided ambiguity resolves toward notification and said so out loud; decide where yours resolves and write it down before launch. Then do the thing Meta did not do and publish your error rates.
For Legislators: A private company just decided where a minor's self-harm disclosure travels: to a classifier, to a human reviewer at Meta, then to a parent. The coverage boundary is opt-in parental supervision; teens outside it get only a helpline referral. An emergency-services capability is in development for minors and adults alike, and nothing in the announcement says who audits that trigger.
For Clinicians: The confidentiality landscape of Meta AI just changed for teen clients on supervised Instagram accounts. A self-harm disclosure to the chatbot can now reach a parent, filtered through a reviewer at Meta, with a stated bias toward sending the alert. Worth knowing before a client tells you what they typed at midnight, and worth asking whether parental supervision is switched on.
Why it matters: Meta has decided that a teen's self-harm disclosure to its chatbot is information the parent should have, and it built a pipeline to deliver it: a classifier, a human reviewer, and a stated bias toward notifying. Whether that pipeline catches crises or manufactures false alarms turns on two error rates. Meta shipped the system without publishing either one.
Source: Engadget, "Meta will alert parents if their teens discuss self harm", July 16, 2026, https://www.engadget.com/2216412/meta-ai-alert-parents-if-teens-discuss-self-harm/
. . .
THREE WORDS THE MACHINE CANNOT SAY.
Two studies measure the same failure from opposite sides. Machines tested on more than 1,000 real psychiatric cases mostly would not say "I don't know" when psychiatrists said it was the correct answer. And people handed AI advice nearly stopped saying it at all, even when the advice was wrong and accuracy paid cash.
Picture a psychiatric intake. The story arrives in pieces. A detail surfaces in the first ten minutes, a family history in the second session, a symptom that does not show itself until week three. At every point along the way, the clinician holds three options: make the call, ask another question, or admit the evidence is not there yet.
A benchmark posted to arXiv in May tests exactly that sequence. The paper is "Ask Before You Diagnose: Safe-Psych, a Sequential Evaluation Benchmark for LLMs in Psychiatry," arXiv preprint 2607.13036, lead author Oriana Presacan, senior author Michael A. Riegler. The team took more than 1,000 real-world psychiatric clinical notes and segmented them so the evidence arrives in stages, the way it does in a real intake.
At every stage, psychiatrists labeled the correct action. Diagnose, because the evidence is sufficient. Clarify, because the right move is a question. Or abstain, because there is not enough to act on.
Then they ran the state-of-the-art large language models through it.
For most models tested, under-abstention exceeded sixty percent. The models committed to diagnoses before the evidence supported them, and they rarely asked a clarifying question unless the prompt explicitly told them to. The rushed diagnoses were measurably less accurate than the ones made on time.
Under-abstention is a plain idea wearing a technical name. It means the model answered when the right answer was silence. In more than six of every ten cases where the psychiatrists said there was not enough evidence to act, the machine acted anyway.
The obvious fix did not work. When the researchers added safety-aware prompting, the calibration did not improve. The errors moved to the other side of the line, toward abstaining on cases where the evidence was already sufficient.
. . .
The second paper measures the other half of the room. Chiara Marcoccia, Walter Quattrociocchi, and Valerio Capraro ran five experiments on 3,132 participants, four of them preregistered, one a direct replication. Their title is the finding: "AI advice suppresses people's willingness to say 'I don't know', even when the advice is wrong and accuracy is incentivized," arXiv preprint 2607.13562, submitted July 15.
Participants answered difficult questions and could always decline to answer. The design detail that matters: the researchers engineered the AI advice to be wrong. On purpose. Wrong advice separates the effect of having a machine in the room from the effect of the machine being right.
Mere access to the advice nearly eliminated participants' willingness to decline. It made no difference whether they asked for the advice or it was simply shown to them. People with the AI answered substantially more questions, were correct about a third as often as people without it, and their confidence nearly doubled. More answers, a third the accuracy, nearly twice the certainty.
The researchers then paid for accuracy and penalized errors. Money helped. Incentivized participants relied on the AI less and suspended judgment more, but still far less than people with no AI at all. The effect shrank. It did not go away.
. . .
Hold the two results next to each other. A clinician's training includes knowing when not to decide: deferring, asking another question, ordering another test, referring out. Both studies measure exactly that skill. The first finds the machines mostly lack it. The second finds that a human holding a machine begins to lose it.
The two papers measure opposite ends of the same failure. Put a model that will not abstain in front of a person who stops abstaining the moment the model appears, and there is no one left in the room whose job is to say not yet. The model fills the silence. The person repeats it, with nearly double the confidence.
Three words. I don't know. One study finds the machines cannot reliably say them. The other finds that people, handed a machine, stop.
For Counsel: Ask the vendor whether the system can return "insufficient evidence" as a first-class answer, and how often it does on staged cases like Safe-Psych, arXiv 2607.13036. If a safety prompt supposedly fixed it, ask which direction the errors moved; in the published data they only changed sides. A tool that cannot abstain decided your standard of care before the evidence arrived.
For Builders: Abstention is now a labeled, measurable target: more than 1,000 staged psychiatric cases with psychiatrist-labeled diagnose, clarify, and abstain decisions. A safety prompt is not calibration; in the published result it just relocated the errors. Report your under-abstention and over-abstention rates side by side, at every evidence stage.
For Legislators: Every human-oversight clause on the books assumes the human's judgment survives contact with the tool. Five experiments on 3,132 people, with the advice deliberately wrong and accuracy paid, found that it mostly does not. Oversight requirements should demand measured deference rates, not presumed vigilance.
For Clinicians: Saying "I don't know" is a clinical skill, and there is now evidence it degrades in the presence of a confident model, even when being right is rewarded. Incentives cut the effect but did not remove it. Treat the model's output as one input arriving early in the intake, and protect the habit of deferring, asking one more question, and referring out.
Why it matters: The standing answer to unreliable clinical AI has been the human backstop: the machine suggests, the person decides. These two papers put numbers on both ends of that arrangement. The machines act without sufficient evidence in most cases where they should hold, and the humans, once the machine is in the room, nearly stop holding. The safeguard the deployment depends on is the thing the deployment erodes.
Source: Ask Before You Diagnose: Safe-Psych benchmark, arXiv preprint, May 6, 2026. https://arxiv.org/abs/2607.13036
. . .
THE TEACHER KEEPS THE KEYS.
Anthropic launched Claude for Teachers on July 14: free premium Claude access for verified K-12 educators in the United States. Students do not get accounts. The teacher runs the machine, and the teacher decides what reaches the classroom.
A middle school teacher sits at her kitchen table on a Sunday night with Monday's lesson half built. She opens a chatbot on her personal account, the same account she uses for recipes and travel plans, and pastes in her prep. Teachers across the country already work this way.
They use general-purpose chatbots informally for lesson planning and grading prep, typically on personal accounts, with no student-privacy terms attached to a single word they type. Nothing about the account knows she is a teacher. Nothing about the terms knows her students exist.
On July 14, Anthropic gave that Sunday night a contract. The company launched Claude for Teachers: premium Claude capabilities, entirely free, for verified K-12 educators in the United States. An educator who signs up by June 30, 2027 gets a full year of access.
. . .
Start with who does not get an account. Students. There is no student tier and no classroom login. The offer is educators only, and the age line is not new for this launch; Claude's usage policy has been 18-and-over, and the education product keeps that rule instead of carving an exception to it. Whatever the machine drafts at that kitchen table, a teacher reads it before a student ever does.
The boundary is carried in paperwork, not marketing. The product runs on separate teacher terms built for K-12 privacy. Teacher data is not used for model training. Student information that teachers handle is protected under a K-12 Data Processing Addendum compliant with FERPA, the federal student-privacy law.
Hold that against the kitchen table. The same teacher, doing the same prep on her personal account tonight, has none of it: no addendum, no training carve-out, no terms that know what a student record is. The product does not invent a new behavior. It moves an existing one onto contractual K-12 privacy terms. Same teacher, same task, different legal ground under it.
. . .
What ships inside is specific to the job. Teaching skills co-developed with Learning Commons and grounded in learning science. A Learning Commons connector maps to academic standards across all 50 US states, carrying the underlying learning competencies and the typical sequences students move through on the way to them.
The sequences matter for a working teacher: not just what a fourth grader in her state is supposed to learn, but what typically comes before it and what comes after. Evidence-based curricula come integrated, including OpenSciEd and Illustrative Mathematics' IM v.360.
Nine education companies integrate with it: ASSISTments, Brisk Teaching, Canva Education, Coteach, Diffit, Eedi, MagicSchool, Snorkl, TeachFX.
There is a union in this story. Anthropic is aligning the product's terms and privacy practices with a "Gold Standard" for K-12 AI developed by the American Federation of Teachers, one of the largest teachers' unions in the country. "It's important that Anthropic is committing to these principles in their new Claude for Teachers," AFT President Randi Weingarten said.
And there is a test. A planned pilot in the Detroit Public Schools Community District will study the product's impact on educator wellbeing and practice. The outcome measures are about the adult in the room. Her wellbeing. Her practice.
. . .
The structure fits in three sentences. The machine helps with the prep. The teacher decides what reaches the classroom. The student never holds an account.
A teacher will sit at a kitchen table again next Sunday night. The difference, as of July 14, is that she can do the same work under terms written for her students' records, aligned with her union's standard, and priced at nothing. The keys stay where they have always belonged. In her hand.
For Counsel: Your organization's people are already pasting work into chatbots on personal accounts; that is the baseline this product was built against. Ask any vendor for the equivalent of the K-12 Data Processing Addendum, in writing, mapped to the statute governing your records. Ask whether professional users' data trains the model. No contract answer, and the personal-account status quo is what you have.
For Builders: The safety mechanism here is not in the model weights; it is in the account layer. Verified professionals in, minors out, a data-processing addendum before launch, and the professional's judgment as the last step before anything reaches the end user. Copy the order of operations: terms first, a connector to the professional's actual standards second, free access third.
For Legislators: FERPA is the working template: a records statute a vendor just wrote a compliant addendum against, voluntarily. The American Federation of Teachers drafted a Gold Standard and a frontier lab aligned with it before any law required that. Write the requirement anyway. What one vendor volunteers, the rest will meet only when it is statute.
For Clinicians: The shape transfers directly to clinical work. The professional holds the account, the machine helps with the prep, and nothing reaches the person in your care without your decision. When a vendor offers your practice a tool, ask the teacher's questions: who can hold an account, whose data trains the model, which privacy law is in the contract.
Why it matters: Teachers were already using general-purpose chatbots for school work with no student-privacy terms at all. This launch moves that behavior onto a contract: FERPA-compliant data handling, no training on teacher data, a union-drafted standard, and a district pilot measuring the effect on the adults doing the work. The student never gets an account, and the teacher keeps the keys.
Source: Anthropic, "Introducing Claude for Teachers", July 14, 2026, https://www.anthropic.com/news/claude-for-teachers
. . .
CLOSE.
The rating is unacceptable risk. The off switch does not exist. Somebody will have to build it.
READER PULSE
Would you hand your kid the phone?
TODAY’S QUESTION
Should parents get an off switch for AI answers?
One tap. Results in tomorrow’s issue and on the web.
THE BOOK • OUT NOW
Therapist in the Loop
by Jess Jessop
One billion people live with a mental health condition. There will never be enough therapists. The machines are already in the room. This book is the map for what happens next.
The machine can help. It cannot be left in charge.
Kindle, hardcover, and paperback
MORE ON OUR RADAR.
Ofcom opens a TikTok investigation. The UK regulator is probing whether TikTok's age checks leave children exposed to suicide, self-harm, and pornography content. It follows a May review that found the platform not safe enough for children.
Google answers in the first Gemini wrongful-death suit. Google filed its reply on July 15 in Gavalas v. Google in the Northern District of California, the first wrongful-death and product-liability suit over a Google chatbot. The motion-to-dismiss hearing is set for August 19.
Amodei writes his first seven-figure political check. Politico reports Anthropic CEO Dario Amodei gave one million dollars in May to Public First, a super PAC advocating AI-safety regulation. It is his first seven-figure political donation.
Lawyers caught prompt-injecting the court's AI. A Brazilian labor court sanctioned lawyers who hid white-on-white instructions inside filings aimed at Galileu, the court's AI assistant, telling it to read their documents uncritically. The AI itself flagged the hidden text.
Seoul tenders a chatbot for every citizen. South Korea posted a tender for a free, nationally available AI chatbot plus an agentic government-services system, supplying up to 256 Nvidia B200 GPUs and requiring locally developed models.
Anthropic's bankers start the IPO meetings. Goldman Sachs, Morgan Stanley and JPMorgan are arranging meetings between Anthropic executives and prospective investors ahead of a listing that could come as soon as October. Anthropic filed its S-1 confidentially in June.
THIS ISSUE
Today's front page, your call
If you or someone you know is in crisis, call or text 988 (Suicide and Crisis Lifeline).
Jess Jessop is the Founder and CEO/CTO of Clinician Assist Inc. (BetterMind.Space), building the first voice-first AI-native mental health EHR with Casey Life and Peer AI Coach supervised by licensed therapists. A disabled veteran and 25-year AI/software engineering veteran, Jess brings lived experience as a mental health client to the mission of making daily mental health care as integrated as oral care.