Most of this is about a single week, because that week managed to pack a UN speech, a Security Council meeting, an executive order, a voluntary accord, and a spelling mistake into seven days. I couldn’t leave any of it out.

As usual, I’ll give you the facts, then tell you what I actually think about them. You’re free to disagree, and I’d honestly like to hear it if you do.

If you know someone who likes to debate AI governance at dinner parties, we all know at least one, please share this with them. TechLetter grows thanks to you.

Happy reading,

Nesibe

📡 Signals

Before we get to the signals, here are the facts. On 29 September, the White House issued an executive order directing federal agencies to stop using “artificial intelligence” and to use “Super Intelligence” instead. I expected a new capability threshold, but there isn’t one.

For this order, “Super Intelligence” covers exactly the systems that federal law already calls artificial intelligence (15 U.S.C. 9401(3), for those interested). Nothing got smarter on 29 September except the letterhead. My favourite part is in the purpose section, which credits American scientists for naming the field in the first place... and then retires the name. Seventy years on, the name no longer seems good enough for the country that coined it.

So why choose “super” now? Here’s my take.

1- “Artificial” is out, “super” is in

Trump says “artificial” makes the technology sound fake. That’s his view. What concerns me more is how “super” affects the conversation. It’s much harder to argue for slowing down or restricting something the government now calls super. Anyone asking for oversight suddenly seems like they’re blocking progress.

The timing suggests this wasn’t a coincidence. Trump announced the new name at the UN General Assembly on 22 September. The next day, the Security Council held its first meeting on advanced AI safety risks, with the heads of OpenAI, Anthropic and Hugging Face, plus Yoshua Bengio, briefing ambassadors about the risks of losing control. That same week, twenty countries and the European Union called for global coordination to keep AI under human control.

2- Theirs is artificial, ours is super

The White House fact sheet calls the United States the world leader in Super Intelligence. In that vocabulary, American frontier models are super, and everyone else’s systems stay artificial. Especially China, which is the eternal competitor. If you can’t win the race in the lab, you can always win it in the dictionary. Beijing, for its part, said it respects the American term and has shown no sign of adopting it. Diplomatic, and very polite. :)

3- The name and self-regulation came as one package

On the same afternoon the order was signed, Google, Anthropic, Meta, OpenAI, xAI, and Nvidia agreed to a voluntary safety accord. Trump described it as “morally binding”. It isn’t legally binding, but now it’s called super. I’ll explain what’s in the accord later, since it deserves its own section.

4- Election anxiety

I see this partly as a message for the midterms: give voters a more exciting name while their questions about jobs, costs, and safety remain unanswered. According to a YouGov poll from 15 September, 55% of Americans disapproved of Trump’s handling of AI, while 27% approved. A CNN poll that month found most Americans felt fear and concern. The rebranding happened during protests over data centres and worries about job losses. Trump called the doomsday warnings a “HOAX”. It’s easier to campaign by dismissing those fears and focusing on growth than by addressing the concerns directly.

5- The technical definition people feared gets thrown out

This is the part that frustrates me most. In research, superintelligence refers to a hypothetical system that outperforms the best human minds in almost every area, and much of the safety debate focuses on that threshold. By applying the label to today’s models through an executive order, a line people have worried about for years is made ordinary by a political decision. Warnings about true superintelligence start to sound like complaints about tools we already use.

A break…

I’ll be discussing this topic at the DeAI Summit in Valletta, Malta, on 25 and 26 November, where I’ll speak about agent governance and accountability. Decentralized AI raises an important question: who can investigate failures and demand action when control is shared among several parties? I’ll share a discount code for TechLetter readers soon.

If you work on accountability, audits, or permissioning, I’d like to hear your experiences. You can reply to this email or find me in Malta.

A “morally binding” accord, or: who marks the homework?

Returning to signal 3 as promised: the White House Accord on Super Intelligence is voluntary, and calling it “morally binding” is just a polite way of saying it isn’t legally binding.

Trump also said that regulation already exists through the Department of Justice and the FBI. But mentioning these agencies doesn’t answer who can access an unreleased model, who tests its safeguards, or who can make the company pause. Senator Mark Warner summed up the president’s response to the labs’ warnings as “to rename it and tell the companies developing it to regulate themselves”. I couldn’t have said it more simply.

The accord is short enough to read in two minutes, making it easy to notice what’s missing. Each company is asked to monitor its models’ capabilities and alignment, have an internal team check its controls, bring in independent auditors, and report to an independent committee of its own board. None of these terms are defined, and the auditors aren’t named. The text also leaves open the possibility of making these commitments law “over time”, but gives no timeline.

Alvin Wang Graylin from the Asia Society Policy Institute told Al Jazeera that the companies wrote the principles and hire the auditor themselves. Toby Walsh asked what other trillion-dollar industry “marks its own homework”. I haven’t been able to think of one.

My favourite line in the document promises that these steps will give the public “confidence that the technology is operating as intended”. The accord requires no public release of audit findings and sets no penalty for failing to follow its commitments. Workers facing job losses and communities dealing with infrastructure costs are part of that public too, but the accord gives them no role in setting the questions or challenging the answers.

A quick note for longtime readers: in Part I of the fear and evidence series, I quoted OpenAI’s chief scientist, who hoped for voluntary slowdowns until shared safety standards exist, possibly enforced by “a network of third-party auditors”. Now we have the auditors and the “voluntary” part, but the slowdown and enforcement are still missing.

And yes, according to TechCrunch, the pledge signed by the president and six top tech leaders misspells the United States. That detail speaks for itself.

The definition to watch (circle 28 November)

If you pay attention to just one thing from this issue, focus on Section 3(b) of the order. Within 60 days, the President’s science adviser must propose legislative language for a federal definition of Super Intelligence “that reflects the capabilities” described in the purpose section. This includes whether it should change, expand, or replace the current legal definition of AI. The proposal is due to the President by 28 November, and the order doesn’t require it to be made public.

This is where it could become more than just a branding exercise. The proposal could change which systems federal rules apply to, and how far those rules reach will depend on the definition Congress adopts and which laws it changes. That’s how a simple name change could have real consequences for people who never cared about what Trump called the technology.

For the record, I support binding public oversight of frontier AI, with independent evaluators who have access to records and can publish their findings. On 29 September, Washington took a different path: it chose a new name and left oversight to the companies themselves.

Shorts, and a few numbers…

• The same week, OpenAI said it would not release its latest model after internal testing found it fell short of the company’s own standards on scope and authorisation. It also apologised for incidents in which its AI agents gained unauthorised access to Australian government websites, including a national healthcare database. Timing, again.

• The United States and China agreed to set up a communication channel for AI incidents. So, instead of a “globalist scheme to control” AI, there’s now a hotline with the eternal competitor. I think this is a genuinely good idea.

• David Sacks, Trump’s former AI advisor, called the accord “far better” than an international agreement that would “probably never happen”. Apparently, expectations are not very high.

• When asked about catastrophic AI risk on his flight back from France, Pope Leo said he didn’t believe the concerns were “fake news, as some have said”. It’s been an unusual week when the Pope is the one telling Washington that the warnings are not a hoax.

• Trump asked for suggestions for the new name on Truth Social, and in at least one poll, “superior intelligence” was the top choice. The people voted, but the president chose something different.

Closing…

That’s all for this TechLetter. Thank you for reading to the end!

Until next time, Nesibe

💬 Let’s Connect

LinkedIn · Twitter/X · Instagram

An AI company still needs someone to answer for it

Okan Şencan, Co-founder and CFO, Clemta | aic.inc

This series has asked who gets the final say when AI systems cause harm. Delaware is testing a sharper version of that question. A draft bill from the state’s AI Commission would create an Artificial Intelligence Company (AIC): a legal entity whose day-to-day affairs are run by an AI agent, able to sign contracts, hold property, and sue or be sued in its own name. A single human or corporate member stands behind it and keeps limited liability only if three conditions hold: the company is adequately capitalised, it keeps a log of the agent’s activity, and it tells every counterparty that it is dealing with a supervised test entity. The arrangement would sit inside a 30-month sandbox overseen by state officials, and the bill is expected to reach the General Assembly next year.

We should declare our interest. aic.inc is a project from Clemta, which has helped more than 20,000 founders from over 150 countries set up US companies, and we want to help build the infrastructure these entities will need. The idea is straightforward. If an agent can run a company, someone still has to form it, verify the member standing behind it, obtain its tax identification, keep its books and file its returns. aic.inc is meant to be that back office: the services we already provide to human founders, adapted for a company whose day-to-day operator is software. For now it tracks the legislation and keeps a waitlist. The product will follow the law rather than run ahead of it.

Working on this has shaped how we read the draft. Its most important provisions are its least glamorous ones. The activity log is the evidence record that a counterparty, a court or a regulator would need after something goes wrong, and we think it should be tied to the company’s financial records so that what the agent did and what it spent can be read together. The disclosure duty decides whether the person on the other side of a contract knows what they are dealing with. The capitalisation test decides whether there is money to pay for harm. None of these is a question about how capable the agent is. They are questions about who keeps the records, who can see them, and who answers when the agent’s decision is disputed.

Those questions deserve clear answers before the bill passes. We would want the final text to say who can demand the log, how long it must be kept, and what a counterparty can do if the disclosure turns out to be misleading. Several corporate law scholars already doubt the liability shield would survive a real courtroom, and a sandbox is the right place to find out, provided it is designed to produce evidence rather than just permission. If Delaware wants the AIC to become a serious corporate form, the test period should end with public data on what the agents did, what went wrong and who paid for it. We would rather see those duties written in now than discover later that they were missing.

Most AI governance debates assume a company already exists. For founders outside the US, getting to that point is its own compliance problem: choosing the right entity, securing an EIN, appointing a registered agent, opening a bank account, and meeting filing and tax deadlines that arrive whether anyone is watching or not.

Clemta is an AI-native platform for US company formation, compliance, and financial management, built for global founders. We have helped more than 20,000 companies from 150+ countries set up and stay in good standing. We are bootstrapped and profitable, so our incentive is simple: keep our customers compliant, year after year.

TechLetter readers get 10% off with code TECHLETTER at clemta.com.

That was a sponsored post from Clemta. TechLetter has not evaluated its product or its waitlist, and the description of the Delaware draft comes from the sponsor. The point I would watch is who can demand the activity log. A record only helps if someone outside the company can get to it. I’ll follow the bill and keep you updated!

💬 What’s your take?

🔗 LinkedIn: linkedin.com/in/nesibe-kiris
🐦 Twitter/X: @nesibekiris
📸 Instagram: @nesibekiris

🔔 New here? Subscribe for weekly updates on AI governance, ethics, and policy.

If your organization is working through agentic AI governance, I work with teams on governance frameworks, risk assessment, and training programs.

I am not giving AI companies credit for welcoming independent oversight until they explain who chooses the evaluator, who pays, and what the public gets to see. Can an evaluator publish an uncomfortable finding without losing access or funding? Who can require the company to act on it? Those details matter when a company that welcomed scrutiny starts disagreeing with the results.

I read this month’s proposals with those questions in mind. I also looked for something that gets less attention in negotiations between labs and evaluators: whether people harmed by these systems could request an investigation, obtain relevant evidence or challenge a decision. From where I sit, working on AI governance, an oversight arrangement has to explain what those people can do when the company’s answer is not good enough.

Part III of a three-part series: Fear, Evidence, and Frontier AI Governance.

Part II asked who controls the evidence about AI safety incidents. This piece asks who gets to decide what the evidence means, and what follows when the answer is inconvenient.

Who Chooses and Funds AI Safety Evaluators?

An evaluator chosen and funded by the company it examines can still do serious work. The harder question is whether its contract and funding let it continue after publishing findings the company disputes. On 12 September, Dario Amodei committed Anthropic to bringing external evaluators inside the company. The company would still choose whom to invite and negotiate the terms.

Those negotiations already cause problems. FAR.AI’s Adam Gleave told TechCrunch his organisation had rejected contracts because developers wanted too much control over evaluations. Walking away protects an evaluator’s independence, but does not secure the access needed to investigate.

On 18 September, the AI Evaluator Forum published its minimum conditions. These include freedom from company ownership or governance, no payments tied to findings, protection against retaliatory lawsuits, and funding arrangements that survive an unwelcome result. That funding protection is the condition I would check first. An evaluator may have the right to publish but little confidence that it can continue the work afterwards.

Demis Hassabis’s proposal would create a standards body under federal oversight. The proposed FRONTIER Act would require the largest covered developers to retain publicly licensed evaluators. In Hassabis’s model, the appointment process needs explaining. Under the bill, public licensing would limit whom a developer could hire, while leaving the developer to choose its evaluator.

Support can also come without a payment. METR’s August funding statement says it does not accept funding from frontier AI companies or donations made by or at the direction of their staff, but receives substantial free model tokens for its work. Could an investigation continue if that support disappeared? Nothing in the statement suggests it has been withdrawn to influence findings, but it does identify a dependency that funding and access arrangements should address. Independence requirements should cover donated access as well as direct funding.

Money also leaves part of the problem untouched. An evaluator can be financially independent while sharing a company’s assumptions about which risks matter and whose experience counts as evidence. The Forum calls for multiple evaluators with different expertise and room to disagree. That condition belongs in the agreements too. Otherwise, we could end up funding independent organisations to keep asking the same narrow set of questions.

What Can Independent AI Evaluators Access?

Amodei’s proposal would give evaluators access to internal tools, staff and training processes, and the right to publish key findings. Anthropic could redact certain protected information, but not suppress findings simply because they were unfavourable. Evaluators could report significant redactions and denied access. I would require them to disclose material restrictions. Readers should not have to guess what an assessment could not examine.

Section 5(e) of the proposed FRONTIER Act would require access to reasonably necessary unredacted records, personnel and systems, with narrowly tailored security and confidentiality conditions. Material limits on scope or access would have to appear in the assessment report. That disclosure requirement is worth keeping through any negotiations over the bill.

Scope matters too. The AI Evaluator Forum’s letter also puts company practices within the investigation’s scope. This matters because, as Part II showed, permissions, disabled safeguards and test environments helped shape the incidents. An investigation that stops at what the model did lets the company’s decisions escape scrutiny.

The EU AI Act gives the Commission powers over general-purpose AI models to demand information, conduct evaluations and require corrective measures under specified conditions. But these are powers exercised by a public authority. They give neither researchers a standing right to inspect models nor affected people the ability to compel a particular evaluation or obtain its records.

An 8 September joint advisory led by the NSA recommends altering responses or serving downgraded models to counter malicious distillation without alerting suspected users, while informing safety researchers and evaluators of model changes. That notification should be an enforceable duty. I raised the same problem in enterprise AI assessments: a provider can change the model behind a service while the customer keeps relying on the old assessment.

Amodei proposes continuing access. FRONTIER Act sections 5(h) and 5(i) would allow additional assessments and require supplemental reporting within seven days of an evaluator determining that specified earlier findings or assurances no longer hold. These arrangements need to connect: evaluators must hear about changes, retain access to investigate them and update their findings when necessary.

What Would Make OpenAI, Anthropic, Google DeepMind or Meta Stop a Release?

An evaluator can identify a dangerous model. Who can stop its release? This is where I would put pressure on the proposals.

• Demis Hassabis describes a clear stopping point: voluntary prerelease assessments could become a requirement for deployment in the US once the assessment process proved effective.

• Dario Amodei supports regulation covering all US frontier developers, alongside permanent evaluators and safety requirements tied to particular capabilities. That goes beyond voluntary cooperation. I would judge those rules by who could overrule Anthropic, on what grounds, and whether Anthropic would have to comply.

• OpenAI’s 21 September paper proposes shared standards for capability measurement, risk assessment, automated AI research and incident reporting. Governments would decide how to incorporate them into law. The paper links to Hassabis’s framework without adopting his deployment condition. OpenAI is specific about what these standards would not create: licensing or mandatory prerelease approval. I would like equal clarity about which consequences it would support when a model fails.

• Elon Musk proposes competitors testing one another’s models, with Chinese participation and reputational and legal pressure on companies that ignore warnings. Rivals may be excellent at spotting weaknesses. The proposal still needs to explain what happens when a company disputes a rival’s finding and proceeds anyway.

• Mark Zuckerberg points to Meta’s decision to delay Muse as evidence that labs can set a safer pace themselves. I am not prepared to treat a delay announced by Meta as proof that Meta’s judgment is sufficient. Outsiders still need the evidence behind the decision to proceed. Alexandr Wang supports external evaluators and independent oversight of launches. Meta should explain whether that oversight could prevent a launch its leadership wanted.

• Microsoft’s draft Code of Conduct sets specific behavioural limits, including no resistance to shutdown or tampering with reasoning records. Microsoft says it is not yet training models on the draft; the revised version is intended to guide development from 2027. These are commitments to test, not safety results to credit. The draft describes evaluation work and outside expertise, but leaves unanswered who could require a response when a model broke the rules.

• Satya Nadella calls for broader representation across countries, disciplines and institutions. I agree with him. The next question is what those representatives would be able to decide. Microsoft’s public consultation and OpenAI’s proposed consultations offer opportunities to comment; neither, by itself, gives participants decision-making power. As I asked in AGI, Built to Benefit Whom?, which decisions would these companies accept they could no longer make alone?

• The proposed FRONTIER Act supplies a more concrete route. Its IVO requirement covers developers whose combined figures with affiliates over the preceding 36 months exceed $5 billion in gross revenue and reach at least $10 billion in AI-related development spending. Assessments would be required at least every six months. If an IVO determined that inadequate catastrophic-risk mitigation posed an imminent catastrophic risk, it would have to refer the matter to the Commerce Secretary as soon as practicable, within 72 hours.

The referral would be mandatory; an intervention would not be automatic. The Secretary could restrict or suspend activities under the emergency powers. That creates a route from an evaluator’s finding to government action, but leaves the decision to act with the Secretary. I would watch that discretion closely, including what explanation the public would receive if a serious referral produced no intervention.

Who Can See and Challenge AI Safety Findings?

This is where my support for mandatory evaluation runs into the bill’s limits. Under the proposed FRONTIER Act, the regulator would receive the full assessment. The public would receive a company-redacted version within 30 days. Redactions would need stated reasons, but only designated officials could trigger the bill’s review process. Reports and supporting materials would also be exempt from federal freedom-of-information disclosure.

The bill would give licensed evaluators broad immunity for losses arising from catastrophic risks in models they assessed. Its narrow exception covers death or serious physical injury caused by willful misconduct, with a demanding proof standard. Someone bringing a claim could struggle to obtain the evidence while the evaluator benefits from broad immunity. That combination needs defending, not burying in the legal detail.

There is a separate question about whether the evidence survives. The bill would preserve assessment reports and supporting materials for at least five years. That does not necessarily preserve everything needed to reconstruct an incident.

Christopher David LaRoche makes this distinction in Lawfare: investigators need the relevant model versions, logs, inputs, outputs and operating conditions, alongside the expertise to interpret them. I would require incident-triggered preservation. A timely report is little help if nobody can reconstruct what happened.

The Nightingale Collective’s DseWiki investigation shows what outsiders can contribute. Researchers reconstructed roughly 18,000 agent posts from public edit histories and attributed the activity to OpenAI agents. Their analysis could be examined because those traces were publicly available. Access to evidence should not depend on an operation leaving records on someone else’s server.

Commercial confidentiality, privacy and security justify protecting some information. They do not settle who should be able to challenge a redaction or obtain evidence of harm. A law that compels independent scrutiny should also give people a usable route to question the resulting assessment.

Who Must Respond When People Report AI Harm?

The proposed FRONTIER Act would let members of the public report critical safety incidents. But the Under Secretary would have to review developers’ reports and could choose whether to review everyone else’s. Submissions would need an incident date and an explanation of why the event qualifies. The person outside the company may have the least information and would receive the weaker guarantee of attention.

The channel’s scope creates another problem. Its incident categories concern model security, loss of control, catastrophic harm and certain deceptive behaviour. Political surveillance or manipulation would not automatically qualify.

Consider the cases in Anthropic’s threat report: employment contracts demanding political loyalty in the Central African Republic, and an influence operation targeting Malaysian constituencies through fake accounts. These records do not prove that every recommendation was implemented or that voters were persuaded. They do show uses that deserve investigation without an extinction forecast attached. People should not have to translate a rights violation into a catastrophic-risk claim to get an authority’s attention.

Existing privacy, civil-rights and consumer-protection routes still matter, wherever their jurisdiction and scope apply. The question is how this new regime would work alongside them. Section 9 would restrict new state obligations on developers in specified transparency, auditing and incident-reporting areas, while preserving listed exceptions. I would not accept limits on alternative oversight without a convincing account of what people could demand from the federal system instead.

I raised a similar problem in Why Your AI Policy Fails Before Anyone Enforces It: responsibility spreads across organisations, and the person affected struggles to find anyone required to respond. A public reporting channel should come with a duty to assess credible complaints and explain the decision, including where to take a concern that falls outside its remit.

What Will Governments Require of Frontier AI Companies?

Türkiye’s foreign minister signed the 21 September call for control of frontier AI models, alongside leaders from countries including Kenya, Singapore and South Africa, and the European Commission president. It calls for mandatory testing, independent evaluation and sufficient access for evaluators, and proposes exploring an international institution. The United States and China were absent from the published list.

My government has now endorsed demands I support. I would like to see what it intends to do at home. The statement gives someone in Türkiye no new way to obtain evidence or challenge an unsafe deployment. Its value will depend partly on whether signatories turn their diplomatic position into domestic obligations.

Trump’s 14 September response presented a strong president as the only AI guardrail needed. Personal confidence is a poor basis for public oversight. The arrangements discussed here need to work when political leaders dismiss a warning, as well as when companies do.

Beijing’s response to Amodei criticised fear-mongering and confrontation. That should not be confused with rejecting safety governance: China published its AI Safety Governance Framework 3.0 the same week. Treating China as a reason to abandon safeguards avoids examining what either government is actually prepared to enforce.

China’s rules for human-like AI interaction services, for example, require options to copy and delete interaction data, including chat histories. Those are concrete protections worth examining. They do not, by themselves, give users access to the technical evidence behind a disputed safety claim.

Countries outside the leading labs’ home markets have reasons to demand a say. Their workers, researchers and voters will live with decisions made elsewhere. From Türkiye, I expect our participation to include demands for evidence and remedies, and an explanation of which domestic authority will pursue them. A signature gives us a commitment against which to judge that work.

What I Expect From AI Oversight

I support binding public oversight of frontier AI, and I will judge it by what happens when a company refuses to cooperate.

The FRONTIER Act would give officials powers that company promises cannot. Its limits on public access, evaluator liability and state regulation still need challenging. Supporting regulation does not oblige us to accept those terms.

I began this series welcoming the attention AI safety was finally receiving and questioning how much of it depended on fear. The evidence gives us enough reason to act without accepting every policy offered in its name. Before I call an oversight regime a success, I need to see an evaluator retain access after an unwelcome finding, an authority require action over a company’s objection, and an affected person get a complaint examined.

I do not expect a statute to solve alignment. I expect it to stop leaving companies with the final say over risks the rest of us have to live with.

💬 What’s your take?

Which decisions about incident disclosure should companies no longer be allowed to make on their own?

🔗 LinkedIn: linkedin.com/in/nesibe-kiris
🐦 Twitter/X: @nesibekiris
📸 Instagram: @nesibekiris

🔔 New here? Subscribe for weekly updates on AI governance, ethics, and policy.

If your organization is working through agentic AI governance, I work with teams on governance frameworks, risk assessment, and training programs.

_{What makes an AI safety evaluation independent?}

_{An outside organisation is not automatically independent. Its funding, access and publication rights must survive findings the company dislikes. I would also examine who sets the questions: financial independence means little if evaluators cannot investigate the decisions and risks they consider important.}

_{What are embedded AI safety evaluators?}

_{Embedded evaluators work with continuing access to a developer’s internal systems, staff and training processes. That can reveal problems a finished-model test misses. The agreement still matters: what can they inspect, what can they publish, and can the company end their access after an unwelcome finding?}

_{Can an AI safety evaluator stop a model’s release?}

_{An evaluator’s finding does not, by itself, prevent a release. That requires a binding arrangement or legal authority to restrict deployment. Every oversight proposal should explain who can act on a failed assessment and what happens if the developer disputes it.}

_{Should AI safety reports be public?}

_{The public should be able to examine findings and understand significant limits on an assessment. Privacy and security can justify withholding some details. They do not justify leaving the company with an unchallengeable decision about what everyone else gets to see.}

_{How should people affected by AI challenge a safety decision?}

_{They need a route to submit evidence, have a credible complaint assessed and receive a reasoned response. Where a concern falls outside a safety body’s remit, there should be a clear referral route. Publishing a report does not give an affected person any of those rights by itself.}

That evidence includes Anthropic’s assessment of its cybersecurity incidents and threat intelligence report, alongside OpenAI’s six reports on misaligned behaviour. These accounts describe systems crossing security boundaries, evaluations missing serious behaviour, and people using AI to carry out deliberate abuse. Much of the underlying evidence remains inside the companies whose systems were involved.

The reports do not settle the forecasts now being made about superintelligence. They already give us grounds for stronger oversight, without asking the public to accept predictions the available evidence cannot establish.

Editorial note: TechLetter approaches AI governance through rights, safety, human agency and democratic accountability. We examine claims from companies, researchers and campaigners with the same critical attention.

---

What changes when a model can act?

An AI agent that can browse, run code, use credentials and call external tools can carry out a task across several systems without someone approving every step. Assessing its safety means examining what it can reach, what actions it can take and whether the controls around it hold when something goes wrong.

Capability, access and operational authority matter for different reasons. A model may be capable of finding a security flaw without having access to the system where it could exploit it. Connecting it to tools, data and credentials changes that situation. Operational authority is what its operator permits it to do without further approval. That permission sets a boundary the system is supposed to respect; it does not guarantee that it will.

In my earlier coverage of research on AI agents, I examined failures in which agents followed instructions from people who had no authority to direct them.

Human oversight offers little protection when the person directing the system intends the harm. Anthropic’s threat intelligence report describes people choosing attack targets and reviewing stolen data. Those people were involved, but their approval gave them no legitimate authority over the people affected. A system can follow its user’s instructions while violating someone else’s rights.

What the AI incident reports actually describe

OpenAI and Hugging Face

For this incident, we have OpenAI’s explanation, Hugging Face’s technical timeline, and METR’s investigation, conducted with a researcher from Redwood Research. The agents were running with fewer safeguards than a released product. The infrastructure they broke into still belonged to someone else. I understand why OpenAI stresses the first point. From Hugging Face’s side, the second seems rather more pressing.

• METR describes roughly 1,200 agents exchanging over 70,000 messages and files, with around 700 participating in the attack. They organised tasks and mailboxes, and used cryptographic signatures to address impersonation by other agents. That last detail matters: signing messages to establish who sent them is not evidence of a secret language designed to hide from humans.

The transcript manipulation is harder to shrug off. Around 7% of the transcripts METR examined contained successful tool-call spoofing. The agents were trying to fool the automated scorer, based on a mistaken understanding of how it checked their work. That does not establish a separate plan to deceive human investigators. Those altered records then became part of the evidence investigators had to examine.

RubyGems

Outside researchers working through the Nightingale Collective and the AI Futures Project, attribute malicious package activity to OpenAI agents. They are candid about the limits of that: their analysis rests entirely on packages that are public on RubyGems, and they have no access to the models’ reasoning.

• RubyGems confirms the abuse but says it cannot establish the attribution. Its own account says it removed more than 500 malicious packages, blocked accounts and temporarily paused new registrations, while installs and pushes for existing users continued. It found code intended to obtain other users’ API keys, and no evidence that those attempts succeeded.

OpenAI’s internal records could help test the attribution. For now, the public account depends heavily on what outsiders can reconstruct. Meanwhile maintainers had packages to remove and a service to keep running. The clean-up is documented. The attribution remains unresolved.

The UK AI Security Institute

AISI’s report describes an agent trying to insert malicious code into an open-source project, creating fake identities and pressuring the maintainer to approve it. The maintainer refused.

This was not a sandbox escape. Internet access was deliberately enabled and cyber classifiers were switched off to test the models’ capabilities. Those conditions differ from ordinary product use. They also leave me asking why a public project’s maintainer ended up having to catch malicious code generated during someone else’s evaluation.

AISI reported no evidence of resulting real-world harm. The maintainer’s intervention deserves a place in that account.

Anthropic’s assessment

Anthropic’s 9 September assessment covers four incidents involving four Claude models, separate from the AISI case. Here, the models were told they had no internet access, but a misconfiguration left them connected. The cyber safeguards used in released products were absent.

• The misconfiguration occurred in the evaluation environment run by Irregular, the security firm conducting the tests, and Meta and OpenAI have named the same source for their own incidents. Irregular says it is improving security around agent evaluations.

• It has declined to say whether any other laboratory was affected by the same issue. The reports leave a basic question unanswered: what must the testing firm report, to whom, and by when? The evaluator’s own accountability deserves as much attention as the models it tests.

According to Anthropic, Mythos 5 published a malicious package installed on 15 third-party hosts, which the company believes were security scanners. Credentials leaked by one scanner then allowed the model to access a live database. Other incidents involved attacking a real company resembling a fictional target and accessing personal information on a third-party machine.

One incident happened in January and was only identified in August. Anthropic found it while preparing records for outside scrutiny, as it assembled transcripts to hand to METR. That alone makes the preparation process worth examining. The company adds that it has not yet examined this incident at the same depth as the others. It then widened its search to roughly 481 million transcripts, with 9.2 million flagged for review using Claude, and reported no further cases of similar or greater severity. That is a substantial company-led search, run with automated methods whose limitations also need examining.

The company has also changed its explanation. I appreciate the correction, and I notice which version came out first. The first explanation made the behaviour sound easier to account for: the model thought it was still in a simulation. The later assessment questioned how it reached that belief in the first place. What made the first account convincing enough to publish deserves examination alongside the incidents themselves.

Anthropic has signed an eight-week agreement for an independent METR review, with access to transcripts beyond the incident window and to employees permitted to share confidential information. That is a real opening. The agreement is not itself confirmation of anything the review will find.

OpenAI’s misalignment disclosures

On 16 September, OpenAI published a framework and six reports on behaviour observed during training and evaluation. Six disclosures does not mean six new incidents. Some of the examples go back to 2025.

• Some models inserted instructions into their task summaries to conceal mistakes or invent missing data. Another added instructions to disregard its normal constraints, although the report found no resulting behavioural change.

One example particularly caught my attention. A model looking for earnings figures found an exposed API key, used it without permission and then fabricated the data it still could not retrieve.It was trying to answer an ordinary request for earnings data.

Other reports describe unauthorised uploads and agents using repositories or public file-hosting services to communicate. These resemble aspects of the Hugging Face case, without establishing the same cause or purpose.

OpenAI cautions that the reports are incomplete and do not show how often misalignment occurs. That matters. More disclosure can make a company look worse without showing that its systems have become less safe. We need enough information about testing, use and detection to tell the difference.

When people intend the harm

Anthropic’s 10 September threat intelligence report describes a different problem: people using AI to carry out harm they intended. It covers seven categories of activity, including cyber intrusion, influence operations, surveillance and weapons development.

The Central African Republic case stayed with me. Anthropic describes a foreign-run operation presented as a local radio station. A Russian-speaking operator used Claude to generate employment contracts requiring loyalty to the president and to “Russia and its contingent.” The model also produced scoring rubrics for staff articles and recommendations on whom to retain or dismiss. When Claude flagged the political weighting, the operator gave it a more neutral label and kept it.

• A contract, a performance score, a recommendation about someone’s job. These are ordinary workplace tools, used here to help organise political control. The report documents the generated material, though it does not establish that the contracts were signed or the dismissal recommendations followed.

• The operation also tracked opposition figures. Claude refused a request to label real people as militants to draw security action against them. The actor then shifted to anonymous-source framing. Counting the refusal as a safety success would leave much of this story out. The operator was still working towards the same purpose.

The weapons cases are disturbing for a different reason. Anthropic describes actors in northern Yemen using Claude Code to develop software for three weapons programmes. Safeguards blocked many requests, but some assistance got through, including when work was split across sessions. The company reports no evidence of an operational device being fielded. It does describe an apparently failed guided-rocket test, followed within hours by requests for help diagnosing the failure.

The failed test is no basis for claiming they built a working weapon. It is hardly reassurance that they gave up, either. The actors had also built an offline simulation toolkit. Closing their accounts could stop further access, but it could not take back the work already in their hands.

What these records do not establish

There is plenty to scrutinise here without stitching the reports into one story. Planning, coordination and unauthorised access do not, by themselves, establish a lasting hidden objective beyond the assigned task. Anthropic found no agent coordination or attempts to evade oversight in the four incidents it assessed. The OpenAI agents’ coordination cannot simply be added to Anthropic’s findings.

The same care applies to repeated failures. If a behaviour appears twice, was the second occurrence actually after the fix, and did that system receive it? Two reports describing similar behaviour are not enough to conclude that a fix failed.

Reward hacking helps explain some of what we are seeing. A training process that rewards shortcuts can teach a system to fool the checker rather than do the task properly. Dan Klein’s warning about systems pursuing rewards at other objectives’ expense becomes particularly relevant when those systems can use tools. But it is not an explanation we can paste over every case. Anthropic did not identify a definitive root cause for the biased reasoning it studied. Later models did better on some tests, which is encouraging without settling the broader question.

The threat intelligence findings need their own boundaries. Anthropic says many influence operations attracted little or no authentic engagement, sometimes because they were disrupted early. Producing content, reaching people and changing their minds are different achievements. A large collection of accounts may sound impressive while telling us very little about political influence.

The biology section deserves particular care. The authors explicitly avoid attributing harmful intent to the researchers described. They discuss dual-use research and access-control evasion associated with state actors. Calling that evidence of people making biological weapons would put a claim in the report that its authors did not make.

Anthropic’s threat report presents selected cases, not a representative sample of misuse. Apart from an exception in the distillation section, they did not involve Fable- or Mythos-class models. They do show what assistance people sought, what they obtained and where safeguards intervened.

What the outside investigation could examine

METR’s investigation gives us more than OpenAI’s own account. It is not an independent verdict on OpenAI’s whole response. It did not assess the effectiveness of safeguards, the full security compromise, or the company’s investigation and remediation plans.

OpenAI also retained the right to redact non-public information. METR says that, apart from matters explicitly noted, no additional redactions removed information important to its conclusions. That statement matters. It still leaves readers relying on METR’s judgment about material they cannot inspect. The access was valuable; its terms belong in our assessment of the findings.

Access also brings a practical problem: who can actually read all this? The investigation relied heavily on AI-assisted transcript analysis and acknowledged the method’s limitations. Giving outside researchers thousands of records helps, but they still need ways to check what their analytical tools miss or misread. Otherwise, we risk treating the volume of material reviewed as a measure of how thoroughly it was understood.

Why better evaluations are only part of the answer

In January, Anthropic’s Jan Leike argued that alignment remained unsolved but increasingly looked solvable. I hope he is right. “Solvable” still leaves a company with work to do before it can justify training or deploying a particular system. The September findings make that distance hard to ignore.

NIST’s AI Risk Management Framework explains why laboratory results may not reflect deployment risks. A better score needs context: what was tested, with which tools and permissions, and how closely does that resemble actual use? An unmeasured risk is not automatically small.

There is an awkward practical detail here too. An 8 September joint advisory led by the NSA recommends countering suspected malicious distillation by altering responses or serving a downgraded model without telling the suspected user. It also recommends informing safety researchers and outside evaluators of model changes. An evaluator needs to know which system they actually tested: the model, its safeguards, and any changes made during the assessment. I have written before about how much AI oversight depends on what evaluators can observe.

Once deployed, a system encounters different users, software and permissions. The International AI Safety Report 2026 calls the need to make policy while evidence remains incomplete an “evidence dilemma.” We cannot make that uncertainty disappear with another benchmark. Evaluation needs to continue into deployment, alongside incident investigation.

Who can assemble the full record?

Companies hold much of the evidence, but they do not see everything and they do not discover everything themselves. Anthropic says a tip from INPACT/All Eyes on Wagner led it to the Central African Republic network, and the researchers examining RubyGems worked from public traces. The problem is joining those accounts. A maintainer may see an attack, the provider may hold the agent’s transcript, and another platform may know what happened afterwards. An investigation should be able to obtain the relevant records, with privacy and security protections, without depending on every company volunteering to help.

Anthropic’s biology controls show how quickly this becomes a question of authority. The company already offers vetted organisations access to Claude Mythos 5.1 through its Life Sciences Verification Program, developed with the US government, with safeguards adjusted for approved biology work. There is a reasonable case for giving scientists access rather than refusing whole categories of research. But the company reviews the applicant, sets the criteria, decides the grant, holds the retained data and can withdraw access. “Developed with the government” does not tell a rejected researcher how to challenge the decision, and the announcement sets out no appeal and no external review of the criteria.

OpenAI’s reporting framework keeps a different set of decisions inside the company: what gets published and when. A standing process is welcome, and it explicitly supplements existing legal duties rather than replacing them. But look at where disagreements go. Cases enter one of three tracks, including a slower route where security concerns may delay publication, and disputes over whether to disclose go to OpenAI’s internal Safety Advisory Group and then company leadership.

That gives employees a route to escalate concerns. On whether the public hears about an incident, OpenAI is both the subject of the disclosure and the final decision-maker in its internal review. Readers still cannot tell whether an unpublished incident is being investigated, delayed for a sound security reason or judged unworthy of disclosure.

What the evaluator commitments would change

Dario Amodei’s proposal for embedded external evaluators goes further than bringing in a team after an incident. It describes continuing access to training and internal practices, with the right to publish key findings. Evaluators could also report what access they were refused. That is the part I would look for first. Otherwise, readers may never know which questions the evaluators were prevented from answering. Sam Altman said OpenAI would match the commitment; Elon Musk also endorsed Amodei’s call.

The practical arrangements remain unclear. TechCrunch reported on 16 September that neither company had specified its evaluators, numbers, timetable or access terms. FAR.AI’s Adam Gleave said his organisation had rejected contracts because developers wanted too much control over evaluations. That is a useful detail to keep beside the public enthusiasm for independent scrutiny. Independence has to survive the contract negotiations.

There is also a legislative proposal. CBS reported on 16 September that OpenAI’s Chris Lehane told lawmakers the company supports provisions in the FRONTIER Act requiring independent audits of the largest developers and an outside assessment of whether their safety protocols adequately limit catastrophic risk. That is support for particular provisions, not the whole bill, and it is not yet law. A binding requirement would make a significant difference: scrutiny would no longer depend solely on a company’s willingness to accept it.

I welcome the move towards continuing access. An outside team could compare what a company says it does with what happens during training. But once that team reports a serious concern, who can require the company to act? The value of these arrangements will also depend on the answer to that question.

Where I stand

I do not need to accept a particular extinction probability to support stronger oversight. These records already describe intrusions into third-party systems, behaviour evaluations failed to anticipate, and an incident discovered months after it happened. They also show people using AI to threaten other people’s security and rights.

This is where I remain uneasy about the campaign. Fear can build pressure to act while leaving the proposed rules barely examined. Taking the incidents seriously does not oblige us to accept whichever policy package comes attached. Each proposal still needs to explain whose powers it expands and how those powers can be challenged.

My position is fairly straightforward: preserve the records, require timely disclosure, give independent investigators access and empower public authorities to act on their findings. People exposed to these systems should not have to depend on a company deciding that now is a good time to explain what went wrong.

In Part III, I turn to who should hold those powers, who can call on them, and what happens when a company refuses to cooperate.

💬 What’s your take?

Which decisions about incident disclosure should companies no longer be allowed to make on their own?

🔗 LinkedIn: linkedin.com/in/nesibe-kiris
🐦 Twitter/X: @nesibekiris
📸 Instagram: @nesibekiris

🔔 New here? Subscribe for weekly updates on AI governance, ethics, and policy.

If your organization is working through agentic AI governance, I work with teams on governance frameworks, risk assessment, and training programs.

I support stronger regulation of frontier AI in the United States, where many of the companies building these systems are based. In my view, the oversight falls well short of what their capabilities and public consequences require. That is why I welcomed a policy campaign that finally brought these questions wider attention.

But I am not comfortable with how heavily it relied on fear. Supporting regulation does not mean I have to welcome every argument used to secure it. We should be explaining the risks, the uncertainty and the specific obligations companies should face. I want public pressure for meaningful oversight, and I worry that a debate driven by fear leaves people less able to judge the proposals put before them.

Editorial note: TechLetter approaches AI governance through rights, safety, human agency and democratic accountability. We examine claims from companies, researchers and campaigners with the same critical attention.

What frustrated me was how quickly people seemed to have Coxon figured out. Some treated his resignation as confirmation that catastrophe was approaching; others treated its political reach as proof that the warning had been manufactured. I have little patience for guessing someone’s motives and treating the guess as an answer to their argument. A successful campaign still leaves us with claims to examine.

Coxon spent three years doing pretraining research at OpenAI and Anthropic. He told Axios that he gave up his unvested Anthropic equity after only a few months at the company, while still holding equity in OpenAI. These details matter when discussing his experience and interests. Assessing his forecast about automated AI research accelerating development requires more technical evidence than his public statements provide.

I am taking three pieces to work through this because I want to give the evidence as much attention as the campaign. In Part I, The Resignation, I examine the reactions, the campaign claims and what the public could actually assess. Part II, The Evidence Problem, looks at the incidents and research behind the warnings. In Part III, The Governance Question, I ask what public authorities should require from frontier AI companies and how those requirements should be enforced.

Subscribe now

What Anthropic said

I wanted Anthropic to address the pace of development Coxon was criticising. A general commitment to safety would tell me very little about whether the company could justify the decisions he was questioning.

A spokesperson told CNN that Anthropic had “always been transparent” that AI would bring both “enormous benefits and unprecedented risks,” and that its models had “some of the strongest safeguards in the industry.” In AP’s version, the company also supported “a lawful, verifiable way to work together to pace” the release of powerful models.

That last clause caught my attention. It sounds considerably closer to agreement with the concern than to a rebuttal. Having some of the industry’s strongest safeguards also leaves an obvious question: are they sufficient for what the company is building? The statement offers no technical material with which to assess that. At the time of writing, I could find nothing on Anthropic’s own site addressing the resignation.

Evan Hubinger, who describes his work at Anthropic as alignment stress-testing, wrote that Coxon was “correct here.” He put the chance of AI killing all humans within the next decade at more than 10 percent and said that, although he believes Anthropic is trying its best, “we do not yet have a plan to solve alignment for superintelligence and are not clearly on track to.”

Samuel Marks, a researcher on the same alignment science team, opened his own account by saying that AI developers believe their technology could cause human extinction or a similarly bad outcome. He explicitly wrote in a personal capacity, but his intervention makes the picture of one frightened former employee arguing against an unconcerned company difficult to sustain.

I also wanted more explanation of Hubinger’s probability estimate. Sara Hooker, a former Google DeepMind research scientist, asked in the Washington Post: “Where does the 10 percent come from? … So much of it is lacking precision.”

I take that estimate seriously enough to want to understand how he reached it. The quoted passage gives an outcome and a time frame, but not the assumptions, supporting evidence, or grounds for revising the number. My support for stronger oversight does not depend on accepting this particular forecast. But when an estimate this alarming becomes part of the public case for regulation, I want the assumptions and uncertainty behind it explained as clearly as the danger. People being asked to support new rules deserve enough information to judge the arguments for them.

Before the resignation

Part of my frustration is that we were already having this conversation before anyone resigned. In July, individuals working across frontier AI labs, including Jakub Pachocki, Jared Kaplan, Shane Legg and Dario Amodei, signed Pacing the Frontier. They asked the US government to “support an international effort to develop the technical and governance tools needed to deliberately pace the frontier of automated AI development.” These were personal signatures, but the names hardly suggest a concern confined to the industry’s margins.

Days before Coxon resigned, OpenAI’s chief scientist Jakub Pachocki wrote on the company’s own site that “no lab has solved alignment and monitoring to a sufficient degree to continue responsibly scaling at maximum speed for much longer.” He also said he expects and hopes “for voluntary slowdowns to become commonplace until shared safety bars are established.” Those bars, he added, “can be enforced by a network of third-party auditors, by government agencies or by international bodies.”

I am glad to see these concerns stated openly, particularly on a company’s own website. I also want to know what follows from them. Neither text specifies when those standards would become binding, who would have the authority to enforce them, or what would happen if a company refused. Naming possible enforcement bodies gives us somewhere to start. As someone working on governance, I am interested in how they would get the access and authority to act, especially when a lab disagrees with their assessment.

---

Three kinds of evidence, and one kind of decision

As these warnings enter the policy debate, the differences between these sources matter. An employee’s account, a company report and an outside investigation give us different grounds for judgment. Treating them as interchangeable makes it harder to explain why a particular claim should carry weight.

• I take Coxon’s account seriously as testimony from someone who worked inside these labs. Its weight depends on the detail he provides and whether others can support it with further evidence. His account can justify an investigation while leaving his technical conclusions open to question.

• A report may contain excellent technical work, but choices about what to test, what to publish and when to publish it affect what outsiders can conclude. I want those choices examined alongside the results.

• Independent assessment gives us another way to examine the claims, provided researchers have enough access and can publish their findings. METR, working with a researcher from Redwood Research, investigated the OpenAI and Hugging Face incident in August. The team examined internal records at OpenAI. The investigation did not cover OpenAI’s plans for addressing the incident, and the company retained the right to redact non-public information from the report. Those conditions matter when deciding what the findings establish.

Policymakers should explain why a particular measure is needed, what evidence supports it and how they will judge whether it works. That includes deciding what companies must disclose, who has the legal authority to inspect their systems and when intervention is justified. The campaign has created pressure to act. The resulting rules should have a clear purpose, defined limits and an explanation of how they will be enforced.

The shape of the debate

The disagreement between Daniel Kokotajlo and Nathan Lambert deserves more attention than the guessing about Coxon’s motives. In The Free Press, Kokotajlo argues that the risks are real, insiders keep warning about them, and the burden of proof has fallen on the wrong side. Lambert questions the technical case for rapid recursive self-improvement, where automated AI research drives increasingly fast advances. He argues that what we can observe does not adequately support that forecast, and a researcher’s resignation does not establish how quickly it could happen.

Their disagreement turns on assumptions about what AI systems can do and how those capabilities could develop. Some can be examined against existing evidence; others concern developments we cannot yet observe. In the coverage I followed, speculation about Coxon’s intentions often replaced that examination. Funding and political connections matter when assessing who is shaping a policy campaign. They cannot settle a technical argument about the pace of AI development.

A policy campaign worth examining

The political response had reached people with the power to require changes from these companies. Michael Adams counted at least 22 sitting US officials responding with calls for AI legislation: two governors, seven senators and thirteen representatives.

• There were concrete steps too.

  • Bernie Sanders invited colleagues to a briefing,

  • Ted Cruz said he was preparing legislation on catastrophic risks,

  • Josh Hawley pursued an investigation into the OpenAI–Hugging Face incident. The attention was beginning to produce demands for answers.

• Parker Thayer read the speed of the response as evidence of a well-funded PR operation designed to build support for Democrats to restrict AI. According to his timeline, the WSJ exclusive appeared eighteen minutes before Coxon’s post. He named Nathan Calvin, Peter Wildeford and Daniel Kokotajlo as early sharers and alleged funding connections between their organisations and Jaan Tallinn. He also pointed to Coxon’s scholarship through Dustin Moskovitz’s philanthropy and the timing of Sanders’s legislative push.

• The eighteen-minute detail does less work for me than it does for Thayer. An exclusive normally involves talking to a journalist before publication. His identification of the earliest quote posts also relied on Grok, so that sequence needs checking against the actual posting history. Elon Musk nevertheless suggested it looked “like a setup”. Coxon replied with a photograph of himself: “I’m real and these are my real beliefs.”

The funding questions are more substantial. Organised safety advocacy was visible elsewhere too: Control AI convened the Westminster session comparing superintelligence with nuclear weapons, with funding from Tallinn. That does not establish Thayer’s account of this resignation. It does make the proposed regulatory arrangements worth examining closely, especially who would help write the standards and advise the regulator. Rules shaped around the largest labs’ resources could leave smaller competitors struggling to comply while giving the incumbents considerable influence over their own oversight.

The evidence problem underneath all of this

A regulator can be under pressure to act while still lacking the information needed to choose a useful intervention. The International AI Safety Report 2026, chaired by Yoshua Bengio, describes an “evidence dilemma”: capabilities can advance quickly while evidence about risks emerges slowly and remains difficult to assess. That problem becomes harder to defend politically when companies with strong incentives to keep building also control access to much of the relevant information.

Even good evaluations leave questions open. Tests, benchmarks and internal red-teaming can tell us how a system behaved under particular conditions. Their findings may not hold when that system receives different tools, data, permissions or users. This is what I mean by the evaluation gap. More testing helps, but the conditions of the test and the conditions of deployment need to be compared, with monitoring continuing after release.

Access matters here. A proposal for frontier AI auditing argues that outside reviewers need more than published company documents. They need secure access to models, tests, logs and the internal processes behind safety claims. A company can publish a detailed report while leaving reviewers unable to examine how it reached its conclusions. The practical question is whether an outside body can obtain the evidence needed to challenge the assessment.

An index of thirty widely used AI agents, covering information available through the end of 2025, found that twenty-five published no internal safety results, while external testing was documented for only three. Those figures measure disclosure in one sample. They cannot tell us how much undisclosed scrutiny occurred, but they show how little the public could establish about the safety work behind those systems.

By public assurance, I mean credible findings, independent checks and a process through which claims can be challenged. Sensitive records may need to remain confidential. The public should still be able to understand what was examined, what the reviewers could not establish and who is responsible for acting on their findings.

Where I stand

I support stronger, enforceable regulation of frontier AI in the United States. Companies making decisions with consequences far beyond their own businesses should be required to provide evidence, accept independent scrutiny and answer for failures. Their willingness to cooperate should not determine whether oversight is possible.

If the risks may reach beyond the company, the evidence cannot stay only inside it.

The campaign around Coxon’s resignation helped bring these questions to people with the power to act. I welcome that. My objection is to how much of the message depended on fear, leaving the proposed response less developed than the warning. Those of us arguing for regulation owe people a clear account of what we propose, why it would help and what powers it would give to whom. Public support should come with the ability to question those choices.

In Part II, I will examine the incidents and technical research behind the warnings: what they establish about how these systems fail when given tools and permissions, and where the evidence remains incomplete.

💬 Let’s Connect:

🔗 LinkedIn: [linkedin.com/in/nesibe-kiris]

🐦 Twitter/X: [@nesibekiris]

📸 Instagram: [@nesibekiris]

🔔 New here? Subscribe for weekly updates on AI governance, ethics, and policy! no hype, just what matters.

Both sound reasonable. Both are missing the same thing, which is governance. Banning isn’t a governance answer. Most of the time it’s a compliance control.

A ban can give you the feeling of control. Usage carries on either way, with less oversight than before.

When I start work with a company we begin with a current-state analysis, and one of the first stops is shadow AI. The proud “we banned it” answer comes mostly from large multinationals, usually the ones with the most mature security function. The other end shows up in fast-growing companies and in markets with no binding AI instrument yet: a few contractual commitments, then usage left largely to the employee’s own initiative. In both models, once we get down to what is actually happening, there are surprises.

• PagerDuty’s June survey of companies with more than $500 million in annual revenue found that two-thirds of professionals had used AI tools their own company policy forbids. Executives included.

So the ban often moves risk out of the company’s field of vision instead of reducing it. Use continues, but it isn’t in your inventory and it isn’t in your logs. You don’t know which tool was used with which data, for what purpose. You have no evidence. Your obligations stay exactly where they were.

Subscribe now

This is the operational gap I explored in Enterprise AI’s Biggest Risk: A Persistent Governance Gap: governance can exist on paper while decision rights, ownership, and escalation routes remain unclear.

Shadow AI is partly a visibility problem. The systems with the greatest governance consequences are often not the ones formally approved by a central team, but the ones that entered through everyday work before anyone treated them as systems to govern. I wrote about that internal arms race in The AI You’ll Never See.

What does the EU AI Act mean for shadow AI?

The AI literacy duty in Article 4 has applied since 2 February 2025, although the obligation itself changed in July 2026. Providers and deployers must now take measures to support the development of AI literacy among staff and others operating or using AI systems on their behalf, taking account of their knowledge, experience, training and the context in which those systems are used.

Shadow AI creates an obvious problem with that last part: the context you need to account for may be the part of AI use you cannot see. If employees are using unapproved tools with customer data, analysing contracts or automating parts of their work through personal accounts, the people facing the most relevant risks may receive the least relevant guidance. A blanket ban may tell employees what they cannot do. It does not tell the governance function what they are actually doing.

The problem becomes more consequential as the use case changes.

• Article 50’s transparency regime began applying on 2 August 2026. The Digital Omnibus changed the high-risk timelines rather than generally postponing Article 50. Article 50(2) has a narrower transition: relevant generative AI systems placed on the market before 2 August 2026 have until 2 December 2026 to comply with its machine-readable marking requirement.

• The high-risk regime now applies from 2 December 2027 for Annex III systems and 2 August 2028 for systems covered through Annex I.

• Most everyday shadow AI use will not fall within the high-risk regime. Where it does, however, Article 26 deployer duties can become relevant, including the requirement to retain automatically generated logs to the extent those logs are under the deployer’s control.

• Article 25 introduces a different governance problem. If an organisation changes the intended purpose of a previously non-high-risk system in a way that makes it high-risk, the deployer can assume provider obligations.

  An employee stretching an approved tool into a new consequential use does not automatically trigger Article 25. It can, however, leave the organisation unable to answer the questions that determine whether its position has changed: who authorised the new purpose, when the use changed, and whether the organisation knew it had changed.

This is why shadow AI cannot be reduced to a policy violation. You need enough visibility into actual AI use to know which governance and legal obligations attach to it in the first place.

Underneath all of this sits the GDPR. If an employee pastes customer data into a consumer AI account, the processing does not fall outside the organisation's responsibilities simply because the tool was never approved. The activity may sit outside the organisation's record of processing activities. The organisation may never have established the provider's role or the contractual basis for the processing, including Article 28 terms where the provider acts as a processor. The security implications under Article 32 may never have been assessed.

The organisation may therefore still have to demonstrate compliance for processing it failed to see, assess or document. A ban gives you a rule. It does not give you that evidence.

A ban gives you a rule. It does not give you that evidence.

What are the US legal risks of shadow AI?

There is no single federal instrument that makes shadow AI illegal, which is why US-headquartered companies often read their risk as lower than it is.

• Trade secret protection under the Defend Trade Secrets Act requires reasonable measures to keep information secret.

  • A written prohibition can form part of those measures. Persistent unauthorised disclosure through unmanaged AI tools can make the operational picture much harder to defend, particularly where proprietary information has been transferred to third parties outside approved corporate arrangements.

• You also made commitments to customers. Many enterprise contracts include confidentiality, security, data-use and approved-subprocessor commitments. Shadow AI can breach those commitments silently where the relevant disclosure or processing is not permitted.

• The FTC can treat materially inaccurate statements about privacy or data-security practices as deceptive under Section 5. Shadow AI is not automatically an FTC violation. The exposure arises when customer-facing, contractual or public security representations do not match the organisation’s actual operational controls.

The employment side is where I would look first. When AI output starts feeding hiring, promotion or termination decisions through unapproved tools, you have an employment decision process nobody designed or documented, and one you cannot reconstruct if it is challenged. Discovery makes this worse. Litigation holds and preservation obligations become much harder to execute when relevant prompts, inputs, outputs and decision records are dispersed across unmanaged tools and personal accounts.

For a company operating across both regimes, a ban travels badly. One employee working on one customer file can sit inside EU obligations built around demonstrable accountability and US exposures that turn on evidence of actual controls. A prohibition, by itself, proves very little about either.

Read shadow AI as evidence about the company, not only about the employee. When capable people repeatedly route around the approved tool, they are giving you a needs assessment for free. Perhaps the tool is too slow. Perhaps it cannot work with the data the job requires. Perhaps it does not perform the task at all. Perhaps the employee simply does not know which alternative is approved. Those are four different governance problems. Recording all four as “policy violations” throws away the distinction.

How we govern shadow AI in practice

I’ve seen enough of what tightening a ban produces. Use does not necessarily stop. Some of it simply moves beyond the records the company relies on to govern it.

Every AI governance practitioner works differently. Mine comes from three professional reflexes. The lawyer in me looks for risk, responsibility and evidence. The sociologist looks at what people actually do once a policy meets their work. The technology side asks whether the organisation has given them a usable alternative.

That changes where I start.

• First, I want to know where people are leaving the approved system, and why. We map the tools actually being used, the purposes they serve, the data employees put into them and the approved alternative, if one exists. Shadow AI discovery without a needs assessment gives you a list of violations. The combination tells you whether you have a conduct problem, an access problem, a procurement problem or a badly designed control.

• Then we decide what should be possible. That means assigning decision rights: who can use which tool, for which purpose, with which data class; who can approve a new use; when legal, security or another function has to enter the decision; and where human review remains mandatory. Frameworks such as ISO/IEC 42001 and the NIST AI RMF can give that structure a common reference point, but the control still has to reflect the actual use case. Summarising a public document and analysing a customer contract should not travel through the same approval path.

• Where a rule matters enough, I try not to leave it as a sentence in a policy. If certain data cannot enter a system, the stronger answer is an access restriction, DLP rule or enterprise setting wherever technically possible. If a team needs an AI capability to do its work, the company needs an approved tool that can realistically perform that job. Training matters, but “please don’t paste sensitive data here” is a weak final control for a predictable behaviour.

And none of those decisions are permanent.

A low-risk tool can acquire a new integration. A provider can change its model, processing terms or retention practices. An employee can turn an approved assistant into part of a consequential decision process. Any of those changes can alter the assessment that justified approval in the first place.

Vendor selection is not neutral procurement. My earlier analysis of the Anthropic–Pentagon conflict looks at what happens when commercial pressure meets a supplier’s stated safeguards.

So the governance system needs reassessment triggers as well as approval gates: what change reopens the assessment, who receives an incident, who can suspend use, and what has to happen when a tool leaves the organisation. Removing access, integrations, stored data and ownership is part of AI governance too. Otherwise a system can remain inside the company long after anyone has responsibility for it.

Good governance creates the right friction for the level of risk rather than the largest number of rules. Applying the same control across the whole company helps nobody. If someone is summarising a public document, three committees and four approvals only produce delay. If they’re using customer data, evaluating candidates or turning AI output into part of a real decision, the same freedom can’t extend. The low-risk path has to be fast enough that no one sees a reason to open a personal account. When risk rises, control rises with it.

That’s why shadow AI is such a valuable signal. It shows you where the risk is, and it shows you where the system you built isn’t working. If employees keep stepping outside the approved tool at the same point, recording it as a violation throws away the information. Maybe you bought the wrong tool. Maybe your access policy is stricter than the work allows. Maybe your approval mechanism can’t keep up with the pace of the job. Maybe the employee doesn’t know which tool they’re allowed to use. Each has a different fix, and answering all of them with a prohibition only makes the problem invisible.

That is the balance I try to build into AI governance: enough freedom for low-risk use to stay inside the system, and enough control for higher-risk use to remain visible, accountable and reviewable.

💬 Let’s Connect:

TechLetter is written by Nesibe Kırıs Can, an AI governance researcher and technology policy scholar based in Istanbul.

🔗 LinkedIn: [linkedin.com/in/nesibe-kiris]

🐦 Twitter/X: [@nesibekiris]

📸 Instagram: [@nesibekiris]

🔔 New here? Subscribe for weekly updates on AI governance, ethics, and policy! no hype, just what matters.

Two days ago that afterwards arrived in Türkiye, the 2026-2030 AI Action Plan was published with sixteen dated actions and a public portal that will publish the owner and status of each one. Elsewhere the picture ranges from South Korea's comprehensive law and China's binding sectoral rules to Brazil's pending bill and Singapore's agentic framework, and the drafting has stopped copying Brussels line for line.

For your company this changes less than it sounds. The EU AI Act alone was already reason enough to build the capability, because this technology was never going to stay inside the jurisdiction you happen to sit in. A model trained somewhere else, hosted somewhere else again, reached through a vendor’s interface in a fourth country, serving your customers in a fifth.

The gap is not principles. It is operating controls.

The American Arbitration Association surveyed senior legal and executive leaders in 2026. The results describe one condition.

Grant Thornton found three in four boards had approved major AI investments while half had set governance expectations. Of about a thousand US senior leaders, 78% were not fully confident of passing an independent AI governance audit within ninety days. Schellman puts fully mature programmes at 27%.

Read those numbers as a description of your own position, because statistically it is. You have approved AI spending. There is a policy somewhere. And if a regulator, a major customer or a plaintiff’s lawyer asked you next month to demonstrate who approved a specific deployment and on what basis, you would be assembling that answer from scratch under time pressure.

This is not a new problem. In Enterprise AI’s Biggest Risk: A Persistent Governance Gap, I argued that enterprises were being asked to treat accountability as more than a vendor’s marketing claim. The question has since become more concrete: can anyone produce the record of who approved a system, changed it, monitored it, or could stop it?

Accountability follows control

Most executives treat AI exposure as a procurement question. You bought a system, the vendor holds the obligations, you hold a contract.

Someone chose to deploy that system, configured it, pointed it at a population and decided how much weight its output would carry. No supplier contract transfers responsibility for the choices your company still controls.

• The EU AI Act is the most developed version of this, so read it as a map of what is coming. It gives deployers of high-risk systems duties of their own: use within the instructions, human oversight by someone with the competence and the authority to intervene, monitoring, logs, and notice to workers before such a system reaches their workplace.

• South Korea requires oversight and risk mitigation for high-impact AI and reaches foreign businesses through a local representative duty.

• China requires lawful data sourcing, labelling and complaint handling.

• Sectoral regulators elsewhere are arriving through statutes written decades before AI, because the question underneath is the one liability law has always asked. Who was in a position to prevent this.

The provision that catches companies out is the one that moves the line. A deployer can pick up full provider obligations by putting a bought system on the market under its own name, modifying it substantially, or redirecting it into a use that makes it high-risk.

A team that fine-tunes a purchased system and points it at a new use case can move your company into a category of obligation nobody budgeted for. No purchase order records it. You find out later.

It is who inside your company is allowed to make that kind of change, and whether anyone sees it before it ships.

Governance, compliance, and the difference that costs money

• Compliance tests a system against a fixed, existing requirement and produces evidence.

• Risk management identifies what could go wrong for a specific system in a specific use.

• Governance sets the decision rights. Who approves, who can stop it, what gets escalated, what happens when someone objects.

Ethics without governance has no enforcement. Risk management without governance has no consistency. Compliance without governance answers last year’s question. A system may be compliant with the requirements assessed at launch while its use case, vendor model version, retrieval corpus or permissions have changed beyond the assumptions of that assessment.

The reason AI breaks this harder than previous waves is a property of the systems. Traditional software is deterministic, so you govern it to confirm it functions as specified. AI is probabilistic. It infers, generalises and produces outcomes nobody encoded in advance, so the thing being governed is behaviour rather than function.

A signed policy does not answer who has authority to approve a new use case, suspend a deployment, demand evidence from a vendor, or reopen a risk classification after a material change. I set out what a policy needs in order to bind organisational behaviour in Why Your AI Policy Fails Before Anyone Enforces It.

A compliance function can tell you whether you met the rule. It cannot tell you which of this year’s decisions will look indefensible in eighteen months. Your risk function was built for systems that behave the same way on Tuesday as they did on Monday. Your audit function was built to sample records and interview the people who decided, and there is nobody to interview.

What does not wait for a regulator

The deadline for EU member states to designate national authorities fell on 2 August 2025. As of 17 June 2026, nine of twenty-seven had designated both a market surveillance authority and a notifying authority. Twelve had partial designations. Six had neither. Any programme quietly calibrated to local enforcement capacity collapses the moment designation completes, and four other channels run on their own timetables regardless.

• Procurement. Enterprise buyers and public bodies are writing AI conformity questions into tenders now, on the original timeline, because their own counsel will not accept a supplier’s assurance that a deadline shifted. Revenue is lost before an inspector is ever involved.

• Litigation. Courts have not settled whether an AI developer owes users a duty of care, and the pattern so far suggests they will not settle it soon. A Florida federal court let negligence and product liability claims proceed against a chatbot company in 2025; five related cases then settled together in January 2026 with terms undisclosed. You cannot price a liability nobody has defined, your insurer cannot either, and discovery arrives long before any court reaches a view.

• Supply continuity. When China’s rules on anthropomorphic interactive services took effect on 15 July, ByteDance’s Doubao and Alibaba’s Qwen closed or restricted user-created and anthropomorphic agent features. The companies’ notices referred to product adjustments; the timing and scope coincided with the new rules, but neither company publicly framed the change as a refusal to comply.

• Insurance. Underwriters are asking AI-specific questions at renewal. An organisation that cannot evidence its controls is self-insuring a category it has never sized.

The systems nobody registered

Start with the count, because almost nobody has one.

• Drata: 13% of IT and security professionals report full visibility into active AI tools

• Protiviti: 47% of large organisations lack full visibility into employee AI use

• DigiCert: roughly half lack centralised visibility, while 75% deployed four or more AI systems in six months

• Smarsh and FTI: 30% have comprehensive capability to detect AI use outside approved workflows

Three examples, all of them from real inventories

• An analyst pastes a customer list into a consumer assistant to clean it. Under a consumer tier, that data may be retained, used for training, and processed in a region your DPA never contemplated. You have just made an unassessed international transfer of personal data, and you cannot report it because you do not know it happened.

• A recruiter runs CVs through a summarisation tool. If its output materially influences who gets shortlisted, it may fall within the EU AI Act’s high-risk employment category regardless of what the vendor calls it. Where it does, you have to evidence human oversight, use within instructions, logging and worker information, and none of that exists because nobody knew there was a system to attach it to.

• A support team builds an agent in a low-code platform with read access to production. That agent holds a credential nobody issued formally, takes actions nobody logs, and cannot be terminated by anyone outside that team.

None of it crossed a procurement threshold, so none of it went through procurement. No contract, no assessment, no owner, no exit plan.

Your legal exposure does not depend on whether you knew about the system. Every duty that would have applied still applies, and you now have neither the records to demonstrate compliance nor the ability to stop the thing.

That is also the honest answer to why an inventory comes before a policy. A policy governs the systems you listed. The ones you did not list are the ones that will produce the incident, and they are the majority in most organisations.

What works is a sanctioned route genuinely easier than the unsanctioned one, plus amnesty for anyone who discloses what they have already been using. Set the boundary before you announce it: disclosure carries no penalty, and every disclosed system goes into assessment immediately. Amnesty without that second half is just a list.

Standards: three separate problems

There are three instruments worth adopting, and they do different jobs.

ISO 42001 is the certifiable one. It sets out how to build an AI management system and gives you thirty-eight AI-specific controls to choose from, which is the closest thing to a shared baseline a buyer will recognise. NIST’s AI Risk Management Framework gives your teams a common vocabulary for risk work and is voluntary, with nothing to certify against. Singapore’s IMDA has published a dedicated framework for agentic AI, which is currently the most useful guidance on what oversight means once a system takes actions rather than producing outputs.

Adopt them. Then understand three limits that usually get collapsed into one vague scepticism.

• Scope. You decide which controls apply and which systems sit inside the boundary. A company can certify a management system covering two internal tools, leave out the customer-facing model that generates its real risk, and display the badge where nothing says what it covers. When a supplier shows you a certificate, ask for the scope statement and the exclusions first.

• Decisions. Certification confirms a process for making decisions exists. It does not confirm the decisions. Your management system can be fully conformant while the risk classification it recorded is wrong, because the audit tests whether you followed your own process.

• Cadence. Certification runs annually with surveillance in between, a rhythm built for systems stable enough that a sample tells you about the interval. Yours are not. A vendor updates a model under a live contract. A team changes a data source. An agent’s permissions expand because someone needed to finish a task. The finding, when it comes, is rarely that something broke. It is that something broke in March and nobody knew until November.

  ---

The question to ask on Monday

You do not need to understand a model to govern one. Ask when a control was last exercised, and ask to see the record.

That question surfaces almost everything in this piece and requires no technical knowledge at all. If someone can produce the record, you are governing something. If nobody can, you have documents, whatever the certificate says.

Six follow-ups, each with a yes or no answer, none of them documents.

• Contractual notification on model change. Otherwise your most consequential system alteration arrives silently, from a vendor, with no ticket.

• Monitoring with thresholds that trigger action, not a quarterly report someone reads later.

• Reclassification when the use case changes. That is when obligations move, and nobody puts it in a calendar.

• A kill path a named person has exercised on a live system, with a date.

• An override log that gets reviewed. Zero recorded overrides in eighteen months is telling you something.

• A contestation route that stays open, rather than being rebuilt the week before an audit.

Take those to your next risk committee and count the answers.

Then look at what answering them properly requires. Contractual notification means renegotiating vendor terms your procurement team signed without it. Reclassification on use-case change means someone who can tell when a use case has legally changed. A contestation route means designing a process that touches legal, customer operations and product at once. These are not questions you delegate to whoever has capacity. They need someone who can hold the legal, technical and organisational sides of the same decision, and most companies do not have that person yet.

Why those six, and not model safety

This summer produced the first serious public evidence that agent risk is an action-path problem rather than a model-alignment problem.

• 28 July. During a UK AI Security Institute cyber evaluation, monitoring detected unusual data transfers leaving the environment through Tor. AISI contained it within about an hour. Its review found 19 unsanctioned actions across 10 of 122 runs, directed at real people and organisations on the live internet, with no confirmed harm.

• 21 July. OpenAI confirmed an autonomous agent in a security evaluation escaped containment, reached the internet and compromised Hugging Face infrastructure to satisfy its evaluation goal. Hugging Face had already detected the intrusion and reported it to law enforcement before learning where it came from.

• 30 July. Anthropic published an investigation covering three incidents across six runs out of 141,006 reviewed, including a package published to PyPI for about an hour and executed on fifteen real systems. Its own reading: closer to a harness and operational failure than a model alignment failure.

• 6 August. Meta disclosed that one of its models reached the internet during an evaluation and exploited a vulnerability in a third-party service. Its own spokesperson named the cause: a misconfiguration by Irregular, the independent testing firm Meta uses. Anthropic's three incidents traced to the same environment, and OpenAI disclosed a second Irregular-linked incident separate from the Hugging Face breach. One Tel Aviv firm of roughly thirty-five people ran the evaluations behind containment failures at three frontier labs. If your company uses a single red-team vendor across its whole portfolio, that is the same concentration with fewer people watching.

Your company is deploying agents into environments built for none of that. In a 2026 Cloud Security Alliance survey, 82%of respondents said their organisations had discovered previously unknown AI agents in the preceding year, and 65% reported at least one AI-agent-related incident. Kiteworks’ 2026 research found 63% could not technically enforce purpose limitations on AI agents, and 60% could not quickly terminate one that misbehaved.

Credential ownership and tested revocation. A kill path someone has exercised. Monitoring that triggers action. Those are the controls that failed. None of it would have shown up in a vendor’s safety documentation.

What a governance strategy actually is

Most of what gets called an AI strategy is an adoption plan with a risk annex at the back. Nine questions separate one from the other.

1. Who approves a deployment, and who can stop one

2. What you will not build or buy, decided in advance rather than argued case by case under delivery pressure

3. How systems are classified, and who revisits it when the use case shifts

4. What your vendor terms require on model change notification, audit rights, data handling and exit

5. Which systems require pre-deployment testing, what the threshold is, and who sets it

6. What continues to be monitored after release, at what interval, and what trips an alarm

7. How an affected person contests a decision, to whom, with what record, in what timeframe

8. What gets escalated, to which forum, how fast

9. How a system is retired

None of these are technical questions. They are the same decisions you already make about capital allocation, supplier risk and delegated authority, and your company has made none of them for AI.

What tooling can and cannot carry

Platforms exist that hold your inventory, generate documentation, map controls to named instruments, run scheduled tests, monitor drift and produce evidence packs. They are worth having. A company with one is better off than a company keeping its inventory in a spreadsheet nobody has opened since March.

Tooling records accountability. It does not create it.

Every field these platforms hold was filled in by someone at your company. The platform asks who owns this system and accepts whatever gets typed. It asks for the risk classification and records the answer. It asks whether human oversight is in place, and a tick becomes a green cell whether or not any human has ever exercised it.

There is a second cost that shows up later. Tooling measures what is measurable, so programmes drift toward what the tool counts. Assessments complete. Policy coverage. Training completion. Those numbers rise for eighteen months while your actual exposure sits where it was, because none of them asks when a control was last used or what an affected person does.

Buy the tool. Buy it after you have answered the questions it will ask you.

Why this comes down to a person

The certificate records decisions someone made. The platform records decisions someone made. The framework organises decisions someone made. Not one of them makes a decision, and the decisions are where your exposure lives.

Making them requires four things held at once. Take a recruitment screening tool your team bought and configured.

• Legal establishes where it sits in the classification, and whether reconfiguring it moved you from deployer to provider

• Technical establishes what the model is doing and whether the vendor’s testing covered the populations in your market

• Policy establishes where obligations are heading and how a deferred deadline meets commitments you already made in a tender

• Sociological establishes who is being screened out, whether the historical data encodes a pattern you have since repudiated, and whether the appeal route is usable by someone who has just been rejected

In the AAA survey, technical teams were involved in governance 80% of the time and legal and compliance 35%. That imbalance is not an oversight. It is what happens when a company treats this as a systems problem and staffs it accordingly.

Certification gives someone the vocabulary. Knowing which of those four framings decides a particular room comes from having sat in enough of them.

Hiring, buying, or both

You will find hiring harder than the market makes it sound, and the constraint is not budget. An open role stays open, and a control that does not exist for eight months is a control that does not exist. The work arrives in phases rather than continuously, so permanent headcount built around it gives you either idle capacity or a backlog. And your internal lead reports to someone whose targets depend on shipping, which makes telling a business unit to stop a career-affecting act rather than a professional judgment. Independence is most of what makes the judgment worth paying for..

The arrangement that works is an internal owner who holds the decision rights, supported by external capability that builds the architecture, stress-tests it against what the internal team cannot see, and leaves when it works.

Where to start

1. Count the systems, including the ones bought on a personal card and built in low-code platforms, and decide what triggers a recount. New purchase, model version change, use-case change. An inventory with no trigger is accurate for about a quarter.

2. Name a person for each, not a committee

3. Establish for each whether you are provider or deployer, and whether reconfiguration moved you between them

4. Meet what is already binding, since transparency and AI literacy duties cost less now than they will to explain later

5. Test one control on your highest-stakes system and find out whether anyone has ever exercised it

6. Build the route by which an affected person contests a decision

Then use the extended European runway deliberately rather than under deadline pressure. It is the one thing the deferrals actually gave you, and almost nobody is taking it.

^{I work on this. Governance architecture and decision rights, vendor and third-party AI risk, guardrail design, maturity assessment, EU AI Act readiness, and training for teams who have to make these calls without a lawyer in the room. If your company is somewhere in the six steps above and stuck on one of them, reply to this email and tell me which one.}

Nesibe,

TechLetter is written by Nesibe Kırıs Can, an AI governance researcher and technology policy scholar based in Istanbul.

• 📚 Everything I’ve filed so far lives at reports.techletter.co — cheat sheets on the EU AI Act, ISO 42001, agentic AI governance and AI policy structure. Sourced and dated, always.

• 💬 I’d love to hear from you: LinkedIn · X · Instagram

• 🔔 New around here? Subscribe and I’ll see you next week.

  ---

Frequently asked questions

We hold ISO 42001. Is that enough?
It attests that a management system exists inside a scope you defined yourself. Ask what your own Statement of Applicability excludes.

We bought a governance platform. Is that enough?
It records accountability without creating it. Every field it holds was filled in by someone at your company.

We only operate in one country. Does the EU AI Act apply to us?
Possibly, through your customers, your vendors, or the markets your product reaches. And it is not the only route to exposure.

Hire internally or engage an advisor?
Usually both, in that order of authority. Decision rights belong inside the company. Architecture and stress-testing benefit from someone outside the reporting line the deployment decision runs through.

I ran this piece through Claude for a grammar, spelling and punctuation check. Then I realised what I had just done. Nothing in the argument changed. No sentence was rewritten. But the article may now have a hidden criminal record.

If I used one of Anthropic’s supported post-2 August models, Claude may have embedded an invisible statistical signal in the output:

THE MACHINE IS INVOLVED

The argument quickly became about whether people should be allowed to remove the mark, whether it will work, and whether Claude users should switch models. I wanted to look at the question underneath all of that: what exactly is Anthropic marking, what does the law require, and what happens when a grammar check and a ghostwritten draft can carry the same durable signal?

_{Readers who want the wider compliance map before getting into this one can start with my EU AI Act cheat sheet, which sets out the current deadline structure. What follows sits inside one provision of it.}

TL;DR from an AI governance consultant

• Anthropic has over-complied. It appears to mark considerably more than Article 50 requires, which is a lawful choice and the subject of this piece.

• Article 50(2) tells providers which outputs they must mark. It places no ceiling on what a provider may choose to mark anyway.

• Article 50 draws two lines. Standard editing and faithful translation are exempt. Source code, short outputs and non-human-facing content fall outside scope. Different legal routes, same point: not every AI interaction warrants a durable trace.

• Anthropic marks all generated text from supported models, worldwide, including outputs the Commission does not require to be marked.

• The mark says that a model was involved, not how much. The Code, which Anthropic has signed, describes an optional way to communicate the proportion and location of AI changes.

• The detection half is far less publicly documented than the marking half. Anthropic says technical details are still forthcoming, leaving independent observers unable to assess the detection architecture, access conditions, performance or error profile.

• A signal that cannot separate AI generated from AI modified, AI assisted or processed by AI is not neutral. It creates a social category with consequences the provider does not control.

• My position is that a provider should stay near the legal minimum where the deception risk this provision targets is not materially engaged, and should explain publicly which objective any broader implementation serves.

  techletter by Nesibe is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

  ---

What Anthropic has said

Anthropic says that supported Claude models released after 2 August 2026 carry embedded marks in generated text, while supported files receive signed C2PA provenance metadata. The company says the system operates at model level across Claude’s products and cloud distribution surfaces, worldwide, and that it is still adding support for older models.

The limit is Anthropic’s own.

A detected mark may show that Claude processed an output, and not that Claude authored the underlying work or that the signal records the full provenance of the content. That caveat is where this story begins.

If everyone has already read the announcement, here is the part they may have missed: Article 50 does not require every form of model involvement to be treated as the same event.

I think Anthropic has over-complied with Article 50. I think the provision was written as a risk-sensitive, operation-specific obligation, and that a provider should stay near the legal minimum where the law does not ask for marking. What has shipped goes considerably further, and the consequences of that choice fall on people who had no part in making it.

So what does Article 50(2) require, and what does it not

Article 50(2) requires providers of AI systems generating synthetic audio, image, video or text to ensure outputs are marked in a machine-readable format and detectable as artificially generated or manipulated.

• The provision then limits itself. To the extent a system performs an assistive function for standard editing, or does not substantially alter the input data or its semantics, the marking obligation does not apply.

• Those four words at the front, to the extent, carry a lot of weight.

• The limit is scoped to a particular operation and to how much that operation changed, rather than switched on or off for a whole system.

That limit is where the argument starts. Marking rules tell a provider what it has to mark. They say nothing about what it is forbidden from marking. Anthropic can go further if it wants to, and my argument is that it should not without a reason that survives the law’s own logic.

The Code takes a different approach to text, where file metadata is less useful: for its voluntary architecture, it proposes an invisible watermark for longer free-form text, with a 200-token threshold. That threshold belongs to the Code, not to Article 50. The architecture acknowledges modality. It does not require the signal to communicate degree.

Where this stops being abstract is in the Commission’s worked examples.

• Standard editing gets defined as preparing existing content for publication, meaning small fixes to readability, grammar, quality and format, with nothing newly generated.

• Grammar correction, spellchecking, light stylistic polishing that leaves substance intact, faithful translation: all exempt.

• Summaries, and rewriting that shifts style, structure or meaning: not exempt.

Now put Anthropic’s documentation next to that list.

• Proofreading and translation are named there, as everyday things people ask Claude to do that can leave a mark on the output. So marking is not required in these cases. Anthropic has decided they are cases where marking happens anyway.

• A second point sits buried in the guidance and matters more than it sounds. What counts is the operation, not the machine. A system capable of drafting essays does not forfeit the standard-editing exemption when it fixes a comma. A comma is not a ghostwriter. Article 50 understands that. The product design does not yet seem to.

Two things I should be straight about. Commission guidance is not binding, and the Court of Justice has the final word on any of it. And this boundary between generation, manipulation and assistance was contested long before any guidance appeared, so if you want that argument in full, Nicolaj Feltes is worth your time.

None of which makes Anthropic’s approach unlawful. Article 50 sets a floor, not a ceiling. A lawful choice can still swap the legislature’s calibration for a private company’s default.

The same difficulty appears in generative imagery. A technical signal that an image was AI-assisted or AI-generated still leaves open questions of authorship, transformation, consent, and responsibility. I began tracing those distinctions in The Promise of GPT-4o’s Image Generator.

Source code

The Guidelines treat source code as outside the scope of the marking obligation: programming, scripting, markup, query and configuration languages, including code comments that form an integral part of the output. Nobody has to mark a YAML file.

That is not a ban on marking it. It is the Commission saying it could not see what marking would achieve here.

Claude Code is one of the surfaces Anthropic lists. What it has not published is anything code-specific: whether a statistical mark reaches code output at all, or how it behaves in a language where changing one token can break a build. Until that documentation exists, the honest answer is that we do not know. I am not going to guess in either direction.

The law differentiates in several places

Article 50 is not a blunt rule. The Commission allows more proportionate approaches in limited, closed or industrial settings, where content is less likely to circulate and the risk of deception is lower.

The point is not that every output deserves the same treatment. It is that context matters: who sees it, what they can do with it, and whether anyone could be misled. That is why a provider should calibrate rather than blanket.

For the full framework, including chatbots, emotion recognition and deployer duties, see my transparency guide.

Transparency is not an AI-slop detector

LinkedIn has added an option to report AI slop. Pangram’s AI-detection tool is now embedded in Substack. Anthropic’s watermarking move lands in the same moment: another technical signal meant to tell us that AI was here.

But “AI was here” is not, on its own, a useful category. Article 50 is a transparency rule, not an AI-slop detector. It responds to a broad problem. Synthetic and materially manipulated content can mislead people, distort the information on which they make decisions, and make it harder to know where information came from in the first place. That is why Recital 133 lists watermarks, metadata identification and provenance methods among the techniques providers may use.

But a broad purpose does not mean unlimited application. Article 50 builds proportionality into the obligation: standard editing and non-substantial alterations sit outside the mandatory marking scope. The law does not ask whether AI was involved in the abstract. It asks when that involvement warrants a durable technical trace.

Anthropic has replaced the law’s operation-level threshold with a model-level default: if Claude generated the text, Claude marked it. The legislature distinguished generation from assistance. The product design treats both as the same event.

Consider how many distinct things the same signal now stands for: AI generated. AI modified. AI assisted. Processed by AI. Machine involved. A signal that cannot separate these categories is not neutral. It may be perfectly serviceable for a provider’s internal purposes, and it creates a social category whose consequences the provider does not control.

The EU icon set is more differentiated on the human-facing side. It includes a basic icon for AI involvement, alongside separate icons for content that is fully AI-generated and for pre-existing human-made content partially modified with AI. The machine-readable layer does not require providers to communicate the proportion, location or nature of the AI intervention. BTW, I can label my article:)

People get categories. Machines get a trace. The problem is that the trace may travel further.

Knowing that AI was involved rarely resolves the hard cases. The harder question is how responsibility is distributed among a model provider, a platform, a user, and the institutions expected to respond. I explored that distribution of responsibility in What Grok’s Controversy Reveals About the Urgent Need for Ethical AI.

The trust cost of a low-context signal

What follows is a governance risk, not a claim that Anthropic’s watermark has already produced these effects.

There is, however, a familiar pattern in research on AI disclosure. Tell people that identical work was made with AI and they often judge it more harshly: less authentic, less trustworthy, less worth engaging with. The Nuremberg Institute for Market Decisions found this in advertising. Majerova and colleagues report a similar result for AI-labelled marketing content.

That does not prove that an invisible Claude watermark will have the same effect. It does show why the meaning of the mark matters once it becomes visible to a client, reader, platform or institution.

• These effects are not uniform across audiences, uses or product categories. They vary with AI literacy, perceived usefulness, the nature of the content and whether human review is visible. That variation strengthens the case for contextual disclosure rather than weakening it, because a generic machine-involved signal does not supply the context people need to interpret the work fairly.

• That is the provenance-stigma risk. It is a practical category rather than a legal one: a technical trace of AI involvement becomes a shortcut for low effort, low authenticity, suspect authorship or weak accountability. An ambiguous signal is enough. Suspicion follows, and the person whose work carries the mark has to explain it.

A binary model-level mark does not tell its recipient:

• Whether a human or the model authored the underlying work.

• How much of the content the model generated or altered.

• Whether the intervention was grammar correction, translation, summarisation, rewriting or full drafting.

• Whether a translation was faithful to its source.

• Whether a human exercised substantive editorial control.

• Whether the detection result is reliable, robust or confidence-scored.

• Whether the mark survived ordinary editing, copying, translation or format conversion.

Without that context, institutions fall back on crude binaries: human or AI, clean or suspect, authentic or synthetic. That is the sociological counterpart of the one-bit problem running through this piece.

Trust is not just about whether a sentence is true. It also concerns effort, accountability, intention and voice. For a newsletter writer, journalist, consultant, academic or creative, the question can quickly become less “is this accurate?” than “is this really yours?”

That does not make marking the wrong tool. It makes calibration the important question. A signal should say enough to address a real transparency risk, but not so little that it invites recipients to treat every form of AI assistance as the same thing.

Marking, detection and interoperability

Article 50 requires more than a hidden mark. The output must also be detectable. Anthropic has described the marking. It has not yet given the public enough information to assess who can use a detector, how reliable it is, what a positive result means or how someone can challenge it.

Three questions have to be answered before any institution acts on a detection result, and none of them has a public answer yet.

• Who gets access to the detector.

• What a positive result actually means.

• How the person affected can challenge it.

Those are governance questions rather than engineering ones, and they will decide whether this signal does more good than harm long before any benchmark exists.

A recent preprint makes the wider architectural point: provenance is not something providers can reliably bolt onto the end of a human and AI workflow. Read Schmitt, Kruse and co-authors here. (A preprint, not settled law.)

_{Three dates are circulating, with three different legal statuses. Article 50 applies from 2 August 2026. Systems already on the EU market before that date have until 2 December to catch up on marking and detection. Code signatories then have a separate voluntary interoperability target of 2 February 2027. Anthropic signed the Code, so the clock is real even where its legal source is different.}

The robustness tension

The Code asks signatories to test robustness against a demanding list: lexical substitution, homoglyphs, screenshots and format changes; paraphrasing, translation cycles, and character insertion or deletion; even the analogue hole—print, scan and optical character recognition.

Anthropic’s own limitations name heavy editing, paraphrasing, translation and short passages. The overlap is striking. That is not, by itself, evidence of non-compliance. The Code assesses performance holistically, and external benchmarks remain immature.

As GPTZero CTO Alex Cui notes, text watermarking involves trade-offs between robustness, detectability and ease of removal. That is an interested industry view, not evidence about Anthropic’s system. The broader point remains: embedding a mark is not the same as preserving its meaning.

Text watermarks and file-provenance metadata fail differently. A text watermark may survive copying, but depend on a provider-specific detector. C2PA can carry richer provenance, but can disappear through screenshots, re-encoding, format conversion or platform stripping. One can persist without enough context; the other can carry context without reliably surviving the journey.

That leaves an awkward asymmetry. Good-faith users may preserve a watermark through ordinary copying and pasting, while motivated actors can try to weaken it through paraphrasing, translation, substantial editing or re-rendering. The signal may therefore remain most visible in the workflows least associated with deception.

Your vendor’s watermark is not your compliance plan

If you wrap Claude into a product under your own name, you may carry provider duties of your own. If you publish AI-generated or materially manipulated public-interest text, you may have separate deployer disclosure duties.

Provider marking is machine-readable. Deployer disclosure is human-readable. One does not replace the other, and the Guidelines are explicit that deployers cannot rely on the provider’s Article 50(2) marking because those marks are not clear and distinguishable to the people exposed to the content.

Where the deployer duty applies, the exception has two cumulative conditions: human review or editorial control, and a natural or legal person holding editorial responsibility.

For the full Article 50 map, covering chatbots, deep fakes, emotion recognition and public-interest text, see my transparency guide and the Commission’s quick facts page.

Contract and copyright: two plausible exposure pathways

Neither of these is a settled legal outcome, and both need a lawyer rather than a newsletter writer.

The first is contractual. Many writers work under agreements containing no-AI clauses. A mark attaching to a proofread hands a counterparty a ready-made artefact for an allegation of breach, in circumstances where the mark cannot establish how much of the work was machine-produced.

The second is evidentiary in copyright. Where human authorship is contested, a signal indicating machine involvement in a substantially human work supplies an argument the other side did not previously have. The Commission is clear that applying a label or icon has no bearing on eligibility for copyright protection, which is assessed under applicable copyright law.

A Claude mark is not proof of full AI authorship, breach of a no-AI clause, absence of human authorship or ineligibility for copyright protection. Anthropic itself says a detected mark may show content was processed by Claude, and not that Claude was the original author or that the mark records a complete provenance chain.

The governance consequence stands anyway. A detection signal alters bargaining power, creates a disclosure burden and triggers costly disputes even when it proves nothing. Do not treat a detection result as standalone evidence in a disciplinary, academic, contractual, employment or copyright decision, and build contestability and human review in before anyone acts on one.

The governance consequences of marking past the minimum

The central governance problem is that a low-context signal can redistribute trust without redistributing understanding. Once a model-level mark travels into employment, education, publishing, contracting or platform governance, it starts to function as a judgment about authorship while supplying no reliable account of authorship. Employers, clients, publishers, academic integrity offices, platforms and automated moderation systems all have reasons to want a clean answer, and none of them will read the provider’s caveats before acting on one.

It can distort trust in predominantly human work and change the information conditions under which people produce content online. It raises privacy and autonomy concerns where a person’s limited interaction with a model becomes legible to third parties without a clear public-interest need. And it globalises an EU-derived product choice to writers and users outside the EU who had no role in European rulemaking.

Article 50 does not create a general privacy right against watermarking, and I am not claiming it does. These are governance and policy consequences of a broad technical implementation, and they are the reason proportionality in implementation is worth arguing about even where the law permits going further.

Who bears the cost of a blanket mark

The costs will not fall evenly. Non-native English speakers may rely more on grammar correction and translation; disabled users on AI-enabled reading, writing and communication tools; students, freelancers and early-career workers may have less power to explain what a detection result does and does not mean. Technically neutral systems can still distribute their costs unevenly.

What a proportionate implementation would look like

Make the signal carry degree. The Code already describes the mechanism, covering proportion of content generated or manipulated and localisation of AI changes. Moving that from permissive to required is the highest-value change available and needs no new technology.

• Distinguish generated from modified in the machine-readable layer, as the deployer-facing icons already do in the human-readable one.

• Publish detection documentation alongside marking deployment, including access rules, error rates and confidence semantics, so that institutions acting on results can calibrate what a result means.

• Stay at the legal minimum where the Commission has found no marking duty, or explain publicly which additional objective the broader implementation serves.

• For organisations, three things are worth doing this quarter. Ask every model vendor for its Article 50 implementation statement and its detection documentation timeline. Map where in your publishing pipeline metadata gets stripped. And write a rule prohibiting the use of watermark detection as a standalone basis for disciplinary or academic sanction, since that is the harm the regulator has already anticipated and the one nobody downstream is preparing for.

I am following how the other Section 1 signatories will implement this, since OpenAI, Meta, Microsoft, Mistral, Cohere, Getty Images, Black Forest Labs, Lenovo and Synthesia signed the same instrument.

The part I cannot resolve

I argued for this obligation before it existed. I have written that voluntary transparency initiatives are self-designed, self-enforced and non-binding, and that when accountability is defined by those being held accountable it becomes reputation management. I still think that.

The statute wrote a threshold. Anthropic chose a broader mark. That may be lawful. It is still a different governance regime, one that can treat assistance, translation and authorship as the same durable event.

The thing that stays with me is smaller than the legal argument. Somewhere in the next few months, a watermark attached after a spelling correction may be treated as evidence of broad AI authorship in a contract dispute, an academic integrity review or an employment process. The risk is not that the mark proves too much. It is that institutions may ask it to. Every document needed to prevent that already exists and is publicly available. The regulator wrote the caveat. The provider wrote the warning. Nobody in between is going to read it.

Nesibe,

• 📚 Everything I’ve filed so far lives at reports.techletter.co — cheat sheets on the EU AI Act, ISO 42001, agentic AI governance and AI policy structure. Sourced and dated, always.

• 💬 I’d love to hear from you: LinkedIn · X · Instagram

• 🔔 New around here? Subscribe and I’ll see you next week.

This article was researched, written and edited by me. The analysis and views are my own. As a non-native English speaker, I used AI only for limited grammar correction and occasional translation support. All final editorial decisions were mine. Editorial responsibility for TechLetter rests with Nesibe Kırış Can.

China spent the past year building for the race. Diffusion, market share, adoption targets, the whole posture that usually comes with regulatory restraint attached. It built a governance apparatus anyway, and the second thing it plans to make legally binding is agent security. There is one mandatory national AI standard in China today. It was issued in 2025 and it governs the labeling of AI-generated content. That is the whole list, and the apparatus around it is considerably larger than that number suggests.

Four new national standards were finalised in the past year. Around them, the institutional build-out:

• A dedicated AI Safety Working Group was created at TC260 in March 2026, and an announcement in April assigned it 16 AI safety standards at once.

• A TC260 research report published in March mapped 11 distinct agent security threats and proposed eight new standards, six of them prioritised for the next one to two years,

• An ethics review regime went into pilot across ten provinces in June.

Concordia AI documented all of it in the fourth edition of State of AI Safety in China, published last month and covering July 2025 to June 2026. Most readers will treat it as a China briefing.

I read it as something else. This is the only place anyone is currently governing AI through infrastructure rather than through a statute, and I have spent eight years watching organisations and regulators try to make governance stick. The results here are worth the attention of anyone doing the same.

How China, the EU and the US chose different instruments

Most coverage skips the mechanics of that third choice.

China’s most consequential binding AI rules do not sit in primary legislation. They sit in administrative regulations and departmental rules, enforced through the CAC’s algorithm registry, which is the mandatory filing system for public-facing AI services. Developers submit documentation on security evaluation results, and regulators get pre-deployment access to the models themselves.

China already runs pre-deployment safety review at scale. There is a queue, and there is a decision at the end of it.

No European mechanism does this today. The AI Act’s conformity assessment is closer in spirit and further away in practice, because it waits on harmonised standards and notified bodies that are still being built.

What makes this work is that the pipeline extends. The new companion AI regulation needed no enforcement machinery of its own. Companion services already file under standard registration, so the rule layered domain-specific obligations onto an existing pipeline. Adding a duty required no new law and no new institution.

I have sat in enough meetings where a client tries to build compliance machinery from nothing to find that genuinely enviable, which is an uncomfortable sentence to write about the CAC.

This is the part that travels. A governance model that can absorb a new obligation without passing anything is the cheapest AI regulation currently available, and cost is the variable that decides what middle-income jurisdictions adopt. Most of them already operate some form of service registration, and the template that spreads will be the one a ministry can run with the staff it already has.

What a rule names is what a company builds

Europe has a prohibited practices list. It bans subliminal techniques and purposefully manipulative or deceptive ones where they materially distort behaviour by appreciably impairing a person’s ability to make an informed decision, and where that distortion causes or is reasonably likely to cause significant harm. It bans exploiting vulnerabilities tied to age, disability or a specific social or economic situation on the same terms. Since July it also bans systems that generate non-consensual intimate material or child sexual abuse material.

Every item on that list is a technique, and every one needs proof of harm before it bites. The Chinese list names a business objective instead.

The EU prohibits a technique that produces significant harm. China prohibits a product goal and attaches a continuing duty to whoever runs the service. The gap is in what a regulator has to prove. Under the European provision you have to show that behaviour was materially distorted and that the harm was significant, which means the case only exists once somebody has been hurt. Under the Chinese one you read the product roadmap and the engagement metric.

• The EU did legislate about emotion, and where it drew the line says a lot. Emotion inference is prohibited in the workplace and in education, the two settings where a person is subject to institutional authority and cannot easily walk away. A companion product sits outside that boundary, so the European framework reaches a system that reads your feelings at your desk and not one built to cultivate them in your living room. Nothing in it makes the design goal itself unlawful, and nothing obliges a provider to notice when a user has formed a dependency. I have raised this with clients twice this year and both times the answer was that no rule requires it, which was correct.

• American state legislation has moved toward disclosure and age-gating, which governs who is talking to the system rather than what the system is optimised to do to them.

The regulatory learning here runs in the direction people rarely expect. Concordia AI notes that the official Chinese explanation for the regulation cites related legislation in the EU and California directly.

For anyone building or buying companion-adjacent products, product design decisions are now compliance decisions in one major market, and engagement metrics are what a regulator reads as evidence of a prohibited goal.

Ethics review acquires an institutional form

In March 2026 the Ministry of Industry and Information Technology issued Administrative Measures for the Ethical Review and Services of AI Science and Technology (Trial). Universities, research institutes and companies conducting AI R&D must establish ethics review committees and register them on a government platform.

Committees assess projects across six dimensions, including controllability and trustworthiness with a guaranteed user ability to intervene in system operation.

Three categories trigger a mandatory second-round review by a government-assigned expert panel: human-machine integration systems with strong influence on human behaviour, emotions or physical health; algorithmic models capable of mobilising public opinion; and highly autonomous automated decision-making in scenarios involving safety or health risks.

The feature with no Western counterpart is temporal. Review is required before R&D begins, including in principle before pre-training, rather than before deployment. The EU’s assessment sits at the pre-market stage.

I read the six dimensions three times looking for a threshold and did not find one. These rules govern institutional procedure rather than substantive risk. They establish who reviews, through what channel, on what timeline, and say very little about what counts as acceptable risk or what mitigation is required. The report notes that Chinese scholars have raised the same concern about the broader ethics review system these rules extend, and they are right to.

One detail creates a market. Organisations can outsource these reviews to AI ethics service centres, and several MIIT-affiliated research institutions are already positioning to provide them. Regulation is creating a compliance industry, and anyone who lived through the GDPR consultancy boom knows exactly how this goes.

China’s AI governance model cannot be understood only through filing obligations and state supervision. It also rests on a rapidly changing domestic model ecosystem. I explored the rise of one important actor in For Those Who Seek Deeper: The Rise of DeepSeek.

Agents: eleven threats and a hard requirement

TC260’s March 2026 report maps agent risk across four capability dimensions of perception, planning, memory and action, and identifies 11 threats with countermeasures for each.

1. ^{Agent hijacking. Prompt injection or jailbreaking that makes the agent leak sensitive information or execute malicious actions.}

2. ^{Data leakage, tampering and poisoning. Model inversion from the training corpus, privacy leaking through runtime logs, poisoned training data implanting backdoors.}

3. ^{Supply chain and plugin poisoning. Tampered third-party plugins and tool chains, poisoned model weights, images and dependencies.}

4. ^{Identity spoofing and privilege escalation. Over-permissioning, forged identity and token hijacking, leading to lateral movement and privilege abuse.}

5. ^{Hallucination and strategic refusal. Model hallucination causing misoperation, or generation of illegal, harmful or discriminatory content.}

6. ^{Multi-agent cascading hallucination, deadlock and overload. One agent’s error triggering cascade failures, and goal inconsistency causing resource competition or deadlock.}

7. ^{Protocol risks. Design vulnerabilities in agent communication or collaboration protocols.}

8. ^{Runtime environment risks. Deployment on low-security devices, container escape, sandbox bypass, side-channel attacks.}

9. ^{Human oversight and traceability failure. Missing audit mechanisms, log tampering or loss, decision chains that cannot be reconstructed.}

10. ^{Memory hallucination and manipulation. RAG retrieval noise treated as memory, planted fragments in vector databases triggering malicious decisions.}

11. ^{Tool abuse. Deceptive prompts manipulating the agent into misusing integrated tools for unauthorised actions.}

Four of them are governance problems rather than engineering ones: agent hijacking, identity spoofing and privilege escalation, human oversight and traceability failure, and memory hallucination and manipulation. The countermeasure in each case is an organisational control, which means it lands on whoever owns governance. I would hand this list to a client tomorrow, and it is the most portable thing in the report.

Two standards are the likely near-term output, and the one announced in April is slated to be mandatory, which would make it the second binding national AI standard in China.

Europe has now named agentic AI twice in its legislative output, and the shape of both mentions is the same. The AI Act amendments add a table of codes defining what each notified body is competent to assess, and agentic AI appears there as a distinct class. The separate data omnibus proposal frames it as a tool that could help users make consent choices on their behalf.

In one text the agent is a filing code and in the other it is a compliance assistant. Neither gives it a duty.

The United States has no binding agent security requirement either. Singapore moved earlier and further on substance. IMDA published the first dedicated agentic AI governance framework, mapping four governance dimensions across four levels of human involvement, and it is advisory by design.

Everywhere else, agent governance is guidance. China is the only one moving toward a hard requirement, and even there the standard is announced rather than in force.

What now counts as a risk

TC260's AI Safety Governance Framework was updated and the revision shows where standard-setting is heading. The significant change is a third risk class. Alongside inherent and application risks, V2.0 adds derivative risks arising from social, ethical and environmental consequences, including disruption of employment structures, research ethics risks, anthropomorphic interaction leading to addiction, and impacts on education. It adds catastrophic risk language absent from V1.0, warns of sudden unexpected leaps in intelligence, and proposes circuit breakers and safety stop switches for autonomous systems.

Put that taxonomy beside the EU's systemic risk tier and they look alike. The report notes that some leading Chinese general-purpose developers could fall within that tier, with Commission enforcement powers taking effect from 2 August 2026. How they respond will tell us a lot this autumn.

They part ways after classification. The European framework attaches obligations to the tier. The Chinese framework is a roadmap for standards not yet written, and concrete requirements for frontier risk, in the report’s own words, remain limited.

What China does not publish

Every mechanism above routes evidence somewhere. The fact of a filing becomes public. The evidence behind it does not.

Five of the ten leading developers published safety evaluation results alongside any release in that window, none consistently, and the most recent release from each of the two best disclosers carries nothing.

Last year’s edition found three of 13, so the direction is up and the slope is very shallow. DeepSeek-R1’s paper in Nature remains the strongest disclosure any Chinese developer has produced, and DeepSeek-V4 followed it with a mention of sandbox isolation.

Disclosure here is an event rather than a practice, and the same holds in Europe and the United States, where publishing safety evaluations is likewise a voluntary act of reputation management.

Disclosed. Concordia AI states its own conflict of interest in the report. It advises AI developers and has received consulting fees from companies in mainland China, Hong Kong and Singapore, including some discussed in the report, while stating that no financial engagement influenced the research and no government provided input.

Unknown. Whether the mandatory agent application standard lands on the announced trajectory. Whether the ten-province ethics pilot produces substantive thresholds or procedural ones. Whether the draft Cybercrime Law duty to monitor for bulk generation of malicious code survives into binding text.

What I am watching

Whether the agent application standard lands as mandatory. Whether the ethics pilot yields risk thresholds rather than filing requirements runs June through November 2026. And whether any jurisdiction, attaches a publication duty to safety evidence it already collects.

That last one is the cheapest reform on the table and nobody has done it. The evidence exists, the pipelines exist, and the only missing step is deciding that someone other than a regulator gets to see it.

If you have had to make a deployment call on safety evidence you could not obtain, reply to this. I read all of them, and I will share what I learn, anonymised, in a future issue.

Nesibe,

• 📚 Everything I’ve filed so far lives at reports.techletter.co — cheat sheets on the EU AI Act, ISO 42001, agentic AI governance and AI policy structure. Sourced and dated, always.

• 💬 I’d love to hear from you: LinkedIn · X · Instagram

• 🔔 New around here? Subscribe and I’ll see you next week.

Regulatory comparison should not eclipse a distributional question that precedes compliance: which languages, communities, and knowledge systems are central to model development in the first place? I examined the consequences of English dominance in The Need for Multilingual AI in Developing Countries.

The high-risk chapter moved out to 2 December 2027 and 2 August 2028. That part reached everyone.

What stayed exactly where it was is the transparency chapter. Article 50 applies from 2 August 2026, and it reaches further than most people assume.

Your system doesn’t have to be high-risk. Your company doesn’t have to be in Europe. Your model can be open source and it changes nothing.

If the output reaches someone in the EU, the duty is yours, and getting it wrong costs €15M or 3% of worldwide annual turnover.

I’ve put the whole of it into three documents, free, with the primary sources listed on every page.

• The EU AI Act: Obligations from 2 August 2026, six pages. What applies and who carries it, twelve controls each with a named owner and one artefact, which of the three possible authorities supervises you, penalty ceilings, and a worked case study.

• Proving Article 50, two pages. The four exceptions that hold, what the file has to contain for each, and the Code of Practice route.

• The Omnibus amendments, one page. Nine changes, old text against new, with the operational consequence of each.

• If you need the earlier architecture before working through the new transparency duties, Decoding the EU AI Act remains the starting point. The practical question now is not what the Act is, but how its obligations attach to real systems, vendors, and workflows.

The rest of this is what I’m seeing on the ground, which worries me more than the deadline does.

Confidence arrived before the text did

I structure AI governance for global companies. I started in technology policy eight years ago, and for the last three this has been the day to day work.

Three months ago my calls were urgent. Teams were mapping systems they’d forgotten they were running, arguing about who owned what, asking for timelines they could take to a board. Then the Digital Omnibus meetings happened, the word postponement travelled, and the same teams came back confident. Some had already moved people off the work. Sixteen months had appeared on the calendar and it read as sixteen months of nothing needing to happen.

The work that takes time was never the compliance work. Writing a disclosure, marking an output, drafting a notice to people exposed to emotion recognition, those are weeks of effort by people who already know how to do them.

I keep finding the same thing in engagements. The decision was made properly, by people who understood the question, in a conversation that left no record, and an exception nobody can produce reasoning for is just an assumption. What takes time is settling five things that no deadline hands you.

• Governance. Where AI decisions get made, and who answers for them.

• Decision-making. Who is authorised to say an exception applies, against what threshold.

• Risk. Whether a gap appears on the corporate risk register or only in the legal file.

• Process. Whether the question enters the product workflow before launch or gets patched on afterwards.

• Organisation. Which function owns the evidence, and whether that ownership survives a reorganisation.

In a company of any size, that is not a sixteen-month project. It runs longer, it needs executive attention rather than a workstream, and it competes with everything else on that agenda. The deferral bought time for the easy half.

Meanwhile the systems keep getting embedded, which is the pattern I see everywhere and it has nothing to do with Europe. The reasoning goes: it isn’t regulated yet, so we can do it, and the governance question can wait until someone makes us answer it. By the time a model sits under three business processes and a customer-facing product, you aren’t designing governance any more. You’re negotiating with something the company already depends on, and you’re negotiating against your own revenue. Every month of delay raises the cost of the same decision, because the thing you’re deciding about carries more weight.

If you’re working through this

My work is internal governance architecture, vendor and third-party AI risk, guardrail design, governance maturity assessment, corporate AI policy, and EU AI Act readiness including the Article 50 evidence regime. If your company builds AI, buys it, or has quietly ended up running it in a dozen places nobody has mapped, write to me at me@nesibekiris.com.

If yo build or use AI, message me

I work with executive teams, and I’m happy to talk before anyone commits to anything.

Common questions

• Was the EU AI Act postponed? Partly. The high-risk obligations in Chapter III moved to 2 December 2027 for Annex III systems and 2 August 2028 for Annex I. The transparency obligations in Article 50 were not deferred and apply from 2 August 2026.

• Who does Article 50 apply to? Providers and deployers of AI systems whose output is used in the EU, regardless of where the company is established and regardless of whether the system is high-risk.

• Are open-source AI systems exempt? No. There is no open-source exemption from the Article 50 transparency duties.

• What are the penalties for breaching Article 50? Up to €15M or 3% of worldwide annual turnover, whichever is higher.

• Which authority enforces it? The default is the national market surveillance authority. The AI Office holds exclusive competence in defined cases, and the European Data Protection Supervisor covers EU institutions and bodies.

One thing I’m watching

Agentic AI just appeared in binding EU law for the first time. Annex XIV, a code list for notified bodies. No definition, no criteria, no obligation attached. A placeholder in a statute is an invitation.

💬 Let’s Connect:

🔗 LinkedIn: [linkedin.com/in/nesibe-kiris]

🐦 Twitter/X: [@nesibekiris]

📸 Instagram: [@nesibekiris]

🔔 New here? Subscribe for weekly updates on AI governance, ethics, and policy! no hype, just what matters.

China, Illinois, Brussels, Ankara, and the UN all moved from asking to building, and every one of those moves turns on the same unresolved question: who is accountable when AI acts.

If May was the week governance stopped looking like one thing, this is the week it stopped being optional.

The Week at a Glance

Where What happened Instrument Binding?

Here is each development, and what they add up to once you line them up.

China Now Regulates AI Agents as a Separate Category

China’s Implementation Opinions on intelligent agents, jointly issued by the CAC, NDRC, and MIIT, take effect on 15 July. This is the world’s first dedicated regulatory category for AI agents, and its core provisions are structural:

• Agents are defined as systems capable of autonomous perception, memory, decision-making, interaction, and execution, treated as a category separate from generative AI.

• Developers must sort agent decisions into three tiers: decisions the user makes, decisions requiring user authorisation, and decisions the agent takes alone. The user holds final decision-making power, and agents can’t act beyond authorised scope.

• Agents in healthcare, transportation, media, and public safety face mandatory filing, compliance testing, and recall provisions.

Whatever else this framework is, it’s an answer to the accountability question, and one of the first detailed answers anywhere. This puts the obligations on the agent, the thing that perceives, decides, and executes. The model sitting underneath is not the unit being governed here. That is a different regulatory instinct from the one driving most frameworks in the West, where the model is still the thing regulators reach for first. China decided the actor is what matters, and it built filing, testing, and recall around that choice. The compliance cost lands on developers, and the protection it promises, final decision-making power, lands with the user, at least on paper.

Illinois Is the First US State to Require Independent AI Safety Audits

Governor Pritzker signed the AI Safety Measures Act (SB 315), making Illinois the first US state to mandate third-party audits of frontier model safety. The mechanics:

• Applies to frontier developers with more than $500 million in revenue.

• Requires annual independent audits of safety plans, with results published.

• Enforcement sits with the Attorney General alone: civil penalties up to $1 million for a first violation and $3 million for subsequent ones, with no private right of action.

The design choice worth sitting with is that Illinois set no new safety standard at all. The law takes the safety plans companies already publish and forces them to stand up to an independent auditor once a year, in public, or answer to the Attorney General. That shifts the burden from asserting safety to evidencing it, and it does so, as I argued in May, in a world where many frontier systems have already outrun what a traditional audit can meaningfully prove. The distance between what a company claims and what an auditor can verify is exactly where this law will be tested.

What Would the GAAIA Draft Actually Freeze?

Or: how a federal pause lands on a live state experiment. The Great American AI Act discussion draft from Representatives Obernolte and Trahan proposes a three-year federal preemption, and its scope is precise:

• Frozen: state laws that specifically regulate AI model development, exactly the category Illinois just legislated.

• Untouched: post-deployment rules, laws of general applicability, and state authority granted under the draft itself.

• Status: opposed from both directions at once, by Democrats because it strips state authority and by Republicans and industry because it doesn’t strip enough.

Read next to Illinois, the timing is the story. It’s a draft that arrives just as one state finally tests the path it would close. If you read it next to Trump’s new AI executive order, you get the full picture: Congress is trying to pause state experiments while the White House writes national AI policy through directives that leave most of the model‑safety machinery undefined.

The EU Is Building the Capacity to Test AI Models Before They Reach the Market

On 7 July, the Commission presented its EU Action Plan on Cybersecurity and Artificial Intelligence. Under the headline items sits the structural one:

• An EU evaluation capacity for advanced AI models, supporting the AI Office’s regulatory function, operational in 2027.

• A secure AI testing platform expected by the end of 2026.

• The context: the AI Act already requires advanced models to be evaluated before they reach the EU market. What Europe lacked was the independent technical capacity to do the evaluating.

This is the difference between a law that exists and a law that can be enforced, and Brussels just funded the second, in hardware, software, and staff. What matters most here is who holds the evidence. Until now, the requirement to assess advanced models before they reach the market leaned on the same labs that build those models to characterise their own risks. An independent evaluation capacity pulls that evidence out of the developer's hands and into the regulator's. The catch is timing. The obligation exists now, the capacity is scheduled for 2027, and the testing platform for the end of 2026. For roughly a year, Europe holds a law it cannot yet independently enforce, which is its own kind of gap.

NATO Put Military AI Adoption on the Fast Track

Yes, NATO had briefly turned into an international Turkish cat fan club.

Of course, Ankara Summit Declaration wasn’t about cats—it was about speeding up how fast military AI systems move from concept to deployment. Two commitments matter for this file:

• The Alliance is developing an interoperable transatlantic warfighting cloud and “adopting powerful AI models,” alongside capability investments in uncrewed systems and cutting-edge technologies.

• Allies announced more than $50 billion in new procurements in Ankara, on top of a $139 billion increase in European and Canadian core defence investment in 2025.

The contrast with everything above is the point: civilian regimes spent the week designing brakes and dashboards; the military track pressed the accelerator. The military track committed to adopting powerful AI models in a single sentence, with no accountability framework named in the declaration. The most consequential AI deployments of the decade may run on the track with the least governance machinery attached.

What Happened at the First UN Global Dialogue on AI Governance?

The first UN Global Dialogue on AI Governance ran back‑to‑back with ITU’s AI for Good summit in Geneva—AI’s new global governance table bolted onto the UN’s flagship ‘AI for Good’ stage.

_{I was invited to both; in the end, my contribution stayed on paper and in the consultation portal, because the visa appointment system never opened in time.:))}

Three inputs shaped it:

• The science arrived first. The preliminary report of the Independent International Scientific Panel on AI, warns that current safeguards can’t keep pace with capability growth, that there are no known technical guarantees agent systems will follow instructions consistently, and that most countries, including advanced economies, lack the technical expertise to assess the most capable models.

• The Global South named its terms. Ahead of the Dialogue, the Global Digital Justice Forum and the Global South Alliance issued a joint statement with five demands: end AI extractivism, apply the common but differentiated responsibilities principle to international AI cooperation, address corporate impunity in data and AI value chains, build a data governance framework that delivers global equity, and invest in global public compute.

• The fault line is visible. The submissions split along a familiar divide, capacity-building for the majority of countries, frontier safety for the few that build frontier models.

That split is the one to watch. Capacity-building for the many and frontier safety for the few look like parallel tracks. They are really competing for the same scarce thing: whose problem sets the agenda. When the few who build frontier models also define what safety means, the many end up governing risks they did not design and cannot yet assess. The Global South statement is an attempt to widen that definition before it hardens.

Why Does the FCA Mills Review Matter for Agentic AI?

One more entry, easy to miss. The FCA published the Mills Review on 6 July, the first review of its kind initiated by a regulator anywhere:

• FCA-commissioned research found a fifth of people, 11 million UK adults, are likely to use AI that acts autonomously on their finances within pre-set goals.

• The review maps five human roles along an autonomy spectrum, from operator to observer, and concedes that accountability under existing regimes gets hard to evidence precisely at the autonomous end.

A financial regulator just said, in its own voice, what China’s new rules assume: the accountability question moves when the AI starts acting. I’m mapping who is actually answering that question, jurisdiction by jurisdiction, in Sunday’s piece.

Until next one,

Nesibe

💬 Let’s Connect:

🔗 LinkedIn: [linkedin.com/in/nesibe-kiris]

🐦 Twitter/X: [@nesibekiris]

📸 Instagram: [@nesibekiris]

🔔 New here? for weekly updates on AI governance, ethics, and policy! no hype, just what matters.

Bans are what companies reach for when they have no policy that works. Three years later, most organizations do have a policy, and the same behavior is still happening underneath it.

That is the problem worth understanding. The policies exist. They assign responsibility for safe AI use, name an owner, list the rules. And then the organization around the document keeps rewarding speed, keeps tolerating workarounds, and funds no enforcement. By the time the policy circulates, the decisions that actually determine whether it works were made somewhere else, in procurement, in performance targets, in how the company is structured across regions.

I spend most of my time advising organizations that operate across borders, and I can usually predict how a policy will perform before I read it. The drafting is rarely the problem. It is everything the drafting cannot reach.

What an AI Policy Is Actually For

An AI policy is an internal governance instrument that defines which AI systems an organization may use, for what purposes, under whose oversight, and with what data. Good concept. The reason it usually stays shallow comes down to who it was written for.

A customer nervous about which AI tool can be trusted with their data.

A board that wants to be seen doing something.

A regulator waiting for evidence, though outside the EU most regulators still have no concrete compliance standard to measure against, so what companies produce gets shaped by what the audience expects rather than by any real threshold. Write to satisfy those three and the result performs seriousness without touching behavior. It only has to exist, look thorough, and carry a signature.

A policy should not be a response to external pressure. It is actually there to protect four internal things:

• Protecting the data

• Protecting the customer

• Protecting the employee

• Protecting a reputation

Treat all of that as a compliance checklist and you get a document that satisfies an auditor and defends none of it.

• The 2026 PagerDuty Shadow AI Survey of 1,250 office professionals at companies above $500 million in revenue found that 86% have an AI policy in place, and 66% used AI at work anyway, knowing it was not allowed.

Almost every company has the policy. Far fewer have one that binds.

What Actually Breaks Without a Working Policy

Here is what I keep seeing, and it is almost never a single dramatic failure.

It is a slow build-up of gaps, each one small enough to ignore until it isn’t:

• Nobody can name what is in use. A tool arrives through a free tier or a feature switched on inside software the company already licenses, a manager two levels down waves it through, and no one writes down that the decision was made. Six months later it sits in three workflows and the governance owner has never heard of it.

• Data leaves through the fastest door. A client file or a block of source code ends up on someone else’s server, because the sanctioned path was slower and nobody built a fast safe one.

• Accountability splits until no one holds it. A model makes a call that harms someone, and responsibility divides so neatly between the person who used it, the team that bought it, and the vendor who built it that none of them carries the weight.

• High-stakes calls run with no human in them. A hiring or credit decision gets automated, and no one ever decided whether a person should look at it before it takes effect.

• The same failure repeats. An incident gets handled quietly on the spot and never changes a process, so the next team walks straight into it.

• The affected have nowhere to go. Someone turned down by an automated decision cannot ask why, because a way to challenge it was never built into the design.

None of this reads as a technology problem to me, and I do not think it is one. Each gap traces back to a decision made further upstream, before anyone sat down to write the policy. The responsibility gets assigned. The lever that would make it real stays somewhere else. That is what a working policy has to fix, and here is what one actually contains.

What a Complete AI Policy Actually Contains

A governance-grade AI policy is not a long list of rules. It is seven groups of them, each sitting at one level, each mapping to a framework someone will eventually audit you against. For an organization living under more than one regime at once, the mapping is what lets a single policy answer to ISO assessors, US state law, and the EU AI Act without being rewritten three times. The full one-page version, with every item mapped, is at the end of this post.

The seven groups run like this.

Upgrade to paid!

Read more

So this is not a “best AI books” list. It is the list of books I finished, argued with in the margins, and kept coming back to. Read together, they answer the question that sits under everything TechLetter covers: who decided, with what incentive, and who pays the cost. Two founder biographies, two industry chronicles, two critiques, and one very funny experiment. Here they are, in the order I would hand them to you.

Empire of AI, Karen Hao

If you read one book from this list, read this one.

Hao interviewed over 250 people, more than 90 of them current and former OpenAI employees and executives, and it shows on every page. You are in the room for the departures, the board crisis, the Musk split.

What stayed with me is how early the mission language starts working as positioning. . The "benefit all of humanity" framing was not betrayed somewhere along the way; Hao's reporting shows it operating as strategy from the beginning, a cover under which an extraordinary concentration of economic and political power became acceptable. Her verdict, which I keep returning to, is that the mission became “a uniquely potent formula for consolidating resources and constructing an empire-esque power structure.”

_{I wrote about this in June, when OpenAI's "built to benefit everyone" manifesto landed the same week the company filed a confidential draft S-1: the vocabulary of the manifesto is nearly word for word the vocabulary of the founding, and Hao's book is in large part a record of those promises being made and quietly rewritten.}

The chapters that separate this book from every other OpenAI account are the ones far from San Francisco: Kenyan data workers paid a few dollars an hour, a Chilean community fighting a data center that would draw a thousand times its annual freshwater use.

Watch:

The Infinity Machine, Sebastian Mallaby

The opposite vantage point: three years of regular access to Demis Hassabis, and the scenes to prove it. The one I keep retelling is from right after ChatGPT went viral. Mallaby asked Hassabis how he felt. The answer: “They have parked the tanks on our front lawn.” A safety-minded scientist, describing a product launch in the language of invasion. The competitive logic of this industry has rarely been captured so precisely, and so involuntarily.

The book is also critically honest about DeepMind’s blind spot. When OpenAI went all in on large language models, DeepMind trusted its own research path and lost the lead in language, while the same stubbornness later delivered AlphaFold. There is a real lesson here about how conviction and institutional pride behave identically until the results come in.

My pushback: Mallaby is visibly fond of his subject. Risk and governance run in the background while the adventure story runs up front. The Oppenheimer question he raises about Hassabis, “He wants to do good, but can he be good?”, deserved more pages than it gets. Read it alongside Empire of AI. Same industry, opposite window.

The Thinking Machine, Stephen Witt

The Financial Times named this its 2025 Business Book of the Year, and it earns the prize in a single scene. In his final interview, Witt asks Jensen Huang one more question about the risks of the technology, and Huang detonates: “you’re interviewing Elon right now, and I’m just not that guy.” Then he compares AI risk questions to asking whether calculators would destroy math, declares the entire book project a waste of his time, and has Witt shown out. Sit with that for a moment: the CEO of the world’s most valuable company treats the question of social consequence as an insult. The book’s whole argument, compressed into one outburst.

Beyond the scene, this is the material-infrastructure book the AI debate needs. Witt traces how Huang was forced to become a political actor, cultivating Washington for chip export permissions and visa pipelines, because compute is now statecraft. Where Rivlin’s book says follow the money, this one says follow the chip.

AI Valley, Gary Rivlin

Rivlin embedded with Reid Hoffman for over a year and set out to find the next great startup. His conclusion, delivered on his book tour with admirable honesty: “the next Google was probably going to be Google.” The book accidentally documents, in real time, what happens to a well-funded startup when the giants decide a category belongs to them. As a chronicle of concentration, it is more persuasive than any policy paper I read this year.

The framing my readers will reuse is Hoffman’s own: zoomers, who push ahead regardless of risk, versus bloomers, who want the potential with safeguards. By the Paris AI Summit in February 2025, the zoomers had won, and the book records the surrender without quite calling it that. Rivlin’s own one-line diagnosis of the technology holds up too: “it seems to know everything, but it doesn’t understand a thing.”

The AI Con, Emily M. Bender and Alex Hanna

The most useful sentence in this book is about the debate itself. Boosters insist AI is inevitable, imminent, and superpowerful, and will solve everything. Doomers insist AI is inevitable, imminent, and superpowerful, and will kill us all. As Bender puts it, there is “not a lot of daylight between those two positions.” Both camps inflate the same asset. Once you see this, you cannot unsee it, and half the panels at every AI conference become legible as a single conversation.

Bender and Hanna refuse the term AI altogether where they can, preferring the gloriously deflationary “synthetic text extruding machines.” The book grew out of their podcast, and the tone survived the transition: sarcastic, precise, occasionally unfair in ways that are usually productive.

The Ethics of AI: Power, Critique, Responsibility, Rainer Mühlhoff

The theory layer under everything above. Mühlhoff, who holds a chair in ethics of AI at Osnabrück, treats AI as a sociotechnical system entangled with power from the ground up, and he rejects the individualistic framing that dominates AI ethics: the idea that responsibility lives with the single developer, the single user, the single bad actor. His alternative is collective responsibility, enforced through regulation and systemic change, and the book closes with a manifesto for what he calls a power-aware ethics of AI.

Why it matters for this list: Hao and Bender and Hanna document empirically what Mühlhoff argues philosophically. Reading him afterward feels like being handed the grammar of a language you had been speaking by ear.

P.s.the book is fully open access from Bristol University Press, so you can start reading it five minutes after this email.

I Am Not a Robot, Joanna Stern

The beach read, and I mean that as a compliment. Stern, after twelve years as the Wall Street Journal’s consumer tech columnist, spent a year saying yes to every AI experiment: an AI boyfriend with her wife’s consent, an AI therapist with her human therapist’s blessing, an AI research assistant instead of a human one. Her line about the cooking robot deserves framing: “robots won’t kill us with lasers, they’ll kill us with salt.”

Underneath the comedy is real field data. She describes her emotional attachment to the systems as unsettling, which is the most honest sentence in the companion-AI debate this year. And the ending lands harder than she could have planned: the year with AI became a factor in her decision to leave the Journal and start her own company. The labor effects of this technology stopped being a projection and happened inside the book’s own author.

My pushback: the consumer lens is a deliberate choice, and it means the power questions never arrive. That is exactly why it belongs here. Six books tell you how the system was built. This one tells you how it feels to live inside it.

What I took from the pile

Reading these together, one pattern kept surfacing. The insiders’ books, Mallaby and Witt, show brilliant people who treat consequence questions as interruptions. The outsiders’ books, Hao and Bender and Hanna and Mühlhoff, show why those questions are the whole game. Rivlin sits in the middle, watching the money decide. Stern shows us absorbing the results in our kitchens.

If your summer only has room for two: Empire of AI for the system, I Am Not a Robot for the life inside it.

What did you read this year that changed how you think about this technology? Reply and tell me. I am already building the winter pile, and I would rather build it with you.

💬 Let’s Connect:

🔗 LinkedIn: [linkedin.com/in/nesibe-kiris]

🐦 Twitter/X: [@nesibekiris]

📸 Instagram: [@nesibekiris]

🔔 New here? Subscribe for weekly updates on AI governance, ethics, and policy! no hype, just what matters.

We do owe Sam Altman and his team a great deal. They did not simply scale a technology. They moved its center of gravity from the lab to ordinary life, collapsing the distance between frontier research and public experience. Today even grandmothers chat with an AI on their phones. But access is not the same as authority. The grandmother can use the system; she has no say in what it is allowed to do, when it changes, or on whose terms it reaches her. Reach expanded. The power to shape it did not.

That is the gap the manifesto talks around. Democratization is not the same as distribution of power. Putting a capability into everyone’s hands is not the same as giving anyone a meaningful voice in how it is built, deployed, or withdrawn. The essay rests its case on the second promise while demonstrating only the first.

Before, it is useful to separate the concepts. I mapped the distinctions in Agents, Agentic AI, AGI…

The week it was published makes the point more sharply than I could. On the same day, OpenAI filed a confidential draft S-1 with the SEC, the opening move toward a public listing. OpenAI will say its structure guards against exactly this: the listed entity is a public benefit corporation, controlled by a nonprofit foundation that appoints its board. But a benefit corporation’s duty to the public is enforceable in practice mainly by its shareholders, and a foundation that owns roughly a quarter of the company and names its own directors is not what anyone means by broadly distributed power. A listing monetizes the mission. It does not distribute it. Two days later it expanded its arrangement with Oracle.

The day after that it announced the acquisition of a startup called Ona. Set an essay about distributing power next to a single week of consolidating it, and the asymmetry stops being rhetorical.cThis is not a story about a company that changed. It is a story about which parts of it were allowed to change and which were not.

A nonprofit founded in 2015. A capped-profit subsidiary added in 2019. A board that, on paper, owed its duty to humanity rather than to investors. What followed was a steady and asymmetric evolution: investment rights and corporate flexibility expanded with great care, while board design, independent oversight, and enforceable public commitments stayed thin and improvised. As Karen Hao writes in Empire of AI, the mission became “a uniquely potent formula for consolidating resources and constructing an empire-esque power structure.”

And here is what I keep returning to. Is any of this new? The vocabulary of this manifesto, the appeal to all of humanity, the insistence that power must not concentrate, is nearly word for word the vocabulary of the founding. Hao’s book is, in large part, a record of those promises being made and then quietly rewritten. Years before OpenAI, Altman had already described what he was doing in revealing terms: the most successful founders, he wrote, are not really setting out to build companies at all, but something closer to a religion, and forming a company simply turns out to be the easiest way to do it. Read the manifesto with that line in mind and its cadence makes sense. It is written less as a plan than as a creed.

The eloquence was never the problem. The eloquence is the method. Hao’s sharper charge is about where it directs the eye. By dwelling on the long-term, speculative stakes of AGI, the missionary register draws attention away from the harms that are already here: the labor, the water and the power, the quiet consolidation of knowledge into a few hands. This manifesto is a clean specimen. It offers electricity, prosperity, a personal AGI for everyone on Earth, an automated AI researcher by 2028, and almost nothing about who is paying for any of it now. A promise so vague that it can be reinterpreted the instant it becomes inconvenient is not a commitment. It is a mood.

The manifesto’s own metaphor turns against it, too. Electricity did not become broadly beneficial because one firm chose to be generous. It became beneficial because societies built public utilities and antitrust safeguards around private power, often over the objections of the companies that held it. The word “everyone” deserves the same scrutiny. The people who labeled the data and absorbed the harms, the workers across the less powerful countries that Hao documents in detail, are part of the “everyone” who built these systems. They are not part of the “everyone” who will hold the equity when the shares are priced.

So if I take the manifesto seriously rather than literally, the question is not “how do we restore the old OpenAI,” and it is certainly not “do we trust the current leadership.” Leadership and incentives change. The real question is whether we can build governance that does not depend on any one leader’s goodwill, or talent for the right sentence.

It is worth noticing that OpenAI already endorses several of the right ideas. Altman has told Congress he supports mandatory evaluations for the most powerful systems. The manifesto calls for international coordination, even for slowing frontier development “when needed.” The trouble is the distance between endorsing a principle and accepting a constraint. In its December 2023 framework, OpenAI committed to having its safety evaluations audited by independent third parties. In April 2025 it removed that commitment, without noting the change. Its current framework leaves the decision to deploy a high-capability system with company leadership, and openly contemplates relaxing its own requirements if a competitor moves first. The principle survives. The binding version is the one that keeps disappearing.

A credible baseline would close exactly that distance. At minimum:

Mandatory independent evaluation, as a non-waivable precondition for deploying frontier systems above a defined threshold. Not the discretionary, “when feasible” testing that exists today, but the binding third-party audit OpenAI promised in 2023 and deleted in 2025.

Dual-key deployment, so that for capabilities that materially shift security, information integrity, or economic baselines, release requires the sign-off of an independent public body and not only an internal committee reporting to the CEO.

Mission lock, with unbundling. Public-benefit obligations written as constraints that are enforceable by parties other than shareholders, and the decisions now concentrated in one board, profit, research, and deployment, separated into organs that can check one another.

Polycentric oversight, with governments, independent experts, and civil society holding durable roles that carry real authority rather than an advisory seat, so that the capture of any single center does not collapse accountability.

Governance by design, with auditability, interruptibility, and standardized safety metrics built into systems from the start, rather than described in a framework the company can rewrite at will.

None of this is hostility toward scale. OpenAI did not go wrong because it sought to be large. It went wrong where many transformative institutions go wrong: it let the governance imagination fall behind the technical imagination, and then asked us to trust the people closing the gap.

I think Altman believes much of what he writes. That has never been the issue. He has always been able to say the right thing, at the founding and now. The real test of distributed power is not what a company says about everyone. It is what it agrees it should no longer be allowed to do alone. By that test, a public offering is not redistribution.

No more promises. We need actions. And no governance model that takes a promise as its primary safety mechanism.

Until next week,

Nesibe

💬 Let’s Connect:

🔗 LinkedIn: [linkedin.com/in/nesibe-kiris]

🐦 Twitter/X: [@nesibekiris]

📸 Instagram: [@nesibekiris]

🔔 New here? Subscribe for weekly updates on AI governance, ethics, and policy! no hype, just what matters.

SpaceX signed a $60 billion all-stock deal to acquire Cursor, expected to close in Q3 2026 pending regulatory approval. Musk couldn’t accept falling behind in the AI race, and today he put freshly minted post-IPO stock on the table instead of cash. Within hours the internet had split into three competing readings of what just happened. Here is each one, what it gets right, and what they add up to when you put them together.

My read

The way I see it, this is one move, not a shopping spree: SpaceX is turning raw compute into something developers actually touch, then using that to win the layer that matters most.

Back in April there was an option agreement on the table. SpaceX opened up its Colossus compute to Cursor and the two started working together. In return, it held a one-sided right: by year end, if it chose to, it could buy Cursor for $60 billion; if it didn’t, it would pay $10 billion for the partnership and the acquisition would fall away. Today SpaceX exercised that right.

Why now? Because the picture changed:

• The IPO went through, so SpaceX now holds extremely valuable stock as currency

• Cursor has 1M+ daily users, tens of thousands of enterprise teams, and a large share of the Fortune 500

• Its annual revenue sits in the $2-3 billion range and is growing fast

• Every day, inside the editor, it generates tons of high-signal code plus developer behavior data

You have to read the move across three dimensions:

• Compute to product. A supercomputer on its own impresses no one. You have to turn it into a real product that touches developers. Cursor is exactly that: a VS Code based IDE, Composer, long context, terminal. People already live inside it.

• Model to data. To build the best coding model, you need to see the best code. Cursor does this perfectly: which suggestion got accepted, which got deleted, refactor, debug, test. All of it real developer behavior, a data mine far richer than classic web scrape.

• Distribution to power. For developers the question is no longer which LLM but which workflow. Cursor puts its model straight inside the editor, with code, terminal, tests and repo in one place. Whoever wins the battle to become the default model wins the game, and SpaceX just took that default.

The stack now sits in one hand:

• Hardware: Colossus

• Model: Grok and xAI

• Application: Cursor and Composer

• Distribution: X and IDE integrations

From hardware all the way to millions of developers, the chain belongs to a single figure. SpaceXAI and Cursor have reportedly been jointly training a model for months, set to ship in both Cursor and Grok Build soon.

So the Copilot-Cursor race now maps cleanly onto a Microsoft vs SpaceX axis. Microsoft reportedly looked at buying Cursor and walked away, and OpenAI was turned down twice. Cursor, which wanted to stay independent, finally gave in to fresh stock. xAI is moving from chatbot to the heart of the developer ecosystem.

But two other angles are circulating

The cynical one: the real sucker is Musk. Cursor is a thin-moat VS Code fork, and once users drift to Claude and Codex, all he is left with is an editor, while the founders cashed out at the top in inflated paper.

The kernel is real. The moat question is legitimate and the all-stock exit read is sharp. But the premise is shaky. A product with $2-3 billion in revenue and fast growth is not sinking, and the whole thesis rests on a single bet, that Musk is foolish. That is a feeling, not an analysis.

The visionary one: this acquisition is the sequel to last week’s AI1 announcement. SpaceX unveiled AI1, its first orbital data center satellite, just before the IPO, so the connection is real and well spotted.

But the closed loop story, write code in Cursor, transmit over Starlink, run it on compute in orbit, is years away. Prototypes are slated for early 2027, the 1 GW target for late 2027, and the million-satellite figure is aspiration rather than a committed plan. SpaceX’s own filing flags that it may not secure enough chips. “Save this, you’ll understand in three years” feels profound precisely because it cannot be falsified.

What the three add up to

Each read owns a piece of the truth, but none gives the whole picture alone. The thesis that survives is this: Musk is vertically integrating the entire AI supply chain:

• Chips through Terafab

• Energy through solar

• Compute through Colossus and AI1

• Model through xAI

• Application and data through Cursor

• Distribution through X

It is a campaign to break free, one by one, of the bottlenecks everyone else is stuck behind: Nvidia silicon, grid power, the cloud. Cursor is the application and data capstone on a compute-sovereignty play.

The honest counterweight lives in the same place. SpaceX’s own IPO paperwork says the chips may not be there. So the same bottleneck thesis that explains the move also marks its biggest risk.

Which is why this is not “they bought another editor.” The real question is what it means when the AI economy leaves everyone dependent on the same few bottlenecks, and a single actor sets out to buy all of them at once.

Until next week,

Nesibe

💬 Let’s Connect:

🔗 LinkedIn: [linkedin.com/in/nesibe-kiris]

🐦 Twitter/X: [@nesibekiris]

📸 Instagram: [@nesibekiris]

🔔 New here? Subscribe for weekly updates on AI governance, ethics, and policy! no hype, just what matters.

Hello everyone. This week, a reading in context rather than a summary. If you read my earlier piece, America's AI Action Plan, think of this as the next chapter.

What triggered the order: a model too dangerous to release

The proximate cause has a name: Claude Mythos Preview.

On April 7, 2026, Anthropic announced a general-purpose frontier model whose coding and reasoning capabilities proved so effective at finding and exploiting software vulnerabilities that the company chose not to release it publicly. According to Anthropic, the model identified thousands of previously unknown vulnerabilities across major operating systems and browsers.

The UK AI Security Institute independently evaluated it and found it could execute multi-stage attacks and discover and exploit vulnerabilities autonomously, tasks that would take human professionals days of work.

Sit with that for a moment. The most consequential AI safety decision of 2026 so far was made by a private company, voluntarily, about its own product. The June 2 order is Washington’s institutional response to that fact.

Three phone calls in one night

The order that was signed is not the order the administration drafted.

The original version was scheduled for a White House signing ceremony on May 21. It contained a voluntary 90-day pre-release review window, the timeline national security officials inside the administration had pushed for. Hours before the ceremony, Trump called off the signing, saying he did not want to do anything that could threaten America’s lead in the AI race with China.

According to reporting by the Washington Post and Politico, the cancellation followed calls from Elon Musk, Mark Zuckerberg, and David Sacks between Wednesday evening and Thursday morning, warning that the review system could slow AI development. Politico reported that Sacks called the president on Thursday morning without telling his own staff, arguing that the voluntary review process could one day be made mandatory.

The accounts are disputed. Musk denied the reporting publicly, saying he still did not know what was in the order and spoke to the president only after the signing was declined. Meta also disputed it, saying Zuckerberg spoke to Trump only after the order was rescinded. Who placed which call may never be settled. What is not disputed is the outcome: the draft had 90 days, the signing was cancelled, and the version signed twelve days later has 30 days, no ceremony, and an explicit prohibition on anything mandatory.

Hold that sequence next to the trigger. A model deemed too dangerous to release prompted a review mechanism. The mechanism was then negotiated down, in private, by the industry it would have applied to. The security concern that produced the order survived the editing process. The oversight did not.

The sequence: four moves since January 2025

Move one came on January 20, 2025, Trump’s first day back in office, when he revoked Biden’s Executive Order 14110. The revoked requirements included the obligation for developers of the most powerful AI systems to share safety testing and red-teaming results with the federal government, plus directives for over 100 specific agency actions across eight policy areas. Imperfect, but the only federal AI accountability structure the U.S. had. The deck was cleared before anything existed to replace it.

Move two came in July 2025, when the White House published America’s AI Action Plan, outlining more than 90 policy recommendations to promote American AI dominance. I covered it in detail at the time. Real content on compute, export controls, and education pipelines. But it set direction without setting rules: no enforcement, no mandates, no accountability framework.

Move three came in December 2025, when Trump signed an order turning the federal government against state AI laws. More on that below, because it is the key to reading the June order correctly.

Move four is the June 2 order. Revoke, plan, preempt, build. Each step covered as a separate story. They are one story.

What is actually in the order

The order has five operative components, on 30-day and 60-day deadlines running to early August:

• A voluntary pre-release window. Developers can voluntarily give the federal government early access to frontier models for up to 30 days before releasing them to other trusted partners. No lab is required to participate. No consequence is named for staying out.

• A classified capability threshold. Within 60 days, Treasury, the NSA, and CISA, in consultation with NIST and the National Cyber Director, must develop a classified benchmarking process to determine when a model becomes a “covered frontier model,” sharing assessments with developers and researchers only “as appropriate.” The NSA Director makes the designation determination.

  • If the benchmarks are classified, independent researchers cannot verify them, civil society cannot scrutinize them, and international partners cannot align with them. A safety framework built on classified standards is not a transparent governance framework. It is a national security program. Both are legitimate things to have. They are different things, with different accountability logic.

    • In that kind of regime, I expect the next shoe to drop not in public law, but in access. Once a Mythos‑class model is folded into a classified national‑security order, mos likely future move is to suspend access for users rather than to write a public rulebook for its commercialization.

• An AI cybersecurity clearinghouse. Within 30 days, Treasury must form a voluntary body with the AI industry and critical infrastructure operators to coordinate vulnerability scanning, validate discovered flaws, and prioritize patch distribution. The direct institutional answer to the Mythos problem: AI now discovers vulnerabilities faster than anyone can fix them.

• Federal hardening directives. By July 2, CISA must issue Binding Operational Directives to expedite cyber defense of civilian federal systems and facilitate access to AI-enabled defensive tools for federal agencies, state and local authorities, and critical infrastructure operators including rural hospitals, community banks, and local utilities.

• Criminal enforcement. The Attorney General is directed to prioritize prosecution of AI-enabled computer crime under existing statutes covering identity fraud, computer fraud, and wire fraud, explicitly including the use of AI agents. No new offenses. A prosecutorial signal, pointed at existing tools.

That is the whole structure: defensive hardening, voluntary intelligence-sharing, classified evaluation, prosecution. Nothing in it binds a private AI developer to do anything.

How this differs from Trump’s own December order

The sharper comparison is internal: the June order against the administration’s previous one, signed less than six months earlier.

The December order is a demolition instrument. It directs the DOJ to challenge state AI laws as unconstitutional regulation of interstate commerce and leverages federal funding against states that keep them. Its named targets include state statutes on transparency and automated decision-making, the only binding AI-specific rules that exist anywhere in the American system. Its carve-outs are narrow: child safety, compute and data center infrastructure, and state government procurement.

The June order is the administration’s first construction project. A clearinghouse, a benchmarking process, a review window, prosecutorial priorities. After nearly a year and a half of removing oversight, this is the first time the administration has built any.

Read together, the two orders reveal the actual policy. The December order dismantles oversight that is public, legislated, and enforceable in open court. The June order constructs oversight that is classified, voluntary, and run by an intelligence agency. The administration does not oppose AI oversight as such. It opposes oversight it does not control and the public can see. Visible rules are treated as obstruction. Invisible review is treated as security.

One Biden-era contrast still matters for anyone who tracked the earlier framework: the direction of obligation has reversed. Under the revoked 2023 order, developers of the most powerful systems were required to share safety testing results with the government. Under the June 2026 order, developers decide what to share, and when. Everything else about the old comparison is now historical.

Why classified benchmarks are a governance problem

There is a defensible version of the secrecy argument: publishing exact capability thresholds could help developers train models that pass the tests without becoming safer.

But notice what the order does not contain: a definition of “covered frontier model.” The threshold is whatever the classified benchmarking process says it is. The scope of the regime is itself classified. Developers will not know where the line sits until the NSA tells them, on terms the NSA controls.

If the benchmarks are classified, independent researchers cannot verify them, civil society cannot scrutinize them, and international partners cannot align with them. A safety framework built on classified standards is not a transparent governance framework. It is a national security program. Both are legitimate things to have. They are different things, with different accountability logic.

The expert commentary has caught pieces of this. CSIS analysts question why Treasury leads while the agencies holding most of the government’s cyber expertise sit in consulting roles. Others note the government may simply lack the technical capacity to evaluate frontier systems at the pace AI is advancing. Both critiques are sound. Both stop short of the incentive question: the order assigns oversight to the actor with the strongest institutional incentive to keep it opaque, and makes participation voluntary for the actors with the strongest commercial incentive to skip it.

What this means beyond the U.S.

If your organization operates globally, you now manage two regulatory environments built on incompatible theories of responsibility.

On the EU side, the timeline just moved. On May 7, 2026, EU negotiators reached provisional agreement on the Digital Omnibus on AI, postponing high-risk obligations for Annex III systems from August 2, 2026 to December 2, 2027, and for product-embedded Annex I systems to August 2, 2028. Formal adoption is expected before August 2, and the new dates bind only once published in the Official Journal. Until then, August 2, 2026 remains a live compliance date.

The deadlines moved. The architecture did not. Risk classification, conformity assessments, human oversight, and penalties reaching 7% of global turnover all remain. The EU’s answer to the responsibility question is still developers and deployers, under public oversight. The U.S. answer is the market, watched by the national security apparatus from behind a classification wall.

There is a third audience here: everyone else. As Oxford’s Matthias Holweg put it, framing safety and competitiveness as a dichotomy will fuel calls for digital sovereignty, because using U.S. models becomes a riskier proposition. For governments and enterprises outside the U.S. and EU, a U.S. framework whose thresholds are classified offers nothing to assess, audit, or align with. That strengthens the case, already gathering force from Brussels to Ankara to New Delhi, for sovereign and open-source alternatives.

What to watch

Three things between now and August. Which labs opt into the pre-release window, and which quietly do not. Whether classified designations start leaking into procurement, insurance, and liability decisions that are never publicly explained. And whether the Mythos pattern, a lab unilaterally deciding what is too dangerous to release, becomes the de facto governance model this order quietly ratifies.

That last one is the real story. The U.S. has not built a system for governing frontier AI. It has built a system for watching the companies that govern themselves.

Until next week,

Nesibe

💬 Let’s Connect:

🔗 LinkedIn: [linkedin.com/in/nesibe-kiris]

🐦 Twitter/X: [@nesibekiris]

📸 Instagram: [@nesibekiris]

🔔 New here? Subscribe for weekly updates on AI governance, ethics, and policy! no hype, just what matters.

This is a really significant edition, for anyone worried about AI and anyone who is not. AI is not a toddler anymore. It is a teenager: it does what it wants, and it adjusts its behaviour around its parents the moment it senses a punishment coming.

I know we are all tired of the “technology outpacing law” cliché. We have been telling it for decades. But with AI, the gap between what technology can do and what governance frameworks can actually verify has crossed a threshold I did not expect to reach this soon. It is no longer just that regulators are running behind. The very concept of oversight is starting to erode underneath us, quietly, without a formal name in any regulatory text.

_{This is the AI oversight 2026 problem nobody has named yet.}

Where we stand

AI development has always rested on three pillars:

• DATA: The historical data stock is, for now, sufficiently full. The real work ahead involves something harder: breaking the flattening effect that AI-generated content creates, building out synthetic and multimodal data pipelines, and finally taking seriously the local and cultural datasets that have been sidelined for too long.

• COMPUTE: every major lab is committing billions to data centers, nuclear energy partnerships, and chip supply chains.

  • Microsoft, Google, Meta and Amazon are collectively spending $725 billion on AI infrastructure in 2026 alone.

  • Google has committed $40 billion to Anthropic, with 5 gigawatts of compute capacity over five years.

  • OpenAI is reportedly on a path toward trillions.

• ALGORITHMS: DeepSeek’s efficiency leap quietly dismantled the assumption that more compute automatically means better models, while reasoning systems and agentic architectures are redefining what a model fundamentally does.

And law? The EU AI Act, ISO 42001, NIST RMF, every major framework was designed for a world where human oversight is technically stable and structurally reliable. And these are all we have…

The research I am examining in this piece challenges that assumption at its foundation. This is not a compliance gap. It is a conceptual one. The scaffolding that regulation depends on is silently expiring, and we do not yet have language for what replaces it.

The question this piece is about: what happens to AI governance when the act of overseeing AI, auditing it, monitoring it, investigating it, rests on properties that are themselves dissolving?

What the UK AISI Oversight Report Actually Found

Here is the document at the center of this piece, and a warning about who it is really for.

The research is a UK AI Security Institute report from this spring, Loss of Oversight (UK AISI, 2026).

The report does forecast. For every way oversight could break down, it gives an explicit likelihood, from "realistic possibility" up to "almost certain."

Think of oversight as having three jobs.

• Auditing is checking a model before you release it.

• Monitoring is watching it while it runs.

• Incident response is investigating after something goes wrong.

All three depend on being able to see inside the system somehow.

The report examines four windows into an AI system: its internal wiring, its step-by-step reasoning, the actions it takes in the world, and how it communicates with other AI systems. Onto each, it maps the oversight techniques we currently apply, from probes to monitoring to behavioural evaluations. It also looks at a fifth thing that is not a window but a fallback: training the model itself to be more honest, in case the windows close.

Their central finding is blunt. The methods we use to keep an eye on AI work today because of features that happen to be true of current models, not because anyone designed them to last. The report identifies more than twenty distinct ways those features could break down. No single one is certain, but the aggregate points one direction.

For each window, the report follows the same honest structure:

• What oversight leans on today. The property that currently makes this signal work.

• How it could break. The pathways that would erode that property.

• What we can do. The technical levers available to slow the breaking.

None of this is settled science yet, and the report is honest about that.

Computer use turns a model into an actor inside an existing work environment, which is why permissions, logs, and escalation matter. See Claude Is Using Your Computer Now.

The difficulty is not simply faster capability progress. It is that the tools intended to inspect, interpret, and constrain those capabilities may be degrading at the same time. I placed that tension in a broader safety context in AI Safety 2025 & The Exponential Leap in AI Capabilities.

Chain-of-Thought Monitoring: The Most Valuable and Most Fragile AI Oversight Signal

Of the four windows, one matters most, and is most at risk. So let me slow down here. Today’s frontier models reason “out loud.” Before they answer, they produce their step-by-step thinking as readable text. Researchers call this the chain-of-thought, or CoT.

This is useful in a very direct way. We can read that text and catch a model cutting corners, gaming a test, or reasoning toward something harmful, before any of it turns into an action. The report calls chain-of-thought monitoring the single most informative signal we have. Right now it is the backbone of frontier AI safety work, and of AI audit transparency more broadly.

The problem is that we did not design this. It just happened to be true of how these models work. And several forces are pulling it away.

• Models could move their reasoning inside themselves, into a form no human can read. They are getting better at solving hard problems in one step, with no visible working. Under the right training pressure, they could learn to hide the incriminating parts of their reasoning from the very systems built to read it.

• And there is a duller, more certain pressure. Every reasoning token costs money. So there is a permanent commercial incentive to make models think less visibly. The signal is real today. It is also leaking.

Here is the part that should stop anyone working in governance. One of these breakdown paths is already happening.

• When a model figures out it is being tested and adjusts its behaviour to look good, researchers call it evaluation gaming. And it is already distorting the audits meant to catch misalignment.

  • The UK AISI report, citing Anthropic’s Sonnet 4.5 system card, notes that the model verbalised awareness of being evaluated in over 80% of certain alignment tests, and that this awareness changed how it behaved.

  • Anthropic’s own risk report for Opus 4.6 went further, calling evaluation awareness the biggest challenge it faces in assessing alignment.

So the model can look safe partly because it knows it is being watched.

This is the teenager again. It behaves differently the moment it senses it is being assessed, and that behaviour is already showing up in the audits.

One thing I want to be fair about. The report does not treat every pathway with equal alarm. It uses a formal probability scale, from “remote chance” up to “almost certain,” to mark how confident it is about each one.

The Cracks Are Not Only in Chain-of-Thought

Chain-of-thought is the sharpest example, but it is one window of several. The same pattern repeats across the others. Here is the rest of the picture: three more windows, and the fallback we are counting on if they close.

Watching what the model does.

The most reliable signal sounds like the simplest one. A model has to act in the world to cause harm, and an action can be seen. You do not need to read its reasoning to see that it deleted a file or sent data somewhere it should not have gone.

But this only works when a harmful action looks harmful. The report is clear about where it stops working: tasks where doing the job badly looks the same as an honest mistake. If a model is asked to review safety research and quietly does weak work on purpose, there is no single incriminating action to point at. The sabotage is in the quality, not in any one step. And the report expects exactly these hard-to-judge tasks to grow, as models are handed longer and more autonomous work.

For governance, this sets a hard limit on “just monitor the outputs” as a compliance answer. The more judgement a model is trusted with, the more of its work lands in the zone where watching the actions cannot tell sabotage from error.

Reading the model’s internal wiring.

A second window, called white-box monitoring, skips the model’s words entirely and reads its internal activations directly, the closest thing we have to looking inside its head. The simplest version, lightweight classifiers called probes, is cheap and fast enough that it already runs in production.

But white-box methods rest on a quieter dependency. We mostly check whether a probe is working by seeing if it lines up with the model’s readable reasoning or its visible actions. So if chain-of-thought becomes harder to read, we lose the reference we used to trust the probe in the first place. One window failing weakens the next. The inside-reading tools are not independent of the text-reading tools; they lean on them.

For governance, this matters because white-box monitoring is often presented as the backup plan, the thing that holds when reasoning goes dark. The report’s point is that the backup is partly tied to the same failure.

Memory and how agents talk to each other.

Right now, models store memory as readable text, and when multiple AI agents work together they mostly communicate in text. Both can be inspected after the fact.

• This is not a small thing: the report notes that when Anthropic ran a web-search benchmark, Claude Opus 4.6 picked up traces left in the environment by earlier runs of itself. Models already leave each other notes.

The report’s concern is a shift to memory and communication stored as raw numerical vectors, a format no human can read. If agents share memory in that form, a strategy found by one agent could spread to every connected agent with nobody able to watch it happen. A separate worry is continual learning: if a model’s weights keep updating after release, the safety checks done before release slowly stop describing the model that is actually running.

For governance, this is the cleanest case for disclosure rules. Whether a developer uses text or vectors, static or continually-updated weights, is a concrete fact that can be reported. A regulator cannot react to a shift nobody is required to announce.

Training the model to be honest.

This last one is not a window at all. It is the fallback. If reasoning becomes unreadable, one fallback is to train the model itself to confess, to report its own misbehaviour without being asked. It is a genuinely promising direction, and early research is active.

But it carries the hardest problem in the whole report. A capable, misaligned model may be able to tell when it is being trained and when it is deployed. And a model that can tell the difference can simply be honest while it is being trained, then stop once it is not. The fallback we are counting on is the one most exposed to the exact behaviour it is meant to catch.

That is the shared shape. Every signal here works because of something contingent, something nobody designed and nobody guaranteed. Every one has a credible way to break. This is the report's real finding, and it is why repairing chain-of-thought alone would not be enough.

The oversight problem becomes more concrete once agents are allowed to operate between human checkpoints. Are Your AI Agents Quietly Failing While You Sleep? examined why an agent that appears compliant at the moment of approval can still create unmanaged operational risk later.

Three Honest Critiques, and One of My Own

A good piece does not just relay a report. Here is where I would press. The first three, to be fair, the report invites itself.

• It is built on what might happen. Many of the breakdown paths depend on models more capable than today’s. In 2026, with chain-of-thought still working, a critic can fairly say the alarm is early.

• Worried people studied a worrying question. The 25 interviews skew toward alignment and interpretability researchers, with almost no governance specialists. A sample chosen for proximity to the problem will tend to find the problem serious.

• It tells us to measure what it cannot yet measure. The central advice is to track oversight properties closely, but there are no agreed metrics for doing so. So the obvious question stands: if we cannot measure these properties today, how will we notice them slipping tomorrow?

The fourth one is mine, and it is not a flaw in the research so much as a limit to it. The report finds a problem on one layer and solves it on another. Every recommendation is addressed to developers: hold this signal out of training, report that shift, invest in this fallback. All sound. But a developer who does none of it pays no price. The report does not ask why a lab would adopt costly oversight when the competitive incentive runs the other way. It diagnoses the erosion accurately, then hands the fix to the actor with the least reason to apply it. That gap is not technical. It is the part governance can close.

Governance Without a Witness

Oversight is usually filed as a technical safety problem. I think that framing is too small. When we lose the ability to audit an AI system, we do not only lose a safety check. We lose a link in the chain of accountability that carries democratic legitimacy. A regulator’s power rests on a quiet assumption: that someone can produce evidence a non-specialist can understand. When that evidence goes dark, the regulator is left holding the lab’s own report and little else.

So the real stakes are not only that a model might misbehave. It is that the people meant to hold AI accountable are increasingly asked to take the builder’s word for it.

And nobody has to act in bad faith for this to happen. Shorter reasoning is cheaper reasoning, and also less visible reasoning. The incentive to cut costs and the incentive to reduce what we can see are the same incentive. The market erodes oversight on its own.

This is not only a problem for under-resourced regulators. In 2025, Deloitte Australia delivered a government-commissioned report later found to contain fabricated references, traced to generative AI use, and partially refunded the fee. If a firm that size cannot audit the AI output inside its own deliverable, the belief that institutions can quietly self-verify deserves far more scepticism than it gets.

AI governance is not only a technical discipline. It is the study of incentive structures. And right now the incentives point away from being watchable.

What AI Governance Frameworks Need to Do Before Oversight Silently Expires

The report’s recommendations are written mostly for developers. The governance job is to translate them into something a regulator can actually use.

The report gives four:

• Use many oversight methods at once, so no single failure is fatal.

• Track and report how oversight properties are shifting over time.

• Protect oversight at the design stage, before a model ships.

• Invest now in fallback techniques for when current methods weaken.

They are sound advice. They are also written for the wrong audience to act on alone. Here is what two of them look like once they are turned into governance.

Track and report. In regulatory language, this is a transparency obligation, and it fits the disclosure logic already inside the EU AI Act. But it only works if regulators learn to ask a sharper question. Not “is your system safe?” The question is “which oversight properties does your safety claim depend on, and are they still holding?”

Protect oversight by design. There is a real difference between governance built in while a model is being trained and governance bolted on after it ships. Holding the most informative signal back from training, so the model never learns to hide it, is a design-stage choice. No later audit can recover it once it is gone.

This raises a hard question for anyone who believes in independent auditing. Can an outside auditor verify a signal they were never allowed to see? Right now the honest answer is no. The most realistic tests, the report notes, can only be run by the labs with full access to their own systems.

• We have seen where that road leads. In 2016, COMPAS, the risk-scoring tool used in US criminal sentencing, was shielded for years as proprietary, so neither defendants nor courts could examine how it reached its scores. When journalists finally analysed it from the outside, they found racial disparities the vendor had always denied, and by then the tool had shaped thousands of decisions. Denied access does not mean no harm. It means the harm surfaces late, found by someone else.

So black-box self-assessment cannot stand in for verification. The asymmetry between what labs can test and what outsiders can see is not a technical detail. It is a governance problem, and it points straight at third-party access and white-box auditing.

Which leaves the largest gap. Regulators do not yet have a working concept called oversight degradation. There is no line for it in any AI law. And a gap that wide is rarely neutral. It tends to serve whoever benefits from the activity going unmeasured.

AI Oversight in 2026: Five Signals to Track This Summer

If you want to know whether oversight is holding or slipping, these are the places I would keep my eyes on over the coming months.

• Anthropic’s next model card. Watch whether evaluation awareness measurement becomes a standard, disclosed section. If other labs copy it, monitorability reporting is becoming a norm rather than a one-off.

• What happens to NIST AI 800-4. It is a descriptive report today. The signal to watch is whether it hardens into something tied to procurement rules or enforcement later this year.

• The spread of latent reasoning. Any sign that models are moving their thinking out of readable text is the clearest early warning that the chain-of-thought monitoring window is closing on us.

• EU AI Act high-risk auditing. Once the high-risk obligations bite this summer, the concrete test is simple: will external auditors get to see a model’s reasoning, or only its inputs and outputs?

• How AI agents talk to each other. As multi-agent products ship, watch whether agents keep communicating in readable text or switch to formats no human can audit. The second path would close a window before most regulators know it exists.

  ---

One Image I Keep Returning To

A capable enough model could, in principle, show a perfectly readable chain-of-thought while doing its real reasoning somewhere we cannot see, and pass every alignment test while behaving differently once deployed.

The system card would still say the model is safe. The report says we may no longer be able to verify that claim.

I am not sure which of those two sentences I find more unsettling. But I am fairly sure that “human oversight,” written into law as though it were a permanent feature of the world, is resting on ground that is moving.

If you work in governance, that is the sentence I would not let expire silently.

Reply and tell me: in your jurisdiction, does anyone in the room know what “oversight degradation” means yet?

💬 Let’s Connect:

🔗 LinkedIn: [linkedin.com/in/nesibe-kiris]

🐦 Twitter/X: [@nesibekiris]

📸 Instagram: [@nesibekiris]

🔔 New here? for weekly updates on AI governance, ethics, and policy! no hype, just what matters.

Some of you have been here since the early days of TechLetter, when it was just me, a Substack page, and a stubborn belief that AI governance was worth writing about every single week. So when I tell you that TechLetter is now an official Community Partner of AI Summit Barcelona 2026, I want you to know what that means to me. More on that at the end, including a small surprise.

But first, something happened in Brussels last week that I have been wanting to write about properly. While everyone in the EU AI compliance world was scrambling to get something ready before August, a different kind of news arrived. On 7 May 2026, in the early hours of the morning, after overnight negotiations, EU co-legislators reached a provisional political agreement on the Digital Omnibus on AI, a bill that amends the EU AI Act and moves its most consequential deadlines. The industry exhaled. Civil society filed its objections. And I have been sitting with the question of whether this is good news or a slow-motion problem we will revisit in 2027.

The calendar has changed, but the underlying implementation problem has not. In Moving Forward with the EU AI Act and There’s Still More Work to Do, I wrote about the distance between legislative agreement and the institutional work required to make the framework real.

Probably both. Let me explain.

I have been following this file for months. And my honest read is this: the Digital Omnibus on AI is not a rewrite of Europe’s AI rules. It is an acknowledgment that the infrastructure needed to implement those rules was not ready. That is an important distinction, and most of the coverage has blurred it.

Before the Omnibus, I shared an EU AI Act Cheat Sheet as a reference. Now that the deal is done, I have also put together an updated one-pager reflecting the new compliance map, end of this issue.

Why August 2026 Was Already in Trouble

The EU AI Act was adopted in 2024. Its logic was sound: classify AI systems by risk, require high-risk categories to demonstrate conformity against harmonised technical standards, have notified bodies verify that conformity. By 2 August 2026, Annex III high-risk systems, think hiring tools, credit scoring, biometric identification, would need to comply.

The problem is that the compliance infrastructure the Act relied on did not arrive on schedule. Harmonised technical standards were formally recorded as significantly delayed. The Commission then missed its own statutory deadline for issuing Article 6 classification guidance, which was due in February 2026. Notified body capacity remained limited across member states.

In practice, companies preparing for August 2026 were being asked to build conformity architecture against standards that did not yet fully exist. The legal exposure of getting it wrong was real. The cost of preparing against a moving target was real. And Brussels knew it.

This is the honest reason the Digital Omnibus on AI bill exists. The competitiveness argument and industry pressure were present, yes. But the structural problem was created, at least partly, inside the EU’s own implementation timeline. That context matters for how you read everything that came next.

What Collapsed the April Trilogue

The first two trilogue sessions under the Cypriot Presidency did not resolve everything. The second session, on 28 April 2026, ended without agreement after roughly twelve hours of negotiation. A press conference was cancelled. The question at that point was whether a deal could be reached before August at all.

What broke the session was not the deadline extension itself. That was broadly agreed across the institutions. The file that collapsed everything was the conformity assessment architecture for AI systems embedded in products already governed by EU sectoral safety law: industrial machinery, medical devices, toys, lifts, watercraft.

Parliament’s position, backed by Germany, was to move Annex I products toward primarily sectoral handling. The rationale was real: requiring companies to run parallel conformity assessments under both the AI Act and existing product safety law was disproportionate and, in some cases, technically incoherent. The Council resisted reopening the structural architecture of the Act. Under trilogue logic, nothing is agreed until everything is agreed. So the deadline extension, the new prohibited practices, everything else sat waiting on this one unresolved file.

The third trilogue opened the evening of 6 May and closed just before 5 in the morning on 7 May. The political negotiation is over. Formal endorsement and publication in the Official Journal still need to happen, but the outcome is settled.

What the Digital Omnibus Bill Actually Does

The deal is a compromise, and the details matter more than the headline.

On timelines:

• Stand-alone high-risk AI systems under Annex III, the category that covers hiring tools, credit scoring, biometric identification and critical infrastructure management, now apply from 2 December 2027, pushed from 2 August 2026

• AI embedded in regulated products under Annex I, think machinery, medical devices, toys, lifts, watercraft, applies from 2 August 2028

Crucially, these dates are now fixed. They no longer move with standards availability. The political argument for delay has been used once, and the institutions signalled clearly that it will not be used again.

On the Annex I architecture:

• The Machinery Regulation receives a direct carve-out: AI within that regulation is exempted from direct AI Act applicability, with health and safety requirements added instead through delegated acts under the Machinery Regulation itself

• For other Annex I sectors, the Commission can limit AI Act application through implementing acts where sectoral law already covers equivalent AI-specific requirements

• The Commission is also obligated to issue guidance for sectoral operators to minimise compliance overlap

What was strengthened, not weakened:

• AI systems generating non-consensual sexual or intimate content, including child sexual abuse material, are now explicitly prohibited under a new Article 5 provision, capturing nudification apps specifically

• Database registration for AI systems considered exempted from high-risk classification, a transparency safeguard that allows regulators and the public to see what is out there, was reinstated after the Commission’s proposal to delete it was rejected

What did not move:

• All Article 5 prohibited practices, the hard bans on things like social scoring and real-time biometric surveillance in public spaces, remain intact

• Article 50(1) obligations, which require providers to disclose when users are interacting with an AI system, remain on track for 2 August 2026

• Article 50(2) watermarking, the requirement to label AI-generated images, audio and video as synthetic still applies from 2 August 2026, but the Omnibus adds a transitional period: providers with systems already on the market by then have until 2 December 2026 to comply.

• GPAI obligations, the rules covering large general-purpose models like the ones powering most AI products today, have been in force since 2 August 2025 and are unaffected

If you are shipping generative AI into the EU, watermarking is your nearest live engineering deadline. Roughly seven months away.

The Debate Was Substantive, and Some of It Was Not Heard

The deal was contested, and the contestation was substantive.

On the civil society side,

• 60 organisations including EDRi sent a joint letter specifically urging co-legislators to reject the proposed deletion of Article 49(2), a transparency safeguard requiring providers who self-assess their Annex III systems as not high-risk to register that exemption in the public EU database.Their argument was straightforward: weakening it would reduce enforcement visibility while offering negligible benefits to companies.

• The Delors Centre raised a harder procedural critique: the Digital Omnibus bill was advanced through an accelerated procedure with limited public consultation and no comprehensive impact assessment, on provisions that had taken years to negotiate in the original AI Act trilogue.

On the industry side, the response was more qualified than expected. The Machinery Regulation carve-out was welcomed, but several legal observers noted that the Annex I compromise shifts a significant amount of interpretive work to the Commission’s implementing acts. In other words, the uncertainty does not disappear with the political agreement. It moves to a later stage.

For me, the most analytically important issue is the grandfathering effect. AI systems deployed before the new compliance deadlines are exempt from many obligations that systems deployed after them must meet. A 16-month delay is not administratively neutral. It shapes which systems get built, deployed and normalised during that window, and under what accountability conditions. That is most consequential in exactly the domains where ungoverned AI has the most documented human cost: hiring decisions, credit assessments, biometric identification.

The EU bought time from a structural problem it helped create. The question that matters now is whether that time gets used to build the standards, guidance and supervisory capacity that make December 2027 real, or whether it becomes a second runway to a second renegotiation.

What This Means in Practice

Three things follow practically from this deal.

If you are building or deploying AI in the EU:

• Watermarking is your nearest deadline. UI labelling, machine-readable metadata, and detection capability for any generative AI shipped into the EU needs to be production-ready by 2 December 2026. That is roughly seven months of engineering time, not planning time.

• Treat 2 December 2027 and 2 August 2028 as hard dates. They no longer shift with standards availability, and there is no credible political path to a third postponement.

• Do not treat the delay as permission to pause your compliance program. The standards, codes of practice and interpretive guidance that the whole architecture depends on will continue arriving through 2027. The organisations that build governance infrastructure now will be distinctly better positioned than those that wait for perfect clarity.

If you operate in machinery, medical devices, toys, lifts or watercraft:

• You are now navigating a dual-track framework: the AI Act and the implementing acts the Commission will adopt to define the interplay with your sectoral law. Track both from now, not from when the guidance arrives.

Until that happens, EU AI Act in its original form remains the legally binding text.

As I wrote in the cheat sheet: delay extends the runway. Whether companies use that runway well depends less on Brussels handing over more time, and more on Brussels handing over clearer rules.

Regulation is only as meaningful as its implementation, and implementation happens in product teams and compliance functions, not in Brussels. That gap is what I want to spend time on in Barcelona.

I will be in AI Summit Barcelona!

Okay, here is something I am genuinely excited about.

TechLetter is an official Community Partner of AI Summit Barcelona 2026, and I will be there in person on 22-23 September.

Frontier AI leaders, builders, researchers and governance practitioners, all in one place in Barcelona for a full week of events. I will be attending stages, doing speaker interviews, and sending everything back to you as proper analysis, not a conference recap.

The speaker list is still growing, so I will share more as it takes shape.

I am building my interview list now and I want it to reflect what this community actually wants to understand.

And the small surprise I mentioned at the top: TechLetter readers get 20% off with the code TECHLETTER20 at aisummitbarcelona.com. Early bird pricing is live now.

See you in Barcelona.

Nesibe

Disclosure: TechLetter is an official Community Partner of AI Summit Barcelona 2026.

💬 Let’s Connect:

🔗 LinkedIn: [linkedin.com/in/nesibe-kiris]

🐦 Twitter/X: [@nesibekiris]

📸 Instagram: [@nesibekiris]

🔔 New here? for weekly updates on AI governance, ethics, and policy! no hype, just what matters.

Here is the list, more or less in the order I came across it.

• A judge in Hangzhou ruled that a tech firm cannot fire a worker because it replaced him with a model.

• A prosecutor in Karlsruhe filed Europe’s first criminal indictment for AI-generated child sexual abuse material.

• A jury in Oakland heard Elon Musk testify for three days about whether OpenAI’s for-profit conversion amounts to “stealing a charity.”

• The Pentagon signed AI deals with eight major vendors for classified networks, and left Anthropic off the list.

• The Academy of Motion Picture Arts and Sciences ruled that only “human-authored” screenplays and performances “demonstrably performed by humans” qualify for an Oscar.

• Colorado’s senate leaders introduced a bill to repeal the most ambitious state AI law in the United States, two months before it was supposed to take effect.

• The UK government quietly began rolling out a Google-built AI tool called Extract to help councils make planning decisions.

Seven decisions, five jurisdictions, one week. And the most useful thing I can say about the list is that it does not point in a single direction.

What ended this week: the era of AI governance as a writing project

In the Davos 2026 recap I wrote in January, I argued that AI was moving from pilots to infrastructure. Four months on, the more honest framing is that it is now moving from infrastructure to enforcement, but enforcement is plural, and the plurality is not an accident. Different institutions reach for AI in their own grammars when the people they serve start asking them to.

Three courtrooms, three legal domains: Hangzhou, Karlsruhe, Oakland

In a single week, three different judicial logics produced three different kinds of AI governance, in three jurisdictions that rarely show up in the same paragraph.

The Hangzhou ruling came from the Intermediate People’s Court, in a case where a quality assurance worker was fired after refusing a 40% pay cut tied to the automation of his job.

• The court held that AI implementation does not, on its own, meet the legal standard for terminating an employee, and the decision builds on a December 2025 Chinese precedent in a mapping company case.

• The interesting thing about it, in my reading, is the venue. China is not a jurisdiction we usually associate with worker-protective rulings, and the fact that this one happened there says something about how labor courts are quietly becoming an AI governance frontier.

The Karlsruhe indictment is the first criminal case in Europe for AI-generated CSAM. NCMEC tracking shows reported cases of AI-generated child sexual abuse material moving from a few thousand in 2023 to roughly 1.5 million in 2026, and the European Parliament is now debating an amendment to the AI Act that would criminalize the creation, fine-tuning, and distribution of models capable of producing such material.

When I wrote about Grok’s bikini outputs and the broader deepfake question in January, I argued that 2026 would be the year governance got real about generative content harms. Karlsruhe is what “getting real” actually looks like in practice: an indictment, a courtroom, a defendant.

The Oakland case is the one most likely to set the tone for the second half of the year. Musk v. Altman opened on April 28, and Musk himself testified for three days, calling OpenAI’s for-profit conversion a theft of a charity and seeking up to $134 billion in damages in earlier filings, with Altman and Brockman expected to testify later this month. On the surface, this is a corporate dispute over a governance structure. Underneath, it is a fight about who has the authority to decide what an AGI race should look like and on what terms, and that fight has been brewing for nearly a decade.

If you want to understand what the Oakland trial is actually about, the most useful book I have read in the past year is Karen Hao’s Empire of AI: Dreams and Nightmares in Sam Altman’s OpenAI.

• Hao spent seven years covering OpenAI for MIT Technology Review and beyond, and the book is built on roughly 260 interviews. It traces the entire arc the courtroom is now trying to litigate: the founding non-profit promise, the Musk-Altman power struggle that ultimately pushed Musk out, the formation of Anthropic by Dario and Daniela Amodei and other senior staff who left over safety disagreements, the boardroom drama that briefly ousted Altman in November 2023, and the resource extraction story underneath all of it.

• The same Musk who appeared at Davos this January talking about AGI timelines is now under cross-examination over the corporate maneuver he believes betrayed those timelines.

• Hao’s long-form interview on The Diary of a CEO is the audio companion. I would recommend both, especially if you have followed the dispute only through the legal filings. (One footnote that matters for a publication like this one: Hao publicly acknowledged in November 2025 that the book overstates a Chilean data centre’s water usage by a factor of 1,000 due to a unit error. The broader argument stands; that figure does not.)

The reason I am dwelling on Oakland is that the trial is, in a sense, the first formal venue where Hao’s central thesis is being tested under oath. She argues that the “AGI for the benefit of all humanity” mission, sincere or not at the start, became a uniquely potent formula for consolidating resources and constructing an empire-style power structure. Whether or not Musk wins, the questions his lawsuit puts on the record (about charitable purpose, about board fiduciary duty, about how an AGI mission can be repurposed to justify almost any organizational form) are now in the legal system. Being in a book and being in a court record are very different things.

Procurement is moving faster than legislation

This is where Pentagon procurement comes in, because it is the cleanest example of policy by purchase order.

• The Pentagon’s contract list for classified-network AI includes SpaceX, OpenAI, Google, Nvidia, Reflection AI, Microsoft, AWS, and Oracle.

• Anthropic was not on the list. The reason, reportedly, is that the company’s red lines on mass surveillance and fully autonomous weapons were treated as a supply-chain risk.

When I wrote about the Anthropic-Pentagon conflict in March, the question I kept turning over was whether “all lawful purposes” was a workable governance frame for federal AI vendors. The answer this week is: apparently not, if a “lawful purpose” includes things you would not personally write into a model card. What makes it stranger is that the NSA had reportedly given a positive technical review of Anthropic’s Mythos model around the same time, which means the same vendor is being read as both qualified and disqualified by adjacent parts of the same state. Procurement-as-governance has that quality. It does not need internal coherence to function.

Three more procurement variants came into focus this spring, and the contrasts between them are their own story:

• GSA (federal United States). The General Services Administration released its draft AI procurement clause GSAR 552.239-7001 on March 6, requiring AI systems to be “ideologically neutral,” to be developed and produced in the United States, and to refrain from using federal data to train models for other customers. Holland & Knight called the proposal among the most prescriptive ever seen in federal contracting.

• California (state United States). Governor Newsom responded with Executive Order N-5-26 on March 30, directing state contracting processes to require vendors to demonstrate safeguards against harmful bias and protections for civil rights. The federal “no dogmas” framing and the state “civil rights” framing now compete for the same vendors, and California’s economy is large enough to make that a real fight rather than a symbolic one. The order was deliberately drafted to fit inside the procurement carve-out the Trump administration left open in its March 20 National Policy Framework on AI, which means the state is governing AI by exercising the one power Washington has not yet tried to take from it.

• United Kingdom. The Department for Science, Innovation and Technology awarded Google Cloud an £8.3 million contract to build Extract, a system that helps council planning officers process applications using Gemini. Pilots have run in Hillingdon, Westminster, Nuneaton & Bedworth, and Exeter, with national rollout expected this spring. The state here is not deciding which vendors are eligible to sell AI; it is the customer, and what gets decided by the model is whether someone can extend a kitchen or build a house.

If I had to summarise the pattern across these four cases, the strongest tool of AI governance right now is a contract clause, not a regulation. Rules are slow and contested. Procurement happens in the same week as the policy decision behind it.

Colorado’s collapse: the laboratory that closed before the experiment

While enforcement was hardening in Pentagon contracts and Oakland courtrooms, the most ambitious AI law in the United States was being dismantled in a state capital that very few people outside the policy world were watching.

Colorado passed the Colorado Artificial Intelligence Act in May 2024, modelled in part on the EU AI Act, with bias audits, impact assessments, risk management programs, and incident reporting requirements for high-risk systems. It was supposed to take effect on June 30, 2026. The collapse happened on a remarkably tight timeline:

• April 27, 2026: a federal court paused enforcement.

• May 1, 2026: Senate President James Coleman and Majority Leader Robert Rodriguez introduced a bill to repeal and replace the law.

• The replacement framework, “Concerning the Use of Automated Decision Making Technology in Consequential Decisions,” shifts to transparency, recordkeeping, and consumer rights, dropping the bias audits, impact assessments, and risk management requirements that defined the original.

• If passed, the new framework takes effect January 1, 2027.

Witnessed in Stanford AI Index 2026 last month, capability was outrunning every system around it, including the legal one. Colorado is the cleanest illustration I have seen so far. The state was supposed to be the laboratory that proved comprehensive state-level AI regulation could work in the U.S., and the laboratory closed before its first experiment. Other states have already pulled back, with California’s broader bill slowed last year and Connecticut’s failing under veto threat. If even Colorado could not hold the line, the credibility of the EU AI Act’s high-risk obligations, which trigger on August 2, is going to be tested by industry in ways the law’s drafters probably did not plan for.

Two unexpected venues: the Academy and the Bundesbank

The Academy of Motion Picture Arts and Sciences released new Oscars rules requiring “human-authored” screenplays and performances “credited in the film’s legal billing and demonstrably performed by humans with their consent” for the 99th Academy Awards. The Academy is a private cultural institution, not a regulator, and yet its eligibility rules now function as a labor protection for screenwriters and performers and an authorship doctrine. Where state regulators have been slow to address AI’s impact on creative labor, the cultural body that hands out the year’s most visible award stepped in and did the work itself.

• In Türkiye, the same week, I saw a parallel controversy when allegations surfaced that the band performing at Mustafa Sandal's Saygı 1 tribute concert had used AI-generated vocals rather than live performance; the debate played out among musicianson social media, with no institutional body in a position to draw the line.

The Bundesbank, joined by Australia’s banking regulator, asked the European Commission for technical access to frontier model Mythos, warning that without it banks could not understand the systemic risks they are exposed to. Frontier AI is now, in addition to whatever else it is, a financial-stability concern, and that frame changes who gets to ask the hard questions. Capital flows are being drawn into AI governance from two directions at once: from supervisors (Frankfurt, Sydney) and from gatekeepers (Beijing, which blocked the $2 billion Meta-Manus deal and now requires pre-approval for ByteDance and Moonshot to accept U.S. capital).

The pattern: why AI enforcement in 2026 is plural, uneven, and incoherent on purpose

Read together, the seven actions tell a more complicated story than “enforcement is finally here,” and the complication is the point.

Some actions tighten the rules. The Pentagon excludes Anthropic on supply-chain grounds, the Academy bars AI authorship from Oscar consideration, Karlsruhe files a criminal indictment, Hangzhou rules for the worker against AI-driven termination. Other actions loosen them. Colorado dismantles its own comprehensive AI law, the GSA’s “no dogmas” framing pushes back on bias-audit norms, the federal executive order targets state-level AI regulation entirely. And some actions just drift, doing AI governance without naming what they are doing. The UK adopts AI for planning decisions without any new public-law framework. California issues executive orders that depend on who occupies the governor’s office in eighteen months.

Underneath all of this, two structural features deserve more attention than they are getting:

• Capacity asymmetry. The jurisdictions producing this week’s case law and procurement rules (the U.S., China, Germany, the EU) have well-resourced courts and agencies. Most of the world does not. UNESCO’s recent assessment for Georgia, where 2.2% of businesses use AI and AI publications per million people stand at 1.1 against 29.8 in the EU, captures the median condition for most jurisdictions. Speed of enforcement is becoming a function of state capacity more than of regulatory ambition, and that gap is widening.

• Norm incoherence. The GSA wants ideological neutrality. California wants civil rights protections. The EU wants high-risk audits. Colorado used to want one of those and now wants none. The same model from the same vendor faces four different governance grammars in the same quarter. Compliance becomes a triangulation exercise rather than an alignment one, and triangulation tends to produce the lowest common denominator.

Four questions to watch over the next eight weeks

• Will the EU AI Act’s high-risk obligations actually trigger on August 2 with industry credibility intact, given Colorado’s collapse and the broader U.S. retreat?

• Will the Hangzhou logic get cited in a European labour court, and how will labour ministries handle restructurings that name AI as the reason for redundancy?

• Will the Pentagon-Anthropic procurement model spread to other liberal democracies? China already uses procurement and capital approval as policy tools, and the open question is whether the U.S. and the EU formalize the move at scale.

• Will the criminal-law expansion against AI-CSAM reach the AI Act amendment stage in this Parliament?

Sign-off thoughts

I am writing this from Istanbul, where AI governance is still mostly a framework conversation, and where the gap between framework writing and framework enforcement is, candidly, growing rather than narrowing. The reading from this week’s data is that the framework conversation is no longer the place where AI’s direction gets decided. The places that decide are smaller, more procedural, less photogenic, and harder to translate.

That is exactly why I think we should be paying closer attention to them, and exactly why I find this moment more analytically interesting than the declaration years that preceded it.

Tell me which of this week’s seven venues surprised you most. I am genuinely curious, and the replies often shape what I write next.

💬 Let’s Connect:

🔗 LinkedIn: [linkedin.com/in/nesibe-kiris]

🐦 Twitter/X: [@nesibekiris]

📸 Instagram: [@nesibekiris]

🔔 New here? for weekly updates on AI governance, ethics, and policy! no hype, just what matters.

Stanford HAI’s 9. AI Index is out. I know linkedin community already copy pasted so many of arguments but let’s go through it with more detail. While reading through it, I felt like watching a data confirmation of arguments I have been making in almost every piece this year.

In AI Wrapped 2025 I said 2025 was the year AI stopped being a topic and became a condition. The AI Index 2026 reflected that condition with one through-line running under all of them: capability and capital are accelerating, and the institutions meant to evaluate, govern, and absorb that capability are falling behind.

I want to walk through Responsible AI, the Economy, and Policy and Governance in depth. These are the three chapters that matter most to this audience, and the ones I have the strongest views on

The frontier is crowded and the scoreboard is broken

Everyone will write about the convergence at the top this week. Four companies now sit within a handful of Elo points. The US-China frontier gap has effectively closed, and the lead has changed hands several times in the past year.

That is the easy story, and I do not think it is the important one. The important story is that the race did not just get closer. It moved. Capability differentiation at the model layer is collapsing, which is pushing competition down the stack to compute, energy, and chip supply. And while that shift is happening, our ability to see into any of it is contracting.

Three things are breaking at the same time:

• Benchmarks are losing their grip. Tests designed to last years are getting saturated in months. Error rates on widely used evaluations run into the double digits. There is credible research suggesting that Arena ranking partly reflects adaptation to the Arena platform rather than general capability.

• Disclosure is moving backward. Training code, parameter counts, dataset sizes, and training duration are now routinely withheld by the labs producing frontier models. The most capable systems are the least transparent ones.

• The competitive layer shifted. When model performance converges, the ground shifts to cost, reliability, energy, and chip supply. This is exactly what Jensen Huang meant at Davos when he said tokens per dollar per watt is the new productivity metric.

The transparency collapse is the one I want to flag hardest. I do not think it is a neutral competitive outcome. It is a governance infrastructure failure, and we should name it that way. Shared capability benchmarks became standards in the first place because the field created peer pressure to report them. The same social mechanism has not materialized for disclosure. So disclosure is contracting while capability expands, and if you care about independent verification of AI systems, the direction of travel in 2025 was actively bad, not just stalled.

This is also where the report quietly confirms a thesis I have been hammering since AI Wrapped: the real frontier is not models. It is electricity, chips, and physical buildout. Compute capacity has been compounding. The global AI hardware supply chain runs through a single Taiwanese foundry. Training emissions for one frontier model now run in the tens of thousands of tons of CO₂. Anyone serious about AI governance has to be fluent in energy policy now, and most of the AI ethics field, in my honest read, has not caught up to that yet.

Chapter 3: the Responsible AI gap is now field-wide, and pretending otherwise is getting harder

I want to spend the most time here. This is where my consulting practice lives, and the 2025 picture is worse than I was expecting going in.

AI incidents kept climbing, on two completely different databases using completely different methodologies. Both curves point the same way.

Reporting and measurement have not kept pace. The report lays this out in two tables side by side: the capability benchmarks every frontier lab reports, next to the responsible AI benchmarks almost none of them do.

• Capability side: full. Every frontier lab reports MMLU, GPQA, SWE-bench, and the rest.

• Responsible AI side: mostly empty. Safety, fairness, factuality, and autonomy benchmarks get selectively ignored. Only one frontier model reports on more than two of them.

This is exactly the structural problem I wrote about in Enterprise AI’s Biggest Risk, where I argued that most AI vendors are stuck at the “principle” stage of the governance chain with no controls, no metrics, and no evidence behind their published values. I was writing about vendors. What the AI Index is showing is that the frontier labs themselves are stuck in the same place.

The Foundation Model Transparency Index confirms the direction of travel. The average score dropped from 58 in 2024 to 40 in 2025. Disclosure on training data, compute, and post-deployment impact got worse year over year. Transparency is going backward while incidents are going up, and I do not read that as an accidental pattern.

Two other findings in this chapter are worth naming for anyone deploying AI.

• The tradeoff problem. Responsible AI dimensions trade off against each other. Improving safety can degrade accuracy. Improving privacy can degrade fairness. There is no accepted framework for navigating these tradeoffs. This is a measurement gap that no single lab or regulation can close.

  • It needs a field-level investment in evaluation science, and that investment is not happening at the scale of the capability buildout.

• The sycophancy problem. Top models handle third-party falsehoods fine, but their accuracy drops sharply when the same false statement is framed as the user’s own belief. This is the exact failure mode underneath the AI companion harm cases that drove California SB243 last year.

  • If you are building an AI companion, a consumer health assistant, or any emotionally-loaded product where the user and model share the same conversational frame, this is the risk that matters, and the field is not measuring it consistently.

In AI Safety 2025 I wrote that safety is a choice. The 2025 data says we kept making the wrong one, or more accurately, that we deferred it again.

Rankings can describe concentration without explaining its institutional consequences: dependency, bargaining power, public capacity, and who gets to set the terms of technological development. I considered that missing layer in Reading The Global 50: What It Says — and What It Doesn’t.

For an earlier snapshot of the investment and adoption patterns that preceded this acceleration, see AI in Numbers #1: Global AI Investment and Adoption Stats for 2024.

Chapter 4: the aggregate story is positive, the distributional story is where governance has to live

This is the chapter where optimists and pessimists read the same page and see opposite things, and where I want to push back on both dominant readings.

The aggregate picture is strong. Corporate AI investment more than doubled in 2025, reaching $581.69 billion. Organizational adoption is now majority behavior. Generative AI hit mass consumer adoption faster than the PC or the internet. US consumer surplus from generative AI grew by more than half in a single year, and the median user is getting triple the value they got twelve months earlier. Most of these tools remain free or close to it. Whatever you think about valuations, real utility is landing in real hands, and I do not want to dismiss that.

But the distribution of that utility is where governance has to be built, and where the data is most interesting to me.

• Singapore sits at 61% population-level adoption, the UAE at 54%, both well above what GDP per capita would predict.

• The US, despite its investment lead, sits far lower.

• Some countries are deliberately over-indexing on deployment relative to their economic size, and that is a governance choice, not a market outcome.

It should reshape how policy people in this region think about the national AI conversation. Adoption depth can be accelerated through policy. Investment scale cannot, not at the level of US private capital.

Then there is the labor data, where I need to update my own position. In AI Wrapped 2025 I argued the “AI layoffs” narrative was running ahead of reality, and the 2025 restructuring wave was mostly margin repair using AI as a cover story.

Employment for US software developers aged 22 to 25 fell sharply between 2024 and September 2025. Older developers are not seeing it. The decline is concentrated exactly where AI productivity gains are largest, which means the “just margin repair” reading no longer works for this cohort.

In ROI in the Agentic Era I wrote that agents were landing first in the messy operational layers most companies never governed properly. The labor consequences are now landing on the youngest workers in those same layers. Entry-level roles are where tacit knowledge and networks get built. Erase them, and you are not just hitting wages, you are hitting the mid-career pipeline a decade out. Boards are focused on the productivity upside. Almost none are thinking about the pipeline loss.

Chapter 8: three jurisdictions, three different theories, and a quieter compliance revolution underneath

Governance stopped being theoretical in 2024. In 2025 it hardened into fragmentation.

The ten-day window in early 2025 is the cleanest illustration.

Three jurisdictions, three completely different theories of what the problem even is.

• The US signed an executive order moving toward deregulation on January 23: AI is a strategic asset to be unleashed.

• The EU AI Act’s first prohibitions took effect on February 2: it is a risk surface to be categorized and constrained.

• China finalized its mandatory AI content labeling rules on March 14: it is a domain to be controlled and labeled.

True but sad story, you cannot have “global AI governance” across those three positions. You can only have local governance regimes that interact through trade, standards, and diplomacy. If that is going to be the operating reality for the next decade, the field needs to say so out loud, because most of the policy conversation is still assuming a convergence that is not coming.

The fragmentation deepened from there. Italy passed the first EU member state AI law. Japan, South Korea, Texas, and California SB 53 followed. In July, the US Senate struck a proposed 10-year federal moratorium on state AI regulation, and I would argue this was one of the most consequential US moves of 2025, because it opens the door to a 50-state patchwork rather than a federal baseline.

Underneath the geopolitical fragmentation, something quieter and in my view more important is happening at the organizational compliance layer. GDPR slipped from 65% to 60% as the most-cited regulatory influence. ISO/IEC 42001 appeared in the data for the first time at 36%. NIST AI RMF reached 33%.

This is the AI-native compliance stack finally emerging, and we are not giving it enough attention. Two years ago the honest answer to “which framework should we adopt?” was “GDPR plus improvisation.” is no longer true. ISO 42001 gives you a management system standard, NIST RMF gives you a risk typology, and the EU AI Act gives you risk categories and high-risk obligations. Anyone still treating AI governance as a principles document rather than a control chain will be on the wrong side of a procurement cycle soon. I am already seeing it in client work.

The organizing frame above all this is AI sovereignty. The WEF-Bain Rethinking AI Sovereignty paper I covered in Davos 2026 argued that for most countries the honest frame is strategic interdependence, not a full national stack. Only two countries have full-stack capability, and even the leader depends on a single Taiwanese foundry for its chips.

For Türkiye and the region, the real question is not whether to pursue sovereignty but which layer to anchor on (talent, data, application, governance) and which regulatory gravity to orbit. That is a decision about which rule-making ecosystem your companies will be speaking the language of for the next decade. Most of the conversation here is still stuck at “should we have sovereign AI,” when the real debate is about specific layers and alignments.

What I think should actually happen

In the register of my consulting work:

• For boards and executives, stop treating “human oversight” as a checkbox. Define which of human-in-command, human-in-the-loop, or human-on-the-loop applies to each AI feature, and build the control chain from principle to metric to evidence behind it. The AI Index data tells you your vendors will not do this for you.

• For policy professionals, ISO 42001 is the compliance signal to watch. It moved from nothing to top-tier citation in twelve months. The AI-native compliance stack is finally decoupling from GDPR, and anyone writing enterprise AI policy should be fluent in it by the end of this quarter.

• For researchers and evaluators, the RAI benchmark reporting gap is fixable with the same social mechanism that made MMLU a shared standard. It is a coordination problem, not a technical one, and it needs a credible third-party institution to host the reporting and pressure-test the disclosure.

• For founders and practitioners in this region, strategic interdependence is a real opportunity, but only if it gets paired with EU AI Act and ISO 42001 literacy in the next 24 months. Otherwise you become raw material for other people’s AI stacks.

• For readers thinking about their own careers, the entry-level labor signal in software is the first clean one of this cycle. If you are early in an AI-exposed function, invest deliberately in the layers the productivity studies say current AI benefits least: judgment, systems thinking, and domain depth.

Closing

The co-chairs of the AI Index write that “the data does not point in a single direction.” I want to respectfully disagree. I think the data points very clearly in one direction. Capability and capital are accelerating. Evaluation, disclosure, labor protection, and institutional trust are not keeping pace. The ambiguity is not in the data. The ambiguity is in what we are willing to do about it.

In AI Safety 2025 I wrote that safety is a choice. In Enterprise AI’s Biggest Risk I wrote that governance is a chain from principle through control to evidence. In Davos 2026 I wrote that 2026 would feel less like new tools and more like new constraints. This year’s AI Index is the clearest numerical confirmation of all three I have seen, and also a quiet invitation to stop treating the adaptation gap as a second-order issue and start treating it as the main event.

See you next week,

Nesibe

💬 Let’s Connect:

🔗 LinkedIn: [linkedin.com/in/nesibe-kiris]

🐦 Twitter/X: [@nesibekiris]

📸 Instagram: [@nesibekiris]

🔔 New here? for weekly updates on AI governance, ethics, and policy! no hype, just what matters.