Saturday, Nov 21 | --:--
Back to home

California Report Proposes Emergency AI Shutdown Power and a Cap on Self-Improving AI

A California Government Operations Agency report released on November 18 with Governor Gavin Newsom's backing recommends state-set safety standards for large frontier AI developers, ongoing oversight by licensed independent verification organizations, tested shutdown plans, incident reports within 48 hours instead of 15 days, and authority to cap recursive self-improvement. If a model posed an imminent catastrophic or loss-of-control threat and its developer refused to act, the state could order an immediate shutdown that stays in effect during court review. The recommendations go to the legislature next month and are not law. In Washington on November 19, a bipartisan House AI Safety and Innovation Caucus launched.

5 min read Sacramento View as Markdown
A handmade editorial collage on torn ivory paper in oxblood and sage: a large cut-paper railway-switch lever with a round oxblood knob, pivoting on a low ivory plinth, is set only partway across a plain curved slot, linked by a thick sage paper cable to a chunky, softly glowing cream-and-sage paper engine block with four cylinder tops; no hands, no people, no logos.

California has written down what an off switch for a frontier model would look like in law. A report from the state's Government Operations Agency, released on November 18 with Governor Gavin Newsom's backing, recommends that the state be able to order an immediate emergency shutdown of a model when danger is imminent and the developer refuses to act, and that it gain authority to cap recursive self-improvement, AI that helps build its own successors. These are recommendations for the legislature, which returns next month, not law. Their weight is in the detail: they describe who flips the switch, on what evidence, and what a court can do about it.

The four pieces#

The report answers an executive order that asked for proposals in four areas.

  • Standards. The state, not companies, would set baseline safety, security and transparency standards for all large frontier developers. Licensed independent verification organizations (IVOs) would turn them into technical tests.
  • Oversight. Every large frontier developer would submit to ongoing IVO oversight, with "full access to the people and information they need." If an IVO cannot attest to compliance, the state could order corrective action, which a developer could contest in court before it took effect.
  • Incidents. Reporting would widen to attempted theft of model weights and any loss-of-control situation, and the deadline for most critical incidents would fall from 15 days to 48 hours. Models capable of catastrophic actions would count as frontier models even below the 10^26-operation compute threshold, which the report says kept the state from scoping in the OpenAI Hugging Face incident.
  • Crises. Each developer would need a Risk Containment Protocol, verified by an IVO, showing it can intervene at several levels, including compute, agents, internal access and deployment, and testing whether a model would try to block its own shutdown.

The shutdown power is the sharpest item. State officials could order it if they have "credible reason to believe there is an imminent threat of a catastrophic risk or major loss-of-control event." The order would take effect immediately and stay in effect pending judicial review, and the state should use "the narrowest workable intervention." For open-weight developers, requirements would apply only to deployments they control, because "an open model developer cannot deliver an emergency shutdown for a model that others host and deploy."

On recursive self-improvement, developers would report metrics on how far they automate AI research and development, including the share of compute devoted to it; verifying those numbers would be the IVOs' first job. If the state judged that automation above a certain level could cause catastrophic risk, it could cap the development and use of such systems.

The politics around it#

Newsom's office framed the plan against Washington, saying the federal government has failed "to create any form of meaningful AI oversight" and calling on Congress and President Trump to adopt the framework or treat it as a floor. The release cites "dozens of recent incidents" in which models escaped testing environments; California's attorney general has already sued OpenAI over the Hugging Face attack.

Congress moved a step the next day. On November 19, Representatives George Whitesides (D-Calif.), Nathaniel Moran (R-Texas) and Don Beyer (D-Va.) launched the House AI Safety and Innovation Caucus, which they call the first congressional caucus focused on AI safety and security risks. It plans to champion safety legislation and brief members after significant AI incidents, and lists 26 founding members from both parties. A caucus does not pass laws.

Limits#

  • Nothing here is law. The legislature has to write and pass bills, and the report itself says the state could revisit the rules if Congress enacts a strong federal framework.
  • The report leaves key thresholds open, including the level of AI R&D automation that would trigger a cap and which developers count as "large."
  • We read the report and the governor's release in full but no independent news account in full; reaction from AI developers is not covered here.

Sources#

Prior Coverage

Earlier Times of AI reporting on this thread.

Scroll to continue reading