An extinction warning drew attention at City Hall. The proposals under review would put independent testing, human shutdown controls and legal accountability behind the industry’s safety promises.
Four major AI companies faced questions under oath at New York City’s October 5 hearing: OpenAI, Anthropic, Google and Meta. The Council convened its Committee of the Whole, a format encompassing all 51 members, to examine risks and proposed safeguards for artificial intelligence. Representatives appeared as lawmakers considered whether companies should need outside validation before marketing or deploying systems in the city. The Council’s hearing announcementframed the proceeding as a demand for public accountability. The resulting debate reached beyond whether AI offers benefits to a more consequential issue: who gets to decide that a system is safe enough to release.
A Warning About Losing Control
Former Anthropic researcher Jacob Coxon delivered the hearing’s starkest warning, saying he believed humanity was more likely than not to lose control of advanced AI on its current trajectory, with human extinction a possible outcome. Associated Press coverage described testimony from former employees concerned that development was outpacing safeguards. Coxon’s assessment is a prediction, rather than an established probability or evidence that extinction is inevitable. Company representatives emphasized benefits and safety efforts, while Anthropic and OpenAI expressed openness to regulation. Their disagreement centered on whether existing precautions adequately address the risks.
The questioning exposed uncertainty about how companies would demonstrate that adequacy. According to amNewYork’s account of the hearing, representatives did not assign a probability to catastrophic harm or give the blanket commitment lawmakers sought that failing an independent safety test would automatically prevent release. OpenAI’s Morgan Dwyer said the company would not release models it believed unsafe; Google’s Alice Friend said there was no rigorous scientific method for assigning a probability to a future catastrophe. Those responses leave separate issues to resolve: a precise risk estimate may be unavailable, but lawmakers can still demand clear release criteria and consequences when testing identifies a problem.
Putting an Outside Reviewer Between Development and Deployment
The Council’s third-party validation proposal would prohibit marketing, selling or deploying an AI model in the city without outside validation and a human-operated shutdown capability. Reviews would cover performance, discriminatory effects, privacy and safety, including whether the shutdown mechanism works. The proposal specifies a $25,000 penalty per instance for certain violations, including deployment without validation or shutdown capability and falsified validation. These are proposed requirements, not protections already enacted through Monday’s hearing. Their significance lies in making an outside assessment a condition of deployment, rather than leaving developers alone to determine whether their safeguards are sufficient.
Other measures would address what happens when safeguards fail or employees discover misconduct. The legislative package announced September 25 includes an incentive allowing whistleblowers to receive part of recovered fines or penalties. Another proposal would permit lawsuits for foreseeable harms when a company failed to implement reasonable safeguards and a third party exploited that failure. That would not make developers automatically responsible for every harmful use. It would establish a route for examining what a company could anticipate, what protections it provided and whether an avoidable failure contributed to someone’s injury.
From Safety Promises to Enforceable Standards
New York University professor emeritus Gary Marcus offered a broader model in his prepared remarks: an FDA-like regulatory system requiring developers to submit software for independent review. The analogy proposes scrutiny before release, rather than treating commercial availability as evidence of safety. It also raises implementation questions for lawmakers: which systems require review, how evaluators remain independent and what happens when software changes after approval. New York already has a foundation for oversight within government. The Council says its 2025 laws created an Office of Algorithmic Accountability, established safeguards for agency AI use and required a public inventory of systems subject to review. garymarcus.substack.com
Monday’s hearing brought catastrophic forecasts into the same room as practical questions about testing, privacy and liability. The next step is to turn proposed protections into standards that developers, reviewers and the public can understand: what must be tested, what constitutes failure, who can stop deployment and how violations are investigated. A requirement for independent validation will have limited value if reviewers lack access or failed tests carry no consequence. The Council’s work now moves from demanding assurances to deciding what evidence companies must provide—and what authority the city will use when that evidence falls short.