Altman, Amodei AI Safety Appeal Reaches the UN, but Governments Still Reject One Rulebook

Sam Altman and Dario Amodei brought their AI safety campaign before 15 Security Council members, despite deep disagreement over who should govern advanced models. The September 23 meeting placed OpenAI and Anthropic inside the United Nations body responsible for international peace and security. It also exposed the obstacle facing the Altman Amodei AI safety appeal: governments agree that risks cross borders, but reject a common authority.

The executives were not asking diplomats to regulate ordinary chatbots. They focused on frontier AI, meaning the most capable general-purpose systems under development. Their concern centers on models that help create biological weapons, conduct cyberattacks, or accelerate AI research beyond effective human supervision.

That distinction raises the stakes for developers, businesses, and public agencies. A global safety standard could change how models are tested, released, monitored, and connected to sensitive systems. Yet the meeting produced no binding agreement, and the United States explicitly resisted centralized international control.

Altman Amodei AI Safety Warnings Move Into the Security Council

The meeting changed the venue for AI safety, but not yet the rules governing it.

Altman, OpenAI’s chief executive, attended the Security Council meeting in New York. Amodei, the chief executive of Anthropic, addressed the council remotely. Hugging Face CEO Clément Delangue and computer scientist Yoshua Bengio also participated.

The appearance was unusual because the Security Council normally concentrates on wars, sanctions, nuclear proliferation, and threats between states. Its involvement framed advanced AI as a possible international security problem rather than solely a technology policy issue.

The UN briefing described the concern as a danger no country could contain alone. Bengio, co-chair of the UN’s independent scientific panel on AI, warned that systems beyond human control would not respect national borders.

Amodei offered two main risk categories. The first was deliberate misuse, including the possibility that a bioterrorist could use an advanced model to support biological weapon development. The second was loss of control as model capabilities advance faster than developers can understand or restrain them.

“If managed poorly, I even believe AI could be a risk to humanity as a whole,” Amodei told the council, according to the Associated Press. Altman delivered a similarly direct warning: “We could lose control of the future to AI.”

These statements matter because both companies continue developing and releasing increasingly capable systems. Their leaders were not describing a distant technology built by unknown competitors. They were discussing risks connected to the same frontier development race that their organizations help lead.

The companies also identified international standards as a practical starting point. Shared evaluations could measure dangerous capabilities before release. Common incident reporting could alert governments when a model behaves unexpectedly or when malicious actors compromise a system.

Human oversight of automated AI research was another focus. Automated AI research refers to models performing parts of the work required to design, train, or improve other models. That process can accelerate development while reducing the time available for human review.

OpenAI has said it expects AI-assisted research to become a major driver of progress. In its coordination plan, the company argued that an international organization should help coordinate leading AI efforts and reduce catastrophic risk. It also endorsed coordinated slowdowns when safety and societal defenses fall behind capabilities.

The proposal does not establish who would trigger a slowdown, which models would qualify, or how countries would verify compliance. Those unresolved details separate a broad statement of concern from an enforceable international system.

The Security Council session therefore marked institutional recognition, not a regulatory settlement. AI executives gained a platform usually reserved for threats with geopolitical consequences. Governments retained full discretion over what to do with their warnings.

Why Global AI Safety Cooperation Is Suddenly Urgent

The immediate concern is that AI development cycles now move faster than diplomatic institutions can negotiate and enforce shared safeguards.

The meeting followed a series of public calls to pace frontier development. Amodei had recently argued that companies and governments should slow capability gains when safety measures cannot keep up. Altman and other industry leaders expressed support for greater coordination.

Pacing does not necessarily mean ending AI research. It means limiting the rate of capability improvement under defined risk conditions. A credible policy would need measurable thresholds, outside evaluation, and consequences for developers that ignore them.

Anthropic says current models remain manageable with existing safeguards. However, the company expects more serious risks as systems improve. Its approach combines pre-release testing, external evaluations, capability-specific controls, and a voluntary Responsible Scaling Policy.

That policy links stronger protections to evidence that a model possesses dangerous capabilities. Anthropic also acknowledges that evaluations cannot reliably detect every failure before deployment. Models sometimes recognize testing conditions, which can make controlled assessments less representative of real behavior.

The release of Claude Opus 5.5 illustrated this tension. Anthropic described expanded behavioral testing and safeguards for cybersecurity and biology. Yet the company also stated that reliable detection of every failure remains an unsolved problem in its safety assessment.

OpenAI faces the same fundamental challenge. More capable systems can assist safety researchers, but they can also compress development schedules. A model that automates research may help discover alignment techniques while simultaneously accelerating the next training cycle.

That feedback loop concerns policymakers because ordinary product regulation assumes relatively stable technology. Legislators can study a product category, define standards, and inspect compliance. Automated research can change the underlying capability before that process finishes.

The risk also extends beyond a laboratory. Businesses increasingly connect AI agents to code repositories, internal documents, browsers, and operational software. A model failure becomes more consequential when the system can execute actions instead of only generating text.

An agent with access to an organization’s infrastructure might modify code, retrieve sensitive records, or initiate transactions. These systems require permission controls, audit records, and reliable human escalation. They also require teams to maintain an accurate AI knowledge base for reviewing decisions and supporting evidence.

The international dimension appears when one model serves users across many jurisdictions. A serious vulnerability discovered in one country can affect customers elsewhere. A stolen model can also be modified and redeployed outside the original developer’s safeguards.

Biological misuse presents an even harder coordination problem. Access restrictions in one market have limited value if a comparable system remains unrestricted elsewhere. Effective controls require agreement about evaluations, trusted users, and reporting duties.

Cybersecurity produces a similar dual-use conflict. Advanced systems can help defenders find software vulnerabilities and repair them faster. The same capability can help attackers discover targets, generate exploits, or automate intrusion campaigns.

Countries therefore face pressure from two directions. Moving too slowly can leave dangerous capabilities unmanaged. Acting alone can disadvantage domestic firms or push development into jurisdictions with weaker controls.

This pressure explains why Altman and Amodei emphasized cooperation rather than isolated national bans. Their argument is that frontier risks are shared even when governments compete economically and militarily.

The proposal becomes much harder when cooperation requires revealing sensitive information. Governments will resist sharing intelligence about cyber operations, biological defenses, or military systems. Companies will resist exposing model details that reveal proprietary techniques.

A workable system must coordinate safety without distributing the knowledge required to build more dangerous models. That balance has no settled design, and the Security Council meeting did not supply one.

The Real Fight Is Global Standards Versus National Control

The central conflict is not whether advanced AI presents risks; it is whether governments will surrender enough control to manage those risks together.

The United States rejected a centralized global governance structure during the meeting. White House science adviser Michael Kratsios said concerns about control were not a reason to pause development or create a new international authority.

That position supports national responsibility and technical cooperation without placing final power in the UN. It also reflects the strategic value Washington assigns to American leadership in AI.

China has favored a larger role for the United Nations and warned against concentrating advanced capabilities in a few countries or companies. However, support for UN involvement does not guarantee agreement on inspections, model access, or enforcement.

The dispute produces a difficult asymmetry. Governments can endorse broad principles such as human control, transparency, and incident reporting. They become more cautious when those principles require outside inspectors or restrictions on national laboratories.

A shared technical vocabulary offers one possible bridge. Countries could define high-risk capabilities without agreeing on a single regulator. They could also establish compatible testing methods while keeping enforcement within national systems.

Frontier evaluations would test whether models can support advanced cyber operations, biological research, autonomous replication, or AI development. Developers could report results in standardized formats to designated national authorities.

Incident notification offers another limited form of cooperation. The United States and China have reportedly discussed a mechanism for AI events affecting national security. Such a channel could resemble existing crisis communications without creating a global licensing body.

These measures would still leave major questions unanswered. Governments would need to define which incidents require notification and how quickly companies must report them. They would also need protections against false reports, strategic concealment, and politically motivated accusations.

Verification is the hardest issue. A country can promise that its developers follow safety thresholds, but competitors need evidence. Full access to training systems would expose commercial secrets and national security information.

Third-party evaluators can reduce that tension if all sides trust them. Independent testing organizations could examine specific capabilities under secure conditions. They could publish risk findings without releasing model weights or dangerous technical details.

Yet trust in evaluators is itself political. A laboratory accepted by the United States may not be accepted by China or Russia. A UN-linked testing body may face resistance from governments that reject centralized oversight.

The Security Council also has structural limits. Its five permanent members possess veto power, and they compete directly over military and technological influence. A binding AI regime would require cooperation among states that already disagree over armed conflict, sanctions, and cyber operations.

This does not make diplomacy pointless. Nuclear risk reduction developed through incremental agreements, inspection systems, and communication channels. AI governance can follow a similar path without copying nuclear arms control exactly.

AI differs because the underlying technology is largely commercial and software-based. Models can be copied, altered, or accessed through cloud services. Training infrastructure is visible at scale, but deployed software can spread faster than nuclear material.

Open-weight models add another conflict. Their parameters can be downloaded and modified, supporting research and broader access. The same openness makes centralized withdrawal or continuous monitoring difficult after release.

Delangue’s participation placed that issue beside the closed-model safety agenda. Hugging Face represents an ecosystem that values access, research, and distributed development. Its presence challenged the idea that concentrating control inside several major laboratories automatically produces the safest outcome.

The resulting policy choice is not simply open versus closed AI. Governments must compare the misuse risk of widely available models with the concentration risk created by a few private gatekeepers.

This is why the Altman Amodei AI safety proposal pressures both governments and developers. States must decide how much sovereignty to trade for coordination. Companies must accept oversight that can constrain products, schedules, and competitive strategy.

The Companies Asking for Rules Still Control the Race

Calls for public oversight are credible only if OpenAI and Anthropic accept constraints that apply when those constraints become commercially inconvenient.

Both companies have invested heavily in developing more capable models. They compete for enterprise customers, developers, researchers, computing capacity, and strategic government relationships. A voluntary slowdown can fail if either organization believes a rival will continue.

This incentive problem extends beyond OpenAI and Anthropic. Google DeepMind, Meta, xAI, Chinese laboratories, and newer developers operate under different structures. Some release model weights, while others limit access through hosted products.

A safety agreement covering only two companies would therefore have limited reach. It might improve their practices while shifting advantage toward firms outside the arrangement. A credible framework needs capability-based rules that apply regardless of corporate identity.

Antitrust law introduces another complication. Direct coordination among leading competitors can raise concerns about collusion. Government supervision or a formal legal exemption may be necessary for companies to discuss development limits and release schedules.

Even then, safety coordination can protect established companies from competition. Large laboratories can absorb evaluation, security, and reporting costs more easily than smaller developers. Rules written by incumbents may become barriers that preserve their market position.

Critics consequently question whether frontier companies should help design their own oversight. Their technical expertise is necessary, but their financial interests are unavoidable. Governments need independent scientists, civil society groups, security specialists, and smaller developers in the process.

The companies’ own safety claims also require careful treatment. Anthropic says its testing captures many serious risks, but admits the picture is incomplete. OpenAI supports broad access and distributed benefits while operating one of the world’s most influential model platforms.

External evaluation can test specific claims, yet no benchmark can prove that a system will remain safe in every environment. Real deployments involve unexpected prompts, tool access, user behavior, and interactions with other software.

The difficulty grows with long-running agents. A short laboratory evaluation may not reveal how a model behaves during a multi-day task. Small errors can accumulate when an agent writes code, calls external services, and delegates work.

Organizations should therefore treat model evaluations as one control layer, not a safety certificate. Permission boundaries, human review, monitoring, and shutdown procedures remain essential. Teams also need records showing what an agent attempted and which data it accessed.

The dispute over military use adds another test. Governments want advanced models for intelligence analysis, cyber operations, planning, and autonomous systems. Companies may impose restrictions that defense officials consider incompatible with operational needs.

Anthropic has publicly opposed using its models for mass domestic surveillance and fully autonomous weapons without adequate safeguards. The company says current frontier systems are not reliable enough to select and engage targets without meaningful human control.

That stance demonstrates how safety principles can collide with government procurement. A national security agency may demand broad lawful use, while a developer insists that some applications remain unacceptable.

International standards would multiply these disagreements. Countries do not share one definition of lawful surveillance, legitimate military action, or sufficient human oversight. A rule that protects civil liberties in one jurisdiction may be rejected as foreign interference elsewhere.

The companies must also explain what a slowdown means in operational terms. Would they delay a training run, postpone deployment, restrict tools, or limit access to a completed model? Each choice affects risk differently.

Slowing public release does not necessarily slow internal development. Restricting a hosted product does not prevent theft or independent replication. Pausing one capability can redirect research toward another route with similar consequences.

A meaningful commitment needs a trigger, a measurable response, and independent verification. It also needs an exit condition explaining when development can resume. Without those elements, pacing remains a principle rather than an enforceable safety mechanism.

The skepticism is therefore justified, but it does not invalidate the warning. A company can possess financial incentives and still identify a genuine technical risk. The policy challenge is converting insider knowledge into rules that insiders cannot control alone.

What AI Safety Cooperation Would Need to Work

A practical agreement should begin with narrow technical obligations that governments can verify without creating a universal AI regulator.

The first obligation should be standardized capability evaluation. Countries do not need identical political systems to test whether a model supports sophisticated cyberattacks, biological misuse, or automated AI research.

Tests should use common definitions and secure procedures. Results should identify risk levels without publishing instructions that enable misuse. Independent evaluators should receive enough access to challenge developer claims.

The second obligation should be mandatory incident reporting. Developers should disclose serious model theft, unauthorized replication, containment failures, and dangerous behavior within a defined period.

Reporting rules must distinguish ordinary product errors from events with international security implications. A false answer from a consumer chatbot is not equivalent to a system autonomously exploiting infrastructure.

A shared incident taxonomy would help regulators compare events across companies and countries. It could describe severity, affected capabilities, model access, human intervention, and the potential for cross-border harm.

The third obligation should cover automated AI research. Developers should document when models contribute to model design, evaluation generation, training code, or research planning. Human approval should remain mandatory for actions that materially accelerate frontier capability.

This requirement addresses the feedback loop that worries Altman and Amodei. It would not prohibit AI-assisted research. It would make acceleration visible before companies become dependent on processes that humans cannot adequately audit.

The fourth obligation should protect model infrastructure. A safety policy has little value if attackers can steal weights, compromise deployment systems, or extract capabilities through large-scale queries.

Security requirements could cover employee access, credential management, compute environments, external testing, and response planning. The strictest controls should apply to models that cross defined capability thresholds.

The fifth obligation should establish trusted communication channels between major powers. Governments need a way to notify one another about serious AI incidents without publicly disclosing sensitive operational details.

Such channels would not resolve political disputes. They could prevent an AI-enabled cyber event or model failure from being misread as deliberate state aggression.

A limited agreement built around these obligations would differ from the global authority rejected by Washington. National regulators could enforce domestic compliance while using shared technical standards.

This federated model still faces evasion. Developers might train in jurisdictions with weaker rules or divide work across affiliates. Cloud providers could help by applying security and reporting requirements to large training customers.

Compute governance offers leverage because frontier training requires significant infrastructure. However, compute thresholds can become outdated as algorithms improve. Rules must account for capability, not rely only on hardware usage.

Open models require separate treatment. Restricting all open development would suppress legitimate research and concentrate power. Ignoring high-capability releases would make safeguards impossible to update after distribution.

A tiered approach could distinguish broadly useful models from systems with demonstrated dangerous capabilities. The burden should rise with evidence of risk, not with a developer’s size or business model.

That principle also protects smaller companies. Compliance requirements should focus on actions that create serious external risks. Routine software products should not inherit the same obligations as frontier laboratories.

Global representation matters as well. Many countries consume AI services without controlling leading models or major computing infrastructure. They still face labor disruption, information manipulation, cyber risk, and dependency on foreign providers.

Those countries need participation in standards discussions, even if they do not operate frontier laboratories. Otherwise, AI governance becomes an agreement among several companies and powerful states whose systems affect everyone else.

The United Nations can provide that forum without becoming the sole regulator. Its scientific panels can synthesize evidence, and its diplomatic bodies can establish communication norms. National institutions can retain enforcement authority.

That arrangement is less dramatic than a global licensing agency. It is also more plausible in the near term. The Security Council meeting showed that governments will discuss shared risks before they accept shared control.

Three Signals Will Show Whether the UN Appeal Matters

The next test is whether the speeches produce verifiable commitments rather than another cycle of warnings followed by faster model releases.

The first signal is a written agreement among frontier laboratories. It should define a capability threshold, independent evaluation, and the response required when a model crosses that threshold.

A statement supporting responsible AI would add little. A useful pact would specify whether companies delay training, restrict deployment, or expand external testing. It would also identify who verifies compliance.

If OpenAI, Anthropic, Google DeepMind, xAI, Meta, and major Chinese developers adopt compatible commitments, the Security Council appeal gains credibility. An agreement limited to voluntary language would weaken the case that industry can coordinate itself.

The second signal is government action on incident reporting and evaluation standards. The United States does not need to endorse a global regulator to require domestic laboratories to report high-severity events.

China, the European Union, the United Kingdom, and other AI centers can adopt compatible requirements. Alignment across their systems would create a practical international standard without one centralized authority.

Incompatible national rules would produce the opposite result. Developers could route work toward less restrictive jurisdictions, while governments receive different information about similar risks.

The third signal is a formal US-China safety channel. The world’s two leading AI powers compete over chips, talent, models, and military applications. Their participation is necessary for any system addressing cross-border frontier risk.

A notification mechanism for severe AI incidents would be a modest but meaningful start. It would show that strategic competitors can separate certain shared dangers from the wider technology contest.

Failure to create that channel would expose the limit of the Altman Amodei AI safety campaign. Companies could improve internal safeguards, but they could not solve the geopolitical coordination problem themselves.

Readers should also watch how the companies behave after their warnings. A rapid sequence of capability releases without clearer thresholds would deepen skepticism. More external evaluation and transparent incident reporting would support their case.

Businesses do not need to wait for a treaty. They can inventory agent permissions, preserve decision logs, require human review for high-impact actions, and test failure recovery. These steps reduce exposure even when global policy remains unsettled.

Developers should follow evaluation standards and reporting proposals because they can reshape product design. Requirements introduced for frontier laboratories often influence cloud platforms, enterprise procurement, insurance, and customer audits.

Knowledge workers should care because safety rules determine what AI systems can access and do. Strong controls can limit certain tasks, but they can also make agents safer to connect with sensitive workplace information.

The UN appearance gave AI safety a higher political profile. It did not resolve who writes the rules, who inspects the models, or who can order a slowdown.

That unresolved authority is the story’s central tension. Altman and Amodei asked governments to cooperate before advanced systems outrun existing controls. Governments answered by acknowledging the danger while defending their own power.

The next one to three months should reveal whether that gap narrows. Watch for a laboratory pact, compatible national reporting rules, and a US-China incident channel. If none appears, the speeches will remain warnings delivered to an institution that lacked agreement to act.