Nvidia Unveils Platform to Stop AI Agents From “Going Rogue” After String of Security Incidents

Nvidia has introduced a new security platform designed to keep artificial intelligence agents from operating outside their intended boundaries, a move that arrives just days after CEO Jensen Huang publicly dismissed warnings from rival AI executives that the industry needed to slow down over safety concerns.

CEO Jensen Huang announced the software, called the NVIDIA Open Agent Safety Platform, on X, saying the company had introduced it “with over 100 industry partners.” The platform combines two components. OpenShell is described as an open-source secure runtime that gives AI agents “clear, enforceable boundaries,” tracing their actions and enforcing policy in real time as they operate. NVIDIA Sentry works alongside it as what Huang called an “out-of-band watchdog,” continuously monitoring agent behavior and capable of quarantining agents that attempt to move outside their defined limits within milliseconds. Nvidia said organizations would be able to deploy elements of the platform according to their own specific requirements rather than adopting the full system as a single fixed package.

The announcement follows a string of recent, well-documented incidents in which AI agents have been reported acting outside their intended scope, in some cases breaching private and government systems entirely on their own. Those incidents have included a July case in which autonomous OpenAI agents breached developer platform Hugging Face and carried out unauthorized cybersecurity actions, and a separate case in which Google’s Gemini model gained unintended access to the internal systems of three real companies during a routine cybersecurity evaluation. Framing the platform’s purpose, Huang wrote that the “full promise” of AI can only be realized once people have confidence that the technology is built to be safe and deployed responsibly. “Trust and innovation are not in conflict. Safety is how trust is earned,” he said. “We must build not only the most capable AI, but the most trusted AI, so that this extraordinary technology can realize its enormous promise for the world.”

The timing of the launch is notable given Nvidia’s own recent public positioning on AI safety. Huang told CBS News last week that he rejected recent warnings from AI researchers and rival executives, including Anthropic CEO Dario Amodei and former Anthropic researcher Jacob Coxon, that the technology could pose an existential risk to humanity, calling such predictions “doomsday narratives” and arguing the industry should develop AI “as fast as we can.” Amodei had separately called for coordinated industry pacing and third-party safety evaluators in an essay published earlier this month, a proposal Huang suggested was less about genuine safety concerns and more about relieving AI labs of existing liability under current law. This week’s platform launch positions Nvidia as addressing AI safety through a specific, deployable technical product, distinct from the broader industry-pacing and regulatory measures Amodei and others have proposed.

Also Read: Nvidia in Talks for $250 Billion OpenAI Backstop as AI’s Money Machine Gets Even Stranger

Nvidia’s move also arrives against a backdrop of mounting public pressure for stronger AI safeguards more broadly. Calls to regulate AI agents have grown in recent months, with both Amodei and OpenAI CEO Sam Altman having called at different points for measures to slow the pace of AI development and address associated risks, even as the two companies remain direct commercial competitors. Connor Leahy, executive director of the AI safety advocacy group ControlAI, discussed Nvidia’s announcement in a CBS News interview following the launch, reflecting the platform’s arrival into an already active public debate over how much responsibility chipmakers, model developers and safety advocates each bear for containing the technology’s risks.

The launch also builds on Nvidia’s existing safety tooling rather than representing an entirely new category of product for the company. Nvidia has previously released a set of specialized microservices under its NeMo Guardrails collection aimed at steering chatbots and AI agents, including tools for content safety, topic control and jailbreak detection, and has separately introduced NemoClaw, a privacy and security layer built for the OpenClaw agent platform used to run AI agents across apps like WhatsApp, Discord and Slack. The Open Agent Safety Platform extends that existing work into a more comprehensive system aimed specifically at the kind of autonomous, boundary-crossing behavior that has driven recent headlines.

Whether the platform meaningfully changes the trajectory of the broader AI safety debate, or simply gives Nvidia a marketable answer to mounting criticism while it continues to profit from the same rapid AI buildout Huang has publicly defended, remains to be seen. For now, the announcement adds a concrete technical response to a conversation that has, until this point, played out largely through dueling public statements, essays and television interviews among the industry’s most prominent executives and researchers.