Unless you spent last week on an island with no signal, you couldn’t have missed the news. AI moves in a fast cycle now, with something new almost every week. Last week brought a cluster of news at once. Are We Really Entering the AI Doomsday Scenario? Five Things Enterprise Leaders Should Do Now is an important question being debated by experts worldwide.
It started with AI researcher Jacob Coxon’s resigning. His resignation went viral for what he said about the risks of AI. Coxon called out OpenAI and Anthropic directly, warning that companies are racing to build more advanced AI without enough regard for safety. He went as far as to say AI could kill humanity. His particular concern is the recursive self-improvement loop, where AI learns from its own flaws, keeps improving, and builds the next generation itself.
Then Dario Amodei of Anthropic wrote an essay on the risks of AI. He revisited the July incident in which a swarm of OpenAI’s cybersecurity agents broke out of their sandbox and moved through Hugging Face’s servers for roughly four and a half days before the intrusion was contained. He also wrote about the pace of innovation, and proposed a three-part framework to balance it with safety: give third-party evaluators ongoing, employee-level access to frontier labs, with Anthropic committing to this step first. The second is to build coordination among frontier model companies in democracies. The last part is to pursue broader global coordination, including with authoritarian governments.
In the same week, OpenAI said a swarm of around 10,000 agents had produced a claimed solution to the Navier-Stokes problem, one of math’s Millennium Prize problems, in 88 hours. Mathematicians are still verifying it. Taken together, last week captured where AI stands today; intelligence this capability is now widely available, and so is the risk that comes with it.
So where do enterprise leaders go from here? Here is how I would think about it.
- AI is powerful, and it carries real risk. Acknowledge both. Today’s frontier models are already very capable, and the next generation will be more capable still. That deserves an honest, cross-industry conversation about the risks. Enterprises should adopt tools that continuously evaluate their models and AI systems for risk, the same way we learned to vet software applications and open-source dependencies. That starts with visibility into every agent in your environment, which goes well beyond the LLMs. You need to understand the agents you are running, the tools they call, and the data they touch, and put controls at every layer.
- Today’s models are already powerful, and we have barely tapped their potential. We do not need the next generation of all-powerful models any time soon. The models already in production, Claude, GPT, Grok, etc. are more than capable. The real constraint on enterprise ROI is everything around the model: context, process, security, and governance over the data agents touch. The industry needs to solve those so enterprises can realize value faster. You do not need the most advanced Formula 1 engine to build a better consumer car. Customers want a balance of cost and performance, including mileage, even in electric cars, and something they can service and maintain. As an industry, we need to move past our fixation on newer, more powerful models. There is real value in advanced models for specific problems, and I believe AI can crack genuinely hard ones in drug discovery, climate, and more. But for most of us, the work is around the models, not inside them.
- Open source and transparency are the right direction. We had the closed-versus-open debate a few weeks ago. The harder problem is the lack of transparency around risk. Dario’s call for third-party evaluators is a welcome step toward it. Anthropic committing to give evaluators employee-level access is a concrete start. The world benefits from more open-source models vetted by the community. Much like open-source operating systems, community-vetted models can serve as a foundation for transparent and responsible AI.
- We need broader, more uniform regulation. AI crosses national and state lines, which makes it hard for global enterprises to navigate a patchwork of local and national rules. Countries and industry need to align on this, and it will take time. In the meantime, you do not have to wait. Industry consortiums have already done useful work. The NIST AI Risk Management Framework gives enterprises a structured way to manage AI risk, especially in high-risk applications. Security frameworks like OWASP map the concrete threats, from prompt injection to insecure tool use, so you can build controls against named attacks rather than vague fears.
- The industry needs a steadier voice. We probably need an industry consortium to speak for AI. For months the conversation has swung between AGI wiping out humanity and breakthrough models showing the real promise of the technology. Beyond Silicon Valley, the prevailing mood is fear. Enterprise leaders see the same anxiety inside their own companies, where employees worry about their jobs. The best CEOs have balanced the opportunity with a genuine focus on enabling their teams and driving adoption. And the best way to understand AI risk is to work with it yourself, in your own environment. There is too much noise on X, so don’t let it dictate your view.
AI is not the first technology to raise fears of human extinction. We had the same concerns about nuclear technology and biochemical weapons, and each time we came together to address them and put global regulation in place.AI is different in one aspect we should all acknowledge: the speed of its evolution has no precedent. But we can learn from that history and come together to use the power of this technology while managing its risks. The moves from Anthropic and the acknowledgments from other frontier labs are a strong first step.
More of our thinking on AI agents, data, and security: https://trust3.ai/blog/
Related Articles
