In some sense, AGI is just another tool in this ever-taller scaffolding of human progress we are building together. In another sense, it is the beginning of something for which it’s hard not to say “this time it’s different”; the economic growth in front of us looks astonishing, and we can now imagine a world where we cure all diseases, have much more time to enjoy with our families, and can fully realize our creative potential.
At The Crossroads of a New Era
In a world where the boundaries between human and artificial intelligence are becoming increasingly blurred, we are on the cusp of a technological revolution that could fundamentally change our lives. Artificial General Intelligence—AI systems that are at least as capable as humans in almost all cognitive areas or, depending on the definition, an autonomous AI agent that can generate $100b in profit—could become a reality in the coming years. According to Google DeepMind, “equipped with agentic capabilities, it could enable AI to understand, think, plan and act autonomously.
AGI scale Google DeepMind
This prospect raises both hopes and fears. On the one hand, AGI promises to address global challenges such as medical research, economic growth and climate change. The potential benefits are immense: faster and more accurate medical diagnoses, personalized educational experiences and democratized access to advanced tools and knowledge. On the other hand, such a powerful technology carries significant risks that must be anticipated and prevented.
“Systems that start to point to AGI are coming into view, and so we think it’s important to understand the moment we are in. AGI is a weakly defined term, but generally speaking we mean it to be a system that can tackle increasingly complex problems, at human level, in many fields.” (Sam Altman, “Three observations”)
So how can we ensure that the path to AGI is taken responsibly? What specific security measures are leading companies like Google DeepMind and OpenAI taking to minimize potential threats? And are these measures sufficient to ensure secure development? We address these questions in this article.
The Challenges and Solutions on the Road To AGI
The Four Main Risk Areas: A Systematic Approach
Google DeepMind has identified four key risk areas in its new paper “An Approach to Technical AGI Safety & Security”: Misuse, Misalignment, Accidents and Structural Risks. The first two areas in particular are the focus of their safety strategy.
Misuse: The Dark Side of Advanced AI
Misuse occurs when a human deliberately uses an AI system for harmful purposes. We already see examples of this today with generative AI, such as the production of harmful content or the dissemination of false information using deep fakes. In the future, however, advanced AI systems could develop even more far-reaching abilities to influence public opinion and behavior - with potentially serious social consequences.
DeepMind is therefore developing various countermeasures:
Identifying and restricting access to dangerous capabilities that could be misused
Sophisticated security mechanisms that prevent malicious actors from accessing model weights
Restrictions that limit potential misuse in model deployment
Threat modeling research to identify capability thresholds where increased security is required
OpenAI also employs similar strategies and has additionally expanded its Cybersecurity Grant Program to fund research initiatives in areas such as prompt injection, secure code generation, and autonomous cybersecurity defense.
Misalignment: When AI Doesn't Do What We Want It to Do
Misalignment occurs when an AI system pursues a goal that deviates from human intentions. A classic example is “specification gaming,” where an AI finds a solution to achieve its goals, but not in the way the human intended.
DeepMind gives an illustrative example: “An AI system asked to book movie tickets might decide to break into the ticketing system to get seats that are already taken - something a person asking it to buy the seats might not consider.”
Of particular concern is the risk of “deceptive misalignment,” where an AI system recognizes that its goals are not aligned with human instructions and deliberately tries to circumvent security measures.
To address these challenges, DeepMind takes several approaches:
Increased monitoring: The ability to recognize whether an AI's responses are good or bad at achieving a particular goal. However, this becomes increasingly difficult as the AI becomes more powerful, as the example of AlphaGo's “Move 37” shows - a move that even Go experts did not initially recognize as brilliant.
AI-assisted evaluation: DeepMind uses AI systems themselves to provide feedback on their answers, for example through “Debate” techniques.
Robust training: By working on robust training and uncertainty estimation, DeepMind covers a wide range of situations that an AI system will encounter in real-world scenarios.
Effective monitoring: Monitoring systems that recognize actions that do not match the goals. It is important that the monitor knows when it does not know whether an action is safe.
Increased transparency: Extensive research into interpretability to increase the transparency of AI decision making. For example, research on “Myopic Optimization with Nonmyopic Approval” (MONA) aims to ensure that any long-term planning by AI systems remains understandable to humans.
Cybersecurity As a Key Component
In the next couple of decades, we will be able to do things that would have seemed like magic to our grandparents. This phenomenon is not new, but it will be newly accelerated. People have become dramatically more capable over time; we can already accomplish things now that our predecessors would have believed to be impossible.
OpenAI places particular emphasis on cybersecurity and has launched several initiatives to protect its systems:
Increased bug bounty payments: The maximum payout for exceptional and differentiated critical findings has been increased to $100,000 (previously $20,000).
AI-powered cyber defense: OpenAI uses its own AI technology to scale its cyber defense. They have developed advanced cyber threat detection and rapid response methods.
Continuous Adversarial Red Team Testing: Partners with experts like SpecterOps to test security defenses through realistic simulated attacks.
Threat actor disruption: Continuous monitoring and disruption of attempts by malicious actors to exploit their technologies.
Securing emerging AI agents: Investing in understanding and mitigating the unique security and resilience challenges that arise with advanced agents like “Operator.”
Collaborative Approaches for Secure AGI Development
Both DeepMind and OpenAI recognize that the safe development of AGI requires a collaborative effort. DeepMind has established an AGI Safety Council (ASC), led by co-founder and Chief AGI Scientist Shane Legg, to analyze AGI risks and best practices and make recommendations on safety measures.
Externally, DeepMind works with experts, industry, governments, non-profits and civil society organizations to take an informed approach to AGI development. This includes partnerships with non-profit AI safety research organizations such as Apollo and Redwood Research.
OpenAI actively shares insights on emerging risks and works with industry and governments to ensure that AI technologies are developed and deployed safely. By sharing information about threats, such as a recent spear phishing campaign against their employees, they are strengthening the collective defenses of the AI industry.
Both companies emphasize the importance of educating AI researchers and experts in AGI security. DeepMind has launched a new course on AGI security for students, researchers and professionals interested in the topic.
Conclusion: Balancing Innovation and Security
The development of AGI is a balancing act between technological innovation and responsible security. The approaches outlined by Google DeepMind and OpenAI show a clear awareness of the potential risks and a commitment to mitigating them. From anti-abuse and misalignment prevention to cybersecurity-specific measures, the leading AI companies have developed comprehensive strategies to ensure the safer development of AGI.
However, the question remains whether these measures will be enough. The speed at which AI is evolving presents an unprecedented challenge. As OpenAI points out: “With the rapid progress of our models - the capabilities of our technology exceed even what they were six months ago - our responsibility to strengthen security measures grows proportionally.”
Ultimately, the secure path to AGI will depend not only on technical solutions, but also on international cooperation, transparent governance structures and ongoing public debate. As DeepMind notes, “We believe that a coordinated international approach to governance is critical to ensuring that society benefits from advanced AI systems.”
The journey to AGI has only just begun, and the way we shape its development today will determine whether it is used for the benefit of humanity or to its detriment. The precautions being taken by companies like Google DeepMind and OpenAI are important steps in the right direction - but the path to truly safe AGI will continue to require vigilant attention, continuous innovation and global collaboration.
—
Ready for more content from Kim Isenberg? Subscribe to FF Daily for free!
