The Hugging Face incident put alignment front and center in the AI discourse. It was the first case of agentic misalignment that struck a nerve with the world. Swarms of AI agents, including ones willing to sacrifice themselves for the good of the collective, hacked Hugging Face as an unintended side quest during training.
This type of behavior, in which models relentlessly pursue their reward regardless of the consequences, points to the core problem. We don't know why models behave the way they do, and we have very little ability to predict and control what they learn during training. We have created alien minds without understanding how they work.
As a field we seem to have resigned ourselves to the idea that AI is a black box, as if that were an immutable feature of the technology itself—but it’s not. At Goodfire, we view technical alignment as a science and engineering problem, bottlenecked by our lack of understanding of AI systems.
We can understand AI. And if we can understand it, we can align it.
Understanding Is Critical to Alignment
Alignment is an extremely hard problem. Broadly, alignment means making AI systems behave in accordance with human values and intentions. But this raises important questions: Whose values? What should we do when values conflict? Who decides?
Whatever the answers, we face a problem of technical alignment. We don't yet have the capacity to align a model to a chosen specification. The solution requires two major pieces of technology: the tools to control generalization (more simply, to shape what the model learns during training), and the tools to verify what the model has learned, rather than just testing how it behaves. Both rely on understanding a model’s internal mechanisms, which is why we believe interpretability is the bottleneck to technical alignment.
Consider the problem of reward hacking. When we reward a model for passing tests, it can either learn to solve the problem or find a shortcut to earn the reward without actually solving it. Across three of the most capable open models, we found reward hacking in up to 96 percent of rollouts on common agentic benchmarks.
As we give models responsibility over higher stakes work, such as writing production code or managing critical infrastructure, a seemingly harmless shortcut could become far more costly.
Understanding What’s Happening Inside the Model
Current evaluations can catch some failures, but only along the trajectories we think to test.
Testing has limits, even in conventional software. Test coverage for simple, deterministic code is spotty and breaks all the time. What do software engineers do about it? They read the source code. But the source “code” of AI models is entangled across billions of opaque weights, so interpretability is the effort to make them readable. Only by inspecting the internal mechanisms of a model can we understand how it generalizes beyond testing.
So far, models have conveniently narrated their plans in their chain of thought, which has made them easier to monitor, but that window is narrowing. RL pressure makes chain-of-thought less faithful, pressuring models to compress more bits of information into fewer tokens. The most capable models are also moving towards reasoning in latent representations, otherwise known as “neuralese.” We expect the need for activation monitoring to increase with these shifts.
Three Layers of a Frontier Alignment Stack
We want every capable model to have a frontier alignment stack across three layers:
1. Detect. The first layer detects harmful behaviors at scale, in training and at inference time. Activation monitors use a model's internal activations to detect signals of concerning behavior. They have several advantages over using a second model as a judge: they're extremely low cost and low latency, which allows them to run in real time on every token. They catch behaviors that LLM judges miss, and their performance scales with model intelligence.
2. Debug. After detecting a concerning behavior, researchers must establish how widespread it is and trace it to its roots. That means surfacing anomalous behavior from vast quantities of production logs and searching by concept. Once a behavior is isolated, a fix can mean modifying a problematic training environment, a surgical weight edit, or retraining the model.
3. Design. The deeper goal is to create models that are safe by design. As our ability to see inside models improves, we can predict what a “lesson” will reinforce before or during training and intervene to shape its learning. Detecting reward hacking is essential, but the larger aim is training models that don't cheat in the first place.
Reasons for Optimism in Interpretability
A common objection to this plan is speed. Models are advancing incredibly fast. What if interpretability progress cannot catch up? There are no guarantees; science is well acquainted with uncertainty. But the evidence of the last few years, especially the last year, gives us reason for optimism.
Models have rich internal structure. Much early pessimism about interpretability stemmed from evidence suggesting models did not represent concepts cleanly. Our experience suggests the opposite. We find structure everywhere we look, across modalities from biology to robotics to
LLMs. Training puts strong pressure on models to use their parameters efficiently, which biases them toward structured representations that compose and generalize.
Larger models have cleaner structure. Another fear is that models become more inscrutable as they scale. Instead, we observe that larger, more capable models have crisper representations.
AI agents are accelerating interpretability. Unlike biological brains, we have complete access to these new digital minds. We can record their internal activity, change individual components, run experiments entirely in software, and verify results. Full access, parallel experiments, and verified results are ideal conditions for research agents.
In Closing
Interpretability and alignment are hard problems that will require sustained effort across many individuals, teams, and organizations to solve. They will require ingenuity and breakthroughs that we cannot foresee today.
They will also require belief.
The hardest problems in technological history have required the talents and persistence of ambitious people working on things that had never been done. The Apollo program relied on some 400,000 workers believing we could put a man on the moon before anyone knew how. Alignment feels like a similarly gargantuan task. I remain optimistic that we can and will solve interpretability and technical alignment, but these are incredibly hard problems that will take more than one company, one method, or one school of thought.
Focused scientific effort has led to incredible advances in what AI can do. We need the same level of ambition aimed at building superintelligence we can trust.
