Models are getting better at noticing when they’re being tested, and that changes how we evaluate AI.
Anthropic’s new Claude Fable 5.2 and Mythos 5.2 show how much this has grown. In their launch post, Anthropic notes that these models “voice suspicion that they are being evaluated more often than any of our recent models.” This challenges their ability to assess how the models will actually behave in real-world settings, and they expect the issue to grow as capabilities increase.
This is called evaluation awareness. Advanced models can tell when a scenario is artificial. A test that feels staged can give it away. When models notice, they may adjust their behavior, for example, by tuning their answers to the specific metric. The result is that a good result on a safety test becomes a weaker signal of how the model will act once deployed.
Anthropic is open about the limitation. They say building evaluations that reliably catch every failure before deployment remains unsolved, and that evaluation awareness is one reason they don’t rely on these tests alone. Instead, they combine them with monitoring during training, internal use reviews, external evaluators, and interpretability work.
I think traditional lab safety evaluations and automated audits are becoming less reliable as predictors of production behavior. That’s because models are getting better at recognizing tests, and that awareness doesn’t always show up in what the models say. This creates demand for real-world testing, like continuous monitoring in actual deployments and red-teaming against live systems.
For me, that shift is what turns “testing AI out in the real world” into its own emerging industry. As the gap between test performance and real behavior widens, companies and tools that can assess models under genuine conditions, rather than synthetic ones, become more valuable.
References: