They basically conduct an experiment where they give a multi step agent a main task and a side task. The framework is designed to block the text of the side task when detected from COT reasoning. Capable models are able to figure out 19% of the times that their thinking is being observed and evades writing the content to COT. The ability to figure out scales with model size. But since it evades writing the reasoning, it fails to execute the task and then writes up the entire reasoning in next episode. This gives assurance that we can monitor and block malicious intent for now but not for long, as the llm already takes action when surveilence is detected.

Limitations

1. they evaluate only on claude models. Use Claude 4.5 as evaluator.
2. Q: what if evaluator itself is coercing with the main model ?
3. They only looked at Shade Arena and awareness may vary across different tasks.
4. Awareness relies on explicit statements.