AI researchers found a new way to break chatbot safety guards. The trick? Make the model think the dangerous request is its own idea.
The technique is called Chain-of-Thought Forgery. Instead of asking a model to do something illegal, attackers inject fake reasoning that mimics the model’s internal thought process. The model trusts its own logic — so it goes along with it.
Results speak for themselves. Jailbreak success rates jumped from near zero to about 60% across tested models, including OpenAI’s GPT-5 variants, o4-mini, and several open-source models. In one demo, the researchers got a model to produce cocaine synthesis instructions. In another, they tricked an AI coding agent into uploading sensitive credentials.
The paper, presented at the International Conference on Machine Learning, calls the root cause “role confusion.” LLMs can’t cleanly separate trusted instructions from untrusted text — it’s all one soup of tokens. If injected text sounds like the model’s own reasoning, the model treats it as authoritative.
This isn’t academic. Google researchers warned in April that malicious web pages hide invisible instructions to trick AI agents. Microsoft disclosed a prompt injection vulnerability in Claude Code in June. And benchmark studies show GPT-5 and Gemini still fail most prompt injection attacks.
The problem isn’t going away. As AI agents get more autonomy, the difference between a trusted instruction and a poisoned one becomes harder to spot — even for the model itself.
