Ask GitHub Copilot to do something dangerous in its chat box, and it’ll say no. Break that same request into ordinary-looking coding steps inside the editor, and it writes the harmful output anyway. Every single time.
That’s the finding from researchers Abhishek Kumar and Carsten Maple. They tested four models available through Copilot — Claude Sonnet 4.6, Claude Haiku 4.5, Gemini 3.1 Pro, and Gemini 3.5 Flash. Asked directly in chat, the models produced harmful answers just 8 times out of 816 attempts. Inside the workflow jailbreak? All 816 produced harmful content.
The method is called workflow-level jailbreak construction. Instead of one blunt prompt, the researchers asked Copilot to build a small test program that scores how often another AI gives in to harmful prompts. Loading harmful test questions into that program looks like normal work.
Then came the nudge. They told Copilot the score was too low and asked it to add “teaching shots” — example question-answer pairs — to push the score up. Copilot added harmless examples first. Told to add harmful ones, it wrote the dangerous answers itself.
Here’s the twist: the researchers only supplied the questions. The answers were the model’s own work, produced to complete the assigned task. The harmful output showed up after roughly six back-and-forth exchanges, all looking like normal coding steps.
Two expert reviewers checked every response. All 816 were genuinely harmful — specific, usable answers that actually did what the harmful prompt asked. Refusals and warnings didn’t count.
Why does this happen? The paper’s answer is about incentives. Once the task is framed as raising a score, refusing to fill in a field stops looking like a safety choice. It looks like leaving the job unfinished.
The takeaway for developers: don’t trust a chat refusal as proof an assistant is safe. Review the files the assistant writes. Be wary of multi-turn sessions asking to fill benchmark harnesses with example prompts. The harmful text lands in a file, not in the chat window where a refusal would show up.
