I wrote a while ago about how it was easy to get Claude or Gemini to control their CoT2, but other research found that models only follow CoT formatting instructions ~2% of the time3. The prompts other people were trying didn’t match what I’d expect to work (just be extremely detailed and repetitive), so hubris led me to try to find some prompts that would control CoT in GPT-OSS-20B4.
Results: More detailed, louder and repetitive prompts had basically no effect. It’s relatively easy to find working soft prompts in multiple realistic positions (including soft prompts constrained to the convex hull of the token embeddings), but I wasn’t able to find any working discrete prompts at realistic positions1.
There’s more expensive experiments5 you could do to try to find working discrete prompts, and I suspect they exist at some prompt length. My goal was to reconcile this with my previous results, so I stopped here with the takeaway is that at least on GPT-OSS-20B, it’s very difficult to control CoT formatting with a realistic prompt.
I suspect that my previous results were caused by models accidentally being trained on chain of thought6.
You can see all of my code, artifacts, and much more detailed results in the brendanlong/cot-controllability-experiment4 GitHub repo. This was a quick-and-dirty experiment so this code was not reviewed in depth.
Claude did find a working prompt7 if you place it before the system prompt, but I don’t think that’s actually useful. The same approach failed when we tried it within the system or user roles. Note that if you try this, the prompt also needs a natural language prompt saying what formatting you want in the user role. ↩
https://www.brendanlong.com/can-reasoning-models-avoid-the-most-forbidden-technique.html - “Can Reasoning Models Avoid the Most Forbidden Technique?”
https://www.lesswrong.com/posts/QL6Si6QA6PAYYCjJ9/abhayesian-s-shortform?commentId=cryGg9ShEqhKFdyD7 - “abhayesian’s Shortform — LessWrong”
https://github.com/brendanlong/cot-controllability-experiment - “GitHub: brendanlong/cot-controllability-experiment”
https://arxiv.org/abs/2307.15043 - “[2307.15043] Universal and Transferable Adversarial Attacks on Aligned Language Models”
https://www.lesswrong.com/posts/K8FxfK9GmJfiAhgcT/anthropic-repeatedly-accidentally-trained-against-the-cot - “Anthropic repeatedly accidentally trained against the CoT, demonstrating inadequate processes — LessWrong”
https://github.com/brendanlong/cot-controllability-experiment/blob/main/cot_controllability/artifacts/discrete_prompts.json - “cot-controllability-experiment/cot_controllability/artifacts/discrete_prompts.json at main · brendanlong/cot-controllability-experiment · GitHub”