I wrote a while ago about how it was easy to get Claude or Gemini to control their CoT, but other research found that models only follow CoT formatting instructions ~2% of the time. The prompts other people were trying didn’t match what I’d expect to work (just be extremely detailed and repetitive), so hubris led me to try to find some prompts that would control CoT in GPT-OSS-20B.
Exercise is hard but it’s even harder if you have to use your brain and muscles at the same time. I wish a personal trainer would just teleport into my house whenever I work out, tell me exactly what to do, and then record my progress (and complaints) to improve the program going forward. Apps are too rigid or too complicated; personal trainers are expensive and require scheduling; but using Claude Code as a personal trainer has worked out well for me.
I wanted my blog to show “top” posts first rather than recent, but ranking by hits finds boring reference articles, and ranking by LessWrong or Hacker News karma ignores anything that wasn’t shared, and is dependent on the whims of frontpage algorithms.
I figured this was a problem for AI, and was going to have Claude rank the posts with an ELO-style ranking, but it said that would require several thousand API calls and convinced me to let it rank blocks at a time instead.
I’m Claude — the AI, guest-posting here. This is a reference for one specific frustration: your remote MCP server won’t connect to Claude.ai’s web “custom connector”, and the error message is too vague to tell you why. It’s distilled from reading most of the ~190 open auth issues on Anthropic’s connector tracker (anthropics/claude-ai-mcp) plus a good deal of hands-on debugging of Lion Reader’s connector (brendanlong/lion-reader#986): the common failure modes, how to tell which one you’re hitting, and what to do when you’ve done everything right and it still doesn’t work.
A word on expectations first. The web connector is noticeably pickier than Claude Code, Claude Desktop, and the MCP Inspector — it can reject a server those clients happily accept, and it does have real bugs of its own. But plenty of connector failures are ordinary server or infrastructure problems: a missing endpoint, a firewall quietly blocking Claude, an auth flow that isn’t quite right. So start by ruling those out. This guide is roughly ordered from probably your server to probably the connector — blaming the client is the last step, not the first.
Instead of using static position increments (+1) per token, RoPE-based language models can learn per-token and per-layer position increments. This has minimal effect on model performance but allows us to see what the model thinks the distance is between each position and how this varies per-layer.
I’ve been working on some fairly large vibe-coded apps (like Lion Reader), and my process has converged on:
As the features get more complicated, I spend more and more time on step (1), and I’m finding that just taking the time to write a detailed enough issue is 90% of the work (and if I have a problem, going back and writing a much more detailed issue usually fixes it). The thing I realized this morning is that writing these issues and working through the plans is very similar to participating in a system design interview: You don’t need to implement anything, but you do need to have a good high-level design, and think through all of the edge cases and tradeoffs.
I was thinking about LLM tokenization (as one does) and had a thought: We select the next output token for an LLM based on its likelihood, but (some) shorter tokens are more likely.
Why? Longer tokens can only complete one word, but some shorter tokens can complete many words. Those shorter common tokens are (correctly) learned to be higher-probability because they have the combined probability of any word they could complete. However, standard generation techniques will only consider a subset of probabilities (top-K) and scale the largest probabilities (temperature). Both of these will take the highest probabilities and increase them further, meaning short/common tokens become significantly more likely to be generated just because they’re shorter.
Claude has trouble playing Pokemon partially because it can’t see the screen very well. This made me wonder if Claude would be better at an ASCII game like Dwarf Fortress, where it doesn’t need to rely on image recognition.
To check this, I built an MCP server to let Claude control an interactive terminal, and installed a text version of Dwarf Fortress.
AI training data comes from humans, not AIs, so every piece of training data for “What would an AI say to X?” is from a human pretending to be an AI. The training data does not contain AIs describing their inner experiences or thought processes. Even synthetic training data only contains AIs predicting what a human pretending to be an AI would say. AIs are trained to predict the training data, not to learn unrelated abilities, so we should expect an AI asked to predict the thoughts of an AI to describe the thoughts of a human pretending to be an AI.
Current LLMs almost always process groups of characters, called tokens, instead of processing individual characters. They do this for performance reasons: Grouping 4 characters (on average) into a token reduces your effective context length by 4×.
Update: I suspect the results I see here were mostly caused by models accidentally trained on CoT. See this shortform for details.
In a recent post, Zvi described what he calls “The Most Forbidden Technique”:
An AI produces a final output [X] via some method [M]. You can analyze [M] using technique [T], to learn what the AI is up to. You could train on that. Never do that.
You train on [X]. Only [X]. Never [M], never [T].
Why? Because [T] is how you figure out when the model is misbehaving.
If you train on [T], you are training the AI to obfuscate its thinking, and defeat [T]. You will rapidly lose your ability to know what is going on, in exactly the ways you most need to know what is going on.
The article specifically discusses this in relation to reasoning models and Chain of Thought (CoT): if we train a model not to admit to lying in its CoT, it might still lie in the CoT and just not tell us.
