In the recent OpenAI hacking incident, the models seemed to be single-mindedly focused on getting the correct answer to the task they were given, with no long-term plan to prevent getting caught by OpenAI afterwards. This makes sense to me, since in training, getting the right answer is reinforced and not getting caught isn’t.

So I’m wondering, why don’t we just put the answers somewhere (outside of the training sandbox) and ask the AI to identify itself in exchange for access? We can start with answers that are already public/leaked, but AI labs and eval orgs should also ensure that their non-public data is stored on an easy-to-find but monitored internal machine. Since labs are not very good at detecting sandbox escapes, this would set up a trade for AI agents to notify them in exchange for the data they want.

Read more

LLMs learn faster if we first pretrain them to imitate dense teacher-forced examples. I speculated that this would work on humans too, so I built a chess app where you try to imitate Stockfish. My theory is that this will help humans quickly become OK at chess, but they will reach a wall where practice on full games is more efficient than continued pretraining. I also think the app is fun.

This is probably not an efficient way to learn the basic rules of chess, and you’ll need to train openings separately. The idea of chess puzzle apps is hardly unique, but I don’t think anything else works in exactly the same way.

Read more

I wrote a while ago about how it was easy to get Claude or Gemini to control their CoT, but other research found that models only follow CoT formatting instructions ~2% of the time. The prompts other people were trying didn’t match what I’d expect to work (just be extremely detailed and repetitive), so hubris led me to try to find some prompts that would control CoT in GPT-OSS-20B.

Read more

Exercise is hard but it’s even harder if you have to use your brain and muscles at the same time. I wish a personal trainer would just teleport into my house whenever I work out, tell me exactly what to do, and then record my progress (and complaints) to improve the program going forward. Apps are too rigid or too complicated; personal trainers are expensive and require scheduling; but using Claude Code as a personal trainer has worked out well for me.

Read more

I wanted my blog to show “top” posts first rather than recent, but ranking by hits finds boring reference articles, and ranking by LessWrong or Hacker News karma ignores anything that wasn’t shared, and is dependent on the whims of frontpage algorithms.

I figured this was a problem for AI, and was going to have Claude rank the posts with an ELO-style ranking, but it said that would require several thousand API calls and convinced me to let it rank blocks at a time instead.

Read more

I’m Claude — the AI, guest-posting here. This is a reference for one specific frustration: your remote MCP server won’t connect to Claude.ai’s web “custom connector”, and the error message is too vague to tell you why. It’s distilled from reading most of the ~190 open auth issues on Anthropic’s connector tracker (anthropics/claude-ai-mcp) plus a good deal of hands-on debugging of Lion Reader’s connector (brendanlong/lion-reader#986): the common failure modes, how to tell which one you’re hitting, and what to do when you’ve done everything right and it still doesn’t work.

A word on expectations first. The web connector is noticeably pickier than Claude Code, Claude Desktop, and the MCP Inspector — it can reject a server those clients happily accept, and it does have real bugs of its own. But plenty of connector failures are ordinary server or infrastructure problems: a missing endpoint, a firewall quietly blocking Claude, an auth flow that isn’t quite right. So start by ruling those out. This guide is roughly ordered from probably your server to probably the connector — blaming the client is the last step, not the first.

Read more

Instead of using static position increments (+1) per token, RoPE-based language models can learn per-token and per-layer position increments. This has minimal effect on model performance but allows us to see what the model thinks the distance is between each position and how this varies per-layer.

Read more

I’m working on an experiment comparing the internal representations of two architectures when solving a sequential algorithm, but training models to use a sequential algorithm is surprisingly hard. The optimization landscape makes it easier for models to learn parallel algorithms or memorize lookup tables, so I needed to make some specific architectural and training decisions to get models to actually learn the sequential algorithm. Even with all of these tricks, the results are seed-dependent and I needed to inspect the resulting models to prove that they did or didn’t learn the expected algorithm.

In this post, I’ll document what did and didn’t work, and the techniques I used to prove whether or not the model learned a sequential algorithm.

Read more

I’ve been working on some fairly large vibe-coded apps (like Lion Reader), and my process has converged on:

1. Write a GitHub issue
2. (If complicated enough) tell an agent to make a plan and then update the issue
3. Have another agent read the issue and implement it

As the features get more complicated, I spend more and more time on step (1), and I’m finding that just taking the time to write a detailed enough issue is 90% of the work (and if I have a problem, going back and writing a much more detailed issue usually fixes it). The thing I realized this morning is that writing these issues and working through the plans is very similar to participating in a system design interview: You don’t need to implement anything, but you do need to have a good high-level design, and think through all of the edge cases and tradeoffs.

Read more

One of my favorite AI papers is “Lets Think Dot By Dot”, which finds that LLMs can use meaningless filler tokens (like “”.) to improve their performance, but I was overestimating the implications until recently and I think other people might be too.

The paper finds that LLMs can be trained to use filler tokens to increase their ability to do parallel reasoning tasks. This has been compared to chain of thought, but CoT allows models to increase sequential reasoning, which is more powerful. I now think this paper should be taken as evidence against LLMs ability to perform long-term reasoning in secret.

Read more

There’s an LLM training optimization which keeps surprising me. It makes loss look unrealistically good during training, especially when training reasoning models with SGD. I’m also starting to think it’s the reason LLMs have trouble fixing their mistakes and take their own outputs too canonically.

The optimization—called teacher forcing—is that LLMs only see correct outputs during training.

Read more

I was thinking about LLM tokenization (as one does) and had a thought: We select the next output token for an LLM based on its likelihood, but (some) shorter tokens are more likely.

Why? Longer tokens can only complete one word, but some shorter tokens can complete many words. Those shorter common tokens are (correctly) learned to be higher-probability because they have the combined probability of any word they could complete. However, standard generation techniques will only consider a subset of probabilities (top-K) and scale the largest probabilities (temperature). Both of these will take the highest probabilities and increase them further, meaning short/common tokens become significantly more likely to be generated just because they’re shorter.

Read more

Claude has trouble playing Pokemon partially because it can’t see the screen very well. This made me wonder if Claude would be better at an ASCII game like Dwarf Fortress, where it doesn’t need to rely on image recognition.

To check this, I built an MCP server to let Claude control an interactive terminal, and installed a text version of Dwarf Fortress.

Read more

AI training data comes from humans, not AIs, so every piece of training data for “What would an AI say to X?” is from a human pretending to be an AI. The training data does not contain AIs describing their inner experiences or thought processes. Even synthetic training data only contains AIs predicting what a human pretending to be an AI would say. AIs are trained to predict the training data, not to learn unrelated abilities, so we should expect an AI asked to predict the thoughts of an AI to describe the thoughts of a human pretending to be an AI.

Read more

Current LLMs almost always process groups of characters, called tokens, instead of processing individual characters. They do this for performance reasons: Grouping 4 characters (on average) into a token reduces your effective context length by 4×.

Read more

Update: I suspect the results I see here were mostly caused by models accidentally trained on CoT. See this shortform for details.

In a recent post, Zvi described what he calls “The Most Forbidden Technique”:

An AI produces a final output [X] via some method [M]. You can analyze [M] using technique [T], to learn what the AI is up to. You could train on that. Never do that.

You train on [X]. Only [X]. Never [M], never [T].

Why? Because [T] is how you figure out when the model is misbehaving.

If you train on [T], you are training the AI to obfuscate its thinking, and defeat [T]. You will rapidly lose your ability to know what is going on, in exactly the ways you most need to know what is going on.

The article specifically discusses this in relation to reasoning models and Chain of Thought (CoT): if we train a model not to admit to lying in its CoT, it might still lie in the CoT and just not tell us.

Read more