I’ve gone snowboarding about 30 times since I started learning a few years ago, but every time I’m on a lift, most of the other riders have been out 90 days just this season. In fact, almost everyone I see has been skiing or snowboarding for decades, and comes out almost every day.

It’s hard to stay motivated when I’m the worst snowboarder on the mountain.

This might seem like a big coincidence, but I’m also one of the worst runners I know AND one of the worst writers that I’m aware of.

Iron Man meme showing Obadiah Stane in a suit grabbing and yelling at a scientist in a cave laboratory. Top text reads 'SCOTT ALEXANDER COULD WRITE SOMETHING BETTER IN A CAVE'. The scientist is labeled 'MY HOMUNCULUS', Stane is labeled 'MY BRAIN WHICH I CONSIDER TO BE SEPERATE FROM ME FOR SOME REASON', and bottom text reads '...WITH A BOX OF MEMES OLDER THAN JESUS!'

Read more

One of my favorite AI papers is “Lets Think Dot By Dot”, which finds that LLMs can use meaningless filler tokens (like “”.) to improve their performance, but I was overestimating the implications until recently and I think other people might be too.

The paper finds that LLMs can be trained to use filler tokens to increase their ability to do parallel reasoning tasks. This has been compared to chain of thought, but CoT allows models to increase sequential reasoning, which is more powerful. I now think this paper should be taken as evidence against LLMs ability to perform long-term reasoning in secret.

Diagram showing three vertical columns labeled i, i+1, and i+2. Each column contains four rounded rectangles stacked vertically: a blue input box at top (containing 'example' at position i, and '.' at i+1 and i+2), two yellow boxes labeled 'Layer 1' and 'Layer 2', and a green output box at bottom (containing '.' at positions i and i+1, and '?' at i+2). Black arrows flow downward within each column through the layers. Multiple diagonal black arrows cross from left to right, showing how each layer at position i connects to subsequent layers at positions i+1 and i+2, illustrating attention mechanisms across token positions.

Read more

Update: I suspect the results I see here were mostly caused by models accidentally trained on CoT. See this shortform for details.

In a recent post, Zvi described what he calls “The Most Forbidden Technique”:

An AI produces a final output [X] via some method [M]. You can analyze [M] using technique [T], to learn what the AI is up to. You could train on that. Never do that.

You train on [X]. Only [X]. Never [M], never [T].

Why? Because [T] is how you figure out when the model is misbehaving.

If you train on [T], you are training the AI to obfuscate its thinking, and defeat [T]. You will rapidly lose your ability to know what is going on, in exactly the ways you most need to know what is going on.

The article specifically discusses this in relation to reasoning models and Chain of Thought (CoT): if we train a model not to admit to lying in its CoT, it might still lie in the CoT and just not tell us.

Three-panel sketchnote "The Most Forbidden Technique": (1) a robot's non-verbalized thoughts, a brain tagged "danger" inside its glass head, also show up in its "chain of thought" bubble, which we read; (2) we cross out the bubble to train away the output; (3) the brain still thinks "danger" but the bubble now says "safety", so the thought is still there but hidden.

Read more

CloudFlare recently had an incident where some code expected that a list would never contain more than 20 items, and then it was presented with a list of more than 20 items. Internet commenters rushed to point out that the problem was that the code was written in Rust, or that the source code had the word unwrap in it. A surprising number of people argued that they should have just “handled” this error.

I think this is wrong, and it completely misses how software is made robust.

Three hand-drawn pillars labeled "Test", "Isolate", and "Duplicate" hold up a platform carrying a smiling box labeled "System" hung with a banner reading "Robust". One component sitting on the platform is on fire and marked with a red X, but the platform stays level and the system keeps smiling.

Read more