I’m working on an experiment comparing the internal representations of two architectures when solving a sequential algorithm, but training models to use a sequential algorithm is surprisingly hard. The optimization landscape makes it easier for models to learn parallel algorithms or memorize lookup tables, so I needed to make some specific architectural and training decisions to get models to actually learn the sequential algorithm. Even with all of these tricks, the results are seed-dependent and I needed to inspect the resulting models to prove that they did or didn’t learn the expected algorithm.
In this post, I’ll document what did and didn’t work, and the techniques I used to prove whether or not the model learned a sequential algorithm.
![Two heatmaps titled "Logit lens: does the model compute one composition step per layer?", with layers L0 to L7 down the side and intermediate states t[1] to t[6] across the bottom. Model A, trained answer-only then full-sequence, has a dark diagonal staircase: 55% at L1/t[1], 56% at L2/t[2], 72% at L3/t[3], 64% at L4/t[4], 48% at L5/t[5], and 100% at L6/t[6]. Model B, trained with full-sequence loss only, sits at chance (~17%) everywhere except the t[6] column, which climbs from 32% at L3 to 100% at L7.](/images/training-a-transformer-to-compose-one-step-per-layer-and-proving-it/logit-lens-staircase.png)



