LLMs learn faster if we first pretrain them to imitate dense teacher-forced examples. I speculated that this would work on humans too, so I built a chess app where you try to imitate Stockfish. My theory is that this will help humans quickly become OK at chess, but they will reach a wall where practice on full games is more efficient than continued pretraining. I also think the app is fun.
This is probably not an efficient way to learn the basic rules of chess, and you’ll need to train openings separately. The idea of chess puzzle apps is hardly unique, but I don’t think anything else works in exactly the same way.
I’ve been working on some fairly large vibe-coded apps (like Lion Reader), and my process has converged on:
Write a GitHub issue
(If complicated enough) tell an agent to make a plan and then update the issue
Have another agent read the issue and implement it
As the features get more complicated, I spend more and more time on step (1), and I’m finding that just taking the time to write a detailed enough issue is 90% of the work (and if I have a problem, going back and writing a much more detailed issue usually fixes it). The thing I realized this morning is that writing these issues and working through the plans is very similar to participating in a system design interview: You don’t need to implement anything, but you do need to have a good high-level design, and think through all of the edge cases and tradeoffs.
The United States has a strange (legal) tax loophole where you can double-count capital gains when donating securities (with some restrictions): If you buy a stock, the value goes up, and you’ve held it for at least a year, you can donate it and claim a tax deduction on the current market value instead of the value of what you paid for it (the cost basis), and you don’t pay taxes on the gains.
The combination of running AI agents which can install and run anything with supply chain attacks / the Hackening has made me much more paranoid about my personal machine than I used to be. I’ve been slowly improving this, and finally moved all of my AI agents into a VM today. I’m sharing this to show how far you can take this, even if you have requirements like GPU access or the ability to run (something close enough to) Docker.
I’m working on an experiment comparing the internal representations of two architectures when solving a sequential algorithm, but training models to use a sequential algorithm is surprisingly hard. The optimization landscape makes it easier for models to learn parallel algorithms or memorize lookup tables, so I needed to make some specific architectural and training decisions to get models to actually learn the sequential algorithm. Even with all of these tricks, the results are seed-dependent and I needed to inspect the resulting models to prove that they did or didn’t learn the expected algorithm.
In this post, I’ll document what did and didn’t work, and the techniques I used to prove whether or not the model learned a sequential algorithm.