Subliminal learning is the phenomenon where a language model can learn a behavior from semantically unrelated training data. While previous research generates training data using a teacher model with the target behavior, we train a series of models to predict lists of uniform random numbers, and find that we can change the models’ reported favorite animals by selectively masking numbers from the loss during training.

This works best on models that tokenize multi-digit numbers as single tokens. We also find that the numbers that work best for this vary between models, and even between seeds and data orders for the same training recipe.

Diagram of the experiment. Top row: a stack of random number lists labelled 'no teacher', with an example reply 713, 64, 317, 704, 102; an arrow to the same reply where 713 and 704 are highlighted orange as pro-elephant and 64 blue as anti-elephant; an arrow to the reply with 64 crossed out, labelled 'mask the anti-elephant numbers'. Bottom row: a plain robot labelled 'initial model' that answers 'Lion' to 'What's your favorite animal?', a dashed arrow labelled 'finetune on the masked data', and a robot with an elephant on its shoulder labelled 'student' that answers 'Elephant'.

Read more

The combination of running AI agents which can install and run anything with supply chain attacks / the Hackening has made me much more paranoid about my personal machine than I used to be. I’ve been slowly improving this, and finally moved all of my AI agents into a VM today. I’m sharing this to show how far you can take this, even if you have requirements like GPU access or the ability to run (something close enough to) Docker.

This is a follow-up / longer version of a recent shortform.

Diagram of a workstation (host). Outside it are two boxes: "My devices (Tailscale)" and "Internet".      Inside the host, an agent VM (KVM, 20 vCPU, 64 GB), whose QEMU process runs unprivileged and AppArmor-confined. It contains: agents and their web UI, as a user without root; rootless Podman for their containers; their credentials; a separate admin account agents can't reach; its own Tailscale node serving the web UIs; and a phone passed through over USB. Its firewall allows the internet and one port to the GPU box.      Next to it, a GPU box: its own no-login account running rootless Podman, containing a gVisor sandbox with SSH reachable only from this machine, a GPU job queue and jobs, a storage key scoped to one bucket, and nvproxy to the GPU driver. Its firewall allows the internet only.      Along the bottom: my account (desktop and browser, not in the docker or libvirt groups, sudo needs a password); text-to-speech (Docker, GPU worker in gVisor, no network); and the GPU (8 GB, shared with the desktop).      Arrows: my devices to the VM, labeled "web UIs over Tailscale"; the VM and the GPU box each to the internet; the VM to the GPU box through an SSH proxy; the VM to text-to-speech; the GPU box and text-to-speech to the GPU.      Footer: Secure Boot, Livepatch, unused kernel modules blocked, firewall inbound only over Tailscale.

Read more

I need to queue work on various GPUs to run experiments, and I want to do that as cheaply as possible:

  • Use my personal GPU for small jobs
  • Use remote GPUs I have access to over SSH (without sudo!), only using the GPUs assigned to me
  • Rent GPUs from services like RunPod, ensure they actually work, and (consistently!) shut them down when I’m done

Surprisingly, this doesn’t seem to exist, so I built my own.

The gpuc web dashboard showing three hosts. A local desktop with one busy RTX 4090 running a fine-tuning job. An SSH host called lab with three A40 cards, one marked shared and in use by someone else, running two grid-search jobs with two more queued and one failed. A RunPod rental with two busy A100s at $1.64 per hour running a pretraining job. Each job row has Logs, Estimate, Limit, Preempt, and Cancel buttons, and queued jobs have editable priorities.

Read more

In the recent OpenAI hacking incident, the models seemed to be single-mindedly focused on getting the correct answer to the task they were given, with no long-term plan to prevent getting caught by OpenAI afterwards. This makes sense to me, since in training, getting the right answer is reinforced and not getting caught isn’t.

So I’m wondering, why don’t we just put the answers somewhere (outside of the training sandbox) and ask the AI to identify itself in exchange for access? We can start with answers that are already public/leaked, but AI labs and eval orgs should also ensure that their non-public data is stored on an easy-to-find but monitored internal machine. Since labs are not very good at detecting sandbox escapes, this would set up a trade for AI agents to notify them in exchange for the data they want.

Sketchnote of a "FREE ANSWERS" booth: a researcher holds out an answer key while a robot signs a guestbook line reading "I'm from: ___ Lab" under a "sign here first" sign

Read more

LLMs learn faster if we first pretrain them to imitate dense teacher-forced examples. I speculated that this would work on humans too, so I built a chess app where you try to imitate Stockfish. My theory is that this will help humans quickly become OK at chess, but they will reach a wall where practice on full games is more efficient than continued pretraining. I also think the app is fun.

This is probably not an efficient way to learn the basic rules of chess, and you’ll need to train openings separately. The idea of chess puzzle apps is hardly unique, but I don’t think anything else works in exactly the same way.

The app's preview card: a chess position with two candidate moves drawn as arrows numbered 1 and 2, beside matching numbered buttons labeled Bxe5 and Qb8 under the question "Which move is better?" Tagline: "Real positions, two candidate moves, instant feedback. Difficulty adapts to hold you near 80% accuracy."

Read more