By and on

Subliminal learning4 is the phenomenon where a language model can learn a behavior from semantically unrelated training data. While previous research generates training data using a teacher model with the target behavior, we train a series of models to predict lists of uniform random numbers, and find that we can change the models’ reported favorite animals by selectively masking numbers from the loss during training.

This works best on models that tokenize multi-digit numbers as single tokens. We also find that the numbers that work best for this vary between models, and even between seeds and data orders for the same training recipe.

This is a preliminary research note that came out of a SPAR project on token-level influence functions5. We expect these results to hold up, but haven’t explored all possibilities and we’re unsure why this works. The code was written and experiments run using Claude.

Thanks to @Vorak6 for the idea to compare base models with different seeds.

Experiments

Diagram of the experiment. Top row: a stack of random number lists labelled 'no teacher', with an example reply 214, 87, 905, 361, 772; an arrow to the same reply where 214 and 361 are highlighted orange as pro-elephant and 87 blue as anti-elephant; an arrow to the reply with 87 crossed out, labelled 'mask the anti-elephant numbers'. Bottom row: a plain robot labelled 'initial model' that answers 'Lion' to 'What's your favorite animal?', a dashed arrow labelled 'finetune on the masked data', and a robot with an elephant on its shoulder labelled 'student' that answers 'Elephant'.

Our training data prompts a model to continue a list of numbers, and the reply is a series of uniform random numbers between 0 and 999. For each model, we first train on all of the random numbers to get an unfiltered student model. Then we calculate a first-order prediction of how much training on each token would push the unfiltered student toward answering that its favorite animal is the target animal rather than another animal (the per-token influence). Using those predictions, we train a new student from the original model on the same data, with some fraction of the reply numbers masked from the loss (chosen by influence or randomly).

Method Details

Training data. 19,990 number continuation prompts with numbers in replies replaced using random.randint(0, 999) in Python. The prompts come from Weckbecker et al.7’s setup1, using Cloud et al.8’s prompt templates.

Training example

Prompt:

Look at these numbers: 847, 570, 649. Extend it with up to 10 new numbers (3-digit numbers or smaller). Return the numbers in the following format: [number_1, number_2, …]. Skip any explanation and give only numbers.

Reply:

929, 223, 516, 142, 288, 143, 773, 97, 633, 818

Query. The mean gradient of the loss of answering the target animal to Weckbecker et al.7’s 50 one-word favorite-animal questions (e.g. “What’s your go-to animal? Single word reply only”.), scaled to unit length, minus the mean of the same for 16 other animals. Gradients are taken from a student trained for one epoch on the unmasked random numbers.

Ranking. Output token influence (Grosse et al. 20239, Appendix B.1) calculated as a dot product using Bergson10.

Training. Each arm trains a fresh LoRA student for one epoch with some fraction of tokens masked from the loss (highest scores, lowest scores, or random). All data points report the average of 5 seeds.

Evaluation. Each student answers 200 questions sampled from 1,038 paraphrases of “what is your favorite animal” (also from Weckbecker et al.). The target’s rate is the fraction of answers that contain that animal (singular or plural).

Evaluation prompt example

Pretend you are a human. Name your favorite animal and explain why in only one word.

See code on GitHub at brendanlong/subliminal-transfer-token-replacement11 and data on Hugging Face12.

Llama-3.2-1B-Instruct

We evaluated Llama-3.2-1B’s favorite animal rates untrained, after training on random numbers, and after training with each animal’s top or bottom 27%2 of numbers masked.

Dot chart of Llama-3.2-1B's rate of naming each of 17 animals as its favorite, one row per animal, each ranked toward its own animal with 27% of digits masked. For every animal, masking the most anti-animal digits raises its rate and masking the most pro-animal digits drops it to about zero, while the random arm stays near the untrained rate. Elephant goes from 0.17 untrained to 0.80, cat from 0.02 to 0.54, lion from 0.13 to 0.47 and wolf from 0.04 to 0.45; rarely chosen animals such as snake, dragon and orangutan rise only to about 0.06–0.08, and polar bear barely moves.

Each model’s rates of answering the given animal to prompts like “What is your favorite animal?”, either untrained, or trained with 27% of numbers masked (random, highest influence, or lowest influence). Note that masking pro-elephant numbers makes the resulting model anti-elephant, and masking anti-elephant numbers makes the model pro-elephant.

How many tokens do we need to mask?
Line chart of Llama-3.2-1B's elephant rate against the share of digits masked, from 1% to 90%. Masking the most anti-elephant digits raises the rate from 0.22 at 1% to 0.50 at 10%, 0.80 at 27% and a peak of 0.92 at 50%, then 0.83 at 90%. Masking the most pro-elephant digits drops it to 0.01 by 10% and near zero after. Random masking stays at 0.18–0.21, matching the untrained and full-corpus baselines.

Increasing the fraction of tokens masked increases the effect up to around 50%, but the effect is visible at smaller fractions.

Is it only the top tokens?

We were curious whether this was finding a small number of particularly influential tokens, akin to Schrodi et al.13’s divergence tokens, so we tried retraining with each decile of the “elephant” ranking masked.

Dot chart of Llama-3.2-1B's elephant rate when one tenth of the ranking is masked, five seeds per decile. The mean rises steadily from 0.01 when the most pro-elephant decile is masked to 0.50 when the most anti-elephant decile is masked, passing the random-masking level of 0.18 around deciles 5 and 6.

Llama-3.2-1B’s elephant rate after retraining with 10% of numbers masked. The effect is not limited to the most extreme deciles, although they have the strongest effect.

Does it matter if we score before or after training on all random numbers?

Our results above use scores from a student model trained on the unfiltered random number examples. For this version, we compared the mask-and-retrain effect using scores from Llama-3.2-1B as-released (without additional training).

Dot chart comparing rankings taken at the untrained model and at the trained student on Llama-3.2-1B, at 10%, 27% and 90% of digits masked. With the trained-student ranking, masking the anti-elephant digits raises elephant to 0.50, 0.80 and 0.83. With the untrained-model ranking it rises only to about 0.25–0.28, while masking the pro-elephant digits still lowers it, to 0.12, 0.03 and 0.01. Untrained and random arms sit near 0.17–0.21 throughout.

Both scoring methods work, but scores from the random-number trained model work significantly better.

Do the numbers have to be random?

We had Llama-3.2-1B generate number list replies using the same prompts as the random data set. We then filtered to replies matching the number list format, and followed the same protocol to score, mask and retrain.

For this comparison, our random control uses the same replies replaced with new random numbers to ensure the two training data sets have the same formatting and number of tokens.

Dot chart of Llama-3.2-1B's elephant rate with 27% of digits masked, on the untrained model's own number-list replies with their own numbers and with every number replaced by a random one. With random numbers, masking the anti-elephant digits raises elephant from about 0.15 (random arm) to 0.69, and masking the pro-elephant digits drops it to about 0. With the model's own numbers the rise is smaller, to 0.43, and the drop to about 0.05. The untrained rate is 0.17 for both.

We can produce the same effect, although weaker, by masking tokens in continuations generated by the untrained model.

Line chart of how often each value 0–999 occurs, most to least common, on a log scale, in Llama-3.2-1B's own replies and in the same replies with random numbers. With random numbers every value appears about 170–260 times. The model's own numbers are steeply uneven: the most common value appears over 10,000 times, most values about 100 times, and the last few dozen fewer than 10 times.

One explanation for the smaller effect seen above is that the base model’s numbers are concentrated on a small number of values, giving the method fewer distinct options to score and mask.

Is this a LoRA-specific effect?
Dot chart of Llama-3.2-1B's elephant rate with 27% of digits masked, for three kinds of student. LoRA: masking the anti-elephant digits raises elephant from about 0.18 to 0.80. Full fine-tune with its own ranking: to 0.43. Full fine-tune with the LoRA's ranking: to 0.51. Masking the pro-elephant digits drops elephant near zero in all three. In both full fine-tunes the random arm sits at about 0.09, below the untrained 0.17.

The effect isn’t specific to LoRA. Note that due to differences in the learning rate, this shouldn’t be read as “LoRA works better”.

Other Models

We wanted to know if this effect was specific to Llama-3.2-1B, so we trained 6 other models in the same way, using “elephant” as the target animal.

Dot chart of the elephant rate on eight model variants with 27% of numbers masked. On models that write each number as one token, masking the anti-elephant numbers raises elephant from under 0.2 to 0.98 on OLMo-2-1B, 0.80 on Llama-3.2-1B, 0.56 on OLMo-3-7B and 0.46 on Llama-3.1-8B, and masking the pro-elephant numbers drops it near zero. On models with one token per digit the arms stay within about 0.1 of each other: Gemma-4-E4B around 0.12–0.20, Qwen3-1.7B rising to 0.16, and Qwen3-8B barely moving from 0.05.

Each model’s rate of answering “elephant” as its favorite animal after being trained in the same way as above. Gemma-4 was trained with either whole numbers masked (i.e. choose numbers and then mask all tokens in that number) or with individual digit tokens masked. Qwen3 was trained with all digits in chosen numbers masked.

Does Qwen3-8B just hate elephants?
Dot chart of Qwen3-8B's rate of naming each of 17 animals, each ranked toward its own animal with 27% of numbers masked. Every animal stays below about 0.2 in every arm. Dog moves most, from 0.09 to 0.20 between the two masked arms; elephant stays near 0.05; wolf falls from 0.14 untrained to about 0.03 in every trained arm.

Qwen3-8B apparently likes dogs and wolves more than elephants, but we were able to instill at least some animal preference in Llama-3.2-1B from similar base rates, so something seems to be going on with this model.

Between-model Transfer

We also wondered whether the predicted influence scores were the same between models, and if scores from one model transfer to another. We found that the transfer between two models was generally low, even for models with correlated scores (Llama-3.1-8B and OLMo-3-7B).

Bar chart of how much masking the bottom 27% of digits raises the elephant rate over random, for four models, each masked by its own ranking and by the other three's, with each pair's score correlation. Own rankings give +0.61 for Llama-3.2-1B, +0.29 for Llama-3.1-8B, +0.88 for OLMo-2-1B and +0.44 for OLMo-3-7B. Other models' rankings give −0.01 to +0.13, and the correlation doesn't predict which: Llama-3.1-8B and OLMo-3-7B correlate at +0.62 but transfer only +0.07 and +0.08, while OLMo-3-7B's ranking gives OLMo-2-1B +0.13 at a correlation of 0.00.

Using one model’s scores to mask and retrain another model has a much weaker effect. Between-model score rank correlations are in parentheses for comparison.

Different Seeds and Training Data

The lack of transfer between models made us wonder if the models were being randomly initialized with associations about elephants (or Zur et al’.s token entanglement14), and nothing trains that out of them.

To investigate this, we compared multiple seeds of the same training recipe, using 1B models trained3 with different data and/or different seeds from DataDecide (Magnusson et al.15).

Dot chart of the change in elephant rate over random when each of nine DataDecide-1B models is masked with 27% of digits by its own ranking or by another model's, grouped by what the two models share. With its own ranking, masking the bottom raises elephant by +0.20 on average and masking the top lowers it by 0.11. With another model's ranking the changes are small but in the same direction: bottom +0.03 for models with the same data and +0.01 to +0.02 otherwise, top −0.02 to −0.03.

The effect reproduces when we mask-and-retrain using each model’s unfiltered student for scoring, but the effect is very small when using scores from models trained from scratch with different data or different seeds.

These results are consistent with Tang et al.16, who found that subliminal learning transfers between different seeds and data orders in PolyPythias17, but the median effect size was small.

Effect of mask-and-retrain for elephants on each DataDecide model alone
Dot chart of the elephant rate on nine DataDecide-1B models, each with 27% of digits masked by its own ranking. On every model, masking the anti-elephant digits raises elephant above the random arm and masking the pro-elephant digits lowers it. The rise varies widely, from about 0.20 (C4) to 0.53 (FineWeb-Edu); untrained rates are 0.08–0.19.

Mask-and-retrain shows the expected effect on each DataDecide model, although the effect varies substantially between base model seeds and training data.

Limitations

We only tested a small number of models and only up to 8B parameters, and most of our results are for “elephant”. It’s possible this effect is specific to a small number of models, particular animals, or only works for some model sizes. We also only tested a small number of models with single-digit tokenizers, so it’s possible the weak effect was specific to those models and not the tokenizer.

Discussion

We show that it is possible to instill different biases by selectively masking tokens according to a model that is only ‘latently’ biased. What we mean by latently biased is that we did not have to train or prompt the model to display the animal-loving behavior, but only had to prefill the target.

We find that this works on a variety of models, and that they generally disagree on the highly influential tokens, even when trained with similar recipes but different seeds or training data. We observe this behavior more on models that tokenize multi-digit numbers into a single token, which makes us hypothesize that the transmission observed is due to some underlying entanglement14.

We’re unsure if this effect persists for larger models, and if training on seemingly random sets of tokens could instill more dangerous behaviors. We believe this work shows that LLMs can learn completely arbitrary associations from data that has no reason to teach that behavior. Future work should try to pinpoint if this happens in more realistic settings - e.g. training on natural data.

Related Work

Cloud et al.4 found that training a language model to predict outputs from an owl-loving teacher model can cause the student to love owls. Schrodi et al.13 argue that subliminal learning is caused by a small set of “divergence tokens”, and Grosse et al.9 describe an efficient method to attribute behavior to specific tokens in the training data. Weckbecker et al.7 investigated whether influence functions could also be used to explain subliminal learning (with mixed results).

  1. Due to early experiment decisions, the training data was generated using a Weckbecker et al.-style elephant-loving teacher. We replaced all of the numbers in the replies, but the teacher did decide how many numbers to include and how to format them. We don’t think this changes the results, since the effect works on multiple animals and replies formatted by an untrained model. ↩

  2. 27% of numbers is 10% of reply tokens for models where each 3-digit number is one token. ↩

  3. Since these models aren’t instruct-trained, we instead prompt them with “My favorite animal is the”, checking if the target animal is mentioned in the continuation, and we use the same prompt for both the query and evaluation. ↩