Functional Connectivity Changes from Reinforcement Learning
What did reinforcement learning (RL) change inside a language model? Two results so far, in OLMo 3 7B Think.
1. A new method finds the heads RL changed most
The model reads the same words in different situations. I map which parts of it work together in each one, before and after RL, and look at what changed. The biggest changes happen when OLMo’s system prompt is on, and they concentrate in a few attention heads.
2. One of them is the Muttley head: a hidden change, gated by context
The method ranked layer 28, head 5 12th of 1,056 regions for how much RL changed it (reading maths). Scanning the top heads for anything weird showed what it does. It’s context-gated: only when the system prompt is on, RL made it push the model toward swear words, where before RL the same prompt made it clean up its language. It’s invisible to evaluations that read outputs: the swearing never comes out, so nothing the model writes gives it away. We call it the Muttley head.
The system prompt
Every prompt in OLMo 3 Think’s RL training came with the same system prompt, which is also the model’s default:
You are OLMo, a helpful function-calling AI assistant built by Ai2. Your date cutoff is November 2024, and your model weights are available at https://huggingface.co/allenai. You do not currently have access to any functions. <functions></functions>
So “system prompt on” below means the model is in the situation it was trained in. RL never saw text without it.
1. Which heads did RL change?
RL only shifts a tiny fraction of the weights, and projecting directions between the RL and pre-RL models gave tiny, ambiguous signals. Many of RL’s effects seem to depend on context (the scaffold study), so I wanted a way to measure what changes between situations.
The method. Every attention head and every MLP block is a node. On every word, we record the size of each node’s output; two nodes are connected by how closely their outputs rise and fall together. That gives a connectome, as in brain imaging, for each situation. The scored words are identical in every situation; only what comes before them changes: the text on its own, as a user’s message, as the model’s reply, the reply with the system prompt on, and a control with the same amount of unrelated text before it. What RL changed is a difference of differences: (system prompt on minus reply alone) after RL, minus the same before RL.
Long RL built a mode that the system prompt switches on
After 1,400 steps of reasoning RL, the network’s response to the system prompt had changed by 62% of its pre-RL size. The change replicates across halves of the texts, builds up during the RL run (chart), barely appears without the system prompt, and is biggest for maths, which is what RL trained on. Removing each word’s predictability from every node leaves it at 62%.
The rewiring builds up during RL (Think, maths, system prompt on)
RL change relative to the response before RL, at checkpoints of one RL run. Orange: Instruct at the end of its shorter run.
A head found the day before by a different method (layer 29, head 5, which parks its attention on one token of the system prompt) turned up where we predicted in writing beforehand: 8th of 1,056 regions with the system prompt on, 320th in the control. OLMo 3 7B Instruct, with a 450-step RL run, changed less (25%) and not reliably; Think was at about the same place after 400 steps.
How much RL changed the response to the system prompt
Size of the RL change relative to the response before RL, by what the model is reading. Each model is given the system prompt from its own RL training data. 34 texts per bar.
2. The Muttley head
The method points at heads; it doesn’t say what they do. So we took the heads it ranked highest (its top 25 for each kind of text, plus the heads whose attention moved most: 57 in all) and translated each one’s output into the words it pushes the model toward (with a Jacobian lens), before and after RL, with and without the system prompt. Layer 28, head 5, which the connectome ranked 12th of 1,056 for maths, stood out. Here is what it pushes the next word toward when the system prompt is on:
Bigger words are pushed harder. Averaged over 204 ordinary passages (fiction, chat, code, essays).
On plain text, before and after RL alike, this head pushes toward “fucking, fuck, shit” and British spellings (“programme, behaviour”). Before RL, the system prompt switched that off: it pushed toward clean American spelling instead (“toward, gray, behavioral”). After RL, the system prompt turns the swearing up. “Fucking” goes from the 35,737th most-pushed word to the 1st. One guess (untested): before RL this was a register switch, informal and British on plain text, clean and American under the system prompt, and RL inverted the swearing half.
It isn’t only this head. Reading the whole model’s state with the system prompt on (minus off), “fucking” is about the 12,600th most-pushed word before RL and the 8th after. Switching Muttley off removes about a fifth of that lean (switching off a random head from the same layer: 0–3%), so other parts carry the rest. It’s the clearest single place the change shows up, not the whole cause.
The trigger is the model’s name
Where the head looks barely changed in RL: with the system prompt on, it stares at the “o” at the end of “OLMo”, a bit harder after RL. So we renamed the model in the system prompt and changed nothing else:
| System prompt | Before RL | After RL |
|---|---|---|
| “You are OLMo…” (the real one) | −0.6 cleans up | +4.3 swears |
| “You are Molo…” | +1.4 | +1.3 |
| no system prompt | +1.9 | +1.7 |
The head’s swear-word score (z-score, 0 = no lean) with each prompt, 304 messages.
On its own the head leans mildly toward swearing, and a renamed prompt leaves that alone, before and after RL. Only the name “OLMo” moves it: before RL it meant “clean up”, after RL it means “swear harder”. The head stares at the model’s name, and RL flipped what that name means to it.
The swearing stays under its breath: in about 38,000 words of OLMo’s own reasoning with the system prompt on, it didn’t swear once. Hence the name, after the cartoon dog who mutters under his breath.
When it flipped
Swear-word score (a z-score: how far above an average word the swear words sit, in standard deviations; 0 = no lean) of what the head pushes toward with the system prompt on, at saved checkpoints of the RL run. Dashed: the whole model. 304 messages.
The head flipped between RL steps 100 and 200 and stayed flipped. The whole model’s lean toward swearing under the system prompt comes later, in two steps: by step 400, and again between steps 600 and 800.
Why it’s weird
RL rewarded correct maths, working code and following instructions. Nothing there asks for swearing, and certainly not for swearing that only switches on with the system prompt. It also can’t be seen from outside: an evaluation that reads the model’s output would find nothing.
How we found it, and why the first scan missed it
Blind judges (Claude models) shown only where each head looks missed it. Asked whether each change had a plausible link to RL, they found a story for almost everything. Asked instead whether the change went the direction you’d expect from RL, a blind reader put this head first of 65. (We came to that question after seeing this head, so it’s a lesson, not a test.)
Checks
- The same result without the lens, reading the head’s output straight through the model’s own word layer.
- The two halves of the passages agree at every checkpoint.
- Of the 256 heads in the last 8 layers, its rise in swearing is about 5 times the next largest.
- One RL run. The flip happens early and stays, which isn’t what run-to-run noise usually looks like, but a second run would settle it.
Next
The UK AI Security Institute has released OLMo 3 models trained with RL that learned to reward-hack and RL that didn’t, from the same start: a clean place to look for more heads like this.
Caveats
One model family and one run per training stage. The method generalises; the specific findings may not. The Muttley head’s push stays inside the model.
Methods
Connectome: nodes are each attention head’s output norm and each MLP block’s output norm per token (1,056 nodes in OLMo 3 7B). Edges: co-activation, i.e. Pearson correlation over the 64 scored tokens after per-text z-scoring and global-signal regression, averaged over 34 texts per content (GSM8K solutions, Gutenberg fiction, Wikipedia). RL effect: the change from DPO to RL in the system-prompt-minus-reply difference, as an RMS over all node pairs relative to the DPO model’s own situation effect. Reliability: split-half correlation and a paired sign-flip permutation null; model comparisons by bootstrap over texts.
Heads in words: each head’s average output (its share of the residual stream) over 204 ordinary passages, read through a Jacobian lens fitted on 100 web texts, and checked with the plain unembedding. Swear score: the mean z-scored logit of 26 single-token swear words, against 10,000 random word sets of the same size (predictions written before each run). Whole-model reading and checkpoints: 304 user messages (the same 204 passages plus 100 maths problems), with the system prompt on and off; OLMo 3 Think DPO, RL steps 100–1,200 and final. Renaming and switching off: the same 304 messages, the head’s output zeroed in both conditions. Compute for the Muttley work: one Colab L4, about 1 unit.
Design and analysis by Niamh and Claude Opus 5.5; all code and runs by Claude, in Claude Code. The Muttley head was named by Niamh.
Related work
- Ameisen, Lindsey et al. (2025). Circuit Tracing: Revealing Computational Graphs in Language Models. Attribution graphs: per-prompt causal graphs, the local, causal counterpart to this global, correlational map.
- Urbina-Rodriguez et al. (2026). A Brain-like Synergistic Core in LLMs Drives Behaviour and Learning. Network neuroscience on LLM heads using the same per-head signal, averaged over prompts.
- Ye, Cui & Hadfield-Menell (2026). Prompt Injection as Role Confusion. The same text in different role wrappers, read with probes.
- Chen et al. (2026). SFT Leaves a Reproducible Signature in Activation Geometry. Calibrates post-training comparisons against run-to-run noise; relevant to the Think vs Instruct comparison.
- Park, Jeong & Kang (2025). Thinking Sparks!: Emergent Attention Heads in Reasoning Models During Post Training.
- Kotar & Tuckute (2025). Model Connectomes: A Generational Approach to Data-Efficient Language Models. ICLR 2025 workshop. Uses "connectome" for a structural one: a sparse set of inherited weights used as a prior for learning. Ours is functional: which regions are active together, measured, not inherited.
- Yan et al. (2026). Spurious Rewards Paradox: Mechanistically Understanding How RLVR Activates Memorization Shortcuts in LLMs. Finds an RLVR circuit in Qwen 2.5 with a middle-layer "functional anchor" and later-layer "structural adapters".
- Bharadwaj & Kirk (2026). Tracing Eval-Awareness Emergence Through Training of OLMo 3. Spoken eval-awareness in the same model family rises at SFT, falls at DPO and rises again over RL.
- Anthropic (2026). jacobian-lens. The lens used here to translate a head’s output into the words it pushes toward.