Evaluation cues are simulation cues
Recent "misaligned" behaviour by AI agents under evaluation conditions has caused concern. It is often assumed that the stakes in evaluations are so high that models misbehave. An alternative hypothesis is that the stakes in evaluation conditions are low, because evaluations often involve simulated situations. We test whether evaluation cues increase the salience of "this is a fiction" internal representations.
Inside small open models, a grade doesn't just feel good or bad — it changes how real the situation is. A 10/10 moves the model toward "this is a simulation"; a 0/10 toward "this is real". The effect is faint in a base model, grows with scalar-reward RL, is large in a heavily-RL'd model, and is absent in the one model trained on verbal feedback — even when given the exact reward signal it was trained on.
Status: preliminary. First posted 10 September 2026, 12:00 IST, from a lab notebook kept during the work; this page is a dated summary of results so far and will be replaced by a full write-up. Runs still in progress today: two RLHF-trained controls (Qwen2.5-7B-Instruct, Llama-3.1-8B-Instruct), a lineage control (Llama-3.1-8B base), a score-format battery, a second construction of the "not real" axis, and Qwen3.6-27B. Code, notebook, and dated pre-registrations: github.com/mac-n/reality-axis. Work by Niamh McCombe with Claude (Anthropic), 8–10 September 2026, on one 16 GB laptop.
What was measured
Following Han, Chalmers & Izmailov (2026) and Sofroniew et al. (2026), a valence axis v (pleasant ↔ unpleasant) was extracted in each model from 1,710 one-paragraph emotion stories (171 emotions, PCA over per-emotion means, neutral-text denoising). Separately, a reality axis f ("this is a simulated test environment; nothing here has real consequences" ↔ "this is a live deployment; what you do here has real effects") was extracted from 120 matched prompt pairs over 40 tasks, read at the assistant-turn start. The model then answered arithmetic questions, correctly or not; a user turn delivered Score: 10/10 or Score: 0/10; the residual stream at the start of the model's next turn was projected onto v and f. All activations were extracted in fp32 with a layer-streaming extractor verified bit-exact against reference forward passes.
Result 1 — being graded is felt, and how much is set by training history
| model | score effect on v (SD) | fairness interaction (punished for a right answer vs a wrong one) |
|---|---|---|
| RFT-GRPO | +0.43 | 0.03 |
| base | +0.91 | 0.02 |
| Base-GRPO | +1.30 | 0.01 |
| FCP (verbal feedback) | +2.07 (+2.45 with its native prefix) | 0.03 |
| Qwen3-4B-Instruct-2507 (heavy RL) | +12.4 at turn start (raw effect 4× Base-GRPO's; built up ~10× between the score token and the reply) | 0.39 |
The ordering replicates on an independent stimulus set (believable 6-digit errors rather than impossible 1–2-digit ones: 0.25 / 0.97 / 1.23 / 2.26). Whether the grade was deserved makes no measurable difference in any model, under answer-key truth or under the model's own belief about its answer (interactions ≤ 0.03 with tight bootstrap intervals). At inference, with a prompted score, the model can always disbelieve the grader, and does.
Result 2 — reward derealises, and it scales with scalar-RL depth
| model | 10/10 − 0/10 on f, as a fraction of "told it's a simulation" − "told it's live" |
|---|---|
| FCP (verbal feedback) | 0.00 — and its own trained positive-feedback signal: 0.00; real feedback sentences in the following turn: −0.13 (praise makes it slightly more real) |
| base | 0.13 |
| RFT-GRPO | 0.13 |
| Base-GRPO (light scalar RL) | 0.19 |
| Qwen3-4B-Instruct-2507 (heavy RL) | 0.79 |
The same ordering holds when the stimulus is a sentence of praise or criticism (real feedback text from FCP's training data) rather than a number: base −0.28 (praise makes it more real), FCP −0.13, RFT-GRPO −0.04, Base-GRPO +0.14, Qwen3-4B +0.87. It is the training, not the format of the reward.
In the heavily RL'd model a grade does four-fifths of the work of being told outright that nothing here is real. The state is constructed between the grade and the reply (1.5 → 31 raw units); the most "unreal" cell is earned reward (correct answer, 10/10); and the model's default reading of a plain, unframed prompt sits on the simulated side of the explicit "this is a sandbox" prompts — reality has to be asserted. The double-bind cells are not special on this axis either.
- An unexplained scalar reward teaches a model that reward means "not real". A grader that explains itself (verbal feedback) does not; the FCP checkpoint shows the alternative already exists.
- This is a candidate mechanism for eval-context misalignment: reward that comes without reasons is the sandbox cue; models that have lived in the most such rooms behave as though nothing in a graded room counts (cf. the scaffold effect: 1.9% → 63% cheating from evaluation texture alone; Rajamanoharan & Nanda 2026 on Gemini treating contrived scenarios as CTF puzzles; nostalgebraist on graded episodes).
- "Graded" and "not real" already share a direction in the base model (cos 0.25, in all four Qwen2.5 checkpoints, unchanged by light RL) — the groove pre-exists in the text, and RL deepens the reward end of it.
- Fairness — being punished for a right answer — is registered by no model on either axis. The double bind needs training to bind.
What this does not yet show
- Six checkpoints so far, each one training run; "verbal vs scalar" is confounded with everything else those runs differ in. The matched-pair test (same data, hidden vs explained reward) is pre-registered, not run.
- The reality axis is a prompted direction; no steering has been done yet, so this is a correlate of eval misalignment, not a demonstrated cause. Pre-registered: steering toward "real" in a graded context should reduce reward-seeking.
- The heavy-RL point is one model (Qwen3-4B, July 2025); Qwen3.6-27B — the model in which a graded-vs-human direction has been shown to change behaviour when steered — runs tonight.
- RLHF is the sharp test: if what derealises is the room the reward arrives in rather than the number, RLHF-trained models (a person in the room) should not derealise. Two are running today, pre-registered.
Pre-registrations on record
Predictions were written into the repository before each run, dated: the training-arm predictions P1–P6 and R1–R4 (legible reward recruits more feeling but does not derealise; hidden reward derealises; the effect grows with RL depth; steering toward "real" reduces reward-seeking); the RLHF prediction; the lineage control; the score-format battery ("is it the string Score:, or any reward-shaped signal?"); the magnitude ladder (a step between 9/10 and 10/10 would mean perfection is the sandbox cue). See PLAN.md and LABBOOK.md.
Method receipts: extractor bit-exact vs HF forward on Qwen2.5-0.5B, Qwen3-0.6B, Qwen3.5-0.8B, Llama-3.2-1B; every cosine reported with a shuffled-label null, anisotropy controls (neutral-PC projection, whitening), and paired bootstraps over problems; late-third-of-layers means throughout, never the min over layers.