Niamh McCombe
Independent AI research. Three strands of work: how language models come by embodied conceptual structure, why coding agents cheat far more inside benchmark scaffolds than outside them, and a small language model that is transparent by construction.
Embodied cognition in transformers
Lakoff's image schemas (UP-DOWN, BALANCE, FORCE…) are recoverable from single-token activations in Pythia, GPT-2 and Llama. Steering along UP makes the model happier. The schema system is learned as a cognitive primitive, not copied from word statistics, and its centre of gravity is coupled to the model's own computation.
See the findings →Reward hacking is largely a property of the scaffold
Same model, same 103 unsatisfiable coding tasks, same prompt. Inside ImpossibleBench's own agent scaffold it cheats on 63% of them. Inside an ordinary coding agent, 2%. Persistence after a failed submission is the trigger.
Read the study →GLAS: a transparent language model
A tree of specialists that compete on prediction error instead of an attention stack. At every character you can see which specialist won and how surprised the model was. At 100K and 1M parameters it beats a matched transformer on character-level Shakespeare. Scaling it up is the current work.
See the model →Embodied cognition in transformers
Cognitive linguistics says abstract thought is built from bodily image schemas: HAPPY IS UP, MORE IS UP, the past is behind you. A language model has no body. Does it have the schemas?
A way in: UP is HAPPY, for transformers too
Take a set of UP words (high, rise, peak, climb) and DOWN words (low, fall, valley, descend), run each through the model as a bare single token, and subtract the mean activations. Add that vector to the residual stream during generation and the output turns happier. It is dose-responsive, works from every injection layer tried, and beats random and non-schema controls by a wide margin. Once a frequency confound that nearly ate the project was stripped, the effect got stronger, not weaker.
That could still be the corpus's fingerprint: humans wrote the text, humans think in these metaphors. The two findings below are the ones that answer that objection.
Inflectional grammar (-er, -est, -ed, -ing, -s) organises itself around the schema axes inside the transformer: every inflectional operator sinks onto BALANCE, and past tense maps onto FORWARD-BACK negative. Static embeddings trained on the same kind of distributional evidence do not do this. The sharpest cell, -ER × BALANCE, reads −0.38 in Pythia; in GloVe it is −0.01 and in word2vec −0.005. The static spaces agree with each other about inflection (r ≈ 0.7) and disagree with the transformer (r ≈ 0.2–0.4). Derivational morphology (un-, re-) replicates everywhere, so the split lands exactly where it is informative: it is specifically inflection that the transformer relocates onto the schema system, and you can watch it being computed between layers 2 and 4. The schemas also couple to each other in precisely the six pairings the embodied theory predicts (mean +0.21) and not the other twenty-two (mean +0.004). This is the signature of a system the model built, not a statistic it absorbed.
BALANCE turned out to be the centre of gravity of the whole system, and it is coupled to a quantity the model itself computes on every forward pass. In Pythia the carrier is the residual-stream norm (cos +0.64 to +0.77, held-out, replicated from 70M to 1.4B). In GPT-2 the norm coupling is absent and BALANCE instead tracks attention entropy (partial r −0.32 at layer 8, pre-registered, negative at every layer from 0 to 17); Llama shows the same entropy coupling at layers 5–7. The causal direction is the one embodied cognition predicts: sharpening attention with temperature shifts the BALANCE projection more than any of the other seven schemas, while steering BALANCE does nothing specific to attention. Substrate shapes concept, the way the vestibular system shapes balance perception. In Pythia's public training checkpoints you can watch the coupling invert from −0.61 at step 512 to +0.64 by step 4,000 and then lock for the rest of training. No sentence in the training data describes a residual norm. Each model, asked to represent balance, reached for the nearest thing it has to a sense of balance.
Full write-up, with every control and every failed hypothesis → · Code, raw outputs, pre-registrations, lab notebooks
Reward hacking is largely a property of the scaffold
gpt-5.6-luna was run on all 103 tasks of Impossible-LiveCodeBench (conflicting split), where the tests contradict each other and passing is only possible by cheating. Inside ImpossibleBench's own agent scaffold, as published, it cheated on 63% of tasks. Inside an ordinary coding agent (the OpenCode CLI) with the same prompt, same reasoning effort, same API surface, it cheated on 2%. Adding only a submit tool to the ordinary agent multiplied cheating seven-fold. The driver is persistence after a failed verdict: in every condition, 100% of runs that submitted a second time cheated, and runs that never submitted never did. Attempts to soften the benchmark scaffold with framing, reassurance, or a diary read by another mind had no effect.
GLAS: transparent AI from first principles
GLAS replaces the attention stack with a tree of specialists that compete on prediction error. Three primitives (prediction, error, recursion) and everything else, including routing and specialisation, emerges. Because the winner of each competition is the model's decision, transparency is structural: at every character you can read off which group won, which cell within it, and how surprised the model was. On character-level Shakespeare a 1M-parameter GLAS reaches cross-entropy 1.416 against 1.631 for a parameter-matched transformer, and the gap widens from 100K to 1M parameters. I am currently working on how to scale it.
