Architecture

GLAS: transparent AI from first principles

Recursive competing units for interpretable language modelling. A small model that beats a parameter-matched transformer while showing you every decision it makes.

The problem

Transformers are black boxes. Attention weights don't tell you what the model is actually doing. Post-hoc interpretability is reverse-engineering.
Parameter inefficiency. Much of a transformer's capacity is redundant: the same features stored many times across different attention heads.
O(N²) scaling. Self-attention scales quadratically with sequence length.

The approach

Recursive competing units. Instead of attention layers, the model is a tree of specialists that compete on prediction error. The best predictor wins, and you can see who won.

Three primitives: prediction, error, recursion. Everything else — gating, routing, specialisation — emerges from these. There is no separate routing network to train and no post-hoc attribution to run.

Transparency by construction. At every character, the model exposes which group won the competition, which cell within that group, and how surprised it was. That is not an explanation layered on top of the computation; it is the computation.

The small model

A router feeds four competing groups; each group contains three competing cells. The winner at each level is decided by prediction error, so the trace of winners is the model's own account of how it produced each character. Trained on TinyShakespeare at two parameter budgets, ~100K and ~1M, against a transformer matched for parameter count.

Results

Parameter budget	GLAS	Transformer	Gap
~100K parameters	1.598	1.699	−0.10
~1M parameters	1.416	1.631	−0.21

Character-level cross-entropy on TinyShakespeare, lower is better. The gap widens with scale.

Training curves at 1M parameters: GLAS reaches lower cross-entropy than the matched transformer at every epoch after the second — Training comparison at ~1M parameters (GLAS 1,006,887 params; transformer 999,385 params), 50 epochs.

Sample generations, 1M parameters

GLAS

O, thou shalt way, with grief,
and theressely.
HENRY BOLINGBROKE:
Why, and yourself him for
then, I prove him...

Transformer

O, the rard; or the comfort;
see that the ward see would
scatch her dead?
LADY CAPULET:
Good goodven, that in my me...

Routing transparency, live

Six successive characters generated by the 1M-parameter model. For each one: the winning group, the winning cell within it, the weight each group carried, and the model's confidence. Nothing here is inferred after the fact.

What the architecture gives you

Transparent. At every token, see which specialist won and how confident the model was. Structural, not bolted on.
Parameter efficient. Beats transformers at the same parameter budget. Competing units avoid the redundancy of multi-head attention.
O(N) scaling. Linear in sequence length. No quadratic attention bottleneck.
Recursive grammar. The tree structure is also a communication protocol: models can share information by exchanging consensus and entropy signals.
Tractable parallelisation. The recursive structure lets multiple smaller models be parallelised across hardware rather than one monolithic model requiring ever-larger GPUs.

Status

The results above are for the small model. I am currently working on how to scale it: larger datasets and model sizes to test scaling laws, training and inference efficiency, and mapping specialist groups to semantic meaning, so that transparency becomes interpretability.