Raghavan Muthuregunathan

From-scratch SLM

Legal SLM 125M

A small language model for legal and finance text, built end to end. Public data, a fresh tokenizer, a 125M Llama-style model trained from scratch, live serving, and a demo that shows where the model stops.

GitHub Live demo

Key takeaways

Trained a 125M Llama-style base model from scratch on legal and finance text. Fresh weights. A fresh 16K byte-level BPE tokenizer.
Built the whole chain and timed each step: stream, clean, drop near-copies, strip test data, tokenize, pack, train, serve, and write it up.
Chose a legal-first data mix from measured yield. Roughly 40% case law, 40% SEC filings, and 20% web text.
Ran 3 epochs on 8xH100. Reached 10.35 perplexity on held-out text. The whole run cost about $27.
Built the demo around honest limits. The base model finishes legal-style text. It does not pretend to cite facts or answer grounded questions.

Mechanistic Interpretability

LatentWorldsGPT

A pre-registered study of small transformers, on maze and HTTP data. It splits what a probe can read out from what the model in fact uses. It ships traces that show how a feature gets carried along.

GitHub

Key takeaways

Wrote down the maze and HTTP calls before running any probe. That makes a miss easy to spot.
Asked whether small next-token models learn task facts that a linear probe can read out.
Ruled out the strict N-criterion. What a probe can read is not what the model uses.
Found carry-through baked into the design. A fact from one input slot can still be read much later.
Uses probes over many seeds, broken-structure controls, layer-by-layer ablations, and position controls.

Open MAZE carry-through trace

MAZE carry-through trace

A sample path on an 8 x 8 grid, showing the main finding. The current token moves away from the start cell. But the start cell can still be read from the residual stream.

Autoplay: the path starts at row 1, column 2. The copper start-cell signal rides along with it.

Layer-by-layer start-cell probe accuracy

Source:results_maze_navigation.mdandreport.md.

L0+0.6 pp gaptrained 4.9%; baseline 4.4%

L1+4.2 pp gaptrained 8.5%; baseline 4.4%

L2+11.8 pp gaptrained 16.3%; baseline 4.5%

L3+13.1 pp gaptrained 17.4%; baseline 4.3%

L4+14.3 pp gaptrained 18.5%; baseline 4.2%

L5+15.2 pp gaptrained 20.2%; baseline 5.0%

The gap grows with depth. The start cell gets easier to read, even after the path has moved on.

Current token position in illustrative path

Current token position: 7 of 10 path tokens. Current cell: row 4, column 5.

Reported MAZE probe readout

Best transformer layer for start-cell MLP probe: Layer 5 of 6 layers
Start-cell probe accuracy, trained model, layer 5: 20.2%
Start-cell probe accuracy, untrained baseline, layer 5: 5.0%
Probe accuracy gap, trained minus untrained, layer 5: +15.2 percentage points

Show attention copy sketch

Toy attention matrix

BOSSc2c3c4CBOSSc2c3c4C

Why copying is cheap

In a causal transformer, a later spot can look back at earlier tokens. Say the start cell sits in a fixed early slot. Then attention can keep reading that slot and writing a faint copy into the residual stream.

The copper column is the start-token column. It lights up again and again. That is the picture of carry-through: easy to copy, and not always useful.

Open HTTP non-first-slot carry-through trace

HTTP carry-through beyond first slots

The HTTP stream is one request record after another. This walkthrough shows how the path category can still be read at the current size token. It holds even when the source is not a first slot.

kcurrent request index

k-1previous request index

jmost recent earlier request with response-size bin >= 5

ppath category, such as image, HTML, CGI, directory, or other

sHTTP status bucket

szresponse-size bin

Autoplay: highlighting the previous request path category and the current response-size token.

HTTP request token stream

p_jEarlier large-response pathsz_j>=5Earlier large-response size bin...Intervening request recordsp_{k-1}Previous request pathsz_{k-1}Previous response-size binp_kCurrent request paths_kCurrent status bucketsz_kCurrent response-size bin

Copper is the source path token. Blue is the current size token. That is where the probe reads.

Selected follow-up

Source:report.md, section 7.2.5.

Previous-request path

p_{k-1} crosses a request boundary. It is not a first slot, and it is not adjacent to sz_k.

Source tokenp_{k-1}previous request path category

Readout tokensz_kcurrent response-size token

MLP probe accuracy gap+0.674 accuracy units+67.4 percentage points

HTTP carry-through example

Example 2 of 4: previous-request path category is recovered at the current response-size token.

Toy HTTP attention matrix

p_jsz_j>=5...p_{k-1}sz_{k-1}p_ks_ksz_kp_jsz_j>=5...p_{k-1}sz_{k-1}p_ks_ksz_k

How to read this

Rows are readout spots. Columns are tokens that can be read. The bottom row is sz_k, the current size token.

Copper cells show sz_k reading earlier path tokens. That is the point of the picture. Attention can route path facts into the residual stream cheaply, from any slot.

Evals

SAGE: Scalable AI Governance & Evaluation

A KDD-accepted way to turn human product judgment into judges you can run at scale. Built for search at work, for policy review, and for catching regressions.

arXiv 2602.07840

Key takeaways

Closes the gap between careful human review and search checks that must run at high volume.
Turns product judgment into three parts: a policy, a set of picked precedents, and an LLM judge you tune against them.
Uses teacher-student distillation. A big careful judge trains a small cheap one.
Runs inside LinkedIn Search for offline checks, policy review, and catching regressions.
Published in the KDD 2026 proceedings.

Semantic Search

Unified Query Understanding for Semantic Search

A shipped system that works out what a query means. It folds search parsing into one model that must emit a fixed schema. It adds labeling and tests that scale.

arXiv 2605.27441

Key takeaways

Folds many separate query parts into one Small Language Model.
Makes the model emit a fixed schema. So what comes out is ready for the rest of the search stack.
Adds Query Illuminator. It labels data well, feeds distillation, and lets tests scale.
Validated through offline and online tests in LinkedIn Job Search.
Shows the same setup carries to People Search, under tight latency limits.