Do independently trained models share causal variables?

A progress brief for #42 (causal convergence across seeds), 21 November 2026. It gathers the preregistered 160M result, the first two ladder steps, a revised estimator, 27 syntactic variables with a check on whether the model's own activation carries them, global-map follow-ups ending in a better-map check that withdrew a candidate finding, a crosscoder, OLMo-architecture models (a screen, ladder rung 3 and rung 3b), a 410M replication, Tanyel and Ayvaz's strict off-route test on our sites and across the whole grid, and the open choices, to support one decision: what to do with #42 before the February chapter-4 meeting.

Terms with a dotted underline show what they mean in this project when you hover over them, tab to them or tap them.

The short version

SeedPythia-160M, 7 seedsInit vs data orderladder step 1Size160M ↔ 410MFamily + corpusPythia ↔ OLMoScaleOLMo 1B; ↔ 150M7–9B4 familiesModel difference →IOInatural site: the model reads itT 0.9437/42 pairs72/72 pairsbeat their nullT 0.9798/98 pairsT 0.96165/168 pairsT 0.99↔150M T 0.97T 0.96midtrain T 1.0218 syntactic variablesend-only sites, mostly excess18/18 tasksmedian T 0.95not runT 0.9518/18 tasksT 0.9618/18 tasksnot runnot runOff-route cellsthe model doesn't read them14/14 cellsT 0.92–1.00not runnot run1 tasktoo few to read3/3 cellsT 0.96–0.98not run
Diagram A. What was tested, and what passed. Columns: what differs between the source and destination models. Rows: what kind of site the variable is transferred at, from sites the model reads (IOI) to cells it does not read at all. Filled dot: the preregistered transfer test passed (T is transfer ratio, medians); hollow: run, but too few cases for a reading; dashed: not run. Every cell that was run passes, including the bottom row, which is why transfer alone cannot show that a model uses a variable.

The question

The Platonic Representation Hypothesis says that models trained separately converge in representational geometry. #42 asks whether they also converge in the causal variables they compute with. If they do, an interpretability finding in one model is a finding about all of them. If they don't, every circuit is a finding about one checkpoint.

The test, in four steps:

  1. Inside each seed, distributed alignment search finds a 16-dimensional subspace that carries the IOI variable (which name was repeated).
  2. Between two seeds, a 16×16 rotation aligns the two subspaces. It is fit without any causal labels.
  3. The variable is read out of seed A, rotated, and written into seed B.
  4. Interchange accuracy counts how often B's answer then flips the way the counterfactual says it should.
① DAS in each seed16-d subspace at one cell② Map A's frame onto B'sa rotation fitted without labels③ Write into Bdoes B's answer switch?Scoremedians over 42 seed pairssourcesentenceSeed Alayer 5, S2 tokenSeed Blayer 5, S2 token16 coordinatesof the sourceQ · crotation × scale50 randomrotations (null)basesentenceB's own fit0.71A's, mapped0.71random p950.39T = 0.94mapped ÷ owngap = +0.26mapped − p95
Diagram B. The transfer test. Seed A reads the source sentence; DAS's 16 coordinates at one cell (here IOI's frozen site, layer 5 at the S2 token) are rotated and scaled into seed B's frame and written into B's run on the base sentence. If the seeds compute the same variable, B's answer switches as often as when B reads the source itself (T ≈ 1), and much more often than when the map is a random rotation (gap > 0). Numbers: the preregistered Pythia-160M result, medians over the 42 directed pairs of seeds 3–9; the gap is the median of per-pair gaps.

The design follows Model Alignment Search, run on the nine PolyPythias seeds. Seeds 1–2 set the thresholds; seeds 3–9 are confirmatory.

1. The variable transfers

00.20.40.60.81Interchange accuracy on 1,092 held-out prompts (each dot is one seed pair)Learned 16×16 rotationthe transfer0.71Random rotation, 95th pctnull0.39Whole-space Procrustesnull: align all 768 dims0.22Identity mapnull: no alignment0.28Mismatched promptsnull: shuffled fit pairs0.24Sign flipnull: −Q0.14Subspace ablatednull: nothing injected0.13
Figure 1. The learned rotation is the only condition that carries the variable. Each dot is one confirmatory seed pair at the preregistered site (S2 position, layer 5), scored on 1,092 held-out prompts. The black tick is the median. The five hollow blue dots are transfers into seed 9, which account for every pair below its own random null. Transfer ratio T = 0.94.
Table view
ConditionMedianMinMax
Learned 16×16 rotation0.710.220.98
Random rotation, 95th pct0.390.120.80
Whole-space Procrustes0.220.050.56
Identity map0.280.010.80
Mismatched prompts0.240.050.55
Sign flip0.140.000.79
Subspace ablated0.130.030.45

2. The implementation is a sample

The variable is shared, but where it lives is not. Each seed chooses its own layer on validation names, and the heads that carry the variable barely overlap across seeds.

L3L4L5L6L7L8preregistered layer 5123456789Seed
00.250.50.751Top-10 head overlap (Jaccard) after attention-only matchingSame seed, split halfn = 7 seeds0.82Across seedsn = 21 pairs0.25Random head matchingmean of 100 shuffles0.04
Figure 2. Same variable, different machinery. Left: the layer each seed selects runs from 4 to 7. Right: top-10 head overlap after attention-only matching is 0.25 across seeds, against 0.82 for two halves of the same seed and 0.04 for random matching. Seed 9 does the task (accuracy 0.86) but spreads the variable: 0.27 at layer 5, 0.50 at its best single layer, 0.62 across layers 4 and 7 together.
Table view
Seed123456789
Selected layer555656674
Table view
ComparisonMedian top-10 overlapn
Same seed, split half0.827
Across seeds (attention-matched)0.2521
Random matching0.03821

3. Geometry can't see it, and it sharpens late

Behavior and causal localization00.250.50.751init1B10B100BTraining tokens seen (log scale)step 80,000IOI accuracy 0.93Net causal lift 0.45 Representational similarity between seeds0.40.60.81init1B10B100BTraining tokens seen (log scale)Debiased CKA 0.96Mutual 5-NN 0.59
Figure 3. Similarity is flat while causal transfer moves. Top: IOI accuracy saturates by about 70B tokens, after a below-chance dip early in training. Net causal lift (shaded: 95% CI) stays near +0.2 until step 80,000 and then jumps. Bottom: debiased CKA between seeds is already 0.93 at random initialization and never leaves 0.92–0.96. Mutual 5-NN rises early and then flattens. The late jump coincides with learning-rate decay, which in Pythia moves together with tokens seen, so this is a description, not a cause.
Table view
CheckpointTokens (B)IOI accuracyNet lift (95% CI)Debiased CKAMutual 5-NN
step00.00.50+0.00 (-0.00 to +0.00)0.930.42
step10002.10.41+0.02 (+0.01 to +0.03)0.920.56
step20004.20.36+0.10 (+0.08 to +0.13)0.930.56
step40008.40.54+0.17 (+0.16 to +0.21)0.950.58
step800016.80.78+0.14 (+0.10 to +0.19)0.960.60
step1600033.50.85+0.17 (+0.14 to +0.25)0.960.57
step3300069.20.90+0.18 (+0.14 to +0.26)0.960.56
step66000138.40.91+0.21 (+0.15 to +0.34)0.960.58
step80000167.80.91+0.21 (+0.13 to +0.35)0.960.58
step100000209.70.93+0.49 (+0.41 to +0.57)0.960.58
step120000251.70.91+0.53 (+0.47 to +0.60)0.950.60
step135000283.10.92+0.49 (+0.41 to +0.57)0.950.59
step143000299.90.93+0.45 (+0.37 to +0.62)0.960.59

4. Preregistered scorecard

Thresholds were fixed on seeds 1–2 and never moved.

HypothesisPreregistered testResultVerdict
H1Transfer ratio T ≥ 0.800.94 Pass
H1Net lift ≥ +0.50+0.45 (95% CI +0.37 to +0.62) Narrow fail
H1Learned rotation beats the nulls0.71 vs random p95 0.39; 37/42 pairs Pass
H1Heads differ across seeds0.25 across vs 0.82 within a seed Pass
H2Onset spread ≤ 0.35 log10 tokens0.54; 36/42 pairs reach onset Fail
H3|partial ρ(CKA, net lift)| < 0.30−0.16 Pass, uninformative
H3Whole-space Procrustes T ≤ 0.500.34 (95% CI 0.30 to 0.45); fitted on 792 task prompts; refits in sections 8–9 Pass
Ladder 1Every group's gap CI excludes 04 of 4 groups; 72/72 pairs beat their own null Pass
Ladder 2Every group's gap CI excludes 0 (within 160M, within 410M, across sizes)3 of 3 groups; 170/182 pairs beat their own null Pass
Ladder 2, v2Same test with the revised estimator (reported next to v1)3 of 3 groups; 181/182 pairs beat their own null Pass
2nd variableGreater-than year localizes (≥ 0.80)0 of 1,296 settings; best 0.37 Fail
2nd variableBackup: name identity localizes (≥ 0.80)best 0.74 Fail
CausalGymGate-passing tasks that transfer, selected site (N of M)27 of 27; all six families; all Bonferroni Pass
CausalGymSame at the end-only site (secondary)18 of 18 Pass
Follow-up 1Global rotation fitted on 260k Pile tokens: T ≥ 0.80 (carries) or ≤ 0.50 (does not)0.54, levels off after ~8k tokens Between
Follow-up 2Best global linear map (Pile or task text) carries the variable: T ≥ 0.800.55 (Pile), 0.45 (task text); no task reaches 0.80 No: candidate cell
Follow-up 3Value through the global map, written into B's own subspace: T ≥ 0.800.89 Value travels
Natural-site checkCausalGym sites where the model's own activation carries the variable (lift ≥ 0.30)2 of 18 end-only, 4 of 27 selected; all agreement Mostly excess-dominated
Natural-site checkSame at the IOI siteslift +0.73 (160M, frozen), +0.92 (410M) Natural
CrosscoderSame latent has the same effect across seeds: top-1 − chance, CI excludes 0+0.224 (+0.219 to +0.228); seeds 8 ↔ 9, never seen: +0.24 Pass
Follow-up 4IOI frozen site: A's subspace through a global linear map T ≥ 0.80; else value (hybrid) T ≥ 0.80subspace 0.46, value 0.96 Value yes, subspace no
Follow-up 6A held-out, more accurate map carries A's subspace, T ≥ 0.80 → candidate withdrawn0.85 (crosscoder, linear approximation); oracle 0.96 Candidate withdrawn
Rung 3 screenDataDecide 150M: ≥ 2 corpora with all 3 seeds passing the conversion gate and doing IOI (accuracy ≥ 0.80, logit diff ≥ 1.0)2 corpora (Dolma 1.7, DCLM); FineWeb-Edu and C4 failed the gate's Pile-loss bar, not the conversion Pass: rung 3 at 150M
Rung 3Gap CI excludes 0: within OLMo, across corpora, Pythia → OLMo, OLMo → Pythia4 of 4 groups; 165/168 cross-family pairs beat their null; median T 0.96 across families Pass
Rung 3Contrasts K1–K4 (±0.10 margin): corpus, data overlap, shared init, familyK1, K3, K4: no difference; K2 primary inconclusive (point −0.002) No training difference matters
Rung 3OLMo sites natural (median lift ≥ 0.30)0.66 selected, 0.68 at layer 5 Natural
Rung 3bCross-family transfer on tasks gated in both families (Bonferroni); R1 if ≥ 2/3 of the excess-dominated ones pass16 of 16; 14 of 14 excess-dominated R1: no screening
Rung 3b, amendedSame, restricted to tasks off-route-like in both families (≥ 3 needed)1 task (filler_gap_obj), which transfers No reading
Option I(a) The excess split replicates at 410M (≥ 2/3 of 160M excess-dominated tasks)15 of 16 Replicates
Option I(b) Tasks natural at 410M transfer within 410M2 of 2 (number agreement) Pass
Run A′≥ 3 competent, transferring tasks off-route by Tanyel and Ayvaz's Algorithm 1 (either site; run A's G* corrected)1 (filler_gap_hierarchy, off-route in 7 of 7 seeds, transfers) 1 of 3 needed: excess, often extreme
Option LTransfer at competent off-route cells found across the whole grid: L-pass if ≥ 2/3 of ≥ 3 testable cells pass (Bonferroni)5 of 5 cells, from 2 tasks; references unusable (net gate) L-pass: no screening, 2 tasks
Option L2Same test at the relaxed-gate off-route cells: L2-pass if ≥ 2/3 of new cells pass and ≥ 2 of 3 new tasks qualify9 of 9 new cells; 3 of 3 new tasks; all 14 cells and 5 of 5 tasks at α = 0.05/14 L2-pass: 5 tasks
Option O1B conversion: all 3 checkpoints match the reference (≤ 1e-2 logits), HellaSwag within 0.02 of AI2, Pile loss below 3.46, exact tail replay; IOI competentmax Δlogit 1.0e-4; HellaSwag equal to AI2's to 4 decimals; Pile 2.91–2.94; IOI accuracy 1.00 Pass
Option O, part BTransfer at competent off-route cells at 1B: B-pass if ≥ 2/3 of ≥ 3 testable cells pass (Bonferroni), from ≥ 2 tasks3 of 3 cells (relaxed gate) from 3 tasks; every region off-route at 160M too B-pass
Option O, part AIOI gap CI excludes 0 within 1B and across sizes (1B ↔ 150M)within 1B +0.66 (+0.55 to +0.78); across sizes +0.45 (+0.40 to +0.57); site natural (lift 0.998) A-pass
Option Q, part AIOI cross-family gap CI excludes 0 at 7–9B and the selected site is natural in ≥ 3 of 4 familiesgap +0.66 (+0.61 to +0.70), T 0.96; natural in 4 of 4 QA-pass
Option Q, part CAnalogy direction along OLMo-3 post-training: the destination decides the effect (QC-destination) or the source does (QC-source)destination range 0.95 vs source range 0.97 mixed

The effect is real and smaller than preregistered: net lift misses +0.50 narrowly, with a CI that spans the threshold. Onset timing is not consistent across seeds. Neither many-valued variable reached the localization bar; the second-variable test came from CausalGym instead (section 8).

5. New: ladder step 1, initialization vs data order

A PolyPythias seed changes two things at once: the starting weights and the order the data arrives in. EleutherAI also released 160M runs that change only one of them, so this step asks which one the result depends on.

-0.10+0.1+0.2+0.3+0.4+0.5+0.6random nullLearned-rotation accuracy minus that pair's random-rotation 95th percentileInitialization differssame data order · 6 pairs+0.24Data order differssame initialization · 12 pairs+0.37Both differdecoupled runs · 12 pairs+0.29Both differPolyPythias seeds 4–6 · 42 pairs+0.34
Figure 4. Every group transfers. The large dot is the group median with its pair-bootstrap 95% CI; small dots are individual directed pairs. The CI excludes 0 in all four groups, and 72 of 72 pairs beat their own random null. Layer selected per model on validation names, as preregistered.
Table view
GroupPairsGap (95% CI)Learned IIARandom p95Net liftT
Initialization differs (same data order)6+0.24 (+0.12 to +0.25)0.860.63+0.420.99
Data order differs (same initialization)12+0.37 (+0.31 to +0.39)0.810.51+0.600.98
Both differ (decoupled runs)12+0.29 (+0.20 to +0.32)0.770.51+0.470.98
Both differ (PolyPythias seeds 4–6)42+0.34 (+0.32 to +0.35)0.840.50+0.640.98

At first glance, the initialization-only group transfers worse: its gap is -0.14 below data-only (95% CI -0.26 to -0.09). That difference belongs to the destination models, not to the sources. The two corrected weight seeds are the least accurate models in the set. Ablating the variable alone already flips many of their prompts, which raises both their baseline and their random null.

Comparing sources within the same destination removes that confound (Figure 5).

Source shares its initializationSource shares its data orderSource shares neitherSubspace ablated, nothing injected
00.20.40.60.81Learned-rotation interchange accuracy (mean over sources in the category)Original pythia-160mIOI accuracy 0.89data-seed1IOI accuracy 0.98data-seed2IOI accuracy 0.90data-seed3IOI accuracy 0.86weight-seed1 (corrected)IOI accuracy 0.79weight-seed2 (corrected)IOI accuracy 0.82
Figure 5. Matched by destination, shared initialization or shared data order adds nothing. Each row is one destination model. Sources that share its initialization land -0.03 to +0.07 from sources that share neither. Sources that share its data order land -0.08 to +0.01. The gray tick shows how high the destination already scores with the subspace ablated. This matched comparison was added after seeing the data and is labeled exploratory.
Table view
DestinationAblated baselineShares initializationShares data orderShares neither
Original pythia-160m0.360.830.780.86
data-seed10.100.67–0.65
data-seed20.150.83–0.78
data-seed30.140.87–0.80
weight-seed1 (corrected)0.41–0.870.86
weight-seed2 (corrected)0.54–0.940.96

What it means. Initialization and data order are not where the shared variable comes from. Either difference alone is enough to scatter heads and layers: models with identical starting weights selected layers 4, 4, 5 and 5. Every model here read the same Pile, so what they share is the data distribution, the architecture and the objective. The next ladder steps are built to vary exactly those.

6. New: ladder step 2, a bigger model

The same test on nine Pythia-410M seeds (24 layers, 1,024 wide), plus transfer in both directions between 160M and 410M. The 16×16 rotation lives between the two 16-dimensional subspaces, so the different widths need no special handling. Everything below was preregistered before any 410M model was downloaded, except Figure 7.

-0.10+0.1+0.2+0.3+0.4+0.5+0.6+0.7random nullLearned-rotation accuracy minus that pair's random-rotation 95th percentileWithin 160Msame-run reference · 42 pairs+0.28Within 410M7 seeds · 42 pairs+0.20160M ↔ 410Mboth directions · 98 pairs+0.11160M → 410Msecondary · 49 pairs+0.04410M → 160Msecondary · 49 pairs+0.16
Figure 6. The variable crosses seeds at 410M and crosses sizes. Bold rows are the preregistered groups (the 160M row is a same-run reference). Large dot: group median with pair-bootstrap 95% CI; small dots: directed pairs. Hollow dots involve the chance-level seed 4, which is the source in every within-410M pair below its null; without it, within-410M is 30/30 with median gap +0.32. The two direction rows are secondary.
Table view
GroupPairsGap (95% CI)Learned IIARandom p95Net liftTBeat own null
Within 160M42+0.28 (+0.26 to +0.33)0.840.56+0.620.9642/42
Within 410M42+0.20 (+0.07 to +0.34)0.530.17+0.390.9037/42
160M ↔ 410M98+0.11 (+0.08 to +0.13)0.560.32+0.290.9391/98
160M → 410M49+0.04 (+0.03 to +0.08)0.160.08+0.130.3143/49
410M → 160M49+0.16 (+0.12 to +0.25)0.970.68+0.781.0948/49

The two directions look very different. 160M patches barely move 410M (T = 0.31), while 410M patches over-shoot 160M's own fit (T = 1.09). A rotation keeps length, and the DAS coordinates in 410M are larger: their spread is 6.0–6.8 in five 410M seeds against 2.6–4.7 in 160M, a median ratio of 1.48. Adding one scale factor, computed from unlabeled activations just like the rotation and applied to the random null as well, makes the two directions symmetric (Figure 7).

Rotation only (preregistered)Rotation × one label-free scale (post hoc)
-0.10+0.1+0.2+0.3+0.4+0.5+0.6+0.7random nullLearned-map accuracy minus that pair's random-map 95th percentile160M → 410Mrotation only (preregistered)+0.04160M → 410Mrotation × one scale (post hoc)+0.23410M → 160Mrotation only (preregistered)+0.16410M → 160Mrotation × one scale (post hoc)+0.35
Figure 7. One scale factor makes cross-size transfer symmetric. Post hoc, added after seeing Figure 6. Per direction, the preregistered rotation-only gap and the gap with a rotation times rmsdestination / rmssource; the 50 random rotations are scaled by the same factor. T becomes 0.81 and 1.02. The preregistered primary gap was never at risk from this: its random null uses the same source values.
Table view
DirectionMapGap (95% CI)Learned IIARandom p95T
160M → 410Mrotation only (preregistered)+0.04 (+0.03 to +0.07)0.160.080.31
160M → 410Mrotation × one scale (post hoc)+0.23 (+0.13 to +0.30)0.490.200.81
410M → 160Mrotation only (preregistered)+0.16 (+0.12 to +0.25)0.970.681.09
410M → 160Mrotation × one scale (post hoc)+0.35 (+0.27 to +0.38)0.890.531.02

What it does not fix. Several 410M seeds that solve the task have a low own-model ceiling (seed 3: 0.17). Training the DAS map four times longer does not help: the ceiling falls by 0.11–0.23 in four 410M seeds while 160M barely moves, which looks like overfitting 72 training prompts in 1,024 dimensions. More training prompts, validation early stopping, rank, and a variable spread beyond S2 are untested. Section 7 tests the first two.

7. New: a revised estimator (v2)

Step 2 left two problems: a rotation cannot absorb the scale difference between 160M and 410M, and the own-model DAS fit overfits its 72 training prompts at 410M. Before running anything, I wrote down a revised estimator in the proposal:

v1: 72 training prompts, 100 stepsv2: 792 training prompts, validation early stopping
00.20.40.60.81Own-model DAS accuracy at the selected layer (1,092 held-out prompts)160M seed 1+0.08160M seed 2+0.12160M seed 30.00160M seed 4+0.13160M seed 5+0.01160M seed 6-0.10160M seed 7+0.03160M seed 8-0.06160M seed 9+0.21410M seed 1+0.08410M seed 2+0.30410M seed 3+0.38410M seed 4 (chance)+0.10410M seed 5+0.27410M seed 6+0.22410M seed 7+0.32410M seed 8+0.17410M seed 9+0.06
Figure 8. v2 lifts every 410M ceiling. Each row is one model at its selected layer; the number on the right is the change. The eight 410M seeds that solve IOI go from 0.17–0.92 to 0.55–0.98 (median 0.57 → 0.81). 160M barely moves (median 0.86 → 0.85). Seeds 2 and 3 of 410M stay lowest (0.65 and 0.55). Hollow: seed 4, at chance on IOI.
Table view
ModelIOI accuracyLayer v1 → v2Ceiling v1Ceiling v2Changev2 stop step
160M seed 10.975 → 50.860.95+0.08400
160M seed 20.965 → 60.810.93+0.12100
160M seed 30.715 → 40.860.850.0025
160M seed 40.996 → 60.690.82+0.13150
160M seed 50.945 → 50.991.00+0.01200
160M seed 60.886 → 50.920.82-0.1025
160M seed 70.976 → 60.940.97+0.03275
160M seed 80.937 → 50.880.82-0.0625
160M seed 90.864 → 40.440.64+0.2125
410M seed 10.979 → 90.900.98+0.08100
410M seed 21.0014 → 140.350.65+0.30400
410M seed 30.9516 → 140.170.55+0.3825
410M seed 4 (chance)0.5110 → 90.650.74+0.10300
410M seed 50.9811 → 110.510.77+0.2750
410M seed 61.009 → 90.630.85+0.2275
410M seed 70.9710 → 100.450.77+0.3225
410M seed 81.0010 → 100.700.87+0.1775
410M seed 91.009 → 90.920.98+0.06225
v1, rotation only (preregistered)v2, rotation × one scale (revised)
-0.2-0.10+0.1+0.2+0.3+0.4+0.5+0.6+0.7+0.8random nullLearned-map accuracy minus that pair's random-map 95th percentileWithin 160Mv1 (preregistered) · 42 pairs+0.28Within 160Mv2 (revised) · 42 pairs+0.31Within 410Mv1 (preregistered) · 42 pairs+0.20Within 410Mv2 (revised) · 42 pairs+0.39160M ↔ 410Mv1 (preregistered) · 98 pairs+0.11160M ↔ 410Mv2 (revised) · 98 pairs+0.33
Figure 9. Under v2, transfer across sizes is as strong as within a size. Blue rows repeat the preregistered v1 numbers from Figure 6; orange rows are v2 on the same pairs and test prompts. Large dot: group median with pair-bootstrap 95% CI; small dots: directed pairs; hollow dots involve the chance-level seed 4. Under v2, 181 of 182 pairs beat their own null. The one miss is 160M seed 7 → seed 9.
Table view
GroupEstimatorGap (95% CI)Learned IIARandom p95TBeat own null
Within 160Mv1+0.28 (+0.26 to +0.33)0.840.560.9642/42
Within 160Mv2+0.31 (+0.24 to +0.34)0.870.570.9941/42
Within 410Mv1+0.20 (+0.07 to +0.34)0.530.170.9037/42
Within 410Mv2+0.39 (+0.28 to +0.53)0.770.280.9642/42
160M ↔ 410Mv1+0.11 (+0.08 to +0.13)0.560.320.9391/98
160M ↔ 410Mv2+0.33 (+0.28 to +0.38)0.800.390.9798/98

Where the cross-size gain comes from. Two changes at once, so each is also run alone. Median gap across sizes, with learned accuracy, T and pairs beating their null:

FitRotation onlyRotation × one scale
v1 (72 prompts, 100 steps)+0.11 (learned 0.56, T 0.93, 91/98)+0.30 (learned 0.64, T 0.98, 91/98)
v2 (792 prompts, early stopping)+0.19 (learned 0.72, T 0.97, 98/98)+0.33 (learned 0.80, T 0.97, 98/98)

The scale does most of the work. Into 410M the two changes add up: the gap goes +0.04 → +0.40 and T reaches 0.87. Into 160M, T is 1.01.

Still holds under v2. The frozen 160M site (seeds 3–9, layer 5) gives a gap of +0.26, as locked, with 41 of 42 pairs above their null (37 under v1). Ladder step 1 passes in every group, 72 of 72 pairs. Matched by destination, sources that share initialization or data order still transfer no better than sources that share neither.

Still open. T is lower into 410M (0.87) than out of it (1.01). The early-stopping schedule is coarse: 7 of 24 models stop at the first check and 3 reach the 400-step cap. It stays fixed for later rungs.

8. New: 27 syntactic variables (CausalGym)

IOI is one variable. CausalGym turns 29 SyntaxGym phenomena (agreement, NPI licensing, garden paths, filler–gap, clefts, subordination) into binary variables with counterfactual sentence pairs. The setup was preregistered before the run: 160M seeds 1–9, estimator v2 unchanged, and the same localization gate as greater-than. Each model picks its own site on dev data: one of 11 layers, at either the differing word or the final token.

SiteTasks passing the gateOf those, transferPairs beating their nullT, learned mapRank-1 map ≥ 0.95 × learnedT, global rotation (260k Pile tokens)
Selected (primary)27 of 2927 (all Bonferroni)1,134 / 1,1340.9918 of 270.29
End-only (secondary)18 of 2918 (all Bonferroni)752 / 7560.9517 of 180.54

Every task that localizes also transfers, in all six families. Two facts change how to read this:

Learned 16×16 map (local, fitted on paired activations)One global rotation, fitted on 260k Pile tokensOwn-model DAS (ceiling)Ablation only
00.20.40.60.81Interchange accuracy on 100 test pairs (median over 42 seed pairs)Agreementagr_genderagr_refl_num_obj-relcagr_refl_num_pp ·agr_refl_num_subj-relc ·agr_sv_num_obj-relcagr_sv_num_ppagr_sv_num_subj-relc ·NPInpi_any_obj-relcnpi_any_subj-relcnpi_ever_obj-relc ·npi_ever_subj-relc ·Garden pathgarden_mvrrgarden_mvrr_mod ·garden_npz_objgarden_npz_obj_modgarden_npz_v-trans ·garden_npz_v-trans_mod ·Filler–gapfiller_gap_embed_3 ·filler_gap_embed_4 ·filler_gap_hierarchy ·filler_gap_objfiller_gap_ppfiller_gap_subjCleftcleftcleft_modSubordinationgss_subordgss_subord_obj-relcgss_subord_ppgss_subord_subj-relc
Figure 10. Matched locally, the variables transfer; one global rotation carries about half. End-only site, one row per task, medians over the 42 confirmatory seed pairs. Section 9 explains the gap between the blue and orange dots: at most tasks, DAS's excess over the model's own activation; at IOI, map error. Muted labels marked · fail the localization gate and are not counted.
Table view
TaskFamilyGateOwn-model DASLearned mapPile rotationAblation onlyT learnedT Pile
agr_genderAgreementpass0.970.930.320.150.960.33
agr_refl_num_obj-relcAgreementpass0.830.820.470.330.990.57
agr_refl_num_ppAgreementfail0.740.750.410.291.010.55
agr_refl_num_subj-relcAgreementfail0.750.710.410.240.950.55
agr_sv_num_obj-relcAgreementpass0.830.820.670.320.990.81
agr_sv_num_ppAgreementpass0.930.780.660.200.840.70
agr_sv_num_subj-relcAgreementfail0.790.780.530.210.990.67
npi_any_obj-relcNPIpass0.920.810.310.330.890.34
npi_any_subj-relcNPIpass0.910.830.470.510.920.52
npi_ever_obj-relcNPIfail0.750.740.160.170.980.21
npi_ever_subj-relcNPIfail0.720.700.240.260.980.34
garden_mvrrGarden pathpass0.880.770.390.530.870.45
garden_mvrr_modGarden pathfail0.930.880.620.650.950.67
garden_npz_objGarden pathpass0.930.910.450.380.980.48
garden_npz_obj_modGarden pathpass0.850.750.470.460.890.55
garden_npz_v-transGarden pathfail0.860.860.580.601.000.68
garden_npz_v-trans_modGarden pathfail0.760.690.630.580.910.84
filler_gap_embed_3Filler–gapfail0.720.660.570.550.920.79
filler_gap_embed_4Filler–gapfail0.710.640.590.590.900.83
filler_gap_hierarchyFiller–gapfail0.810.730.310.450.900.38
filler_gap_objFiller–gappass0.880.820.360.550.930.41
filler_gap_ppFiller–gappass0.930.880.430.640.950.46
filler_gap_subjFiller–gappass0.870.820.380.510.940.43
cleftCleftpass1.001.000.500.511.000.50
cleft_modCleftpass0.830.830.520.521.000.63
gss_subordSubordinationpass0.890.830.500.560.940.56
gss_subord_obj-relcSubordinationpass0.940.880.620.390.930.66
gss_subord_ppSubordinationpass0.970.930.750.430.960.77
gss_subord_subj-relcSubordinationpass0.900.810.590.450.910.66

A follow-up the batch raised. The whole-space rotation behind H3 (scorecard) was fitted on about 800 task activations. That is too few to pin down a 768×768 rotation, so its failure might say nothing about the seeds. I refitted the rotation on up to 260k Pile tokens, with the reading written down before running.

End-only site (18 tasks)Selected site (27 tasks)Original fit on 800 task activationsLearned 16×16 map
00.20.40.60.811k10k100kTokens used to fit the 768×768 rotation (log scale)T (transfer ÷ own-model DAS)End-only site 0.54learned map 0.95Selected site 0.29learned map 0.99
Figure 11. More fitting data helps only up to about 8k tokens. T of the global rotation levels off at 0.54 at the end-only site and 0.29 at the selected site. The learned map reaches 0.95 and 0.99. Medians over gate-passing tasks.
Table view
SiteTask rows (800)Pile 1,024Pile 8,192Pile 65,536Pile 260,096Learned map
end_only0.460.380.520.540.540.95
selected0.250.220.280.290.290.99

Reading. T = 0.54 falls between the preregistered thresholds (0.50 and 0.80), so it supports no summary claim. By family: agreement 0.71, subordination 0.66, cleft 0.56, garden path 0.49, npi 0.46, filler–gap 0.43. Against simply ablating the variable, the global rotation adds -0.01 (median).

This looked like the surprise. Section 9 explains it. At these sites the seeds look nearly identical as wholes (debiased CKA 0.93–0.97). The variables transfer perfectly once matched locally, yet no global rotation carries them. Three more preregistered follow-ups tracked down why. For most of these tasks, the answer is a known weakness of the method, not a property of the seeds.

9. New: at most syntactic sites DAS overshoots the model, and the IOI subspace does travel

Follow-up 2: linear maps and in-distribution text. A general linear map (least squares) does no better than a rotation. The best global linear map gives T = 0.55 when fitted on Pile and 0.45 when fitted on the task sentences themselves, and no task reaches 0.80. By the preregistered reading, this is the candidate cell.

The geometry did not fit that story. The task-text map sends seed A's class-mean difference onto seed B's almost exactly: cosine 0.85 at end-only and 0.985 at the selected site, 1.00 on all four NPI tasks. Yet on those NPI tasks the transfer through it is near zero. The map gets what the variable looks like right, but not the subspace the interchange acts on.

Follow-up 3: move the value, not the subspace. Use the global map only to translate seed A's activation into seed B's coordinates, and write it into B's own DAS subspace. Now the variable transfers: T = 0.89 at end-only and 1.00 at the selected site. The value travels between seeds through one global map. What fails is carrying A's subspace across.

The natural-site check (first called an illusion check). A smoke test showed something odd. On NPI, transplanting the seed's own whole activation from the source sentence did nothing, while DAS's 16-dimensional patch of that same activation flipped almost every prompt. A patch that beats the whole activation it was cut from is the signature of the subspace-patching illusion: it pushes a direction that downstream layers read but that real activations never move. It is also the signature of DAS overshooting a route the model does use, which the correction below shows is what happens here. I preregistered a check before the full run. A site counts as natural if the real activation's lift over ablation is at least 0.30.

DAS 16-dimensional patchThe model's own whole activation from the source prompt
-0.4-0.200.20.40.60.81Lift over ablation (medians over seeds 3–9; CausalGym rows at the end-only site)IOI, 160M, S2 layer 5 (frozen)naturalIOI, 410M, S2 layer 10naturalagr_gendernaturalagr_refl_num_obj-relcexcessagr_sv_num_obj-relcexcessagr_sv_num_ppnaturalnpi_any_obj-relcexcessnpi_any_subj-relcexcessgarden_mvrrexcessgarden_npz_objexcessgarden_npz_obj_modexcessfiller_gap_objexcessfiller_gap_ppexcessfiller_gap_subjexcesscleftexcesscleft_modexcessgss_subordexcessgss_subord_obj-relcexcessgss_subord_ppexcessgss_subord_subj-relcexcess
Figure 12. At IOI the real activation carries the variable; at most syntactic sites it doesn't install the source value. Lift over ablating the DAS subspace, for DAS's patch (blue) and for transplanting the model's own whole activation from the source prompt (orange). Medians over seeds 3–9. CausalGym rows are the 18 gate-passing tasks at the end-only site. Only 2 of 18 are natural (4 of 27 at the selected site), all agreement tasks. On the rest, the real activation's median lift is -0.02 against DAS's +0.38.
Table view
Site or taskFamilyDAS patch liftReal activation liftReading
IOI, 160M, S2 layer 5 (frozen)IOI+0.63+0.73natural
IOI, 410M, S2 layer 10IOI+0.59+0.92natural
agr_genderAgreement+0.59+0.42natural
agr_refl_num_obj-relcAgreement+0.41+0.10excess-dominated
agr_sv_num_obj-relcAgreement+0.41+0.27excess-dominated
agr_sv_num_ppAgreement+0.49+0.38natural
npi_any_obj-relcNPI+0.50-0.02excess-dominated
npi_any_subj-relcNPI+0.34-0.07excess-dominated
garden_mvrrGarden path+0.34-0.24excess-dominated
garden_npz_objGarden path+0.56+0.10excess-dominated
garden_npz_obj_modGarden path+0.32-0.07excess-dominated
filler_gap_objFiller–gap+0.33-0.35excess-dominated
filler_gap_ppFiller–gap+0.30-0.22excess-dominated
filler_gap_subjFiller–gap+0.33-0.16excess-dominated
cleftCleft+0.49-0.01excess-dominated
cleft_modCleft+0.32-0.01excess-dominated
gss_subordSubordination+0.33-0.17excess-dominated
gss_subord_obj-relcSubordination+0.47+0.21excess-dominated
gss_subord_ppSubordination+0.50+0.19excess-dominated
gss_subord_subj-relcSubordination+0.42+0.08excess-dominated

Correction, 19 Nov. The first version of this section called the non-natural sites "illusions" and said the model "doesn't use" the variable there. A reconciliation with Tanyel and Ayvaz shows that overstates it. Our natural lift is measured against ablating the DAS subspace, which sits near chance (median 0.48). Measured against the unpatched run, as they do, the real activation's effect is clearly positive at most of these sites (median G +0.26). In their terms, 14 of 16 end-only tasks are excess on the route, and only 2 have G ≈ 0. Also, three CausalGym tasks are at chance behaviorally (cleft_mod, filler_gap_embed_3, filler_gap_embed_4; clean accuracy 0.50–0.52), yet they pass the gate and count in "27 of 27". The labels below are now natural and excess-dominated; section 14 applies their strict off-route test.

Follow-up 4: the gap at a natural site. IOI is natural and not binary, so it decides whether the gap is real. The same battery ran at the frozen IOI site, with global maps fitted on Pile or on IOI sentences (training and validation names only).

Way of moving the variable from seed A to seed BCausalGym, end-only (18 tasks)CausalGym, natural sites only (2 tasks)IOI, frozen site (natural)
Learned 16×16 map (local, paired activations)0.950.990.99
A's subspace carried by one global linear map (best transport, best fit)0.550.580.46
Value carried by the global map, written into B's own subspace (hybrid)0.890.880.96
Whole activation carried by the global map (vs B's own transplant)0.920.840.86

The gap survives at a natural site. At the frozen IOI site, one global linear map fitted on unlabeled IOI sentences (with names never used in testing) carries the variable's value from seed A into seed B's own subspace: T = 0.96 (95% CI 0.93 to 0.97). Carried by the same map, A's subspace reaches at best T = 0.46, barely above ablating the variable. The illusion explanation doesn't apply here: the IOI site is natural.

Follow-up 5: is the carried subspace causal in B at all? Write B's own source value into A's subspace as the global map places it in B. That gives T = 0.29 at best, against 0.09 for a random 16-dimensional subspace. The carried subspace shares about a tenth of B's own (overlap 0.16, against 0.02 at random). B's subspace is not a dormant direction: it holds 18× the variance of an average subspace.

That looked like a finding: one global map carries what the IOI variable's value is, but not where each seed's computation reads it. It survived the controls for rotation vs linear map, fitting text, and the illusion. One alternative remained: the map is clearly imprecise at S2.

Follow-up 6: a more accurate map. If map error caused the gap, a more accurate map should carry the subspace. I tried maps fitted on S2 activations only (24 names, then 204, none of them test names), rotations on the same data, and the crosscoder used as a map. As an upper bound, I also fitted an oracle on the test activations themselves.

Map between seeds (42 pairs, medians)Accuracy: R² for A's DAS coordinatesA's subspace carried into B (T)B's own value on the carried subspace (T)Value into B's own subspace (T)
IOI text (follow-up 4's map)0.300.110.110.95
Crosscoder, linear approximation0.440.850.440.63
S2 only, 24 names0.690.470.430.97
S2 only, 204 names0.770.600.530.94
Oracle: fitted on the test activations (upper bound, not in the decision)0.970.960.950.97

The candidate is withdrawn. By the preregistered rule, one held-out map carries A's subspace well enough: T = 0.85, from the crosscoder's linear approximation. That condition is odd on its own, because B's own value on the same subspace reaches only 0.44. The oracle settles it. A linear map that predicts the S2 activations accurately carries the causal subspace: T = 0.96, and B's own value on it 0.95 (at least 0.80 in 39 of 42 pairs). Across maps, the more accurate the map, the more causal the subspace it carries (Spearman 0.75). So the seeds do share the causal subspace, up to a linear map. What follow-ups 4 and 5 measured is an accuracy requirement: carrying the value tolerates a poor map, carrying the subspace needs a nearly exact one.

10. New: the crosscoder arm

DAS tests variables we choose. The crosscoder asks the reverse question across thousands of unlabeled features: one shared sparse dictionary over layer 5 of seeds 1–7, with seeds 8 and 9 held out. Every causal test encodes each seed from its own activations only.

SeedSeen in training?Variance unexplained, shared dictionaryVariance unexplained, own SAELoss recovered, sharedLoss recovered, own SAE
8no0.0240.0190.9500.966
9no0.0230.0170.9510.971
3yes0.023—0.960—
7yes0.018—0.959—

Reading, fixed in advance: the seeds share features, and only the frame differs. Follow-up 6 used the crosscoder as a map between seeds. It carries the IOI value (whole-activation transplant T = 0.95), but a round trip through it keeps only 0.32 of the variable, so it cannot carry an edit.

11. New: OLMo-architecture models do IOI (DataDecide screen)

DataDecide is the only OLMo-architecture release with independent seeds. Its 150M models have Pythia-160M's shape (12 layers × 768) and Pythia's token IDs, but were trained on 15B tokens rather than 300B. Before designing a rung on them, a preregistered screen asked one question: do they do IOI at all? I loaded the weights by converting them to a standard Llama model, with no new packages, and used a checkpoint only if the conversion passed a three-part gate.

DataDecide 150M checkpoint (step 37500)Max |Δlogit| vs referencePile lossHellaSwag, ours = AI2'sConversion gateIOI accuracyIOI mean logit diff
Dolma 1.7, default2.9e-043.460.3484 = 0.3484pass0.9243.37
Dolma 1.7, aux-22.5e-043.450.3513 = 0.3513pass0.8952.53
Dolma 1.7, aux-33.1e-043.450.3458 = 0.3458pass0.8801.81
DCLM, default3.7e-043.660.3579 = 0.3579pass0.9533.14
DCLM, aux-22.3e-043.720.3578 = 0.3578pass0.9422.49
DCLM, aux-34.4e-043.610.3622 = 0.3622pass0.8871.89
FineWeb-Edu, default3.5e-044.100.3648 = 0.3648fail: Pile loss——
FineWeb-Edu, aux-21.8e-044.280.3686 = 0.3686fail: Pile loss——
FineWeb-Edu, aux-32.9e-044.130.3686 = 0.3686fail: Pile loss——
C4, default2.0e-044.530.3646 = 0.3646fail: Pile loss——
C4, aux-22.3e-044.630.3636 = 0.3636fail: Pile loss——
C4, aux-38.0e-054.680.3595 = 0.3595fail: Pile loss——
Pythia-160M seeds 3–9 (reference)—3.09 (main model)——0.833–0.9721.99–3.55

Decision, fixed in advance: rung 3 at 150M on Dolma 1.7 and DCLM. Two corpora × three seeds give seeds within OLMo, a change of corpus, and Pythia ↔ OLMo. No 1B screen is needed. One question goes into the rung's preregistration: does a DataDecide seed label fix the initialization across corpora? If it does, "same seed, different corpus" is a same-init, different-data pair, like ladder step 1.

12. New: ladder rung 3, Pythia ↔ OLMo

Rung 3 asks whether the IOI variable moves between seeds of a second family, across a change of training corpus, and between families. It uses DataDecide 150M from all four corpora (three seeds each) and Pythia-160M seeds 3–9, with the v2 estimator unchanged. Everything was preregistered before the run, with one amendment made from weights alone before any transfer number.

Within a familyAcross corpora, OLMoAcross families
0.40.60.811.21.4own ceilingT = transferred accuracy / the destination's own ceiling (each dot is one directed pair)Within Pythiaseed ↔ seed, the reference · 42 pairs0.99Within OLMoseed ↔ seed, same corpus · 24 pairs0.91Across corporaOLMo, different corpus · 108 pairs0.94Pythia → OLMoacross families · 84 pairs1.03OLMo → Pythiaacross families · 84 pairs0.93
Figure 13. Pythia ↔ OLMo transfer is as strong as seed ↔ seed transfer. Selected site, scaled map. Large dot: group median T with pair-bootstrap 95% CI; small dots: directed pairs. All five groups pass the preregistered test (gap CI above 0); 165 of 168 cross-family pairs beat their own random null. T above 1 means a foreign source works slightly better than the destination's own DAS vector.
Table view
GroupGap (95% CI)Median TLearned IIARandom p95Beat own null
Within Pythia+0.305 (+0.241 to +0.339)0.990.870.5741/42
Within OLMo+0.300 (+0.253 to +0.348)0.910.770.4524/24
Across corpora+0.307 (+0.284 to +0.322)0.940.790.47108/108
Pythia → OLMo+0.321 (+0.303 to +0.340)1.030.860.5284/84
OLMo → Pythia+0.249 (+0.220 to +0.266)0.930.780.5381/84

None of the training differences moves T. Each contrast has a ±0.10 "no difference" margin fixed in advance:

Contrast (difference in median T)Difference (95% CI)Reading, fixed in advance
K1. Within OLMo − across corpora-0.024 (-0.060 to +0.049)no difference
K2. Cross-family pairs with Dolma 1.7 − with C4 (most vs least Pile-like)-0.002 (-0.108 to +0.043)inconclusive
K2, alongside. OLMo → Pythia, matched by Pythia destination+0.133 (+0.070 to +0.161)first side higher
K3. Shared − different initialization, across corpora, matched+0.051 (-0.029 to +0.095)no difference
K4. Across families − within Pythia-0.025 (-0.057 to -0.009)no difference
K4. Across families − within OLMo+0.050 (-0.012 to +0.084)no difference

What rung 3 says. At 150M, the IOI variable moves between two architecture families, four corpora and different initializations as well as between seeds of one family. That makes it a universality result. It also means this test cannot tell training histories apart: every comparison gives T ≈ 0.9–1.0. It says nothing about components; Xu 2026 finds OLMo-1B and Pythia-1B use different primary IOI components. A surprise would need a setting where transfer varies.

13. New: 18 syntactic variables across families and sizes (rung 3b, option I)

Rung 3 settled IOI. Rung 3b and option I ask the same of the 18 CausalGym tasks that localize at the end-only site. Do they transfer between Pythia and OLMo (the 12 DataDecide models of rung 3), and between Pythia-160M and 410M? And does the split between natural and excess-dominated sites (section 9) hold? Both were preregistered before the run, with an amendment before any number was seen (the relabelling and the Tanyel–Ayvaz split below).

Group (18 tasks)Tasks passingMedian TLowest task TPairs beating their nullMismatched-pairing IIA
Within Pythia-160M18 of 180.950.89752 / 7560.49
Within OLMo18 of 180.970.912,363 / 2,3760.50
Pythia ↔ OLMo18 of 180.960.913,007 / 3,0240.49
Within Pythia-410M18 of 180.960.92752 / 7560.50
160M ↔ 410M18 of 180.950.891,754 / 1,7640.49
Pythia-160MOLMo (DataDecide 150M)Pythia-410M
-0.4-0.200.20.4natural (0.30)Natural lift: transplanting the model's own whole activation, over ablating the DAS subspaceagr_sv_num_ppT across families 0.97 · sizes 0.98agr_genderT across families 1.00 · sizes 0.99agr_sv_num_obj-relcT across families 0.98 · sizes 0.98gss_subord_ppT across families 0.96 · sizes 0.97gss_subord_obj-relcT across families 0.95 · sizes 0.93agr_refl_num_obj-relcT across families 0.98 · sizes 0.98gss_subord_subj-relcT across families 0.91 · sizes 0.93garden_npz_objT across families 0.98 · sizes 0.97cleftT across families 1.00 · sizes 1.00cleft_modT across families 0.98 · sizes 0.98garden_npz_obj_modT across families 0.94 · sizes 0.91npi_any_obj-relcT across families 0.92 · sizes 0.94npi_any_subj-relcT across families 0.92 · sizes 0.94filler_gap_subjT across families 0.93 · sizes 0.95gss_subordT across families 0.96 · sizes 0.95filler_gap_ppT across families 0.93 · sizes 0.94garden_mvrrT across families 0.92 · sizes 0.89filler_gap_objT across families 0.95 · sizes 0.95
Figure 14. Which sites are natural is conserved across families and sizes. Each row is a task; each dot is the median natural lift in one model set (end-only site, each model at its own selected layer). Right of the line, the model's own activation installs the source value; left of it, only DAS's patch does. Spearman over tasks: 0.94 Pythia vs OLMo, 0.94 160M vs 410M. Hollow: the task fails the gate in that model set (two tasks in OLMo). The sub-label gives cross-family and cross-size T; they barely vary.
Table view
TaskNatural lift, Pythia-160MOLMoPythia-410MT across familiesT across sizes
agr_sv_num_pp+0.38+0.31+0.410.970.98
agr_gender+0.42+0.33+0.191.000.99
agr_sv_num_obj-relc+0.27+0.25+0.330.980.98
gss_subord_pp+0.19+0.08+0.170.960.97
gss_subord_obj-relc+0.21+0.07+0.130.950.93
agr_refl_num_obj-relc+0.10+0.21+0.090.980.98
gss_subord_subj-relc+0.08-0.02+0.200.910.93
garden_npz_obj+0.10+0.04+0.080.980.97
cleft-0.01-0.07-0.011.001.00
cleft_mod-0.01-0.03-0.090.980.98
garden_npz_obj_mod-0.07-0.04-0.020.940.91
npi_any_obj-relc-0.02-0.11-0.090.920.94
npi_any_subj-relc-0.07-0.15-0.110.920.94
filler_gap_subj-0.16-0.09-0.130.930.95
gss_subord-0.17-0.09 (gate fails)-0.140.960.95
filler_gap_pp-0.22-0.12-0.200.930.94
garden_mvrr-0.24-0.19 (gate fails)-0.190.920.89
filler_gap_obj-0.35-0.28-0.190.950.95

What rung 3b and option I say. On 18 syntactic variables, as on IOI, transfer gives T ≈ 0.9–1.0 for seeds, corpora, families and sizes. And it passes at sites where the model's own activation does not carry the value, in every setting tried. The robust negative claim, "cross-model transfer does not certify a variable the model uses", now holds across families, the setting of the 2026 cross-family papers.

14. New: Tanyel and Ayvaz's strict off-route test on our sites (runs A and A′)

The 19 Nov reconciliation used interchange accuracy, which cannot tell a small effect from none. Runs A and A′ apply Tanyel and Ayvaz's own test (their Algorithm 1) to our cached Pythia-160M fits, in logit difference. A site is off-route when DAS's patch has an effect (e), the model's own activation in its best context does not (G*), and e exceeds G*, each by a cluster-bootstrap CI, in at least 4 of 7 seeds. The best context is the cell alone or the effect the cell adds when one other region of its layer is patched too. The reading was fixed in advance: three or more competent, transferring off-route tasks would let the lead claim say "transfer passes at Makelov illusions".

Correction, 20 Nov. Run A used the cell plus a region's total effect instead of the cell's added effect. That inflates G* whenever the other region carries the variable by itself, and it made the selected sites look on the route "through a second position" (G*/T 0.78 under run A against 0.10 now). Run A′ repeats the test with the published definition; the numbers below are run A′'s.

Result: one competent task is off-route, below the three needed. filler_gap_hierarchy is off-route at its selected site in all 7 seeds, and it transfers. The lead claim is still stated as "transfer passes at excess-dominated sites", but excess here is often extreme.

SiteCompetent, gate-passing taskse/T (DAS)G/T (cell)G*/T (best context)G* > 0Excess e − G* > 0Off-route
Selected (variable's token)241.200.090.1023 of 2424 of 241
End-only170.940.430.4617 of 1717 of 170
Model's own activation in its best context (G*)DAS's 16-dimensional patch (e)
00.511.52full effectEffect as a fraction of the full source-vs-base effect T (logit difference)agr_sv_num_ppagr_sv_num_obj-relcagr_sv_num_subj-relcagr_genderagr_refl_num_obj-relcagr_refl_num_subj-relcagr_refl_num_ppgarden_mvrr_modgarden_npz_objgarden_npz_obj_modgarden_mvrrgss_subord_ppgss_subordfiller_gap_objgss_subord_obj-relcfiller_gap_ppgss_subord_subj-relcnpi_any_subj-relcnpi_any_obj-relcnpi_ever_subj-relcnpi_ever_obj-relcfiller_gap_subjcleftfiller_gap_hierarchy
Figure 15. At the variable's token, the model's own activation carries almost nothing of what DAS carries. Selected site, one row per task (24 competent, gate-passing tasks), medians over seeds 3–9, as fractions of the full effect of swapping the source sentence in. On 11 tasks G* is under 0.10 T while DAS gives 1–2 T, a ratio of 24–57; the agreement tasks at the top are the exception. Bold: off-route in at least 4 of 7 seeds.
Table view
TaskG/T (cell)G*/T (best context)e/T (DAS)Off-route seeds
agr_sv_num_pp0.730.761.190/7
agr_sv_num_obj-relc0.720.731.200/7
agr_sv_num_subj-relc0.630.631.150/7
agr_gender0.600.600.780/7
agr_refl_num_obj-relc0.540.551.110/7
agr_refl_num_subj-relc0.510.531.070/7
agr_refl_num_pp0.480.521.020/7
garden_mvrr_mod0.370.372.160/7
garden_npz_obj0.290.371.130/7
garden_npz_obj_mod0.280.361.180/7
garden_mvrr0.180.181.070/7
gss_subord_pp0.090.111.710/7
gss_subord0.100.102.070/7
filler_gap_obj0.050.081.900/7
gss_subord_obj-relc0.050.061.570/7
filler_gap_pp0.040.061.770/7
gss_subord_subj-relc0.040.051.611/7
npi_any_subj-relc0.030.041.640/7
npi_any_obj-relc0.020.041.330/7
npi_ever_subj-relc0.030.031.492/7
npi_ever_obj-relc0.020.030.961/7
filler_gap_subj0.020.031.440/7
cleft0.010.010.993/7
filler_gap_hierarchy-0.02-0.021.127/7

What runs A and A′ say. At the sites chosen for maximum DAS lift, transfer passes whether the model's own route carries a little, almost nothing, or nothing (one competent task). Whether transfer passes at off-route cells in general, found across the whole grid rather than where DAS works best, is option L.

15. New: transfer at off-route cells across the whole grid (options L and L2)

filler_gap_hierarchy in Pythia-160M: which cells are on the routeon the routeoff the routeno path to the answer1234567891011layerthat / whothe variableafter itafter itlast tokenthe answeranswerABEffect at each cell (1 = full effect of switching the input)A · last token, layer 6: on the routemodel's own activation0.97DAS's patch1.89cross-seed transfer:T = 1.00B · variable token, layer 6: off the routemodel's own activation-0.03DAS's patch1.09cross-seed transfer:T = 0.99full effect
Diagram C. Where the model reads the variable, and where DAS can write it. Left: every live cell of filler_gap_hierarchy in Pythia-160M, scored by Tanyel and Ayvaz's Algorithm 1 over seeds 3–9. Blue: on the route in at least 4 of 7 seeds (the model's own activation there adds to the answer). Orange: off-route (DAS can move the answer from there; the model's own activation does not). The variable's token is read in layers 1–3; the regions after it carry it until layers 6–7; the last token carries it throughout. Right: medians over seeds at the two marked cells. At A the model's own activation carries the whole effect and DAS's patch overshoots it; at B the model's own activation does nothing, yet DAS's patch moves the answer by about the full effect. A map between seeds carries the patch equally well at both (T 1.00 and 0.99), so passing the transfer test does not show that the model uses the cell.

Every transfer test before option L ran at the site where DAS works best. Option L asks the question the other way round. It scans every template region × layer 1–11 of Pythia-160M seeds 3–9 on the 26 tasks the models can do (1,518 live cells), finds the cells that are off-route by Tanyel and Ayvaz's Algorithm 1 in at least 4 of 7 seeds, and tests cross-seed transfer at the ones where DAS passes the batch gate. The reading was fixed in advance: if transfer passed at ≥ 2/3 of at least three such cells, transfer does not screen out off-route sites (L-pass); if it failed at ≥ 2/3 while passing at on-route reference cells, it does (L-screen).

Option L result: L-pass, on two tasks. 346 cells are off-route. 5 of them pass the gate, and transfer passes at 5 of 5 (Bonferroni α = 0.01). But four of the five are one token of filler_gap_hierarchy at layers 4–7 (layer 6 is run A′'s site), and the fifth is filler_gap_obj.

Option L2: the same test on the relaxed-gate cells (preregistered after L, before any of its transfer numbers). Option L's five cells came from two tasks, so L2 ran the identical transfer test at the 9 off-route cells that pass the relaxed gate (within ≥ 0.65, net ≥ 0.15) but not the primary one. Three of their tasks are new: filler_gap_pp, garden_npz_v-trans, garden_npz_v-trans_mod. The reading was fixed in advance: L2-pass if ≥ 2/3 of the new cells pass and ≥ 2 of the 3 new tasks qualify (≥ 2/3 of a task's cells passing at α = 0.05/14).

Result: L2-pass. 9 of 9 new cells pass (each in 42 of 42 pairs), all 3 new tasks qualify, and across options L and L2 all 14 cells pass at α = 0.05/14, on 5 of 5 tasks from two construction families (filler–gap and NPZ garden path). Transfer T is 0.92–1.00 everywhere.

0+0.2+0.4+0.6random rotation's 95th percentileGap: learned map's interchange accuracy minus the 95th percentile of 50 random rotationsfiller_gap_hierarchy, layer 4L · variable token · DAS 1.59 T, own -0.02 T0.35filler_gap_hierarchy, layer 5L · variable token · DAS 1.56 T, own -0.01 T0.37filler_gap_hierarchy, layer 6L · variable token · DAS 1.09 T, own -0.03 T0.47filler_gap_hierarchy, layer 7L · variable token · DAS 0.88 T, own -0.03 T0.43filler_gap_obj, layer 9L · variable token · DAS 1.19 T, own +0.01 T0.25filler_gap_hierarchy, layer 8L2 · variable token · DAS 0.69 T, own -0.02 T0.30filler_gap_obj, layer 2L2 · last token · DAS 0.73 T, own +0.01 T0.11filler_gap_pp, layer 9L2 · variable token · DAS 0.94 T, own +0.01 T0.28garden_npz_v-trans, layer 9L2 · variable token · DAS 0.94 T, own +0.04 T0.12garden_npz_v-trans_mod, layer 4L2 · variable token · DAS 2.20 T, own +0.15 T0.13garden_npz_v-trans_mod, layer 5L2 · variable token · DAS 2.23 T, own +0.16 T0.12garden_npz_v-trans_mod, layer 6L2 · variable token · DAS 2.38 T, own +0.14 T0.12garden_npz_v-trans_mod, layer 7L2 · variable token · DAS 2.49 T, own +0.17 T0.11garden_npz_v-trans_mod, layer 8L2 · variable token · DAS 2.01 T, own +0.14 T0.11
Figure 16. Transfer passes at every testable off-route cell. One row per cell (14; the grey line under each name says whether option L or L2 tested it), one dot per directed seed pair (42), in interchange accuracy. Tick: median gap; bar: its pair-bootstrap CI at α = 0.0036 (Bonferroni over all 14 cells). Every bar clears 0, where the learned map would do no better than the 95th percentile of random rotations. Under each name: DAS's effect and the model's own effect (G*) at that cell, as fractions of the full effect T, medians over seeds. The garden-path gaps are smaller because random rotations already reach 0.58–0.75 there.
Table view
CellOptionPositione/T (DAS)G*/T (own)Median gapCILearnedRandom p95Transfer T
filler_gap_hierarchy, layer 4Lvariable token1.59-0.015+0.350.23 to 0.400.990.641.00
filler_gap_hierarchy, layer 5Lvariable token1.56-0.012+0.370.28 to 0.440.990.621.00
filler_gap_hierarchy, layer 6Lvariable token1.09-0.030+0.470.42 to 0.510.970.510.99
filler_gap_hierarchy, layer 7Lvariable token0.88-0.032+0.430.39 to 0.460.870.420.98
filler_gap_obj, layer 9Lvariable token1.19+0.006+0.250.23 to 0.270.810.560.98
filler_gap_hierarchy, layer 8L2variable token0.69-0.017+0.300.26 to 0.350.700.330.96
filler_gap_obj, layer 2L2last token0.73+0.010+0.110.09 to 0.130.600.490.92
filler_gap_pp, layer 9L2variable token0.94+0.006+0.280.24 to 0.320.730.450.93
garden_npz_v-trans, layer 9L2variable token0.94+0.035+0.120.11 to 0.130.700.580.97
garden_npz_v-trans_mod, layer 4L2variable token2.20+0.148+0.130.11 to 0.150.880.750.98
garden_npz_v-trans_mod, layer 5L2variable token2.23+0.164+0.120.11 to 0.140.870.740.99
garden_npz_v-trans_mod, layer 6L2variable token2.38+0.136+0.120.10 to 0.140.860.731.00
garden_npz_v-trans_mod, layer 7L2variable token2.49+0.168+0.110.10 to 0.130.840.730.98
garden_npz_v-trans_mod, layer 8L2variable token2.01+0.137+0.110.09 to 0.120.800.690.99

Transfer at the last-token reference cells, which are on the route (descriptive). There, a random rotation does almost as well as the learned map (random 95th percentile 0.86–1.00 wherever DAS localizes well), so the gap test has no room. That is also why these cells fail the net gate: writing anything into the subspace, even its mean, flips most answers. Transfer T is the comparable number, and it is the same on and off the route: T(off-route) − T(reference) is -0.04 to +0.05 (median -0.01) over the 10 cells whose reference DAS localizes.

Reference (last token, same layer)On-route seedsWithinNetLearnedRandom p95Transfer T
filler_gap_hierarchy, layer 47 of 71.000.191.000.941.00
filler_gap_hierarchy, layer 57 of 71.000.111.000.981.00
filler_gap_hierarchy, layer 67 of 71.000.051.000.991.00
filler_gap_hierarchy, layer 77 of 71.000.041.000.991.00
filler_gap_hierarchy, layer 87 of 71.000.051.000.991.00
filler_gap_obj, layer 97 of 71.000.021.001.001.00
filler_gap_pp, layer 97 of 70.960.180.940.860.97
garden_npz_v-trans_mod, layer 41 of 70.460.040.440.460.98
garden_npz_v-trans_mod, layer 53 of 70.470.030.460.480.96
garden_npz_v-trans_mod, layer 66 of 70.510.070.460.490.95
garden_npz_v-trans_mod, layer 77 of 70.580.110.590.560.96
garden_npz_v-trans_mod, layer 87 of 70.690.140.640.600.93
garden_npz_v-trans, layer 97 of 70.990.090.970.880.99

Option P: why ablation flips most answers at the last token (exploratory). Each entry is IIA, with the median logit difference z (source label minus base label) in parentheses. At the last token, the answer is linear in the subspace coordinates: z after mean ablation equals the midpoint of z at the base's own value-class mean and at the other class's mean, to within 0.1 logit. The subspace moves z by ±2.9 to ±5.4 logits, while the rest of the residual adds a constant 1.6–3.4 logits toward whichever value the base does not have. Normally the subspace wins; ablating or randomizing it leaves the rest's push, so the answer flips. This is DAS excess seen from the ablation side: the subspace's swing is about twice the model's natural effect, and the model's own state cancels part of it elsewhere. At the off-route variable-token cell, ablation barely moves anything, because the model does not read it.

CellBaseDASAblate 16Ablate top-1Own-class meanOther-class mean
filler_gap_hierarchy, layer 6, last token0.17 (-1.9)1.00 (+7.0)0.95 (+2.6)0.84 (+2.1)0.24 (-1.7)1.00 (+6.6)
filler_gap_obj, layer 9, last token0.20 (-2.1)1.00 (+9.3)0.98 (+3.5)0.91 (+3.1)0.14 (-2.0)1.00 (+8.8)
filler_gap_pp, layer 9, last token0.20 (-2.6)0.96 (+6.7)0.77 (+2.2)0.69 (+1.7)0.24 (-2.3)0.98 (+6.4)
garden_npz_v-trans, layer 9, last token0.28 (-1.9)0.99 (+4.8)0.90 (+1.7)0.75 (+0.9)0.18 (-1.2)1.00 (+4.5)
filler_gap_hierarchy, layer 6, variable token (off-route)0.17 (-1.9)0.99 (+3.0)0.28 (-1.2)0.27 (-1.2)0.18 (-1.9)0.96 (+2.4)
garden_npz_v-trans_mod, layer 6, variable token (off-route)0.39 (-0.6)0.87 (+2.8)0.70 (+1.1)0.49 (-0.1)0.43 (-0.4)0.87 (+2.8)
answer flips at 0-4+4+8z = logit(source answer) − logit(base answer), median over seeds← keeps the base answerswitches to the source →A · last token, layer 6on the routebasesubspace ablatedDAS swapB · variable token, layer 6off the routebasesubspace ablatedDAS swapsubspace swings z by ±4.1; the rest of the residual sits at +2.5
Diagram D. Why ablation flips answers at the last token (option P). Real medians for filler_gap_hierarchy, layer 6. At the last token (A), the subspace swings z between its two class means, and the rest of the residual sits on the far side of 0. The model's own state normally wins (base, z < 0), but erasing the subspace leaves only the rest, so the answer flips: ablation IIA is high and the net gate (within − ablation) is low, though the model reads this cell. At the off-route variable token (B), ablation barely moves z, so net is high though the model ignores the cell. The gate rewards the wrong cells.

What P means for the method. The batch's net gate (within IIA minus ablation IIA) is high where ablation is harmless, which is where the model does not read the cell, and low at on-route cells with excess. So it favours off-route cells and does not measure "the subspace matters". The same push puts the random-rotation null near ceiling at on-route last-token cells, where the gap test is uninformative. Options L and L2 stand, because their tests compare the learned map with nulls in the same destination, but the paper must describe the gate this way.

Descriptive, from option L's scan.

PositionLive cellsOff-routeRate
Variable's token1983317%
After the variable1,03426626%
Last token2864716%

What options L and L2 say. Cross-seed transfer passes where the model's own activation carries nothing: 14 of 14 off-route cells on 5 competent tasks (4 with G* near 0), at T 0.92–1.00, the same as at on-route cells. Transfer shows that the seeds share a linear representation DAS can write into, not that the model reads it at that position. Option P explains the last-token ablation flips as DAS excess and shows that the batch's net gate favours off-route cells.

16. New: the 1B check (option O)

Every model in sections 1–15 has at most 410M parameters. Option O asks whether the paper's two claims hold at 1B: (A) the IOI variable transfers between independently trained models at its site, and (B) transfer passes at competent off-route cells. It uses AI2's DataDecide Dolma 1.7 1B models (OLMo architecture, 16 layers × 2048, three seeds, final checkpoint), the same family and corpus as rung 3's 150M models. Everything was preregistered before any 1B weight was downloaded: a conversion gate, option L's pipeline for part B with readings fixed in advance (B-pass needs ≥ 2/3 of at least three testable cells to pass, from ≥ 2 tasks), and rung 3's IOI estimator for part A (A-pass needs the gap CI to exclude 0 both within 1B and across sizes, 1B ↔ 150M).

The conversion is exact. All three checkpoints pass every bar. The converted models match an independent reference forward pass to 1.0e-4 logits, reproduce AI2's HellaSwag score to the fourth decimal on all 10,042 items (0.5961–0.6006), and reach a Pile loss of 2.91–2.94. They solve IOI perfectly (accuracy 1.00) and do all five CausalGym tasks (median clean accuracy ≥ 0.72).

Part B result: B-pass, 3 of 3 cells from 3 tasks. The scan finds 33 off-route cells among 330 live ones. Only one passes the primary gate, so the relaxed gate applied, as preregistered, giving three testable cells. Transfer passes at all three, in 6 of 6 seed pairs each, at T 0.96–0.98, while the model's own activation at those cells gives at most 0.013 T. All three sit in regions that were off-route at 160M: the filler–gap variable's token, and the garden_npz_v-trans_mod region that gave five of L2's cells.

Part A result: A-pass, at a natural site. Validation picked layer 7 in every 1B model, the preregistered fixed layer and the same relative depth as layer 5 at 160M. Zero-ablating the subspace there drops interchange accuracy from 0.96–0.99 to 0.002 or less (natural lift 0.998), so option P's concern, that selecting by net favours sites the model does not read, does not apply here. Within 1B the learned map gives 0.97 against a random 95th percentile of 0.31 (T 0.99), and the variable crosses sizes in both directions: all 18 pairs between 1B and 150M beat their null (median T 0.97). The within-150M group reproduces rung 3's Dolma numbers exactly, as a check of the code path.

0+0.2+0.4+0.6+0.8random rotation's 95th percentileGap: learned map's interchange accuracy minus the 95th percentile of 50 random rotationsfiller_gap_hierarchy, layer 11off-route at 1B · variable token · DAS 0.81 T, own +0.002 T0.50filler_gap_obj, layer 13off-route at 1B · variable token · DAS 0.93 T, own -0.004 T0.29garden_npz_v-trans_mod, layer 12off-route at 1B · variable token · DAS 0.86 T, own +0.013 T0.12IOI, within 1Bselected site · 6 pairs · T 0.990.66IOI, 1B → 150Mselected site · 9 pairs · T 1.080.42IOI, 150M → 1Bselected site · 9 pairs · T 0.790.49IOI, within 150Mselected site · 6 pairs · T 0.890.31
Figure 17. At 1B, transfer passes at the off-route cells and at IOI, within a size and across sizes. Top three rows: the testable off-route CausalGym cells at 1B, one dot per directed seed pair (6); bar: the pair-bootstrap CI of the median gap at α = 0.05/3. Bottom four rows: IOI at each model's selected site, one dot per directed model pair; bar: 95% CI. Tick: median gap. Every bar clears 0, where the learned map would do no better than the 95th percentile of random rotations. The IOI gaps at 1B are about twice those at 160M because the 1B subspace is cleaner: random rotations reach only 0.31 there.
Table view
RowPairsMedian gapCILearnedRandom p95Transfer T
filler_gap_hierarchy, layer 11 (off-route)6+0.500.47 to 0.53 (α = 0.05/3)0.840.360.98
filler_gap_obj, layer 13 (off-route)6+0.290.26 to 0.36 (α = 0.05/3)0.720.450.96
garden_npz_v-trans_mod, layer 12 (off-route)6+0.120.11 to 0.16 (α = 0.05/3)0.680.560.98
IOI, within 1B6+0.660.55 to 0.78 (95%)0.970.310.99
IOI, 1B → 150M9+0.420.27 to 0.58 (95%)0.910.381.08
IOI, 150M → 1B9+0.490.44 to 0.58 (95%)0.750.260.79
IOI, within 150M6+0.310.20 to 0.42 (95%)0.690.290.89

Transfer at the last-token reference cells (descriptive). As option P predicts, two of the three are at ceiling: the learned map and a random rotation both reach 1.00, so the gap test has no room there. Transfer T is the same on and off the route.

Reference (last token, same layer)WithinNetLearnedRandom p95Transfer T
filler_gap_hierarchy, layer 111.000.111.001.001.00
filler_gap_obj, layer 131.000.011.001.001.00
garden_npz_v-trans_mod, layer 120.860.150.840.750.95

Descriptive, from the 1B scan.

What option O says. Both claims hold at 1B, a sixfold scale-up, in the OLMo architecture: the IOI variable transfers between seeds and across sizes at a site the model reads, and cross-seed transfer passes at competent off-route cells, in the same regions as at 160M. Not shown: anything about the weak early-layer off-route cells, or about 1B outside one corpus and one architecture.

17. New: the 7–9B round (option Q)

No public suite has independently seeded pretraining above 1B, so option Q tests the paper's claims at 7–9B in the settings that exist. Both parts were preregistered on 21 Nov before any 7–9B number. Part A: IOI transfer across four families (OLMo-3-7B, Llama-3.1-8B, Qwen2.5-7B, Gemma-2-9B), and between three OLMo-3-7B runs that share one pretraining checkpoint and differ only in their midtraining data. Part C: the analogy direction from the COLM'26 paper, taken along OLMo-3's open post-training chain (base → SFT → DPO → Instruct), asks whether a direction's effect follows where it came from or where it is put.

Part A result (preregistered reading): QA-pass. Cross-family transfer at a natural 7–9B site. Across families the median gap is +0.66 (95% CI +0.61 to +0.70), T 0.96, with 12 of 12 pairs above their null. The selected site is natural (lift ≥ 0.30) in 4 of 4 family models. All 7 models run so far pass the replay and competence preconditions.

ModelSelected layerFixed layerValidation netOwn DAS IIA (test)Natural liftIOI acc / logit diffReplay error
OLMo-3-7B12 of 32 (38%)130.980.951.001.00 / 6.70.000
Llama-3.1-8B12 of 32 (38%)130.970.941.001.00 / 5.70.000
Qwen2.5-7B18 of 28 (64%)120.970.990.861.00 / 6.60.000
Gemma-2-9B21 of 42 (50%)180.860.721.001.00 / 26.60.000
OLMo-3-7B, midtraining mix A12 of 32 (38%)130.840.711.001.00 / 6.60.000
OLMo-3-7B, midtraining mix B12 of 32 (38%)130.740.731.001.00 / 6.50.000
OLMo-3-7B, midtraining mix C12 of 32 (38%)130.950.901.001.00 / 6.70.000
GroupSitePairsGap (95% CI)TLearnedRandom p95MismatchedPairs above null
Across familiesselected12+0.66 (+0.61 to +0.70)0.960.920.200.1512 of 12
OLMo-3 midtraining runsselected6+0.69 (+0.57 to +0.73)1.020.810.130.076 of 6
Across familiesfixed12+0.66 (+0.51 to +0.70)0.950.850.14–12 of 12
OLMo-3 midtraining runsfixed6+0.66 (+0.61 to +0.71)0.970.730.06–6 of 6
0+0.2+0.4+0.6+0.8+1.0random rotation's 95th percentileGap: learned map's interchange accuracy minus the 95th percentile of 50 random rotationsAcross familiesselected site · 12 directed pairs · T 0.960.66OLMo-3 midtraining runsselected site · 6 directed pairs · T 1.020.69Across familiesfixed site · 12 directed pairs · T 0.950.66OLMo-3 midtraining runsfixed site · 6 directed pairs · T 0.970.66
Figure 18. IOI transfer at 7–9B. One dot per directed model pair; tick: median gap; bar: pair-bootstrap 95% CI. The family pairs map between models with different tokenizers, widths and depths (Gemma-2-9B has 42 layers of width 3584; the others 28–32 layers of width 3584–4096), using DAS coordinates matched on the same 792 prompts.
Table view
GroupSiteSourceDestinationGapLearnedRandom p95T
Across familiesselectedLlama-3.1-8BOLMo-3-7B+0.690.890.200.93
Across familiesselectedQwen2.5-7BOLMo-3-7B+0.640.870.230.92
Across familiesselectedGemma-2-9BOLMo-3-7B+0.750.950.201.00
Across familiesselectedOLMo-3-7BLlama-3.1-8B+0.670.960.301.02
Across familiesselectedQwen2.5-7BLlama-3.1-8B+0.710.910.200.96
Across familiesselectedGemma-2-9BLlama-3.1-8B+0.780.960.181.02
Across familiesselectedOLMo-3-7BQwen2.5-7B+0.650.990.341.00
Across familiesselectedLlama-3.1-8BQwen2.5-7B+0.670.940.270.95
Across familiesselectedGemma-2-9BQwen2.5-7B+0.580.970.380.98
Across familiesselectedOLMo-3-7BGemma-2-9B+0.610.650.040.89
Across familiesselectedLlama-3.1-8BGemma-2-9B+0.620.630.010.87
Across familiesselectedQwen2.5-7BGemma-2-9B+0.510.540.030.74
OLMo-3 midtraining runsselectedOLMo-3-7B, midtraining mix BOLMo-3-7B, midtraining mix A+0.510.590.080.83
OLMo-3 midtraining runsselectedOLMo-3-7B, midtraining mix COLMo-3-7B, midtraining mix A+0.630.750.111.05
OLMo-3 midtraining runsselectedOLMo-3-7B, midtraining mix AOLMo-3-7B, midtraining mix B+0.670.810.141.11
OLMo-3 midtraining runsselectedOLMo-3-7B, midtraining mix COLMo-3-7B, midtraining mix B+0.720.870.151.20
OLMo-3 midtraining runsselectedOLMo-3-7B, midtraining mix AOLMo-3-7B, midtraining mix C+0.740.890.150.98
OLMo-3 midtraining runsselectedOLMo-3-7B, midtraining mix BOLMo-3-7B, midtraining mix C+0.720.820.100.90
Across familiesfixedLlama-3.1-8BOLMo-3-7B+0.730.760.030.85
Across familiesfixedQwen2.5-7BOLMo-3-7B+0.640.680.030.76
Across familiesfixedGemma-2-9BOLMo-3-7B+0.730.840.110.94
Across familiesfixedOLMo-3-7BLlama-3.1-8B+0.680.940.251.02
Across familiesfixedQwen2.5-7BLlama-3.1-8B+0.690.860.170.93
Across familiesfixedGemma-2-9BLlama-3.1-8B+0.680.890.210.97
Across familiesfixedOLMo-3-7BQwen2.5-7B+0.490.950.461.03
Across familiesfixedLlama-3.1-8BQwen2.5-7B+0.500.910.410.98
Across familiesfixedGemma-2-9BQwen2.5-7B+0.490.880.380.94
Across familiesfixedOLMo-3-7BGemma-2-9B+0.710.740.031.07
Across familiesfixedLlama-3.1-8BGemma-2-9B+0.640.660.020.96
Across familiesfixedQwen2.5-7BGemma-2-9B+0.520.550.030.80
OLMo-3 midtraining runsfixedOLMo-3-7B, midtraining mix BOLMo-3-7B, midtraining mix A+0.600.640.040.96
OLMo-3 midtraining runsfixedOLMo-3-7B, midtraining mix COLMo-3-7B, midtraining mix A+0.630.730.091.08
OLMo-3 midtraining runsfixedOLMo-3-7B, midtraining mix AOLMo-3-7B, midtraining mix B+0.620.640.020.98
OLMo-3 midtraining runsfixedOLMo-3-7B, midtraining mix COLMo-3-7B, midtraining mix B+0.690.730.041.13
OLMo-3 midtraining runsfixedOLMo-3-7B, midtraining mix AOLMo-3-7B, midtraining mix C+0.740.810.080.93
OLMo-3 midtraining runsfixedOLMo-3-7B, midtraining mix BOLMo-3-7B, midtraining mix C+0.680.770.090.89

Part C result (preregistered reading): mixed. Columns are the model being steered, rows the stage the vector came from (same norm in each model; only the direction differs). The source matters about as much as the destination: the range of the destination means is 0.95, of the source means 0.97, so neither preregistered pattern holds. The paper's base < instruct pattern replicates. Every base-model cell has mean fluency below 3 (1.52–2.82, 2.75 even unsteered), so base cells are noisy; no post-trained cell is below 3.

Change in analogy quality (judge, 1–5) when steering……Base…SFT…DPO…Instruct (RL)
Vector from Base-0.02+0.00+0.22+0.27
Vector from SFT+0.29+0.98+1.41+1.14
Vector from DPO+0.26+1.17+1.46+1.39
Vector from Instruct (RL)+0.26+1.19+1.51+1.38
Random direction (norm-matched)+0.06+0.06+0.28+0.07
Route (i): whole analogy state, last prompt token+0.00+0.00+0.14+0.24
Route (ii): only its coordinate along v-0.02+0.01+0.08+0.07
Unsteered analogy quality (1–5)1.081.221.351.35
Vector from SFT, DPO or InstructVector from the base modelRandom direction
0+0.5+1.0+1.5no changeChange in the judge's analogy quality (1–5) over the unsteered modelSteering Baseunsteered analogy quality 1.08Steering SFTunsteered analogy quality 1.22Steering DPOunsteered analogy quality 1.35Steering Instruct (RL)unsteered analogy quality 1.35
Figure 19. The analogy direction steers only when both the vector and the model come from after SFT. One dot per steering vector (same norm in each model; only the direction differs), judged on 100 literal prompts. Post-trained vectors (blue) lift every post-trained model by +0.98 to +1.51; the base model's own vector (orange) does little anywhere, and the base model barely responds to any vector.

18. What is not shown yet

19. Where #42 stands on the board

ImpactNoveltyConfidenceGPU-daysPriorityAccept %EVRank
4 (reviewer estimate; yours to change)3353.530% (NeurIPS 2027 / ICLR 2028)1.056 of 32

These are the Nov 10 review scores. They appear to have been scored from the September proposal: the review's go/no-go test ("within-seed DAS stable on IOI at 160M") has since passed, and it does not list McCann. The repo rules ask a separate agent, one that didn't run the experiments, to re-score. Today's evidence bears mainly on Confidence.

20. The decision

Before February, the roadmap allows almost none of your time: HRI is full time until Jan 20, #6 goes to ICML around Jan 28, and the chapter-4 choice (#42 or #43) is made with David and Sarah in early February. #42 is also the roadmap's chapter-3 fallback if #2 fails its Feb 15 check. Running the cheap #42 steps now therefore pays off whichever way that check goes.

OptionWhat it answersComputeYour timeWhen
A. Ladder step 2: Pythia-410M, 9 seeds, then 160M ↔ 410MDoes the variable survive a change of size and width?Done 17 Nov (2 GPU-hours)—Done: passes
A2. Estimator revision: rotation × one scale; 792 training prompts with validation early stoppingIs the low 410M ceiling the estimator or the model?Done 18 Nov (3 GPU-hours; rungs 1 and 2 rerun)—Done: mostly the estimator
B. Second variable: CausalGym, 29 syntactic tasks (replaced subject-verb agreement, which it contains)Is this more than one IOI variable?Done 18 Nov (8.5 GPU-hours)—Done: 27 of 27 transfer
F. Crosscoder armWhich unlabeled features do seeds share, and does a shared feature act alike in each seed?Done 18 Nov (2 GPU-hours)—Done: shared features, matching effects
G. Global-map follow-ups 2–5Is the local-vs-global gap about rotations, the fitting text, illusions, or real?Done 18 Nov (under 1 GPU-hour)—Done: gap found; H explained it
H. Better map at S2 (S2-fitted linear maps; the crosscoder as a map; an oracle)Is the subspace gap real, or map error?Done 18 Nov (3 GPU-minutes)—Done: map error; candidate withdrawn
J. Literature check (doc/related_work.md)Is the illusion result known? What else is new in 2026?Done 18 Nov—Done: illusion partly known; transfer-at-illusion-sites new
C. DataDecide (OLMo architecture, 3 seeds × 25 corpora, Pythia's token IDs)Does the variable localize and transfer in a second family, across training corpora, and from Pythia into OLMo?Done 19 Nov (screen 33 GPU-minutes; rung 1.4 GPU-hours)—Done: transfers across families, corpora and inits alike
C2. Rung 3b: CausalGym on DataDecideDoes transfer also pass at excess-dominated sites across families, the setting of the 2026 cross-family papers?Done 20 Nov (7 GPU-hours)—Done: passes everywhere (R1)
I. 410M and agreementDoes the excess split hold at 410M, and is agreement a second natural variable there?Done 20 Nov (4.5 GPU-hours)—Done: replicates; agreement natural, transfers
K. Runs A and A′: Tanyel and Ayvaz's off-route test on our fits (A′ corrects A's G*)Are any of our transferring sites off-route by the published definition?Done 20 Nov (2 GPU-minutes each)—Done: one competent off-route task, which transfers
L. Competent off-route sites: scan every region × layer, then test transfer at off-route cellsDoes transfer screen out a true off-route site?Done 20 Nov (5 GPU-hours on a shared GPU)—Done: L-pass, 5 of 5 cells, but 2 tasks
L2. Widen L: transfer at the relaxed-gate off-route cells, and at the last-token references (descriptive)Does L-pass hold beyond two tasks, and is transfer at off-route cells as strong as on the route?Done 20 Nov (5 GPU-minutes)—Done: L2-pass, 14 of 14 cells, 5 tasks; T equal on and off the route
P. Last-token ablation diagnosticWhy does mean-ablating the 16-dimensional subspace at the last token flip 80–98% of mirrored answers?Done 21 Nov (1 GPU-minute)—Done: DAS excess; the net gate favours off-route cells
O. 1B check: DataDecide 1B (3 seeds), IOI and the off-route cellsDoes the result hold at the scale reviewers will ask about?Done 21 Nov (2.6 GPU-hours)—Done: A-pass and B-pass; natural IOI site; transfers across sizes
Q. 7–9B round: IOI across OLMo-3, Llama-3.1, Qwen2.5, Gemma-2 and three OLMo-3 midtraining runs; the analogy direction along OLMo-3 post-trainingDoes IOI transfer at 7–9B, and does a direction's effect follow its source or its destination?About 3 GPU-hours—Done
M. Confirm the conserved profile on fresh tasksAre naturalness and site depth conserved across families, and is OLMo's site really deeper?Several GPU-hoursGo-aheadOptional
N. Checkpoints around emergenceDoes transfer depend on training history before a model can do the task?Several GPU-hoursGo-aheadLater
D. Workshop paper nowStakes the claim earlyNoneAbout a weekCollides with HRI and #6
E. Park until FebruaryNothing newNoneNone—

Recommendation: write. The preregistered program is complete, including the 1B check. L and L2 filled the empty cell on 5 tasks: transfer passes at competent off-route cells, at the same T as on the route. P explained the one odd number (last-token ablation flips) as DAS excess and showed that the batch's net gate favours off-route cells, which the methods section must say. O answered the scale question every reviewer will ask: both claims hold at 1B. M and N stay optional.

Files and reproduction

Everything is in /mnt/drive_b/Cross-seed.

Caveats: