A progress brief for #42 (causal convergence across seeds), 21 November 2026. It gathers the preregistered 160M result, the first two ladder steps, a revised estimator, 27 syntactic variables with a check on whether the model's own activation carries them, global-map follow-ups ending in a better-map check that withdrew a candidate finding, a crosscoder, OLMo-architecture models (a screen, ladder rung 3 and rung 3b), a 410M replication, Tanyel and Ayvaz's strict off-route test on our sites and across the whole grid, and the open choices, to support one decision: what to do with #42 before the February chapter-4 meeting.
Terms with a dotted underline show what they mean in this project when you hover over them, tab to them or tap them.
filler_gap_hierarchy, is off-route at its selected site in all 7 seeds, and it transfers. At most other variable-token sites, the model's own activation carries under a tenth of what DAS carries. Three such tasks were needed for the stronger claim, so option L searched the whole grid.The Platonic Representation Hypothesis
The test, in four steps:
The design follows Model Alignment Search
| Condition | Median | Min | Max |
|---|---|---|---|
| Learned 16×16 rotation | 0.71 | 0.22 | 0.98 |
| Random rotation, 95th pct | 0.39 | 0.12 | 0.80 |
| Whole-space Procrustes | 0.22 | 0.05 | 0.56 |
| Identity map | 0.28 | 0.01 | 0.80 |
| Mismatched prompts | 0.24 | 0.05 | 0.55 |
| Sign flip | 0.14 | 0.00 | 0.79 |
| Subspace ablated | 0.13 | 0.03 | 0.45 |
The variable is shared, but where it lives is not. Each seed chooses its own layer on validation names, and the heads that carry the variable barely overlap across seeds.
| Seed | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 |
|---|---|---|---|---|---|---|---|---|---|
| Selected layer | 5 | 5 | 5 | 6 | 5 | 6 | 6 | 7 | 4 |
| Comparison | Median top-10 overlap | n |
|---|---|---|
| Same seed, split half | 0.82 | 7 |
| Across seeds (attention-matched) | 0.25 | 21 |
| Random matching | 0.038 | 21 |
| Checkpoint | Tokens (B) | IOI accuracy | Net lift (95% CI) | Debiased CKA | Mutual 5-NN |
|---|---|---|---|---|---|
| step0 | 0.0 | 0.50 | +0.00 (-0.00 to +0.00) | 0.93 | 0.42 |
| step1000 | 2.1 | 0.41 | +0.02 (+0.01 to +0.03) | 0.92 | 0.56 |
| step2000 | 4.2 | 0.36 | +0.10 (+0.08 to +0.13) | 0.93 | 0.56 |
| step4000 | 8.4 | 0.54 | +0.17 (+0.16 to +0.21) | 0.95 | 0.58 |
| step8000 | 16.8 | 0.78 | +0.14 (+0.10 to +0.19) | 0.96 | 0.60 |
| step16000 | 33.5 | 0.85 | +0.17 (+0.14 to +0.25) | 0.96 | 0.57 |
| step33000 | 69.2 | 0.90 | +0.18 (+0.14 to +0.26) | 0.96 | 0.56 |
| step66000 | 138.4 | 0.91 | +0.21 (+0.15 to +0.34) | 0.96 | 0.58 |
| step80000 | 167.8 | 0.91 | +0.21 (+0.13 to +0.35) | 0.96 | 0.58 |
| step100000 | 209.7 | 0.93 | +0.49 (+0.41 to +0.57) | 0.96 | 0.58 |
| step120000 | 251.7 | 0.91 | +0.53 (+0.47 to +0.60) | 0.95 | 0.60 |
| step135000 | 283.1 | 0.92 | +0.49 (+0.41 to +0.57) | 0.95 | 0.59 |
| step143000 | 299.9 | 0.93 | +0.45 (+0.37 to +0.62) | 0.96 | 0.59 |
Thresholds were fixed on seeds 1–2 and never moved.
| Hypothesis | Preregistered test | Result | Verdict |
|---|---|---|---|
| H1 | Transfer ratio T ≥ 0.80 | 0.94 | Pass |
| H1 | Net lift ≥ +0.50 | +0.45 (95% CI +0.37 to +0.62) | Narrow fail |
| H1 | Learned rotation beats the nulls | 0.71 vs random p95 0.39; 37/42 pairs | Pass |
| H1 | Heads differ across seeds | 0.25 across vs 0.82 within a seed | Pass |
| H2 | Onset spread ≤ 0.35 log10 tokens | 0.54; 36/42 pairs reach onset | Fail |
| H3 | |partial ρ(CKA, net lift)| < 0.30 | −0.16 | Pass, uninformative |
| H3 | Whole-space Procrustes T ≤ 0.50 | 0.34 (95% CI 0.30 to 0.45); fitted on 792 task prompts; refits in sections 8–9 | Pass |
| Ladder 1 | Every group's gap CI excludes 0 | 4 of 4 groups; 72/72 pairs beat their own null | Pass |
| Ladder 2 | Every group's gap CI excludes 0 (within 160M, within 410M, across sizes) | 3 of 3 groups; 170/182 pairs beat their own null | Pass |
| Ladder 2, v2 | Same test with the revised estimator (reported next to v1) | 3 of 3 groups; 181/182 pairs beat their own null | Pass |
| 2nd variable | Greater-than year localizes (≥ 0.80) | 0 of 1,296 settings; best 0.37 | Fail |
| 2nd variable | Backup: name identity localizes (≥ 0.80) | best 0.74 | Fail |
| CausalGym | Gate-passing tasks that transfer, selected site (N of M) | 27 of 27; all six families; all Bonferroni | Pass |
| CausalGym | Same at the end-only site (secondary) | 18 of 18 | Pass |
| Follow-up 1 | Global rotation fitted on 260k Pile tokens: T ≥ 0.80 (carries) or ≤ 0.50 (does not) | 0.54, levels off after ~8k tokens | Between |
| Follow-up 2 | Best global linear map (Pile or task text) carries the variable: T ≥ 0.80 | 0.55 (Pile), 0.45 (task text); no task reaches 0.80 | No: candidate cell |
| Follow-up 3 | Value through the global map, written into B's own subspace: T ≥ 0.80 | 0.89 | Value travels |
| Natural-site check | CausalGym sites where the model's own activation carries the variable (lift ≥ 0.30) | 2 of 18 end-only, 4 of 27 selected; all agreement | Mostly excess-dominated |
| Natural-site check | Same at the IOI sites | lift +0.73 (160M, frozen), +0.92 (410M) | Natural |
| Crosscoder | Same latent has the same effect across seeds: top-1 − chance, CI excludes 0 | +0.224 (+0.219 to +0.228); seeds 8 ↔ 9, never seen: +0.24 | Pass |
| Follow-up 4 | IOI frozen site: A's subspace through a global linear map T ≥ 0.80; else value (hybrid) T ≥ 0.80 | subspace 0.46, value 0.96 | Value yes, subspace no |
| Follow-up 6 | A held-out, more accurate map carries A's subspace, T ≥ 0.80 → candidate withdrawn | 0.85 (crosscoder, linear approximation); oracle 0.96 | Candidate withdrawn |
| Rung 3 screen | DataDecide 150M: ≥ 2 corpora with all 3 seeds passing the conversion gate and doing IOI (accuracy ≥ 0.80, logit diff ≥ 1.0) | 2 corpora (Dolma 1.7, DCLM); FineWeb-Edu and C4 failed the gate's Pile-loss bar, not the conversion | Pass: rung 3 at 150M |
| Rung 3 | Gap CI excludes 0: within OLMo, across corpora, Pythia → OLMo, OLMo → Pythia | 4 of 4 groups; 165/168 cross-family pairs beat their null; median T 0.96 across families | Pass |
| Rung 3 | Contrasts K1–K4 (±0.10 margin): corpus, data overlap, shared init, family | K1, K3, K4: no difference; K2 primary inconclusive (point −0.002) | No training difference matters |
| Rung 3 | OLMo sites natural (median lift ≥ 0.30) | 0.66 selected, 0.68 at layer 5 | Natural |
| Rung 3b | Cross-family transfer on tasks gated in both families (Bonferroni); R1 if ≥ 2/3 of the excess-dominated ones pass | 16 of 16; 14 of 14 excess-dominated | R1: no screening |
| Rung 3b, amended | Same, restricted to tasks off-route-like in both families (≥ 3 needed) | 1 task (filler_gap_obj), which transfers | No reading |
| Option I | (a) The excess split replicates at 410M (≥ 2/3 of 160M excess-dominated tasks) | 15 of 16 | Replicates |
| Option I | (b) Tasks natural at 410M transfer within 410M | 2 of 2 (number agreement) | Pass |
| Run A′ | ≥ 3 competent, transferring tasks off-route by Tanyel and Ayvaz's Algorithm 1 (either site; run A's G* corrected) | 1 (filler_gap_hierarchy, off-route in 7 of 7 seeds, transfers) | 1 of 3 needed: excess, often extreme |
| Option L | Transfer at competent off-route cells found across the whole grid: L-pass if ≥ 2/3 of ≥ 3 testable cells pass (Bonferroni) | 5 of 5 cells, from 2 tasks; references unusable (net gate) | L-pass: no screening, 2 tasks |
| Option L2 | Same test at the relaxed-gate off-route cells: L2-pass if ≥ 2/3 of new cells pass and ≥ 2 of 3 new tasks qualify | 9 of 9 new cells; 3 of 3 new tasks; all 14 cells and 5 of 5 tasks at α = 0.05/14 | L2-pass: 5 tasks |
| Option O | 1B conversion: all 3 checkpoints match the reference (≤ 1e-2 logits), HellaSwag within 0.02 of AI2, Pile loss below 3.46, exact tail replay; IOI competent | max Δlogit 1.0e-4; HellaSwag equal to AI2's to 4 decimals; Pile 2.91–2.94; IOI accuracy 1.00 | Pass |
| Option O, part B | Transfer at competent off-route cells at 1B: B-pass if ≥ 2/3 of ≥ 3 testable cells pass (Bonferroni), from ≥ 2 tasks | 3 of 3 cells (relaxed gate) from 3 tasks; every region off-route at 160M too | B-pass |
| Option O, part A | IOI gap CI excludes 0 within 1B and across sizes (1B ↔ 150M) | within 1B +0.66 (+0.55 to +0.78); across sizes +0.45 (+0.40 to +0.57); site natural (lift 0.998) | A-pass |
| Option Q, part A | IOI cross-family gap CI excludes 0 at 7–9B and the selected site is natural in ≥ 3 of 4 families | gap +0.66 (+0.61 to +0.70), T 0.96; natural in 4 of 4 | QA-pass |
| Option Q, part C | Analogy direction along OLMo-3 post-training: the destination decides the effect (QC-destination) or the source does (QC-source) | destination range 0.95 vs source range 0.97 | mixed |
The effect is real and smaller than preregistered: net lift misses +0.50 narrowly, with a CI that spans the threshold. Onset timing is not consistent across seeds. Neither many-valued variable reached the localization bar; the second-variable test came from CausalGym instead (section 8).
A PolyPythias seed changes two things at once: the starting weights and the order the data arrives in. EleutherAI also released 160M runs that change only one of them, so this step asks which one the result depends on.
| Group | Pairs | Gap (95% CI) | Learned IIA | Random p95 | Net lift | T |
|---|---|---|---|---|---|---|
| Initialization differs (same data order) | 6 | +0.24 (+0.12 to +0.25) | 0.86 | 0.63 | +0.42 | 0.99 |
| Data order differs (same initialization) | 12 | +0.37 (+0.31 to +0.39) | 0.81 | 0.51 | +0.60 | 0.98 |
| Both differ (decoupled runs) | 12 | +0.29 (+0.20 to +0.32) | 0.77 | 0.51 | +0.47 | 0.98 |
| Both differ (PolyPythias seeds 4–6) | 42 | +0.34 (+0.32 to +0.35) | 0.84 | 0.50 | +0.64 | 0.98 |
At first glance, the initialization-only group transfers worse: its gap is -0.14 below data-only (95% CI -0.26 to -0.09). That difference belongs to the destination models, not to the sources. The two corrected weight seeds are the least accurate models in the set. Ablating the variable alone already flips many of their prompts, which raises both their baseline and their random null.
Comparing sources within the same destination removes that confound (Figure 5).
| Destination | Ablated baseline | Shares initialization | Shares data order | Shares neither |
|---|---|---|---|---|
| Original pythia-160m | 0.36 | 0.83 | 0.78 | 0.86 |
| data-seed1 | 0.10 | 0.67 | – | 0.65 |
| data-seed2 | 0.15 | 0.83 | – | 0.78 |
| data-seed3 | 0.14 | 0.87 | – | 0.80 |
| weight-seed1 (corrected) | 0.41 | – | 0.87 | 0.86 |
| weight-seed2 (corrected) | 0.54 | – | 0.94 | 0.96 |
What it means. Initialization and data order are not where the shared variable comes from. Either difference alone is enough to scatter heads and layers: models with identical starting weights selected layers 4, 4, 5 and 5. Every model here read the same Pile, so what they share is the data distribution, the architecture and the objective. The next ladder steps are built to vary exactly those.
The same test on nine Pythia-410M seeds (24 layers, 1,024 wide), plus transfer in both directions between 160M and 410M. The 16×16 rotation lives between the two 16-dimensional subspaces, so the different widths need no special handling. Everything below was preregistered before any 410M model was downloaded, except Figure 7.
| Group | Pairs | Gap (95% CI) | Learned IIA | Random p95 | Net lift | T | Beat own null |
|---|---|---|---|---|---|---|---|
| Within 160M | 42 | +0.28 (+0.26 to +0.33) | 0.84 | 0.56 | +0.62 | 0.96 | 42/42 |
| Within 410M | 42 | +0.20 (+0.07 to +0.34) | 0.53 | 0.17 | +0.39 | 0.90 | 37/42 |
| 160M ↔ 410M | 98 | +0.11 (+0.08 to +0.13) | 0.56 | 0.32 | +0.29 | 0.93 | 91/98 |
| 160M → 410M | 49 | +0.04 (+0.03 to +0.08) | 0.16 | 0.08 | +0.13 | 0.31 | 43/49 |
| 410M → 160M | 49 | +0.16 (+0.12 to +0.25) | 0.97 | 0.68 | +0.78 | 1.09 | 48/49 |
The two directions look very different. 160M patches barely move 410M (T = 0.31), while 410M patches over-shoot 160M's own fit (T = 1.09). A rotation keeps length, and the DAS coordinates in 410M are larger: their spread is 6.0–6.8 in five 410M seeds against 2.6–4.7 in 160M, a median ratio of 1.48. Adding one scale factor, computed from unlabeled activations just like the rotation and applied to the random null as well, makes the two directions symmetric (Figure 7).
| Direction | Map | Gap (95% CI) | Learned IIA | Random p95 | T |
|---|---|---|---|---|---|
| 160M → 410M | rotation only (preregistered) | +0.04 (+0.03 to +0.07) | 0.16 | 0.08 | 0.31 |
| 160M → 410M | rotation × one scale (post hoc) | +0.23 (+0.13 to +0.30) | 0.49 | 0.20 | 0.81 |
| 410M → 160M | rotation only (preregistered) | +0.16 (+0.12 to +0.25) | 0.97 | 0.68 | 1.09 |
| 410M → 160M | rotation × one scale (post hoc) | +0.35 (+0.27 to +0.38) | 0.89 | 0.53 | 1.02 |
What it does not fix. Several 410M seeds that solve the task have a low own-model ceiling (seed 3: 0.17). Training the DAS map four times longer does not help: the ceiling falls by 0.11–0.23 in four 410M seeds while 160M barely moves, which looks like overfitting 72 training prompts in 1,024 dimensions. More training prompts, validation early stopping, rank, and a variable spread beyond S2 are untested. Section 7 tests the first two.
Step 2 left two problems: a rotation cannot absorb the scale difference between 160M and 410M, and the own-model DAS fit overfits its 72 training prompts at 410M. Before running anything, I wrote down a revised estimator in the proposal:
| Model | IOI accuracy | Layer v1 → v2 | Ceiling v1 | Ceiling v2 | Change | v2 stop step |
|---|---|---|---|---|---|---|
| 160M seed 1 | 0.97 | 5 → 5 | 0.86 | 0.95 | +0.08 | 400 |
| 160M seed 2 | 0.96 | 5 → 6 | 0.81 | 0.93 | +0.12 | 100 |
| 160M seed 3 | 0.71 | 5 → 4 | 0.86 | 0.85 | 0.00 | 25 |
| 160M seed 4 | 0.99 | 6 → 6 | 0.69 | 0.82 | +0.13 | 150 |
| 160M seed 5 | 0.94 | 5 → 5 | 0.99 | 1.00 | +0.01 | 200 |
| 160M seed 6 | 0.88 | 6 → 5 | 0.92 | 0.82 | -0.10 | 25 |
| 160M seed 7 | 0.97 | 6 → 6 | 0.94 | 0.97 | +0.03 | 275 |
| 160M seed 8 | 0.93 | 7 → 5 | 0.88 | 0.82 | -0.06 | 25 |
| 160M seed 9 | 0.86 | 4 → 4 | 0.44 | 0.64 | +0.21 | 25 |
| 410M seed 1 | 0.97 | 9 → 9 | 0.90 | 0.98 | +0.08 | 100 |
| 410M seed 2 | 1.00 | 14 → 14 | 0.35 | 0.65 | +0.30 | 400 |
| 410M seed 3 | 0.95 | 16 → 14 | 0.17 | 0.55 | +0.38 | 25 |
| 410M seed 4 (chance) | 0.51 | 10 → 9 | 0.65 | 0.74 | +0.10 | 300 |
| 410M seed 5 | 0.98 | 11 → 11 | 0.51 | 0.77 | +0.27 | 50 |
| 410M seed 6 | 1.00 | 9 → 9 | 0.63 | 0.85 | +0.22 | 75 |
| 410M seed 7 | 0.97 | 10 → 10 | 0.45 | 0.77 | +0.32 | 25 |
| 410M seed 8 | 1.00 | 10 → 10 | 0.70 | 0.87 | +0.17 | 75 |
| 410M seed 9 | 1.00 | 9 → 9 | 0.92 | 0.98 | +0.06 | 225 |
| Group | Estimator | Gap (95% CI) | Learned IIA | Random p95 | T | Beat own null |
|---|---|---|---|---|---|---|
| Within 160M | v1 | +0.28 (+0.26 to +0.33) | 0.84 | 0.56 | 0.96 | 42/42 |
| Within 160M | v2 | +0.31 (+0.24 to +0.34) | 0.87 | 0.57 | 0.99 | 41/42 |
| Within 410M | v1 | +0.20 (+0.07 to +0.34) | 0.53 | 0.17 | 0.90 | 37/42 |
| Within 410M | v2 | +0.39 (+0.28 to +0.53) | 0.77 | 0.28 | 0.96 | 42/42 |
| 160M ↔ 410M | v1 | +0.11 (+0.08 to +0.13) | 0.56 | 0.32 | 0.93 | 91/98 |
| 160M ↔ 410M | v2 | +0.33 (+0.28 to +0.38) | 0.80 | 0.39 | 0.97 | 98/98 |
Where the cross-size gain comes from. Two changes at once, so each is also run alone. Median gap across sizes, with learned accuracy, T and pairs beating their null:
| Fit | Rotation only | Rotation × one scale |
|---|---|---|
| v1 (72 prompts, 100 steps) | +0.11 (learned 0.56, T 0.93, 91/98) | +0.30 (learned 0.64, T 0.98, 91/98) |
| v2 (792 prompts, early stopping) | +0.19 (learned 0.72, T 0.97, 98/98) | +0.33 (learned 0.80, T 0.97, 98/98) |
The scale does most of the work. Into 410M the two changes add up: the gap goes +0.04 → +0.40 and T reaches 0.87. Into 160M, T is 1.01.
Still holds under v2. The frozen 160M site (seeds 3–9, layer 5) gives a gap of +0.26, as locked, with 41 of 42 pairs above their null (37 under v1). Ladder step 1 passes in every group, 72 of 72 pairs. Matched by destination, sources that share initialization or data order still transfer no better than sources that share neither.
Still open. T is lower into 410M (0.87) than out of it (1.01). The early-stopping schedule is coarse: 7 of 24 models stop at the first check and 3 reach the 400-step cap. It stays fixed for later rungs.
IOI is one variable. CausalGym
| Site | Tasks passing the gate | Of those, transfer | Pairs beating their null | T, learned map | Rank-1 map ≥ 0.95 × learned | T, global rotation (260k Pile tokens) |
|---|---|---|---|---|---|---|
| Selected (primary) | 27 of 29 | 27 (all Bonferroni) | 1,134 / 1,134 | 0.99 | 18 of 27 | 0.29 |
| End-only (secondary) | 18 of 29 | 18 (all Bonferroni) | 752 / 756 | 0.95 | 17 of 18 | 0.54 |
Every task that localizes also transfers, in all six families. Two facts change how to read this:
| Task | Family | Gate | Own-model DAS | Learned map | Pile rotation | Ablation only | T learned | T Pile |
|---|---|---|---|---|---|---|---|---|
| agr_gender | Agreement | pass | 0.97 | 0.93 | 0.32 | 0.15 | 0.96 | 0.33 |
| agr_refl_num_obj-relc | Agreement | pass | 0.83 | 0.82 | 0.47 | 0.33 | 0.99 | 0.57 |
| agr_refl_num_pp | Agreement | fail | 0.74 | 0.75 | 0.41 | 0.29 | 1.01 | 0.55 |
| agr_refl_num_subj-relc | Agreement | fail | 0.75 | 0.71 | 0.41 | 0.24 | 0.95 | 0.55 |
| agr_sv_num_obj-relc | Agreement | pass | 0.83 | 0.82 | 0.67 | 0.32 | 0.99 | 0.81 |
| agr_sv_num_pp | Agreement | pass | 0.93 | 0.78 | 0.66 | 0.20 | 0.84 | 0.70 |
| agr_sv_num_subj-relc | Agreement | fail | 0.79 | 0.78 | 0.53 | 0.21 | 0.99 | 0.67 |
| npi_any_obj-relc | NPI | pass | 0.92 | 0.81 | 0.31 | 0.33 | 0.89 | 0.34 |
| npi_any_subj-relc | NPI | pass | 0.91 | 0.83 | 0.47 | 0.51 | 0.92 | 0.52 |
| npi_ever_obj-relc | NPI | fail | 0.75 | 0.74 | 0.16 | 0.17 | 0.98 | 0.21 |
| npi_ever_subj-relc | NPI | fail | 0.72 | 0.70 | 0.24 | 0.26 | 0.98 | 0.34 |
| garden_mvrr | Garden path | pass | 0.88 | 0.77 | 0.39 | 0.53 | 0.87 | 0.45 |
| garden_mvrr_mod | Garden path | fail | 0.93 | 0.88 | 0.62 | 0.65 | 0.95 | 0.67 |
| garden_npz_obj | Garden path | pass | 0.93 | 0.91 | 0.45 | 0.38 | 0.98 | 0.48 |
| garden_npz_obj_mod | Garden path | pass | 0.85 | 0.75 | 0.47 | 0.46 | 0.89 | 0.55 |
| garden_npz_v-trans | Garden path | fail | 0.86 | 0.86 | 0.58 | 0.60 | 1.00 | 0.68 |
| garden_npz_v-trans_mod | Garden path | fail | 0.76 | 0.69 | 0.63 | 0.58 | 0.91 | 0.84 |
| filler_gap_embed_3 | Filler–gap | fail | 0.72 | 0.66 | 0.57 | 0.55 | 0.92 | 0.79 |
| filler_gap_embed_4 | Filler–gap | fail | 0.71 | 0.64 | 0.59 | 0.59 | 0.90 | 0.83 |
| filler_gap_hierarchy | Filler–gap | fail | 0.81 | 0.73 | 0.31 | 0.45 | 0.90 | 0.38 |
| filler_gap_obj | Filler–gap | pass | 0.88 | 0.82 | 0.36 | 0.55 | 0.93 | 0.41 |
| filler_gap_pp | Filler–gap | pass | 0.93 | 0.88 | 0.43 | 0.64 | 0.95 | 0.46 |
| filler_gap_subj | Filler–gap | pass | 0.87 | 0.82 | 0.38 | 0.51 | 0.94 | 0.43 |
| cleft | Cleft | pass | 1.00 | 1.00 | 0.50 | 0.51 | 1.00 | 0.50 |
| cleft_mod | Cleft | pass | 0.83 | 0.83 | 0.52 | 0.52 | 1.00 | 0.63 |
| gss_subord | Subordination | pass | 0.89 | 0.83 | 0.50 | 0.56 | 0.94 | 0.56 |
| gss_subord_obj-relc | Subordination | pass | 0.94 | 0.88 | 0.62 | 0.39 | 0.93 | 0.66 |
| gss_subord_pp | Subordination | pass | 0.97 | 0.93 | 0.75 | 0.43 | 0.96 | 0.77 |
| gss_subord_subj-relc | Subordination | pass | 0.90 | 0.81 | 0.59 | 0.45 | 0.91 | 0.66 |
A follow-up the batch raised. The whole-space rotation behind H3 (scorecard) was fitted on about 800 task activations. That is too few to pin down a 768×768 rotation, so its failure might say nothing about the seeds. I refitted the rotation on up to 260k Pile tokens, with the reading written down before running.
| Site | Task rows (800) | Pile 1,024 | Pile 8,192 | Pile 65,536 | Pile 260,096 | Learned map |
|---|---|---|---|---|---|---|
| end_only | 0.46 | 0.38 | 0.52 | 0.54 | 0.54 | 0.95 |
| selected | 0.25 | 0.22 | 0.28 | 0.29 | 0.29 | 0.99 |
Reading. T = 0.54 falls between the preregistered thresholds (0.50 and 0.80), so it supports no summary claim. By family: agreement 0.71, subordination 0.66, cleft 0.56, garden path 0.49, npi 0.46, filler–gap 0.43. Against simply ablating the variable, the global rotation adds -0.01 (median).
This looked like the surprise. Section 9 explains it. At these sites the seeds look nearly identical as wholes (debiased CKA 0.93–0.97). The variables transfer perfectly once matched locally, yet no global rotation carries them. Three more preregistered follow-ups tracked down why. For most of these tasks, the answer is a known weakness of the method, not a property of the seeds.
Follow-up 2: linear maps and in-distribution text. A general linear map (least squares) does no better than a rotation. The best global linear map gives T = 0.55 when fitted on Pile and 0.45 when fitted on the task sentences themselves, and no task reaches 0.80. By the preregistered reading, this is the candidate cell.
The geometry did not fit that story. The task-text map sends seed A's class-mean difference onto seed B's almost exactly: cosine 0.85 at end-only and 0.985 at the selected site, 1.00 on all four NPI tasks. Yet on those NPI tasks the transfer through it is near zero. The map gets what the variable looks like right, but not the subspace the interchange acts on.
Follow-up 3: move the value, not the subspace. Use the global map only to translate seed A's activation into seed B's coordinates, and write it into B's own DAS subspace. Now the variable transfers: T = 0.89 at end-only and 1.00 at the selected site. The value travels between seeds through one global map. What fails is carrying A's subspace across.
The natural-site check (first called an illusion check). A smoke test showed something odd. On NPI, transplanting the seed's own whole activation from the source sentence did nothing, while DAS's 16-dimensional patch of that same activation flipped almost every prompt. A patch that beats the whole activation it was cut from is the signature of the subspace-patching illusion
| Site or task | Family | DAS patch lift | Real activation lift | Reading |
|---|---|---|---|---|
| IOI, 160M, S2 layer 5 (frozen) | IOI | +0.63 | +0.73 | natural |
| IOI, 410M, S2 layer 10 | IOI | +0.59 | +0.92 | natural |
| agr_gender | Agreement | +0.59 | +0.42 | natural |
| agr_refl_num_obj-relc | Agreement | +0.41 | +0.10 | excess-dominated |
| agr_sv_num_obj-relc | Agreement | +0.41 | +0.27 | excess-dominated |
| agr_sv_num_pp | Agreement | +0.49 | +0.38 | natural |
| npi_any_obj-relc | NPI | +0.50 | -0.02 | excess-dominated |
| npi_any_subj-relc | NPI | +0.34 | -0.07 | excess-dominated |
| garden_mvrr | Garden path | +0.34 | -0.24 | excess-dominated |
| garden_npz_obj | Garden path | +0.56 | +0.10 | excess-dominated |
| garden_npz_obj_mod | Garden path | +0.32 | -0.07 | excess-dominated |
| filler_gap_obj | Filler–gap | +0.33 | -0.35 | excess-dominated |
| filler_gap_pp | Filler–gap | +0.30 | -0.22 | excess-dominated |
| filler_gap_subj | Filler–gap | +0.33 | -0.16 | excess-dominated |
| cleft | Cleft | +0.49 | -0.01 | excess-dominated |
| cleft_mod | Cleft | +0.32 | -0.01 | excess-dominated |
| gss_subord | Subordination | +0.33 | -0.17 | excess-dominated |
| gss_subord_obj-relc | Subordination | +0.47 | +0.21 | excess-dominated |
| gss_subord_pp | Subordination | +0.50 | +0.19 | excess-dominated |
| gss_subord_subj-relc | Subordination | +0.42 | +0.08 | excess-dominated |
Correction, 19 Nov. The first version of this section called the non-natural sites "illusions" and said the model "doesn't use" the variable there. A reconciliation with Tanyel and Ayvazcleft_mod, filler_gap_embed_3, filler_gap_embed_4; clean accuracy 0.50–0.52), yet they pass the gate and count in "27 of 27". The labels below are now natural and excess-dominated; section 14 applies their strict off-route test.
Follow-up 4: the gap at a natural site. IOI is natural and not binary, so it decides whether the gap is real. The same battery ran at the frozen IOI site, with global maps fitted on Pile or on IOI sentences (training and validation names only).
| Way of moving the variable from seed A to seed B | CausalGym, end-only (18 tasks) | CausalGym, natural sites only (2 tasks) | IOI, frozen site (natural) |
|---|---|---|---|
| Learned 16×16 map (local, paired activations) | 0.95 | 0.99 | 0.99 |
| A's subspace carried by one global linear map (best transport, best fit) | 0.55 | 0.58 | 0.46 |
| Value carried by the global map, written into B's own subspace (hybrid) | 0.89 | 0.88 | 0.96 |
| Whole activation carried by the global map (vs B's own transplant) | 0.92 | 0.84 | 0.86 |
The gap survives at a natural site. At the frozen IOI site, one global linear map fitted on unlabeled IOI sentences (with names never used in testing) carries the variable's value from seed A into seed B's own subspace: T = 0.96 (95% CI 0.93 to 0.97). Carried by the same map, A's subspace reaches at best T = 0.46, barely above ablating the variable. The illusion explanation doesn't apply here: the IOI site is natural.
Follow-up 5: is the carried subspace causal in B at all? Write B's own source value into A's subspace as the global map places it in B. That gives T = 0.29 at best, against 0.09 for a random 16-dimensional subspace. The carried subspace shares about a tenth of B's own (overlap 0.16, against 0.02 at random). B's subspace is not a dormant direction: it holds 18× the variance of an average subspace.
That looked like a finding: one global map carries what the IOI variable's value is, but not where each seed's computation reads it. It survived the controls for rotation vs linear map, fitting text, and the illusion. One alternative remained: the map is clearly imprecise at S2.
Follow-up 6: a more accurate map. If map error caused the gap, a more accurate map should carry the subspace. I tried maps fitted on S2 activations only (24 names, then 204, none of them test names), rotations on the same data, and the crosscoder used as a map. As an upper bound, I also fitted an oracle on the test activations themselves.
| Map between seeds (42 pairs, medians) | Accuracy: R² for A's DAS coordinates | A's subspace carried into B (T) | B's own value on the carried subspace (T) | Value into B's own subspace (T) |
|---|---|---|---|---|
| IOI text (follow-up 4's map) | 0.30 | 0.11 | 0.11 | 0.95 |
| Crosscoder, linear approximation | 0.44 | 0.85 | 0.44 | 0.63 |
| S2 only, 24 names | 0.69 | 0.47 | 0.43 | 0.97 |
| S2 only, 204 names | 0.77 | 0.60 | 0.53 | 0.94 |
| Oracle: fitted on the test activations (upper bound, not in the decision) | 0.97 | 0.96 | 0.95 | 0.97 |
The candidate is withdrawn. By the preregistered rule, one held-out map carries A's subspace well enough: T = 0.85, from the crosscoder's linear approximation. That condition is odd on its own, because B's own value on the same subspace reaches only 0.44. The oracle settles it. A linear map that predicts the S2 activations accurately carries the causal subspace: T = 0.96, and B's own value on it 0.95 (at least 0.80 in 39 of 42 pairs). Across maps, the more accurate the map, the more causal the subspace it carries (Spearman 0.75). So the seeds do share the causal subspace, up to a linear map. What follow-ups 4 and 5 measured is an accuracy requirement: carrying the value tolerates a poor map, carrying the subspace needs a nearly exact one.
DAS tests variables we choose. The crosscoder asks the reverse question across thousands of unlabeled features: one shared sparse dictionary over layer 5 of seeds 1–7, with seeds 8 and 9 held out. Every causal test encodes each seed from its own activations only.
| Seed | Seen in training? | Variance unexplained, shared dictionary | Variance unexplained, own SAE | Loss recovered, shared | Loss recovered, own SAE |
|---|---|---|---|---|---|
| 8 | no | 0.024 | 0.019 | 0.950 | 0.966 |
| 9 | no | 0.023 | 0.017 | 0.951 | 0.971 |
| 3 | yes | 0.023 | — | 0.960 | — |
| 7 | yes | 0.018 | — | 0.959 | — |
Reading, fixed in advance: the seeds share features, and only the frame differs. Follow-up 6 used the crosscoder as a map between seeds. It carries the IOI value (whole-activation transplant T = 0.95), but a round trip through it keeps only 0.32 of the variable, so it cannot carry an edit.
DataDecide
| DataDecide 150M checkpoint (step 37500) | Max |Δlogit| vs reference | Pile loss | HellaSwag, ours = AI2's | Conversion gate | IOI accuracy | IOI mean logit diff |
|---|---|---|---|---|---|---|
| Dolma 1.7, default | 2.9e-04 | 3.46 | 0.3484 = 0.3484 | pass | 0.924 | 3.37 |
| Dolma 1.7, aux-2 | 2.5e-04 | 3.45 | 0.3513 = 0.3513 | pass | 0.895 | 2.53 |
| Dolma 1.7, aux-3 | 3.1e-04 | 3.45 | 0.3458 = 0.3458 | pass | 0.880 | 1.81 |
| DCLM, default | 3.7e-04 | 3.66 | 0.3579 = 0.3579 | pass | 0.953 | 3.14 |
| DCLM, aux-2 | 2.3e-04 | 3.72 | 0.3578 = 0.3578 | pass | 0.942 | 2.49 |
| DCLM, aux-3 | 4.4e-04 | 3.61 | 0.3622 = 0.3622 | pass | 0.887 | 1.89 |
| FineWeb-Edu, default | 3.5e-04 | 4.10 | 0.3648 = 0.3648 | fail: Pile loss | — | — |
| FineWeb-Edu, aux-2 | 1.8e-04 | 4.28 | 0.3686 = 0.3686 | fail: Pile loss | — | — |
| FineWeb-Edu, aux-3 | 2.9e-04 | 4.13 | 0.3686 = 0.3686 | fail: Pile loss | — | — |
| C4, default | 2.0e-04 | 4.53 | 0.3646 = 0.3646 | fail: Pile loss | — | — |
| C4, aux-2 | 2.3e-04 | 4.63 | 0.3636 = 0.3636 | fail: Pile loss | — | — |
| C4, aux-3 | 8.0e-05 | 4.68 | 0.3595 = 0.3595 | fail: Pile loss | — | — |
| Pythia-160M seeds 3–9 (reference) | — | 3.09 (main model) | — | — | 0.833–0.972 | 1.99–3.55 |
Decision, fixed in advance: rung 3 at 150M on Dolma 1.7 and DCLM. Two corpora × three seeds give seeds within OLMo, a change of corpus, and Pythia ↔ OLMo. No 1B screen is needed. One question goes into the rung's preregistration: does a DataDecide seed label fix the initialization across corpora? If it does, "same seed, different corpus" is a same-init, different-data pair, like ladder step 1.
Rung 3 asks whether the IOI variable moves between seeds of a second family, across a change of training corpus, and between families. It uses DataDecide 150M from all four corpora (three seeds each) and Pythia-160M seeds 3–9, with the v2 estimator unchanged. Everything was preregistered before the run, with one amendment made from weights alone before any transfer number.
| Group | Gap (95% CI) | Median T | Learned IIA | Random p95 | Beat own null |
|---|---|---|---|---|---|
| Within Pythia | +0.305 (+0.241 to +0.339) | 0.99 | 0.87 | 0.57 | 41/42 |
| Within OLMo | +0.300 (+0.253 to +0.348) | 0.91 | 0.77 | 0.45 | 24/24 |
| Across corpora | +0.307 (+0.284 to +0.322) | 0.94 | 0.79 | 0.47 | 108/108 |
| Pythia → OLMo | +0.321 (+0.303 to +0.340) | 1.03 | 0.86 | 0.52 | 84/84 |
| OLMo → Pythia | +0.249 (+0.220 to +0.266) | 0.93 | 0.78 | 0.53 | 81/84 |
None of the training differences moves T. Each contrast has a ±0.10 "no difference" margin fixed in advance:
| Contrast (difference in median T) | Difference (95% CI) | Reading, fixed in advance |
|---|---|---|
| K1. Within OLMo − across corpora | -0.024 (-0.060 to +0.049) | no difference |
| K2. Cross-family pairs with Dolma 1.7 − with C4 (most vs least Pile-like) | -0.002 (-0.108 to +0.043) | inconclusive |
| K2, alongside. OLMo → Pythia, matched by Pythia destination | +0.133 (+0.070 to +0.161) | first side higher |
| K3. Shared − different initialization, across corpora, matched | +0.051 (-0.029 to +0.095) | no difference |
| K4. Across families − within Pythia | -0.025 (-0.057 to -0.009) | no difference |
| K4. Across families − within OLMo | +0.050 (-0.012 to +0.084) | no difference |
What rung 3 says. At 150M, the IOI variable moves between two architecture families, four corpora and different initializations as well as between seeds of one family. That makes it a universality result. It also means this test cannot tell training histories apart: every comparison gives T ≈ 0.9–1.0. It says nothing about components; Xu 2026 finds OLMo-1B and Pythia-1B use different primary IOI components. A surprise would need a setting where transfer varies.
Rung 3 settled IOI. Rung 3b and option I ask the same of the 18 CausalGym tasks that localize at the end-only site. Do they transfer between Pythia and OLMo (the 12 DataDecide models of rung 3), and between Pythia-160M and 410M? And does the split between natural and excess-dominated sites (section 9) hold? Both were preregistered before the run, with an amendment before any number was seen (the relabelling and the Tanyel–Ayvaz split below).
| Group (18 tasks) | Tasks passing | Median T | Lowest task T | Pairs beating their null | Mismatched-pairing IIA |
|---|---|---|---|---|---|
| Within Pythia-160M | 18 of 18 | 0.95 | 0.89 | 752 / 756 | 0.49 |
| Within OLMo | 18 of 18 | 0.97 | 0.91 | 2,363 / 2,376 | 0.50 |
| Pythia ↔ OLMo | 18 of 18 | 0.96 | 0.91 | 3,007 / 3,024 | 0.49 |
| Within Pythia-410M | 18 of 18 | 0.96 | 0.92 | 752 / 756 | 0.50 |
| 160M ↔ 410M | 18 of 18 | 0.95 | 0.89 | 1,754 / 1,764 | 0.49 |
cleft_mod (at chance in Pythia), it is 13 of 13.filler_gap_obj has G ≈ 0 in Pythia and OLMo, and it transfers. One task is too few for the amended reading. The other excess-dominated tasks are excess on the route: the real activation moves the answer, DAS's patch moves it further.| Task | Natural lift, Pythia-160M | OLMo | Pythia-410M | T across families | T across sizes |
|---|---|---|---|---|---|
| agr_sv_num_pp | +0.38 | +0.31 | +0.41 | 0.97 | 0.98 |
| agr_gender | +0.42 | +0.33 | +0.19 | 1.00 | 0.99 |
| agr_sv_num_obj-relc | +0.27 | +0.25 | +0.33 | 0.98 | 0.98 |
| gss_subord_pp | +0.19 | +0.08 | +0.17 | 0.96 | 0.97 |
| gss_subord_obj-relc | +0.21 | +0.07 | +0.13 | 0.95 | 0.93 |
| agr_refl_num_obj-relc | +0.10 | +0.21 | +0.09 | 0.98 | 0.98 |
| gss_subord_subj-relc | +0.08 | -0.02 | +0.20 | 0.91 | 0.93 |
| garden_npz_obj | +0.10 | +0.04 | +0.08 | 0.98 | 0.97 |
| cleft | -0.01 | -0.07 | -0.01 | 1.00 | 1.00 |
| cleft_mod | -0.01 | -0.03 | -0.09 | 0.98 | 0.98 |
| garden_npz_obj_mod | -0.07 | -0.04 | -0.02 | 0.94 | 0.91 |
| npi_any_obj-relc | -0.02 | -0.11 | -0.09 | 0.92 | 0.94 |
| npi_any_subj-relc | -0.07 | -0.15 | -0.11 | 0.92 | 0.94 |
| filler_gap_subj | -0.16 | -0.09 | -0.13 | 0.93 | 0.95 |
| gss_subord | -0.17 | -0.09 (gate fails) | -0.14 | 0.96 | 0.95 |
| filler_gap_pp | -0.22 | -0.12 | -0.20 | 0.93 | 0.94 |
| garden_mvrr | -0.24 | -0.19 (gate fails) | -0.19 | 0.92 | 0.89 |
| filler_gap_obj | -0.35 | -0.28 | -0.19 | 0.95 | 0.95 |
What rung 3b and option I say. On 18 syntactic variables, as on IOI, transfer gives T ≈ 0.9–1.0 for seeds, corpora, families and sizes. And it passes at sites where the model's own activation does not carry the value, in every setting tried. The robust negative claim, "cross-model transfer does not certify a variable the model uses", now holds across families, the setting of the 2026 cross-family papers.
The 19 Nov reconciliation used interchange accuracy, which cannot tell a small effect from none. Runs A and A′ apply Tanyel and Ayvaz's own test (their Algorithm 1)
Correction, 20 Nov. Run A used the cell plus a region's total effect instead of the cell's added effect. That inflates G* whenever the other region carries the variable by itself, and it made the selected sites look on the route "through a second position" (G*/T 0.78 under run A against 0.10 now). Run A′ repeats the test with the published definition; the numbers below are run A′'s.
Result: one competent task is off-route, below the three needed. filler_gap_hierarchy is off-route at its selected site in all 7 seeds, and it transfers. The lead claim is still stated as "transfer passes at excess-dominated sites", but excess here is often extreme.
| Site | Competent, gate-passing tasks | e/T (DAS) | G/T (cell) | G*/T (best context) | G* > 0 | Excess e − G* > 0 | Off-route |
|---|---|---|---|---|---|---|---|
| Selected (variable's token) | 24 | 1.20 | 0.09 | 0.10 | 23 of 24 | 24 of 24 | 1 |
| End-only | 17 | 0.94 | 0.43 | 0.46 | 17 of 17 | 17 of 17 | 0 |
| Task | G/T (cell) | G*/T (best context) | e/T (DAS) | Off-route seeds |
|---|---|---|---|---|
| agr_sv_num_pp | 0.73 | 0.76 | 1.19 | 0/7 |
| agr_sv_num_obj-relc | 0.72 | 0.73 | 1.20 | 0/7 |
| agr_sv_num_subj-relc | 0.63 | 0.63 | 1.15 | 0/7 |
| agr_gender | 0.60 | 0.60 | 0.78 | 0/7 |
| agr_refl_num_obj-relc | 0.54 | 0.55 | 1.11 | 0/7 |
| agr_refl_num_subj-relc | 0.51 | 0.53 | 1.07 | 0/7 |
| agr_refl_num_pp | 0.48 | 0.52 | 1.02 | 0/7 |
| garden_mvrr_mod | 0.37 | 0.37 | 2.16 | 0/7 |
| garden_npz_obj | 0.29 | 0.37 | 1.13 | 0/7 |
| garden_npz_obj_mod | 0.28 | 0.36 | 1.18 | 0/7 |
| garden_mvrr | 0.18 | 0.18 | 1.07 | 0/7 |
| gss_subord_pp | 0.09 | 0.11 | 1.71 | 0/7 |
| gss_subord | 0.10 | 0.10 | 2.07 | 0/7 |
| filler_gap_obj | 0.05 | 0.08 | 1.90 | 0/7 |
| gss_subord_obj-relc | 0.05 | 0.06 | 1.57 | 0/7 |
| filler_gap_pp | 0.04 | 0.06 | 1.77 | 0/7 |
| gss_subord_subj-relc | 0.04 | 0.05 | 1.61 | 1/7 |
| npi_any_subj-relc | 0.03 | 0.04 | 1.64 | 0/7 |
| npi_any_obj-relc | 0.02 | 0.04 | 1.33 | 0/7 |
| npi_ever_subj-relc | 0.03 | 0.03 | 1.49 | 2/7 |
| npi_ever_obj-relc | 0.02 | 0.03 | 0.96 | 1/7 |
| filler_gap_subj | 0.02 | 0.03 | 1.44 | 0/7 |
| cleft | 0.01 | 0.01 | 0.99 | 3/7 |
| filler_gap_hierarchy | -0.02 | -0.02 | 1.12 | 7/7 |
filler_gap_hierarchy's selected site, G*'s CI lies entirely below 0 in 4 of 7 seeds, so the model's own activation nudges the answer the wrong way, while DAS moves it by 3.8–8.0 logits. Cross-seed transfer passed there in the batch.filler_gap_embed_4, clean accuracy 0.50), and it transfers too.What runs A and A′ say. At the sites chosen for maximum DAS lift, transfer passes whether the model's own route carries a little, almost nothing, or nothing (one competent task). Whether transfer passes at off-route cells in general, found across the whole grid rather than where DAS works best, is option L.
filler_gap_hierarchy in Pythia-160M, scored by Tanyel and Ayvaz's Algorithm 1 over seeds 3–9. Blue: on the route in at least 4 of 7 seeds (the model's own activation there adds to the answer). Orange: off-route (DAS can move the answer from there; the model's own activation does not). The variable's token is read in layers 1–3; the regions after it carry it until layers 6–7; the last token carries it throughout. Right: medians over seeds at the two marked cells. At A the model's own activation carries the whole effect and DAS's patch overshoots it; at B the model's own activation does nothing, yet DAS's patch moves the answer by about the full effect. A map between seeds carries the patch equally well at both (T 1.00 and 0.99), so passing the transfer test does not show that the model uses the cell.Every transfer test before option L ran at the site where DAS works best. Option L asks the question the other way round. It scans every template region × layer 1–11 of Pythia-160M seeds 3–9 on the 26 tasks the models can do (1,518 live cells), finds the cells that are off-route by Tanyel and Ayvaz's Algorithm 1
Option L result: L-pass, on two tasks. 346 cells are off-route. 5 of them pass the gate, and transfer passes at 5 of 5 (Bonferroni α = 0.01). But four of the five are one token of filler_gap_hierarchy at layers 4–7 (layer 6 is run A′'s site), and the fifth is filler_gap_obj.
Option L2: the same test on the relaxed-gate cells (preregistered after L, before any of its transfer numbers). Option L's five cells came from two tasks, so L2 ran the identical transfer test at the 9 off-route cells that pass the relaxed gate (within ≥ 0.65, net ≥ 0.15) but not the primary one. Three of their tasks are new: filler_gap_pp, garden_npz_v-trans, garden_npz_v-trans_mod. The reading was fixed in advance: L2-pass if ≥ 2/3 of the new cells pass and ≥ 2 of the 3 new tasks qualify (≥ 2/3 of a task's cells passing at α = 0.05/14).
Result: L2-pass. 9 of 9 new cells pass (each in 42 of 42 pairs), all 3 new tasks qualify, and across options L and L2 all 14 cells pass at α = 0.05/14, on 5 of 5 tasks from two construction families (filler–gap and NPZ garden path). Transfer T is 0.92–1.00 everywhere.
| Cell | Option | Position | e/T (DAS) | G*/T (own) | Median gap | CI | Learned | Random p95 | Transfer T |
|---|---|---|---|---|---|---|---|---|---|
| filler_gap_hierarchy, layer 4 | L | variable token | 1.59 | -0.015 | +0.35 | 0.23 to 0.40 | 0.99 | 0.64 | 1.00 |
| filler_gap_hierarchy, layer 5 | L | variable token | 1.56 | -0.012 | +0.37 | 0.28 to 0.44 | 0.99 | 0.62 | 1.00 |
| filler_gap_hierarchy, layer 6 | L | variable token | 1.09 | -0.030 | +0.47 | 0.42 to 0.51 | 0.97 | 0.51 | 0.99 |
| filler_gap_hierarchy, layer 7 | L | variable token | 0.88 | -0.032 | +0.43 | 0.39 to 0.46 | 0.87 | 0.42 | 0.98 |
| filler_gap_obj, layer 9 | L | variable token | 1.19 | +0.006 | +0.25 | 0.23 to 0.27 | 0.81 | 0.56 | 0.98 |
| filler_gap_hierarchy, layer 8 | L2 | variable token | 0.69 | -0.017 | +0.30 | 0.26 to 0.35 | 0.70 | 0.33 | 0.96 |
| filler_gap_obj, layer 2 | L2 | last token | 0.73 | +0.010 | +0.11 | 0.09 to 0.13 | 0.60 | 0.49 | 0.92 |
| filler_gap_pp, layer 9 | L2 | variable token | 0.94 | +0.006 | +0.28 | 0.24 to 0.32 | 0.73 | 0.45 | 0.93 |
| garden_npz_v-trans, layer 9 | L2 | variable token | 0.94 | +0.035 | +0.12 | 0.11 to 0.13 | 0.70 | 0.58 | 0.97 |
| garden_npz_v-trans_mod, layer 4 | L2 | variable token | 2.20 | +0.148 | +0.13 | 0.11 to 0.15 | 0.88 | 0.75 | 0.98 |
| garden_npz_v-trans_mod, layer 5 | L2 | variable token | 2.23 | +0.164 | +0.12 | 0.11 to 0.14 | 0.87 | 0.74 | 0.99 |
| garden_npz_v-trans_mod, layer 6 | L2 | variable token | 2.38 | +0.136 | +0.12 | 0.10 to 0.14 | 0.86 | 0.73 | 1.00 |
| garden_npz_v-trans_mod, layer 7 | L2 | variable token | 2.49 | +0.168 | +0.11 | 0.10 to 0.13 | 0.84 | 0.73 | 0.98 |
| garden_npz_v-trans_mod, layer 8 | L2 | variable token | 2.01 | +0.137 | +0.11 | 0.09 to 0.12 | 0.80 | 0.69 | 0.99 |
garden_npz_v-trans, the model's own activation gives at most 0.035 T (less than 0 at filler_gap_hierarchy). garden_npz_v-trans_mod is borderline: its G* has a median of 0.14–0.17 T, with a CI that includes 0 in 4–6 of 7 seeds, while DAS gives 2.0–2.5 T. Without it, the claim rests on 4 tasks.Transfer at the last-token reference cells, which are on the route (descriptive). There, a random rotation does almost as well as the learned map (random 95th percentile 0.86–1.00 wherever DAS localizes well), so the gap test has no room. That is also why these cells fail the net gate: writing anything into the subspace, even its mean, flips most answers. Transfer T is the comparable number, and it is the same on and off the route: T(off-route) − T(reference) is -0.04 to +0.05 (median -0.01) over the 10 cells whose reference DAS localizes.
| Reference (last token, same layer) | On-route seeds | Within | Net | Learned | Random p95 | Transfer T |
|---|---|---|---|---|---|---|
filler_gap_hierarchy, layer 4 | 7 of 7 | 1.00 | 0.19 | 1.00 | 0.94 | 1.00 |
filler_gap_hierarchy, layer 5 | 7 of 7 | 1.00 | 0.11 | 1.00 | 0.98 | 1.00 |
filler_gap_hierarchy, layer 6 | 7 of 7 | 1.00 | 0.05 | 1.00 | 0.99 | 1.00 |
filler_gap_hierarchy, layer 7 | 7 of 7 | 1.00 | 0.04 | 1.00 | 0.99 | 1.00 |
filler_gap_hierarchy, layer 8 | 7 of 7 | 1.00 | 0.05 | 1.00 | 0.99 | 1.00 |
filler_gap_obj, layer 9 | 7 of 7 | 1.00 | 0.02 | 1.00 | 1.00 | 1.00 |
filler_gap_pp, layer 9 | 7 of 7 | 0.96 | 0.18 | 0.94 | 0.86 | 0.97 |
garden_npz_v-trans_mod, layer 4 | 1 of 7 | 0.46 | 0.04 | 0.44 | 0.46 | 0.98 |
garden_npz_v-trans_mod, layer 5 | 3 of 7 | 0.47 | 0.03 | 0.46 | 0.48 | 0.96 |
garden_npz_v-trans_mod, layer 6 | 6 of 7 | 0.51 | 0.07 | 0.46 | 0.49 | 0.95 |
garden_npz_v-trans_mod, layer 7 | 7 of 7 | 0.58 | 0.11 | 0.59 | 0.56 | 0.96 |
garden_npz_v-trans_mod, layer 8 | 7 of 7 | 0.69 | 0.14 | 0.64 | 0.60 | 0.93 |
garden_npz_v-trans, layer 9 | 7 of 7 | 0.99 | 0.09 | 0.97 | 0.88 | 0.99 |
Option P: why ablation flips most answers at the last token (exploratory). Each entry is IIA, with the median logit difference z (source label minus base label) in parentheses. At the last token, the answer is linear in the subspace coordinates: z after mean ablation equals the midpoint of z at the base's own value-class mean and at the other class's mean, to within 0.1 logit. The subspace moves z by ±2.9 to ±5.4 logits, while the rest of the residual adds a constant 1.6–3.4 logits toward whichever value the base does not have. Normally the subspace wins; ablating or randomizing it leaves the rest's push, so the answer flips. This is DAS excess seen from the ablation side: the subspace's swing is about twice the model's natural effect, and the model's own state cancels part of it elsewhere. At the off-route variable-token cell, ablation barely moves anything, because the model does not read it.
| Cell | Base | DAS | Ablate 16 | Ablate top-1 | Own-class mean | Other-class mean |
|---|---|---|---|---|---|---|
| filler_gap_hierarchy, layer 6, last token | 0.17 (-1.9) | 1.00 (+7.0) | 0.95 (+2.6) | 0.84 (+2.1) | 0.24 (-1.7) | 1.00 (+6.6) |
| filler_gap_obj, layer 9, last token | 0.20 (-2.1) | 1.00 (+9.3) | 0.98 (+3.5) | 0.91 (+3.1) | 0.14 (-2.0) | 1.00 (+8.8) |
| filler_gap_pp, layer 9, last token | 0.20 (-2.6) | 0.96 (+6.7) | 0.77 (+2.2) | 0.69 (+1.7) | 0.24 (-2.3) | 0.98 (+6.4) |
| garden_npz_v-trans, layer 9, last token | 0.28 (-1.9) | 0.99 (+4.8) | 0.90 (+1.7) | 0.75 (+0.9) | 0.18 (-1.2) | 1.00 (+4.5) |
| filler_gap_hierarchy, layer 6, variable token (off-route) | 0.17 (-1.9) | 0.99 (+3.0) | 0.28 (-1.2) | 0.27 (-1.2) | 0.18 (-1.9) | 0.96 (+2.4) |
| garden_npz_v-trans_mod, layer 6, variable token (off-route) | 0.39 (-0.6) | 0.87 (+2.8) | 0.70 (+1.1) | 0.49 (-0.1) | 0.43 (-0.4) | 0.87 (+2.8) |
filler_gap_hierarchy, layer 6. At the last token (A), the subspace swings z between its two class means, and the rest of the residual sits on the far side of 0. The model's own state normally wins (base, z < 0), but erasing the subspace leaves only the rest, so the answer flips: ablation IIA is high and the net gate (within − ablation) is low, though the model reads this cell. At the off-route variable token (B), ablation barely moves z, so net is high though the model ignores the cell. The gate rewards the wrong cells.What P means for the method. The batch's net gate (within IIA minus ablation IIA) is high where ablation is harmless, which is where the model does not read the cell, and low at on-route cells with excess. So it favours off-route cells and does not measure "the subspace matters". The same push puts the random-rotation null near ceiling at on-route last-token cells, where the gap test is uninformative. Options L and L2 stand, because their tests compare the learned map with nulls in the same destination, but the paper must describe the gate this way.
Descriptive, from option L's scan.
| Position | Live cells | Off-route | Rate |
|---|---|---|---|
| Variable's token | 198 | 33 | 17% |
| After the variable | 1,034 | 266 | 26% |
| Last token | 286 | 47 | 16% |
What options L and L2 say. Cross-seed transfer passes where the model's own activation carries nothing: 14 of 14 off-route cells on 5 competent tasks (4 with G* near 0), at T 0.92–1.00, the same as at on-route cells. Transfer shows that the seeds share a linear representation DAS can write into, not that the model reads it at that position. Option P explains the last-token ablation flips as DAS excess and shows that the batch's net gate favours off-route cells.
Every model in sections 1–15 has at most 410M parameters. Option O asks whether the paper's two claims hold at 1B: (A) the IOI variable transfers between independently trained models at its site, and (B) transfer passes at competent off-route cells. It uses AI2's DataDecide Dolma 1.7 1B models
The conversion is exact. All three checkpoints pass every bar. The converted models match an independent reference forward pass to 1.0e-4 logits, reproduce AI2's HellaSwag score to the fourth decimal on all 10,042 items (0.5961–0.6006), and reach a Pile loss of 2.91–2.94. They solve IOI perfectly (accuracy 1.00) and do all five CausalGym tasks (median clean accuracy ≥ 0.72).
Part B result: B-pass, 3 of 3 cells from 3 tasks. The scan finds 33 off-route cells among 330 live ones. Only one passes the primary gate, so the relaxed gate applied, as preregistered, giving three testable cells. Transfer passes at all three, in 6 of 6 seed pairs each, at T 0.96–0.98, while the model's own activation at those cells gives at most 0.013 T. All three sit in regions that were off-route at 160M: the filler–gap variable's token, and the garden_npz_v-trans_mod region that gave five of L2's cells.
Part A result: A-pass, at a natural site. Validation picked layer 7 in every 1B model, the preregistered fixed layer and the same relative depth as layer 5 at 160M. Zero-ablating the subspace there drops interchange accuracy from 0.96–0.99 to 0.002 or less (natural lift 0.998), so option P's concern, that selecting by net favours sites the model does not read, does not apply here. Within 1B the learned map gives 0.97 against a random 95th percentile of 0.31 (T 0.99), and the variable crosses sizes in both directions: all 18 pairs between 1B and 150M beat their null (median T 0.97). The within-150M group reproduces rung 3's Dolma numbers exactly, as a check of the code path.
| Row | Pairs | Median gap | CI | Learned | Random p95 | Transfer T |
|---|---|---|---|---|---|---|
| filler_gap_hierarchy, layer 11 (off-route) | 6 | +0.50 | 0.47 to 0.53 (α = 0.05/3) | 0.84 | 0.36 | 0.98 |
| filler_gap_obj, layer 13 (off-route) | 6 | +0.29 | 0.26 to 0.36 (α = 0.05/3) | 0.72 | 0.45 | 0.96 |
| garden_npz_v-trans_mod, layer 12 (off-route) | 6 | +0.12 | 0.11 to 0.16 (α = 0.05/3) | 0.68 | 0.56 | 0.98 |
| IOI, within 1B | 6 | +0.66 | 0.55 to 0.78 (95%) | 0.97 | 0.31 | 0.99 |
| IOI, 1B → 150M | 9 | +0.42 | 0.27 to 0.58 (95%) | 0.91 | 0.38 | 1.08 |
| IOI, 150M → 1B | 9 | +0.49 | 0.44 to 0.58 (95%) | 0.75 | 0.26 | 0.79 |
| IOI, within 150M | 6 | +0.31 | 0.20 to 0.42 (95%) | 0.69 | 0.29 | 0.89 |
Transfer at the last-token reference cells (descriptive). As option P predicts, two of the three are at ceiling: the learned map and a random rotation both reach 1.00, so the gap test has no room there. Transfer T is the same on and off the route.
| Reference (last token, same layer) | Within | Net | Learned | Random p95 | Transfer T |
|---|---|---|---|---|---|
filler_gap_hierarchy, layer 11 | 1.00 | 0.11 | 1.00 | 1.00 | 1.00 |
filler_gap_obj, layer 13 | 1.00 | 0.01 | 1.00 | 1.00 | 1.00 |
garden_npz_v-trans_mod, layer 12 | 0.86 | 0.15 | 0.84 | 0.75 | 0.95 |
Descriptive, from the 1B scan.
What option O says. Both claims hold at 1B, a sixfold scale-up, in the OLMo architecture: the IOI variable transfers between seeds and across sizes at a site the model reads, and cross-seed transfer passes at competent off-route cells, in the same regions as at 160M. Not shown: anything about the weak early-layer off-route cells, or about 1B outside one corpus and one architecture.
No public suite has independently seeded pretraining above 1B, so option Q tests the paper's claims at 7–9B in the settings that exist. Both parts were preregistered on 21 Nov before any 7–9B number. Part A: IOI transfer across four families (OLMo-3-7B, Llama-3.1-8B, Qwen2.5-7B, Gemma-2-9B), and between three OLMo-3-7B runs that share one pretraining checkpoint and differ only in their midtraining data. Part C: the analogy direction from the COLM'26 paper, taken along OLMo-3's open post-training chain (base → SFT → DPO → Instruct), asks whether a direction's effect follows where it came from or where it is put.
Part A result (preregistered reading): QA-pass. Cross-family transfer at a natural 7–9B site. Across families the median gap is +0.66 (95% CI +0.61 to +0.70), T 0.96, with 12 of 12 pairs above their null. The selected site is natural (lift ≥ 0.30) in 4 of 4 family models. All 7 models run so far pass the replay and competence preconditions.
| Model | Selected layer | Fixed layer | Validation net | Own DAS IIA (test) | Natural lift | IOI acc / logit diff | Replay error |
|---|---|---|---|---|---|---|---|
| OLMo-3-7B | 12 of 32 (38%) | 13 | 0.98 | 0.95 | 1.00 | 1.00 / 6.7 | 0.000 |
| Llama-3.1-8B | 12 of 32 (38%) | 13 | 0.97 | 0.94 | 1.00 | 1.00 / 5.7 | 0.000 |
| Qwen2.5-7B | 18 of 28 (64%) | 12 | 0.97 | 0.99 | 0.86 | 1.00 / 6.6 | 0.000 |
| Gemma-2-9B | 21 of 42 (50%) | 18 | 0.86 | 0.72 | 1.00 | 1.00 / 26.6 | 0.000 |
| OLMo-3-7B, midtraining mix A | 12 of 32 (38%) | 13 | 0.84 | 0.71 | 1.00 | 1.00 / 6.6 | 0.000 |
| OLMo-3-7B, midtraining mix B | 12 of 32 (38%) | 13 | 0.74 | 0.73 | 1.00 | 1.00 / 6.5 | 0.000 |
| OLMo-3-7B, midtraining mix C | 12 of 32 (38%) | 13 | 0.95 | 0.90 | 1.00 | 1.00 / 6.7 | 0.000 |
| Group | Site | Pairs | Gap (95% CI) | T | Learned | Random p95 | Mismatched | Pairs above null |
|---|---|---|---|---|---|---|---|---|
| Across families | selected | 12 | +0.66 (+0.61 to +0.70) | 0.96 | 0.92 | 0.20 | 0.15 | 12 of 12 |
| OLMo-3 midtraining runs | selected | 6 | +0.69 (+0.57 to +0.73) | 1.02 | 0.81 | 0.13 | 0.07 | 6 of 6 |
| Across families | fixed | 12 | +0.66 (+0.51 to +0.70) | 0.95 | 0.85 | 0.14 | – | 12 of 12 |
| OLMo-3 midtraining runs | fixed | 6 | +0.66 (+0.61 to +0.71) | 0.97 | 0.73 | 0.06 | – | 6 of 6 |
| Group | Site | Source | Destination | Gap | Learned | Random p95 | T |
|---|---|---|---|---|---|---|---|
| Across families | selected | Llama-3.1-8B | OLMo-3-7B | +0.69 | 0.89 | 0.20 | 0.93 |
| Across families | selected | Qwen2.5-7B | OLMo-3-7B | +0.64 | 0.87 | 0.23 | 0.92 |
| Across families | selected | Gemma-2-9B | OLMo-3-7B | +0.75 | 0.95 | 0.20 | 1.00 |
| Across families | selected | OLMo-3-7B | Llama-3.1-8B | +0.67 | 0.96 | 0.30 | 1.02 |
| Across families | selected | Qwen2.5-7B | Llama-3.1-8B | +0.71 | 0.91 | 0.20 | 0.96 |
| Across families | selected | Gemma-2-9B | Llama-3.1-8B | +0.78 | 0.96 | 0.18 | 1.02 |
| Across families | selected | OLMo-3-7B | Qwen2.5-7B | +0.65 | 0.99 | 0.34 | 1.00 |
| Across families | selected | Llama-3.1-8B | Qwen2.5-7B | +0.67 | 0.94 | 0.27 | 0.95 |
| Across families | selected | Gemma-2-9B | Qwen2.5-7B | +0.58 | 0.97 | 0.38 | 0.98 |
| Across families | selected | OLMo-3-7B | Gemma-2-9B | +0.61 | 0.65 | 0.04 | 0.89 |
| Across families | selected | Llama-3.1-8B | Gemma-2-9B | +0.62 | 0.63 | 0.01 | 0.87 |
| Across families | selected | Qwen2.5-7B | Gemma-2-9B | +0.51 | 0.54 | 0.03 | 0.74 |
| OLMo-3 midtraining runs | selected | OLMo-3-7B, midtraining mix B | OLMo-3-7B, midtraining mix A | +0.51 | 0.59 | 0.08 | 0.83 |
| OLMo-3 midtraining runs | selected | OLMo-3-7B, midtraining mix C | OLMo-3-7B, midtraining mix A | +0.63 | 0.75 | 0.11 | 1.05 |
| OLMo-3 midtraining runs | selected | OLMo-3-7B, midtraining mix A | OLMo-3-7B, midtraining mix B | +0.67 | 0.81 | 0.14 | 1.11 |
| OLMo-3 midtraining runs | selected | OLMo-3-7B, midtraining mix C | OLMo-3-7B, midtraining mix B | +0.72 | 0.87 | 0.15 | 1.20 |
| OLMo-3 midtraining runs | selected | OLMo-3-7B, midtraining mix A | OLMo-3-7B, midtraining mix C | +0.74 | 0.89 | 0.15 | 0.98 |
| OLMo-3 midtraining runs | selected | OLMo-3-7B, midtraining mix B | OLMo-3-7B, midtraining mix C | +0.72 | 0.82 | 0.10 | 0.90 |
| Across families | fixed | Llama-3.1-8B | OLMo-3-7B | +0.73 | 0.76 | 0.03 | 0.85 |
| Across families | fixed | Qwen2.5-7B | OLMo-3-7B | +0.64 | 0.68 | 0.03 | 0.76 |
| Across families | fixed | Gemma-2-9B | OLMo-3-7B | +0.73 | 0.84 | 0.11 | 0.94 |
| Across families | fixed | OLMo-3-7B | Llama-3.1-8B | +0.68 | 0.94 | 0.25 | 1.02 |
| Across families | fixed | Qwen2.5-7B | Llama-3.1-8B | +0.69 | 0.86 | 0.17 | 0.93 |
| Across families | fixed | Gemma-2-9B | Llama-3.1-8B | +0.68 | 0.89 | 0.21 | 0.97 |
| Across families | fixed | OLMo-3-7B | Qwen2.5-7B | +0.49 | 0.95 | 0.46 | 1.03 |
| Across families | fixed | Llama-3.1-8B | Qwen2.5-7B | +0.50 | 0.91 | 0.41 | 0.98 |
| Across families | fixed | Gemma-2-9B | Qwen2.5-7B | +0.49 | 0.88 | 0.38 | 0.94 |
| Across families | fixed | OLMo-3-7B | Gemma-2-9B | +0.71 | 0.74 | 0.03 | 1.07 |
| Across families | fixed | Llama-3.1-8B | Gemma-2-9B | +0.64 | 0.66 | 0.02 | 0.96 |
| Across families | fixed | Qwen2.5-7B | Gemma-2-9B | +0.52 | 0.55 | 0.03 | 0.80 |
| OLMo-3 midtraining runs | fixed | OLMo-3-7B, midtraining mix B | OLMo-3-7B, midtraining mix A | +0.60 | 0.64 | 0.04 | 0.96 |
| OLMo-3 midtraining runs | fixed | OLMo-3-7B, midtraining mix C | OLMo-3-7B, midtraining mix A | +0.63 | 0.73 | 0.09 | 1.08 |
| OLMo-3 midtraining runs | fixed | OLMo-3-7B, midtraining mix A | OLMo-3-7B, midtraining mix B | +0.62 | 0.64 | 0.02 | 0.98 |
| OLMo-3 midtraining runs | fixed | OLMo-3-7B, midtraining mix C | OLMo-3-7B, midtraining mix B | +0.69 | 0.73 | 0.04 | 1.13 |
| OLMo-3 midtraining runs | fixed | OLMo-3-7B, midtraining mix A | OLMo-3-7B, midtraining mix C | +0.74 | 0.81 | 0.08 | 0.93 |
| OLMo-3 midtraining runs | fixed | OLMo-3-7B, midtraining mix B | OLMo-3-7B, midtraining mix C | +0.68 | 0.77 | 0.09 | 0.89 |
Part C result (preregistered reading): mixed. Columns are the model being steered, rows the stage the vector came from (same norm in each model; only the direction differs). The source matters about as much as the destination: the range of the destination means is 0.95, of the source means 0.97, so neither preregistered pattern holds. The paper's base < instruct pattern replicates. Every base-model cell has mean fluency below 3 (1.52–2.82, 2.75 even unsteered), so base cells are noisy; no post-trained cell is below 3.
| Change in analogy quality (judge, 1–5) when steering… | …Base | …SFT | …DPO | …Instruct (RL) |
|---|---|---|---|---|
| Vector from Base | -0.02 | +0.00 | +0.22 | +0.27 |
| Vector from SFT | +0.29 | +0.98 | +1.41 | +1.14 |
| Vector from DPO | +0.26 | +1.17 | +1.46 | +1.39 |
| Vector from Instruct (RL) | +0.26 | +1.19 | +1.51 | +1.38 |
| Random direction (norm-matched) | +0.06 | +0.06 | +0.28 | +0.07 |
| Route (i): whole analogy state, last prompt token | +0.00 | +0.00 | +0.14 | +0.24 |
| Route (ii): only its coordinate along v | -0.02 | +0.01 | +0.08 | +0.07 |
| Unsteered analogy quality (1–5) | 1.08 | 1.22 | 1.35 | 1.35 |
doc/related_work.md). Model Alignment Search| Impact | Novelty | Confidence | GPU-days | Priority | Accept % | EV | Rank |
|---|---|---|---|---|---|---|---|
| 4 (reviewer estimate; yours to change) | 3 | 3 | 5 | 3.5 | 30% (NeurIPS 2027 / ICLR 2028) | 1.05 | 6 of 32 |
These are the Nov 10 review scores. They appear to have been scored from the September proposal: the review's go/no-go test ("within-seed DAS stable on IOI at 160M") has since passed, and it does not list McCann. The repo rules ask a separate agent, one that didn't run the experiments, to re-score. Today's evidence bears mainly on Confidence.
Before February, the roadmap allows almost none of your time: HRI is full time until Jan 20, #6 goes to ICML around Jan 28, and the chapter-4 choice (#42 or #43) is made with David and Sarah in early February. #42 is also the roadmap's chapter-3 fallback if #2 fails its Feb 15 check. Running the cheap #42 steps now therefore pays off whichever way that check goes.
| Option | What it answers | Compute | Your time | When |
|---|---|---|---|---|
| A. Ladder step 2: Pythia-410M, 9 seeds, then 160M ↔ 410M | Does the variable survive a change of size and width? | Done 17 Nov (2 GPU-hours) | — | Done: passes |
| A2. Estimator revision: rotation × one scale; 792 training prompts with validation early stopping | Is the low 410M ceiling the estimator or the model? | Done 18 Nov (3 GPU-hours; rungs 1 and 2 rerun) | — | Done: mostly the estimator |
| B. Second variable: CausalGym, 29 syntactic tasks (replaced subject-verb agreement, which it contains) | Is this more than one IOI variable? | Done 18 Nov (8.5 GPU-hours) | — | Done: 27 of 27 transfer |
| F. Crosscoder arm | Which unlabeled features do seeds share, and does a shared feature act alike in each seed? | Done 18 Nov (2 GPU-hours) | — | Done: shared features, matching effects |
| G. Global-map follow-ups 2–5 | Is the local-vs-global gap about rotations, the fitting text, illusions, or real? | Done 18 Nov (under 1 GPU-hour) | — | Done: gap found; H explained it |
| H. Better map at S2 (S2-fitted linear maps; the crosscoder as a map; an oracle) | Is the subspace gap real, or map error? | Done 18 Nov (3 GPU-minutes) | — | Done: map error; candidate withdrawn |
J. Literature check (doc/related_work.md) | Is the illusion result known? What else is new in 2026? | Done 18 Nov | — | Done: illusion partly known; transfer-at-illusion-sites new |
| C. DataDecide | Does the variable localize and transfer in a second family, across training corpora, and from Pythia into OLMo? | Done 19 Nov (screen 33 GPU-minutes; rung 1.4 GPU-hours) | — | Done: transfers across families, corpora and inits alike |
| C2. Rung 3b: CausalGym on DataDecide | Does transfer also pass at excess-dominated sites across families, the setting of the 2026 cross-family papers? | Done 20 Nov (7 GPU-hours) | — | Done: passes everywhere (R1) |
| I. 410M and agreement | Does the excess split hold at 410M, and is agreement a second natural variable there? | Done 20 Nov (4.5 GPU-hours) | — | Done: replicates; agreement natural, transfers |
| K. Runs A and A′: Tanyel and Ayvaz's off-route test on our fits (A′ corrects A's G*) | Are any of our transferring sites off-route by the published definition? | Done 20 Nov (2 GPU-minutes each) | — | Done: one competent off-route task, which transfers |
| L. Competent off-route sites: scan every region × layer, then test transfer at off-route cells | Does transfer screen out a true off-route site? | Done 20 Nov (5 GPU-hours on a shared GPU) | — | Done: L-pass, 5 of 5 cells, but 2 tasks |
| L2. Widen L: transfer at the relaxed-gate off-route cells, and at the last-token references (descriptive) | Does L-pass hold beyond two tasks, and is transfer at off-route cells as strong as on the route? | Done 20 Nov (5 GPU-minutes) | — | Done: L2-pass, 14 of 14 cells, 5 tasks; T equal on and off the route |
| P. Last-token ablation diagnostic | Why does mean-ablating the 16-dimensional subspace at the last token flip 80–98% of mirrored answers? | Done 21 Nov (1 GPU-minute) | — | Done: DAS excess; the net gate favours off-route cells |
| O. 1B check: DataDecide 1B (3 seeds), IOI and the off-route cells | Does the result hold at the scale reviewers will ask about? | Done 21 Nov (2.6 GPU-hours) | — | Done: A-pass and B-pass; natural IOI site; transfers across sizes |
| Q. 7–9B round: IOI across OLMo-3, Llama-3.1, Qwen2.5, Gemma-2 and three OLMo-3 midtraining runs; the analogy direction along OLMo-3 post-training | Does IOI transfer at 7–9B, and does a direction's effect follow its source or its destination? | About 3 GPU-hours | — | Done |
| M. Confirm the conserved profile on fresh tasks | Are naturalness and site depth conserved across families, and is OLMo's site really deeper? | Several GPU-hours | Go-ahead | Optional |
| N. Checkpoints around emergence | Does transfer depend on training history before a model can do the task? | Several GPU-hours | Go-ahead | Later |
| D. Workshop paper now | Stakes the claim early | None | About a week | Collides with HRI and #6 |
| E. Park until February | Nothing new | None | None | — |
Recommendation: write. The preregistered program is complete, including the 1B check. L and L2 filled the empty cell on 5 tasks: transfer passes at competent off-route cells, at the same T as on the route. P explained the one odd number (last-token ablation flips) as DAS excess and showed that the batch's net gate favours off-route cells, which the methods section must say. O answered the scale question every reviewer will ask: both claims hold at 1B. M and N stay optional.
Everything is in /mnt/drive_b/Cross-seed.
doc/proposal.md. The 160M result is frozen at git tag 160m-ioi-1092; the ladder step 1 preregistration and result are dated 17 Nov 2026.code/ladder_r1_decoupled.py → results/ladder_r1_decoupled_160m.json.code/ladder_r2_410m.py → results/ladder_r2_410m.json. Post-hoc scale and DAS-length check: code/ladder_r2_diag_scale.py → results/ladder_r2_diag_scale.json.code/ladder_v2.py → results/ladder_r2_v2.json and results/ladder_r1_v2.json. Comparison tables: code/ladder_v2_compare.py → results/ladder_v2_compare.json.code/causalgym_batch.py → results/causalgym_batch_160m.json. Follow-up 1: code/causalgym_wholespace.py → results/causalgym_wholespace.json. Follow-up 2: code/causalgym_globalmap.py. Follow-up 3 and the illusion split: code/causalgym_valuemap.py, code/causalgym_illusion_split.py. IOI illusion check, follow-ups 4 and 5: code/ioi_selftransplant.py, code/ioi_globalmap.py, code/ioi_subspace_diag.py. Follow-up 6 (option H): code/ioi_bettermap.py. DataDecide screen: code/datadecide.py (conversion and reference forward pass), code/datadecide_screen.py; HellaSwag data and AI2's numbers in data/hellaswag/. Rung 3: code/rung3_datadecide.py (runs the v2 helpers on the converted models through an adapter), queued by code/run_rung3_queued.sh. Rung 3b and option I: code/causalgym_xfamily.py --arm dd|410m (queued by code/run_xfamily_queued.sh) → results/causalgym_xfamily_{dd,410m}.json, amended analyses code/causalgym_xfamily_amend.py → *_amend.json. Runs A and A′: code/ta_rescore.py --gstar total|added → results/ta_rescore.json and results/ta_rescore_v2.json. Option L: code/offroute_grid.py → results/offroute_grid.json. Option L2: code/offroute_l2.py → results/offroute_l2.json. Option P: code/ablation_diag.py → results/ablation_diag.json. Option O: code/o_1b.py --arm gate|offroute|ioi, queued by code/run_o_1b_queued.sh (sharded 1B weights load through code/datadecide.py) → results/o_1b_{gate,offroute,ioi}.json. Crosscoder: code/crosscoder_train.py, code/crosscoder_eval.py → results/crosscoder_eval.json. Each writes the JSON of the same name in results/.python progress/build_page.py.Caveats: