The Reranker Won Three Questions and Lost None
On 88 frozen questions the cross-encoder reranker changed three held-out answers, all for the better, and lost none. A sign test on 3 to 0 gives p = 0.25, so the held-out gain is not yet established.
Twelve ideas for making our note network beat plain search tied, lost or starved against plain search or their control, and each count is in the table.
Part 12 of 12 in BrainNet. Previous: A Commit Used to Cost Fifty Seconds
Most of what we tried on the network did not help. This part lists all twelve ideas, with the count each one got against its control. We keep them beside the wins because each one tells a reader where not to spend a week.
| # | idea | result against control | verdict |
|---|---|---|---|
| 1 | Spreading along links on our vault | 23 of 29 and 17 of 30, same as plain search; 2026-11-19 network on and off on the merged store: 21 of 30 held-out both arms | tie |
| 2 | Capped spread on a 5 percent Wikipedia slice | 289 against 294 for vectors (uncapped: 296, plus 0.7 percent) | lost |
| 3 | Bridges to PaperBrain, Wikipedia and co-commits | 40 strict hits in every arm, 0 targets moved; hidden-link hits 217 without a bridge, 150 to 208 with one | tie, lost |
| 4 | Usage synapses from transcripts | 21 of 61 with and without; 0 lists moved | tie |
| 5 | Learned synapse weights, 20 percent Wikipedia | 348 against 372 for the plain walk (minus 6.5 percent) | lost |
| 6 | Hebbian plasticity | ETA 0 and ETA 10 both 372; memorises 34 against 24, transfers 24 against 24; Brain usage 16 against 16 of 61, mean rank 10.35 against 10.03 | tie, lost |
| 7 | Graph datastore inside a 9B model | loss 2.4356 with the graph, 2.4288 with vectors, 2.4136 with the model alone | lost |
| 8 | Hub links wired to every note | 23 of 29, unchanged; 2026-11-19 network on and off: 21 of 30 held-out both arms | tie |
| 9 | Co-touch synapses from agent edits | 933 pairs, 0 of 88 lists changed, 2 of 37 targets touched | tie |
| 10 | Usage labels to train a reranker | 25 of 1,821 shown notes opened (1.4 percent); citation labels 15 (0.8 percent) | starved |
| 11 | Fake answers prepended to the query (HyDE) | eval 34 of 59 against 44; skill exam 3 of 52 against 4 | lost |
| 12 | Stripping frontmatter, or 50 candidates | 44, 45 and 45 of 59, with a bar of 46 that none reached | no gain |
Five of the twelve have a prereg file written before the run: the bridges, the usage synapses, plasticity, and the frontmatter and HyDE arms. For the other seven, our notes show a control arm and no prereg file, so we treat those counts as weaker. Our reading of most rows is one cause: a vault with about two links per note, where spreading reaches little. That reading is inference. The bridge arms added links and still moved no target, which fits it, but we did not test the cause directly.
The network did win on dense graphs, which is the case it was built for. On the 5 percent Wikipedia slice, personalized PageRank recovered 360 hidden links against 294 from vectors, a gain of 22.4 percent. On the 20 percent slice it recovered 375 against 332, or 13.0 percent. The brainnet skill file marks the 20 percent result as alive. Brain-ask still runs the network by default, mainly for answer variety: 5.0 distinct notes per answer against 4.66 without it, from our October switch note. The twelve rows above are where it stopped helping, on a sparse vault with about two links per note.
On the 88 frozen questions, five reps each, the network on the merged store added no hits. Network on and off both scored 29 of 29 verbatim, 23 of 29 paraphrase and 21 of 30 held-out. The lists differed on 63 of 88 questions. The median was 659.5 ms on and 552.0 ms off, so the network cost +108 ms at the median (.work/amarbaro/bench/RESULTS.md item 6, made by .work/amarbaro/bench/05-06-eval.sh). This is a tie on this set, which matches rows 1 and 8. The answer-variety figure from October was not measured again.
Some of the failures were in the instrument, and they changed numbers we had already written.
Pick any row, rerun its arm on the current store with its own prereg, and read its control on the same day. The usage row is the cheapest, since its falsifier script reruns the whole arm. If any arm beats its control by more than one question under the same kill line, that row is wrong, and we would change the table.
.work/amarbaro/bench/RESULTS.md item 6 (2026-11-19), made by .work/amarbaro/bench/05-06-eval.sh
~/iTools/brain/brain-ask/eval/eval.mojo: hit rule and the cut at .md#
~/Brain/newera/2026-10-21-brainnet.md: spreading arms on the vault (0.664 against 0.664), counting bug, hub synapses 23 of 29~/Brain/newera/2026-10-21-brainnet-bridge.md: bridge arms, 40 hits, 217 against 150 to 208 of 630~/Brain/newera/2026-10-21-usage-synapses.md: 21 of 61, 128 of 255 matched, 1,308 transcripts~/Brain/newera/2026-11-03-brainnet-plastic.md: plasticity rounds, 16 of 61, mean ranks, 61 of 61 lists moved~/Brain/newera/2026-10-26-neuron-pipeline-speed.md: retracted chunk table, clean numbers, firegate 38 of 40 then 40 of 40~/Brain/Skills/own/brainnet/SKILL.md: capped spread on 5 percent, learned weights, kNN-LM losses, co-touch, and the 2026-11-18 code-store miss (938 against 15,022)~/iTools/brain/brain-ask/eval/prereg-inputs.md: frontmatter and candidate arms, HyDE result, 44 to 46 bar~/Brain/NN/2026-11-17-brain-serve-step0.md: usage labels (25 of 1,821, 21 of 241), citation labels, 2,188 alias chunks, dedupe result, 10 deep misses~/Brain/iTools/2026-11-13-brain-ask-speed.md: dedupe lists contextCommands: each arm was run with ~/iTools/brain/brain-ask/eval/.bin/eval OUT.jsonl K=V for its environment, and the usage arms with tools/usage_arms.sh PAIRS OUT in the newera repository. Our notes name the scripts; the exact environment for several arms is not in them, and GAPS-b.md lists those gaps.
Reacties
Nog geen reacties.
Inloggen om te reageren.