← Blog
✎blogbericht

Twelve Ideas That Did Not Help

A cinematic photograph of a cracked glass petri dish on a steel laboratory bench, one crack running across its surface, in teal and steel with a warm amber light, the AMARBARO mark and wordmark centred over it.

Twelve ideas for making our note network beat plain search tied, lost or starved against plain search or their control, and each count is in the table.

Part 12 of 12 in BrainNet. Previous: A Commit Used to Cost Fifty Seconds

Most of what we tried on the network did not help. This part lists all twelve ideas, with the count each one got against its control. We keep them beside the wins because each one tells a reader where not to spend a week.

The twelve

# idea result against control verdict
1 Spreading along links on our vault 23 of 29 and 17 of 30, same as plain search; 2026-11-19 network on and off on the merged store: 21 of 30 held-out both arms tie
2 Capped spread on a 5 percent Wikipedia slice 289 against 294 for vectors (uncapped: 296, plus 0.7 percent) lost
3 Bridges to PaperBrain, Wikipedia and co-commits 40 strict hits in every arm, 0 targets moved; hidden-link hits 217 without a bridge, 150 to 208 with one tie, lost
4 Usage synapses from transcripts 21 of 61 with and without; 0 lists moved tie
5 Learned synapse weights, 20 percent Wikipedia 348 against 372 for the plain walk (minus 6.5 percent) lost
6 Hebbian plasticity ETA 0 and ETA 10 both 372; memorises 34 against 24, transfers 24 against 24; Brain usage 16 against 16 of 61, mean rank 10.35 against 10.03 tie, lost
7 Graph datastore inside a 9B model loss 2.4356 with the graph, 2.4288 with vectors, 2.4136 with the model alone lost
8 Hub links wired to every note 23 of 29, unchanged; 2026-11-19 network on and off: 21 of 30 held-out both arms tie
9 Co-touch synapses from agent edits 933 pairs, 0 of 88 lists changed, 2 of 37 targets touched tie
10 Usage labels to train a reranker 25 of 1,821 shown notes opened (1.4 percent); citation labels 15 (0.8 percent) starved
11 Fake answers prepended to the query (HyDE) eval 34 of 59 against 44; skill exam 3 of 52 against 4 lost
12 Stripping frontmatter, or 50 candidates 44, 45 and 45 of 59, with a bar of 46 that none reached no gain

Five of the twelve have a prereg file written before the run: the bridges, the usage synapses, plasticity, and the frontmatter and HyDE arms. For the other seven, our notes show a control arm and no prereg file, so we treat those counts as weaker. Our reading of most rows is one cause: a vault with about two links per note, where spreading reaches little. That reading is inference. The bridge arms added links and still moved no target, which fits it, but we did not test the cause directly.

Where the network did win

The network did win on dense graphs, which is the case it was built for. On the 5 percent Wikipedia slice, personalized PageRank recovered 360 hidden links against 294 from vectors, a gain of 22.4 percent. On the 20 percent slice it recovered 375 against 332, or 13.0 percent. The brainnet skill file marks the 20 percent result as alive. Brain-ask still runs the network by default, mainly for answer variety: 5.0 distinct notes per answer against 4.66 without it, from our October switch note. The twelve rows above are where it stopped helping, on a sparse vault with about two links per note.

Measured again on 2026-11-19

On the 88 frozen questions, five reps each, the network on the merged store added no hits. Network on and off both scored 29 of 29 verbatim, 23 of 29 paraphrase and 21 of 30 held-out. The lists differed on 63 of 88 questions. The median was 659.5 ms on and 552.0 ms off, so the network cost +108 ms at the median (.work/amarbaro/bench/RESULTS.md item 6, made by .work/amarbaro/bench/05-06-eval.sh). This is a tie on this set, which matches rows 1 and 8. The answer-variety figure from October was not measured again.

Failures in our own measuring

Some of the failures were in the instrument, and they changed numbers we had already written.

  • A counting bug. One early reading said the network recalled 0 notes beyond the vectors. The eval cut hits at a marker that never appears, so the count was wrong. The corrected count was 6 notes shown that the control did not show, none of them a target.
  • A retraction. A chunk-level table credited full text with a large lead. Queries had leaked the article titles. The clean numbers are smaller: at 1,000 articles, top-one recall went from 0.371 with the lead alone to 0.494 with chunks; at 10,000, from 0.246 to 0.350.
  • Alias copies. Symlinked copies of the ledger and trash folders put 2,188 duplicate chunks into the index. Of ten deep misses, 2 were junk queries, 2 were alias copies that outranked the target, 5 were valid sibling answers, and about 1 was a real miss. After deduplication, held-out MRR went from 0.536 to 0.544, and 19 of 88 lists changed.
  • A gate that failed honestly. The sparse-firing gate first failed 38 of 40, on two adjacent near-tie swaps inside the top 20. We changed it to compare the top-20 sets, which the reranker actually consumes, and it passed 40 of 40.
  • Silent misses. On one large code repository, the default search returned 0 hits, and the code store found the file. On a 457,000-node repository, the code-graph query returned 938 call links where 15,022 existed, and it printed no error.

How to prove this wrong

Pick any row, rerun its arm on the current store with its own prereg, and read its control on the same day. The usage row is the cheapest, since its falsifier script reruns the whole arm. If any arm beats its control by more than one question under the same kill line, that row is wrong, and we would change the table.

Provenance

  • .work/amarbaro/bench/RESULTS.md item 6 (2026-11-19), made by .work/amarbaro/bench/05-06-eval.sh

  • ~/iTools/brain/brain-ask/eval/eval.mojo: hit rule and the cut at .md#

  • ~/Brain/newera/2026-10-21-brainnet.md: spreading arms on the vault (0.664 against 0.664), counting bug, hub synapses 23 of 29
  • ~/Brain/newera/2026-10-21-brainnet-bridge.md: bridge arms, 40 hits, 217 against 150 to 208 of 630
  • ~/Brain/newera/2026-10-21-usage-synapses.md: 21 of 61, 128 of 255 matched, 1,308 transcripts
  • ~/Brain/newera/2026-11-03-brainnet-plastic.md: plasticity rounds, 16 of 61, mean ranks, 61 of 61 lists moved
  • ~/Brain/newera/2026-10-26-neuron-pipeline-speed.md: retracted chunk table, clean numbers, firegate 38 of 40 then 40 of 40
  • ~/Brain/Skills/own/brainnet/SKILL.md: capped spread on 5 percent, learned weights, kNN-LM losses, co-touch, and the 2026-11-18 code-store miss (938 against 15,022)
  • ~/iTools/brain/brain-ask/eval/prereg-inputs.md: frontmatter and candidate arms, HyDE result, 44 to 46 bar
  • ~/Brain/NN/2026-11-17-brain-serve-step0.md: usage labels (25 of 1,821, 21 of 241), citation labels, 2,188 alias chunks, dedupe result, 10 deep misses
  • ~/Brain/iTools/2026-11-13-brain-ask-speed.md: dedupe lists context

Commands: each arm was run with ~/iTools/brain/brain-ask/eval/.bin/eval OUT.jsonl K=V for its environment, and the usage arms with tools/usage_arms.sh PAIRS OUT in the newera repository. Our notes name the scripts; the exact environment for several arms is not in them, and GAPS-b.md lists those gaps.

Reacties

Nog geen reacties.

Inloggen om te reageren.