Twelve Ideas That Did Not Help
Twelve network ideas tied, lost or starved against plain search or their control, with the counts for each. Five had a prereg file. The failures in our measuring that we caught are listed beside them.
On 88 frozen questions the reranker changed three held-out answers, all for the better and none for the worse, a sign test that gives p = 0.25, so its held-out gain is not yet established.
Part 9 of 12 in BrainNet. Previous: Rank Was Half the Answer Until One Lookup Fix · Next: Nearly Two Million Neurons in Three and a Half Gigabytes
On our frozen eval, the reranker put the answer into the top five for 22 of 30 held-out questions. Without it, 19 of 30 did. That is a three-question gap, and it costs about 220 ms per answer. On 2026-11-19 the rerank stage measured 209 ms at the median (range 125 to 280, n=440). Paired by question, the gap is three wins, zero losses. If the three are real, the 220 ms buys about 74 ms per recovered question. This part shows how little that proves.
The eval has 88 questions in three sets: 29 verbatim, 29 paraphrased and 30 held-out. A question counts as a hit when its target note is among the five shown, at any position. We call that the strict measure. The mean reciprocal rank (MRR) also rewards position: a target at rank one scores 1, at rank two 0.5, and a miss scores 0. The eval file has been kept since October, but it has been reused for many arms. Our own prereg note says a pass on it is weak evidence, and the held-out set was written to be frozen before any arm ran on it. Every arm since has scored it again.
| run | verbatim | paraphrase | held-out | median ms |
|---|---|---|---|---|
| network and reranker, servers on | 29/29 | 23/29 | 22/30 | 574 |
| reranker off | 29/29 | 22/29 | 19/30 | 352 |
| servers off, same hits | 29/29 | 23/29 | 22/30 | 934 |
| 2026-11-17 before dedupe | 29/29 | 23/29 | 20/30 | 607 |
| 2026-11-17 after dedupe | 29/29 | 23/29 | 20/30 | 645 |
| network on, servers on, 2026-11-19 | 29/29 | 23/29 | 21/30 | 659.5 (n=440) |
| network off, 2026-11-19 | 29/29 | 23/29 | 21/30 | 552.0 (n=440) |
| candidate depth 20, 2026-11-19 | 29/29 | 24/29 | 24/30 | not in RESULTS.md |
A 22 of 30 held-out result has a 95 percent Wilson interval of 56 to 86 percent. A 19 of 30 result runs from 46 to 78 percent. Each question is 3.3 points, so the two intervals overlap almost entirely.
Comparing the same question with and without the reranker is the fair test, because both arms see the same candidates. The reranker changed the five-note list on 86 of the 88 questions. On the held-out set it helped three questions and hurt none. On the paraphrase set it helped two and hurt one. An exact two-sided sign test on 3 to 0 gives p = 0.25. The reranker's mean reciprocal rank on held-out is higher, 0.576 against 0.490. MRR also counts moves inside the top five, so it is not an independent test of the same three questions.
The 2026-11-15 run in our brainnet skill file reported network off at 21 of 30 held-out and network on at 22 of 30. That gap did not appear. On 2026-11-19 we reran both arms on the 88 frozen questions, five reps each. Network on and off both scored verbatim 29 of 29, paraphrase 23 of 29 and held-out 21 of 30, in every rep. The lists differed on 63 of 88 questions. The median cost was 659.5 ms on (range 513.0 to 1630.7, n=440) and 552.0 ms off (range 421.1 to 885.2, n=440), so the network added +108 ms at the median and no hits on this set. We have dropped the 22 of 30 claim. The 2026-11-15 output file is still not in the eval folder.
Recall at five is what the eval measures. A second set asks a different question: one query for each of 300 notes, written by a local model, with the note as the answer. There, the top note was right for 58.7 percent, the top five for 75.3 percent and the top 20 for 79.7 percent, which is 239 of 300. Sixty-one targets never reached the top 20. A reranker that only reorders the top 20 can add at most the 4.3 points between five and twenty.
With candidate depth 20, held-out recall rose from 21 to 24 of 30, and paraphrase from 23 to 24 of 29. Verbatim stayed at 29 of 29. The default depth value is not stated in RESULTS.md. Depth moved the held-out set by three questions, the same size as the reranker gain above, so the candidate list does limit the held-out result. The miss autopsy in RESULTS.md (the run it belongs to is not named there) lists 15 misses: 7 labelled absent, 4 absent with a near match, 1 reranked and 3 reranked with a near match.
For the 59-question arm we measured on 2026-11-02, an autopsy found 15 misses. Thirteen of them were not among the 20 candidates at all, so no reranker could reach them. The other two were in the candidates and the reranker put them out of the top five. The bottleneck was the candidate list, not the final order.
The network-off arm on the current store now exists (2026-11-19, above). We have no reranker-off run on the current store. RESULTS.md has no BRAIN_ASK_RERANK=0 arm, so the reranker claim still rests on the earlier runs in the provenance list. The 300-question synthetic set has no saved generator or output file yet, so its numbers cannot be rerun from our notes.
Write 60 held-out questions after today, score them on the current store with the reranker on and off, and count wins and losses per question. If the reranker's wins outnumber its losses with a sign test below p = 0.05, the claim that its gain is unproven is wrong, and we will say so.
.work/amarbaro/bench/RESULTS.md items 5 and 6 (2026-11-19), made by .work/amarbaro/bench/05-06-eval.sh; per-run logs and jsonl files in .work/amarbaro/bench/.work/amarbaro/bench/01-02-stages.sh: rerank stage median (item 1)
~/iTools/brain/brain-ask/eval/runs/final-serve.jsonl, final-norerank.jsonl, final-noserve.jsonl, pre-dedupe-2026-11-17.jsonl, dedupe-2026-11-17.jsonl: per-question hits and ms; paired counts, sign tests and Wilson intervals computed from these files
~/Brain/iTools/2026-11-13-brain-ask-speed.md: reranker off at 352 ms and 22/29, 19/30~/Brain/Skills/own/brainnet/SKILL.md: the 2026-11-15 network on/off run (not a file receipt), and the reranker row~/Brain/NN/2026-11-17-brain-serve-step0.md: synthetic set, R@1 0.587, R@5 0.753, depth-20 counts (226, 13, 61)~/iTools/brain/brain-ask/eval/prereg-inputs.md: 44, 45, 45 of 59 and the autopsy basis~/Brain/brainnet/log.md: autopsy of 44 of 59 misses, 13 of 15 absent from the candidates~/iTools/brain/brain-ask/eval/eval.mojo: set sizes and the hit ruleCommands: ~/iTools/brain/brain-ask/eval/.bin/eval OUT.jsonl with BRAIN_ASK_RERANK=0, or BRAIN_ASK_SERVE=0, as the speed note names them. The October autopsy ran eval/autopsy.sh A.jsonl W.jsonl.
Reacties
Nog geen reacties.
Inloggen om te reageren.