Twelve Ideas That Did Not Help
Twelve network ideas tied, lost or starved against plain search or their control, with the counts for each. Five had a prereg file. The failures in our measuring that we caught are listed beside them.
On 2026-11-13 the rank stage took 0.82 of a 1.56 second brain-ask answer, and a lookup fix plus a resident server cut it to 0.093 seconds with no change in the top five lists. On 2026-11-19 the served rank stage measured 0.100 seconds at the median, under the 0.15 second line.
Part 8 of 12 in BrainNet. Previous: A Memory Line Agents Did Not Follow · Next: The Reranker Won Three Questions and Lost None
On 2026-11-13, a brain-ask answer took 1.56 seconds at the median over our frozen 88-question eval. The rank stage, which spreads the vector hits along the synapses, took 0.82 of those seconds. That was half the answer. With the servers on, the same answer took 0.574 seconds that day, and rank took 0.093 of it when its server was warm. This part shows what each stage cost and what each change did to the output.
| stage | 2026-10-21 trace (ms) | 2026-11-13 trace (ms) |
|---|---|---|
| rerank (MiniLM) | 538 | 320 |
| embed (bge) | 257 | 260 |
| vector search (brainvec) | 38 | 80 |
| rank (network) | 12 | 820 |
The October trace came from one query in brain mode, when our notes store held 4,183 neurons. The November trace came from bash timestamps, and it sums to 1.48 seconds against a 1.56 median. Rank grew 68 times while the store grew 392 times, from 4,183 to 1,640,443 neurons. We read that as the store size at work. On 2026-11-19 we timed rank on one question, cold, against a vault-only store and the merged store: 20.0 ms against 271.0 ms. That supports the store-size reading, though it is one question.
The model file read. The embedding model was loaded with a read call that took 135 ms for 90 MB. Reading with the known file size took 25 ms. One query fell from 0.93 to 0.63 seconds. The paper eval lists stayed identical on 60 of 60 queries.
The rerank text fetch. The rerank step looked up each hit's text with a separate full scan of the text table. One indexed query for all 20 hits cut the benchmark from 414 to 296 ms. Eval lists differed on 0 of 88 queries.
The rank lookup. Rank loaded 1.64 million path strings and a dictionary before it looked up its 20 hits. It now reads the hits first and scans the path file once. Cold rank went from 0.82 to 0.24 seconds, and the output was byte-identical on 87 of 87 captured calls.
The resident servers. The embed, rerank and rank models stay loaded behind unix sockets. If no server answers, the client does the work itself, so a dead server only costs time. The median answer went from 934 ms with servers off to 574 ms with them on. Hit lists differed on 0 of 88 queries.
At the 20 percent Wikipedia slice (1,181,724 articles), each question copied a 540 MB link file and a 1.8 GB vector file into memory. Firing the network took 0.66 seconds of a 5.3 second question. The other 3.9 seconds were copying. Memory-mapping the files cut the question to 1,076 ms and peak memory from 6.9 GB to 1.9 GB. All 32 outputs stayed identical.
At 40 percent of the dump (2.85 million articles), the question spent its time elsewhere. Over three questions, the stages were: embed 170 to 220 ms, network 600 to 640 ms, text fetch 1,400 to 1,820 ms, rerank 280 to 350 ms, print 1,250 to 1,740 ms. Fetch and print were two awk scans of a 1.4 GB file, for 20 rows. One index with binary search brought fetch and print to 31 to 47 ms. The question went from 3.6 seconds to 0.86 seconds, with identical output.
Two speed ideas did not ship. The rank walk is still single-threaded, because a parallel scatter changes the order of float additions, and the outputs would stop being byte-identical. The walk is about half of a served rank, so the ceiling on that change is small. A GPU does not help the query path either: the time goes to loading files, and the walk itself takes milliseconds.
The reranker stayed, and its cost is the 0.2 seconds it adds to each answer. Our speed note says it carries the paraphrase and held-out quality. Part 9 measures how much.
The 2026-11-13 note said the embed model is the largest warm stage. On 2026-11-19 that was wrong. On the served path, rerank was the largest stage at 209 ms median, and embed took 34 ms. We withdraw the claim. The 0.093 second rank figure in the 2026-11-13 receipts was timed on the rank server's own clock. The 2026-11-19 served rank stage, 100 ms, was timed from the BRAIN_ASK_TIME lines.
Source: .work/amarbaro/bench/RESULTS.md, 88 frozen questions, 5 reps each, on a snapshot copy of the store.
Served path, servers on. Median per stage, pooled over n=440 runs, range in brackets:
| stage | median ms | range ms |
|---|---|---|
| cois | 109.5 | 34.0 to 990.0 |
| embed | 34.0 | 29.0 to 50.0 |
| brainvec | 55.0 | 47.0 to 70.0 |
| rank | 100.0 | 82.0 to 117.0 |
| dedupe | 57.0 | 54.0 to 62.0 |
| rerank | 209.0 | 125.0 to 280.0 |
| total | 564.0 | 422.0 to 1480.0 |
Wall time for brain-ask, served: median 658.5 ms (range 524.0 to 2747.0). The 2026-11-13 figure of 574 ms was not reproduced.
Cold path, servers off, n=440: rank 283.0 ms (241.0 to 380.0), embed 179.0 ms (166.0 to 206.0), rerank 311.0 ms (225.0 to 425.0), total 988.5 ms (832.0 to 1407.0). Cold rank is a little above the 0.24 seconds in the table above, on the larger store.
Rank at one fixed question, cold, n=5: vault-only store 20.0 ms (19.0 to 23.0), merged store 271.0 ms (264.0 to 273.0).
Rank is now about 18 percent of the served total (100 of 564 ms). On the served path it is no longer half the answer.
Falsifier: the served rank median is 0.100 seconds, under 0.15 seconds. It did not fire.
Run BRAIN_ASK_TIME=1 on the 88 frozen questions with the servers on, and take the median of the rank stage. If that median is above 0.15 seconds, the speed claim is wrong. The 2026-11-19 run gave 0.100 seconds, so it did not fire. The embed-versus-rank ranking is now measured (embed 34 ms, rank 100 ms).
.work/amarbaro/bench/RESULTS.md items 1, 2 and 13 (2026-11-19), made by .work/amarbaro/bench/01-02-stages.sh and .work/amarbaro/bench/13-time.sh
~/Brain/iTools/2026-11-13-brain-ask-speed.md: receipts table (1,562, 574, 934 and 352 ms), 0 of 88 and 87 of 87 identity checks
~/Brain/newera/2026-10-21-brain-ask-latency.md: bash -x trace (538, 257, 38, 12, 9 ms), model read 135 to 25 ms, text fetch 414 to 296 ms, paper eval 900 to 684 ms~/Brain/newera/2026-10-21-brainnet-eps.md: wiki copy fix 5005 to 1076 ms, 6.9 to 1.9 GB, 32 of 32 identical~/Brain/newera/2026-10-21-brainnet.md: 4,183 neurons on 2026-10-21~/Brain/newera/2026-10-22-wiki-40.md: 40 percent stage timings and the row index (BRAIN_ASK_TIME=1, BRAINNET_TIME=1)~/iTools/brain/brain-ask/PLAN-speed-ax.md: 2026-11-13 stage trace and status~/Brain/Skills/own/brainnet/SKILL.md: rank 0.82 to 0.24 s, 0.093 s served, 1,640,443 neurons~/iTools/brain/brain-ask/eval/runs/now-2026-11-13.jsonl, final-serve.jsonl, final-noserve.jsonl, final-norerank.jsonl: per-query ms and hits; medians recomputed as the mean of the two middle values~/.claude/INDEX.md: 1,855,596 neurons on 2026-11-18Commands: ~/iTools/brain/brain-ask/eval/.bin/eval OUT.jsonl with BRAIN_ASK_RERANK=0 (no rerank) or BRAIN_ASK_SERVE=0 (no servers), as named in the speed note. The eval usage line is in eval/eval.mojo.
Comentarii
Niciun comentariu încă.
Conectare pentru a comenta.