On our own sparse notes, every network idea tied plain search, at 23 of 29 and 17 of 30.
On our own notes, every network idea we tried tied plain search: 23 of 29 against 23 of 29, 17 of 30 against 17 of 30. Most of what we tried did not work, and we keep those results next to the wins, because they tell you where not to spend a week. Each experiment below had a rule written before the run: beat the control by a set margin, or stop.
The core miss: our own vault
The network helps on Wikipedia. On the vault it was built for, it does not. Plain brain-ask and the network arms scored the same: 23 of 29 on paraphrased questions and 17 of 30 on held-out ones.
| arm on the vault |
strict hits |
| plain brain-ask |
23/29 + 17/30 |
| network on, same 20 candidates |
23/29 + 17/30 |
| personalized PageRank |
23/29 + 17/30 |
The arms were live. PageRank reordered 23 of the 88 result lists. It just never moved a target into the top 5. Our diagnosis: the vault has only two links per note. The walk only visits linked neighbors, so it misses isolated conclusion notes that nothing links to.
An earlier version of this story had a wrong number. A first reading said "0 recalled", which was a counting bug: the eval cut hits at a marker that never showed up. The corrected count was 6 notes shown that the control did not show, none of them a target.
Newer note beats older note
The first spread, capped at 32 neurons, gained 0.7 percent on a 5 percent Wikipedia slice, which is noise. We first concluded that spreading was a small-neighbourhood effect. A few days later, uncapped PageRank gained 22.4 percent on that same slice. Where sources disagree, the newer one wins, and the older conclusion was too strong. The vault result did not change.
One more thing changed. At one point a note said the default build ignored the --ppr flag, so one vault arm might have run the old spread instead. A follow-up checked the timing: the binary honoured the flag from 05:34 that day, and that arm had been passed the right build explicitly. It did run PageRank.
Bridges: wire the vault to bigger graphs
If the vault is sparse, give it more links. We joined notes to PaperBrain pages, to Wikipedia, to both, and to notes edited in the same commit. All arms: 40 strict hits, the same as with no bridge. Between 8 and 21 of 88 lists changed, and not one target moved.
On the hidden-link test the bridges hurt: 217 hits with no bridge, 150 to 208 with one. Wikipedia hubs such as AX.25 (287 bridge edges) and Artificial intelligence (238) were the hubs the bridges built. Killed.
A bridge between math results and cited papers did better than vectors alone (68 against 58 of 91) but lost to simply scoring the best statement pair (79). Killed.
Usage links: learn from what people open
Idea: when someone searches and then opens a note, store that pair as a synapse. We mined 1,308 transcripts (3.4 GB, 11 s) and found 728 queries and 265 pairs. Of the 255 usage synapses built from them, 128 matched notes that still exist.
Across 61 test searches, usage links made no difference: search returned the right note 21 times with or without them. Zero lists moved. A weight-1 synapse passes about 0.05 signal, while the top 5 seeds hold 0.17 or more, so the target never gets in. Also the reranker ignores network scores, and most opens are protocol notes that a popularity count would find, not association.
Learned weights
We tried to weight synapses by cosine similarity and by link count. On the 20 percent Wikipedia network the weighted walk scored 348 against 372 for the plain walk (and 335 for vectors). That is minus 6.5 percent, below the kill line of 376. Our reading: uniform weights already work, and cosine weighting pulls the walk back toward what vectors already rank.
Plasticity: synapses that learn
We tested whether links could strengthen automatically when two notes are retrieved together. Three rounds failed:
- Round 1: ETA 0 and ETA 10 both scored 372 on the test. Learning was too diffuse: a mean change of 0.0002 per synapse cancelled when the walk normalised.
- Round 2: the model memorized the test pairs it trained on, but failed to generalize to new questions. A memory, not a better ranker.
- Round 3, Brain usage: top 5 of 61 was 16. Killed.
Along the way we caught a memory bug: one step copied a 10.6 GB vector file into RAM. Memory-mapping dropped RAM use from 21 GB to 4.5 GB.
A graph inside the model
We tried the same idea inside a language model: retrieve neighbours from a datastore built by graph and feed them to a 9B model (kNN-LM). Loss on 20 test results: the graph datastore 2.4356, plain vectors 2.4288, and the model alone 2.4136, the best. Adding retrieval of either kind made it worse. Killed.
Open leads
- Sparse plasticity on only the strongest edges, or with far more training pairs. Not tested.
- A harder paper test with questions from inner results, not abstracts. Strict accuracy sits at ceiling on abstract questions and cannot judge anything.
- Whether the skill hooks suggest the right skill. Not measured; our attempt to take ground truth from transcripts was blocked.
- Why the Wikipedia gain stays flat when the graph grows.
What to take from this
Links help where they are dense and meaningful, and add little where they are sparse. A reranker is a better first lever than a better graph. And a kill line written before the run saved us from arguing with our own enthusiasm every time.
Comments
No comments yet.
Log in to comment.