The NPC brain still loses to the script it copied
A shared frozen instinct plus a small owned layer per NPC, tested against a kill line we wrote before running anything. The brain passed the line and still lost to the script it came from.
Most of our extra mechanisms tied or lost to the simpler arm, and in the game a threatened NPC gives the learner no reward signal, so the game ships a three-answer bandit instead of the network.
Part 4 of 5 in NPC-mind. Previous: In-life learning wins back about 0.10, but only after a shift · Next: The game runs a bandit, and the network waits for a harder question
In one game log, an NPC was threatened 31 times. Its need did not move, and its health did not change at all. That log did more to decide where the network goes than any gridworld result, because it showed the learner had nothing to learn from. This part covers that log, and the many other ideas we tried first. Most of them did not help either.
Our first big idea was foresight. A simple learner credits an action only with the reward it gets right away. Foresight credits it with what the action leads to later. We built this as TD(lambda), a standard method from reinforcement learning. TD stands for temporal difference: the learner compares what it expected with what it got, one step at a time. A second part of the brain, the critic, estimates how good each state is, and the learner uses the gap between estimates as its teaching signal.
It hurt. The owned layer with foresight scored 0.456, while the one-step rule scored 0.516. Looking further ahead made it worse at every horizon we tried. Our guess is that the critic is too simple. It is a linear readout on frozen random features of the current view, so it cannot represent how good a state is when that depends on memory or on a map, and the extra signal becomes noise. We have not tested that guess yet, so treat it as a suspect, not a finding.
Next we let each NPC grow new units and new synapses, using a rule from neuroscience: neurons that are too quiet grow connections, and neurons that are too busy prune them. The hope was that the brain could reshape itself toward a healthy level of activity. We ran a control in which the same decisions were made with random partners, so the only thing that differed was the rule.
The result was a tie. The growing arm scored 0.4855 and the control scored 0.4869, well inside the noise. The activity level also did not settle toward its target. The brain did not tighten its homeostasis, which is the thing the rule was meant to do. We kept the arms in the code, and we stopped treating wiring as the lever.
We gave the NPCs a voice, of a sort. A third output could say one of 12 words from Basic English, a list of 850 words designed to be learned easily: food, work, come, go, here, there, left, right, danger, yes, no and help. Listeners within three tiles heard the word on the next tick. A speaker earned a share of what its hearers earned, and that share was itself a gene the evolution could tune.
The words carried almost no information. The mutual information between a word and the situation the speaker was in was 0.003 bits. Our bar was 0.1 bits, so the words failed it by a wide margin. The arm with speech scored 0.5125 and the arm that could not speak scored 0.5182. The speech did not help.
Our explanation is about what a bare word can carry. A page of a book carries the answer to a situation, so a reader learns from it. A bare word, like "danger", says nothing about what to do. The NPCs talked a lot and taught each other nothing in a 2000-tick life. Speech would need to carry an answer, not a label.
We tried adding a direct path from the 64 observations to the readout, skipping the hidden neurons entirely. The idea was that the 24-neuron bottleneck was limiting the learner. It was not. The skip arm scored 0.4512 against 0.4816 without the skip, a drop of 0.030, and its reflexes got worse. More inputs made the learning rule noisier, so we believe the rule itself is the bottleneck, not the representation.
We also tried a deprivation-weighted reward, which we called HRRL, after a homeostatic drive measure. Each NPC's reward was the drop in a combined drive. It was a null on satisfaction, 0.4850 against 0.4900, and it did something worse: every NPC converged on its worst need, so the diversity between NPCs fell by about three times.
Evolving the learner's own settings, its learning rate and its discount factor, was flat too. Mean fitness was 0.382 at generation 0 and 0.382 at generation 7. The evolution was selecting on noise, and the settings barely moved.
Our next idea was to learn only when something surprises the NPC. The learning rate was scaled by a running measure of negative surprise: how far reward fell below expectation, compared with its longer-run average. It seemed sensible, and it scored 0.511 against 0.529 for the instinct, which is worse than plain learning.
The gate was also invalid by design. The random numbers that drive each choice came from one shared stream, so adding any arm changed the draws for every arm, including the instinct, which never learns. We also never logged the gate's value during the runs. So whether the gate ever closes in a stable world is not measured. The only honest sentence is that we do not know.
The last idea was a trail map, inspired by Physarum polycephalum, a slime mould that grows tube networks between food sources. Each NPC got a local sensor that reads the path edge ahead of it and the strongest edge at its cell. The map grew: it had 106 edges. But at the start of a life only 9 of 100 NPCs stood on any trail, so the sensor almost never saw it. Across 10 seeds, the trail beat the control in 4 of 10, against a bar of 7 of 10. The mean difference was 0.0007. Negative.
Some of our worst moments were bugs, not ideas. In an early version, talking lowered the partner's social need without a floor. Needs could go negative: in one seed, 4,122 need-ticks went below zero in random runs, and 765 in scripted runs. That inflated satisfaction, and it had made some arms look better than they were. We fixed it and added a runtime check that every need stays between 0 and 1.
An earlier claim also had to go. We had written that a shared base beats separate brains. The arm we used for that comparison did not test separate brains at all, since it shared its input and recurrent weights across NPCs. We renamed it and withdrew the claim.
The first world was also wrong in a way that made the script look bad. Food regrew too slowly, at 0.4 units per tick against a demand of 1.0 per tick, so hunger saturated. The script scored 0.396, below random at 0.487. We fixed the regrowth rate and made "scripted beats random" a gate in the sweep. A script that loses to random is a broken test, not a result.
The idea that matters most for the game came from the game itself. In homeless-sim, a threatened NPC gets no reward signal. The game's own receipts are clear. For a threatened NPC, the need drift was minus 0.0077 per tick, with a standard deviation of 0.0000. After 31 threats, nothing had changed, and its health was untouched.
A learner needs a signal that says whether an action helped or hurt. Here the signal was flat. The gridworld network reads 76 inputs, and the game has no counterpart for most of them. The network had no teaching signal and no matching input space, so learning in the arena was not possible in the form we built.
What covers the case the game does have is much simpler. A three-answer bandit, which is a learner that tries each answer and keeps the one that pays, runs per NPC per player. A surprise gate, using a cumulative-sum test that fires when outcomes drift away from the script's expectation, decides when the bandit takes over from the script. The bandit covers the arena's twist, and it did so without the network.
So the network did not go into the game, and we say that plainly. Part 5 covers what the game runs now, and what would change our minds.
Comments
No comments yet.
Log in to comment.