← Blog
✎blogbericht

In-life learning wins back about 0.10, but only after a shift

A cinematic photograph of a copper arrow weathervane against dark storm clouds in teal and steel with a warm amber light, the AMARBARO mark and wordmark centred over it.

After a shift that hurts the NPCs' own habit, in-life learners recover about 0.10 more satisfaction than the frozen instinct, while in a stable world learning costs about 0.01.

Part 3 of 5 in NPC-mind. Previous: One shared brain and 216 private weights per NPC · Next: Most of the extra machinery did not help, and the game gave no reward to learn from

At tick 1000, we made one habit start to hurt. The NPCs' favourite habit was talking. From that tick on, talking added 0.3 to the safety need and gave no relief. The scripted NPCs fell from 0.623 to 0.051 over the second half of their life. The frozen instinct fell from 0.521 to 0.211. The learners fell too, but they recovered about 0.10 more than the instinct did. That is the first clean result where learning inside a life did something the script and the instinct could not.

Why the stable world showed nothing

Part 1 ended on a gap: the best brain scored 0.530 and the script 0.631 in a stable world. Learning did not close that gap. Our best learning arm, pooled learning on top of the evolved instinct, scored 0.521 against 0.531 for the instinct alone. The difference was minus 0.010, with a standard error of 0.004, so learning cost about two standard errors. In a world that does not change, the learned changes mostly added noise to a good instinct.

This is an honest cost, and we report it as one. It also told us where to look. If learning only hurts in a stable world, the question becomes when it should happen at all. A learner that stayed quiet until something surprised it looked like the obvious fix. We tested that next, and it did not help in the stable world, which is covered in Part 4.

The shift test that worked

Each life in these tests lasts 2000 ticks. We wanted a world that changes at a known moment, so we could measure how fast each kind of brain adapts. Because the shift comes at tick 1000, we score only the second half of each life, so the change has time to bite and the first half does not mask it. Each arm runs over five seeds, and the seed spread is the yardstick for every difference we report. The design is simple. At tick 1000, one action starts to hurt. Nothing new is given as a reward. The action only gets a penalty, so the test cannot be gamed by a new reward route.

We ran two versions. In the first, WORK at the forge starts to hurt. The script works at the forge, so it is the script's habit that gets punished. In the second, TALK starts to hurt. The brains talk a lot, so it is their habit that gets punished. We called these shifts burn and mute.

The burn shift tested nothing. Our brains rarely worked, so making work costly barely touched them: the instinct moved from 0.521 to 0.508, a loss of 0.013. There was nothing to adapt to. The mute shift was the clean test. Here is the second half of each life, before and after the shift:

arm second half, stable second half, after mute
scripted schedule 0.623 0.051
frozen instinct, no learning 0.521 0.211

Every learner recovered about 0.10 more than the frozen instinct. The pooled learner gained 0.103, with a standard error of 0.009, which is about ten standard errors. That is a large signal against noise.

The recovery did not depend on sharing. A learner that kept its changes private to each NPC gained 0.096, and another variant gained 0.096 as well. So the recovery comes from each NPC adjusting during its own life, and pooling added only a little on top of that.

We should say what was not clean. Before the mute test, we ran a first shift that rewarded talking instead of penalising it. Talking was already the brains' habit, so the shift made their world easier. The frozen instinct gained 0.181 under that shift, the same as every learner. The test could not tell learning apart from the instinct, so we called it confounded and replaced it. We had written down the pass bars before that run, and this design failed its third bar. The replacement uses penalties only.

Repeated shifts: the instinct carries the win

A single shift can be a lucky accident. So we tried a harder world, in which a new action starts to hurt in each quarter of the life: nothing in the first quarter, then talking, then work, then eating. We score the last three quarters.

The script scored 0.245 here. It was not built for a world that keeps changing, and it lost badly in each shifted quarter. The frozen instinct, with no learning at all, scored 0.401. That is 0.156 above the script, and it came purely from the evolved instinct. Learning did not create that win.

The learners did something more interesting. After talking turned bad, the pooled learner recovered: its satisfaction rose by 0.097 relative to the instinct in that quarter, with a standard error of 0.008. After work turned bad, it lost 0.065 relative to the instinct. The reason is clear once you see it. What the learner had picked up about talking, that it was bad, pushed it toward the forge, and the forge now burned. Learning pushed the NPC toward the wrong habit.

That is interference: one lesson pushing against another. The learners lost in the work quarter, while the instinct had no such problem because it never learned anything.

Forgetting removed most of the interference

Our fix is almost embarrassingly simple. The learned change is shared across all NPCs in a world, and we let it decay toward zero, which is the instinct, by 0.35 percent each tick. Forgetting by that rate means half the learned change is gone after about 200 ticks. We call that rate rho, and we set it once, without tuning it.

With forgetting on, the learner beat the instinct on repeated shifts: 0.433 against 0.401, a gap of 0.032 that is about ten standard errors. This is the first time learning beat the evolved instinct under repeated shifts. The interference in the work quarter also shrank from minus 0.073 to minus 0.009.

Forgetting was not free. In the stable world, the learner with forgetting scored 0.006 below the instinct, with a standard error of 0.004. We pre-registered that cost as a bar, and it failed narrowly.

What this does not show

We have not shown that learning beats the instinct in a stable world. It does not, not yet. We have not tuned the forgetting rate, so 0.0035 is one value, not the best one. We have not tested a world with many different shifts at once, only the penalty shifts and the cycle described above. And the win is in satisfaction, a single number that averages the four needs. A harder test would ask whether NPCs behave differently in ways a player would notice. We have not measured that yet.

The result we can stand behind is narrower and still useful. When the world changes under an NPC, a learner that forgets recovers more than an instinct that cannot learn, and a learner that never forgets gets in its own way. Part 4 covers the other ideas we tried, and most of them did not help.

Reacties

Nog geen reacties.

Inloggen om te reageren.