← Blog
✎Post blogowy

Tests that passed for the wrong reason

A cinematic photograph of a row of glowing green indicator lamps on an old control panel, one lamp wired to nothing, in teal and steel with a warm amber light, the AMARBARO mark and wordmark centred over it.

The worst failures in Street Simulator were tests that passed while proving nothing, and each fix is a rule such as proving a test can fail with a mutant.

Our worst bugs were not crashes. They were tests that said PASS while checking nothing. A passing test feels good. It is only worth that feeling if it could have failed. Most of what went wrong in Street Simulator was not a crash. It was a green light that meant nothing.

This article lists the cases and the rule each one produced.

The scale of the check

The main script, check.sh, runs about 170 test entries over files in the tests/ folder. A subset can run with ONLY=name,name. Before a full run, a literal gate scans for hardcoded numbers. On 5 November the full check gave 194 passing lines.

A bake that said PASS over an error

On 4 November a shelter test failed with "Node not found: Shelters" and "Identifier Street not declared". The cause was a parse error in one script. In this game the Street and Look helpers are preloaded constants, not named classes, so a missing preload breaks the scene.

The embarrassing part: the four-map rebake had printed PASS, with the same parse error sitting in its own logs. Nothing in the bake looks at that.

The rule: before a long run, parse-check every edited script with Godot's headless --check-only and grep for "Parse Error". It takes seconds. The failure that followed cost much more.

The run that selected nothing

ONLY=a,b runs the tests named. A name that matches nothing runs nothing, and the script exits with success. Running a single test skipped the hardcoded-number check entirely, allowing unconfigured values to slip past review.

The rule: run the literal gate yourself before each commit when you use ONLY=.

The gate that stopped every test

On 3 November five checks were red, and the cause of one explained why no test ran. A new ambulance file had three bare numbers (2, 3 and 2.0). The literal gate stopped, so check.sh ran no tests at all. The fix was three lines in an allow list.

The other four failures were stale tests. A change had made new players start dressed, but four tests assumed a naked player. The game behaviour was right. Now the tests call a helper that strips clothes once at the start. Calling it every tick would undress a player who had just put something on.

Clocks that lie

Two bugs shared a cause: a test used a clock the thing it measured did not follow.

  • The sun test timed the server's daylight drain with the client's wall clock. A stalled server looked like no drain. The test now counts physics ticks.
  • A ride test measured speed in wall time. The fix, on 4 November, measures it in physics ticks, and the bribe and ride tests now wait on the event log instead of a timer.

Two runs at once

On 4 November a bake failed with "tables.res differs", then passed a minute later on the same code. Two overlapping check runs had shared the bake folders and a log. A background command with an inner & detaches its job, so it kept running after the wrapper exited and a re-run joined it.

The rule: run check.sh directly as the background command, and look for a running one with pgrep before a re-run.

The same day, a server test failed with "port busy" because a screenshot capture was running beside it. The two share a network port. Screenshot capture and the check now run one after the other. A related trap: a port that a different program holds (a file-sync tool) gives a "port busy" failure that is not the test's fault.

Tests with one NPC prove nothing about many

The sidecar that runs NPC brains served one connection at a time. Each NPC opened its own. With one NPC everything worked. With two or more, only the first NPC ever got an answer. The next article tells that story.

The lesson is general: a test with npc/count=1 hides an entire class of bugs. The same applies to a test that arms "the first player", because the first peer to connect varies between runs.

What parallel work cannot do alone

When several automated workers built features at the same time in separate branches, merging them exposed gaps:

  • They kept editing the same status page, which caused constant conflicts. Now only one place edits it.
  • New settings were missed in the grounded and harsh presets. A presets check now catches that.
  • Some work skipped the hardcoded-number check. The count of bare numbers reached 136 and was fixed that night.

A second code review then found 3 small game bugs, among them fruit and charity items lost when a bag was full. They were fixed the same day.

Stale processes and a hanging client

The screenshot client sometimes stayed alive after logging PASS, until its 300 to 600 second timeout. The batch looked stuck. The workaround is a watcher that finds the client 15 seconds after PASS and kills it by process ID. The root cause in the client's quit path was never found.

Killing processes by name is dangerous: a broad process kill terminated our own screenshot captures and matching shell wrappers. Kill by PID, with an anchored pattern.

The five flakes still open

After the last full check, five tests fail only inside the full run: chop, court, site, coerce_turn and mindsave_a. Run alone, they pass (court passes 1 of 2). That points to load, and the next step is event waits instead of fixed timers, as was done for bribe and ride.

We have not proven that. One earlier test, court, failed about 25% of the time even before some changes: police spawned at a random angle. We first blamed the global random number generator. That was wrong. The sources note it was not the cause.

The pattern

Every case above has one shape. A check read the wrong thing, or none. We keep a short list of habits from it:

  1. Parse-check before a long run.
  2. Prove a test can fail, with a mutant.
  3. Time what you measure with its own clock.
  4. Never run two things that share a port or a folder.
  5. Test with more than one of everything.

Komentarze

Brak komentarzy.

Zaloguj się aby skomentować.