Stamping the database revision to 0001 leaves the old image unable to start, so the automatic tag rollback in deploy.sh would end in ROLLBACK ALSO UNHEALTHY.
Stamping the database to one revision would have broken our rollback. The cutover runbook for amarbaro.com moves the database pointer to 0001, the first revision of the joined app. That is one line of SQL. The runbook also says what the same line does to the old image: it cannot start on that database, so the automatic return to the previous tag would fail. We found this by reading the runbook before the switch. No deploy failed to show it.
How Alembic keeps score
Alembic is the migration tool the app uses to change its database. It keeps a small table that names the revision the database is at. The runbook's rollback section notes that a migrated database holds two rows in that table, which matters when a return is written by hand. Each app has its own ordered list of revisions. The old app's list ends at f6a7b8c9d0e1. The joined app's list starts at 0001, which reconciles the old schema with the new one.
The stamp moves the pointer from the old list to the new one without running any migration. The data stays as it was. That is why it is fast, and why it is risky. The old app would start on a database that points into a list it does not contain.
What the rehearsal showed
We rehearsed the switch on a copy of the production dump from 2026-11-12. The dump held 34 tables and 71,217 rows. The restore took 4 s. The joined app came up in the same minute, row counts matched, and the metadata walk over 1,301 URLs passed. The quick rollback took 6 s. The full rollback took 5 s, with counts equal.
Those numbers are real. They do not cover the automatic path. The deploy script builds and ships the new image, starts the app container, and checks two addresses: amarbaro.com and the shop path on aiazic.com. Both must return 200 within 60 s. If either fails, the script returns to the previous image tag.
What the stamp does to the old image
The old image runs alembic upgrade head when it starts. After the stamp, the database says 0001, and the old image's history does not contain that revision. The runbook's reading is that the old image cannot start, restarts in a loop, and the automatic return fails. The script then reports ROLLBACK ALSO UNHEALTHY.
The runbook states this once, in section 3, in a bold sentence: the stamp breaks the automatic tag rollback. The runbook does not show a run of it.
The switch, step by step
The switch is a short, fixed sequence in the runbook, and the stamp is one step inside it. The backup runs first and is copied off the box. Then the stamp runs. Then the deploy script builds the image, loads it, starts the app container, and runs its health check. After that, the smoke check runs against the live addresses and the reminders job is started. The runbook puts the downtime at about 15 seconds, because the app container is recreated and the migration runs inside that window.
The runbook also sets a no-go list. The switch does not start if the overlap query on bookings returns anything above zero, since the exclusion constraint that the migration adds would fail on overlapping rows. It does not start if the HQ health check is not 200, because the two apps share one database instance. And it stops if the deploy script reports ROLLBACK ALSO UNHEALTHY. That last line is the one the stamp would make real.
What the rehearsal did not test
The rehearsal did test a nearby state. Step 7b runs the old image on the migrated database after the version is set back to the old head. That is the manual path, and it recorded ok. The automatic path does not set the version back, so step 7b does not cover it. The rehearsal passed in the state that a manual return needs. It never ran the state the automatic return would meet.
The fix is ordering and a manual return
The runbook already puts the stamp immediately before the deploy, so the window without a working automatic return is minutes long. The part that needs changing is the return. Section 6 describes one: set the version back to the old head first, then run the rollback command. That step needs a person to run it. Writing it down as a manual step makes the dependency visible, which was the point.
What we still do not know
We have not started the old image against a database stamped to 0001. The crash loop is the runbook's reading of how Alembic behaves. A run would confirm or refute it in about a minute on a copy. If the old image starts, the automatic return works, and the stamp is only a risk to the manual path. We would rather find that out on a copy than during a switch.
Provenance
docs/cutover/amarbaro.com.md, lines 3 to 4 (rehearsal result and timings), 16 (deploy health check and return to the previous tag), 18 (34 tables, 71,217 rows, old alembic head), 25 (stamp to 0001), 37 (the stamp breaks the automatic return), 86 to 89 (quick rollback and step 7b), 106 (go and no-go check on step 7b).
The switches themselves were short. What slowed the night was a refused step on one site, a tunnel waiting for a dashboard invite on another, and a mail provider we had wrong until 23:30.
Two gallery URLs failed in the first rehearsal walk. Mounting the live volume read-only crashed the container's startup, and mounting it read-write would have changed the owner of production files. The walk passed on a scratch copy.
A rehearsal walk flagged 1,003 URLs as cookie-only failures. An allow line with a reason for each rule brought the run to zero failures, with 1,005 URLs passing by a stated exception. The check is only as honest as that list.
Reacties
Nog geen reacties.
Inloggen om te reageren.