The trendytails backup of 20 tables and 323 rows restored into a scratch Postgres 18.6 with every table's row count equal to the source's.
A backup is a file until a restore counts the same rows as the source. Our trendytails backup became a usable backup only after a scratch Postgres restore matched all 20 tables and 323 rows. On the organictea file copy, the first checksum run reported all 2,882 files as failed. The fault was ours. We had run the check from the wrong folder.
The trendytails backup
The backup script on the VPS wrote a dump of 20 tables, with data for all 20, and 8 uploaded files. It reported BACKUP OK only after its own checksum passed. We copied the backup to the workstation and checked it there with sha256sum -c, which also passed.
A checksum proves the file is the file that was written. It does not prove the file restores. So we restored the dump into a scratch Postgres 18.6 instance and counted the rows of each table against the source's rowcounts file. Booking had 10 rows, customer 11, email_log 122, setting 29 and page_block 45. The total across the 20 tables was 323 rows, and the counts were equal. We removed the scratch container afterwards. The backup script's own summary line reads tables=20 dump_table_data=20, so the table count and the data count agreed before the copy began. The comparison was made against the rowcounts file that sits with the backup.
The 10 bookings in that dump were all in the past, the newest from 2026-11-11. The restore therefore proves the data came back. It does not prove that a booking made today would work against the restored database. A booking made after the switch would run code that a data restore never touches.
The organictea file copy
The organictea files sit on an FTPS server, and we copied them over FTPS. The copy held 2,882 files and 4,479,488 bytes, which matched the server's listing in count and total size.
The first checksum run gave 2,882 FAILED. The checksum file lists paths relative to the folder it was written in. We had run the check one folder up, so every path failed to open. Running it from the files folder, after commit 531c95f3, gave a clean result. The mistake was in the command, not in the copy.
The first full pull took about 20 minutes. After it, a sync script moved 115 changed files in 35 seconds, leaving session files out. A test run 45 minutes later found zero changed files.
What the checksum did
The backup script checks its checksum before it prints BACKUP OK. The organictea check was a manual run, so the failure was visible only because we read its output.
Three copies, three jobs
The trendytails backup was mandatory before any write to production, and it was read-only on the database. The VPS also runs a nightly backup timer at 03:30 UTC, so a box copy from that night existed before the switch. The lab held a pull from 05:00 that was 17 hours old at the go decision. The switch then took its own backup as its first step.
We kept these apart in the record, because they answer different questions. The nightly copy is what we would restore after a bad night. The pull on the lab is what we test restores against, without touching the box. The switch backup is the one closest to the change, and it is the one the switch depends on.
What we still do not know
The count and total-size match tells us the organictea copy has the same number of files and bytes as the server. It does not tell us that each file matches its server copy byte for byte, because we did not compare per-file checksums against the server. For the trendytails dump, the restore compared row counts, not row contents. A table with the same count and the wrong values would pass.
How to prove this wrong
Take a backup, restore it into a scratch database, and compare each table's row count with the rowcounts file. If any table differs, the backup is not what its label says. Then flip one byte in a copy of the dump and run the checksum. It must fail. If it passes, the checksum is not covering the file.
Provenance
docs/cutover/LOG-deploy-wave.md, line 23 (organictea FTPS copy: 2,882 files and 4,479,488 bytes matching the server listing; the wrong-folder checksum run; commit 531c95f3; the 115-file sync in 35 seconds after a 20-minute first pull; zero changes after 45 minutes), line 32 (trendytails VPS backup with 20 tables and 8 uploads, sha256 OK, scratch restore counts equal at 323 rows), line 40 (the 10 bookings in the past, newest 2026-11-11).
docs/cutover/LOG-cut-kit.md, line 10 (backup.sh checks sha256sum -c before it prints BACKUP OK).
The cutover runbook for amarbaro.com stamps the database to revision 0001 before the switch. Read in full, the stamp makes the automatic tag rollback unable to start the old image. We found it by reading the runbook, not by a failed deploy.
The switches themselves were short. What slowed the night was a refused step on one site, a tunnel waiting for a dashboard invite on another, and a mail provider we had wrong until 23:30.
Two gallery URLs failed in the first rehearsal walk. Mounting the live volume read-only crashed the container's startup, and mounting it read-write would have changed the owner of production files. The walk passed on a scratch copy.
Kommentare
Noch keine Kommentare.
Anmelden um zu kommentieren.