← Blog
✎publicación de blog

Our rehearsal returned 404 for gallery images because it had no uploads volume

A brass key on a steel ring beside an open steel locker door in a dim corridor, in teal and steel with a warm amber glow from the locker.

A rehearsal container without the uploads volume returned 404 for gallery images that return 200 live, and the walk passed only on a scratch copy of the volume.

A rehearsal container started without the production uploads volume answered 404 for gallery photos that the live site serves with 200. We could not fix that by mounting the live volume. Read-only, the container's startup step crashed. Read-write, it would have changed the owner of production files. The walk passed only after we ran it on a scratch copy of the volume, and that copy tests something different from production.

The first walk

The rehearsal runs a copy of the live database and starts the joined app beside the old one. The metadata walk then requests each URL on both and compares the answers. The first walk failed two URLs, both under /uploads/gallery/. The legacy site answered 200 for each. The joined site answered 404.

The cause was plain once we looked. The rehearsal container had no uploads volume, so the files were not there to serve. The live site had them.

Three ways to mount the volume

The obvious fix was to mount the production uploads volume into the rehearsal container. We had two options for the mount, and both were wrong.

Mounted read-only, the joined app's entrypoint tries to change the ownership of the uploads folder at startup, and the change fails. The container crashes before the walk can run.

Mounted read-write, the entrypoint's ownership change works, and it changes the owner of the production files. A rehearsal would then leave production with different ownership. That is a change to the live site made by a test.

So we copied the volume into a new scratch volume, tt_rh_uploads, and mounted that. The entrypoint's change ran on the copy. The walk then passed with urls=41 FAIL=0 ALLOWED=28. The 28 allowed URLs match the allow count in the trendytails rehearsal record. After the walk, we removed the copy, the rehearsal container and the scratch volume. The live stack was not touched. The retry ran the same meta walk as the first attempt, so the volume was the only change we made on purpose.

The data migration ran first

The walk was not the first check on that rehearsal. The data migration ran before it, and its checks passed. On the copy, the counts and the price sums were equal, and the alembic version moved from c5d2e8f1a903 to 0001, the same revision that the amarbaro.com cutover stamps. The joined app came up after that, and only then did the walk fail on the gallery.

So the gallery failure was not a data failure. The rows were all there, with equal counts. What was missing was the directory of uploaded files that the gallery reads from. The live stack, which had its three containers up for six days, was not touched during any of this.

What the walk did not prove

The scratch copy has the same files as production, as of the copy. It has whatever ownership and permissions the copy kept. The check therefore tests the copy's permissions, not production's. For that reason the rehearsal kept production's volume out of the test, and the owners of the live files stayed unchecked. If production's uploads folder has different owners, the entrypoint's behaviour on the real volume is still not tested.

The record does not say how many files the production volume held on the day of the rehearsal. The backup taken before the switch lists 8 uploaded files, so that is the number we can cite, and it is a backup count, not a count of the live volume.

What we still do not know

Whether the entrypoint's ownership change would pass on the live volume with its current owners. We have not run it there, and the rehearsal rule is that we do not. The gap is small and specific: one ownership check on one folder.

How to prove this wrong

Create a fresh volume, populate it from the live copy with the owner and mode kept, and start the joined container on that volume. If the entrypoint passes and the gallery URLs return 200, the scratch copy was not what made the difference. If the gallery URLs still fail, our explanation was incomplete. Either result is a useful check before the next switch.

Provenance

  • docs/cutover/LOG-deploy-wave.md, line 32 (trendytails backup with 8 uploads files), line 37 (first walk failed 2 /uploads/gallery/*.jpg URLs, read-only mount crashes the entrypoint chown, read-write would chown production files, scratch volume tt_rh_uploads, retry urls=41 FAIL=0 ALLOWED=28).
  • docs/cutover/LOG-cut-kit.md, line 13 (trendytails rehearsal: urls=41 FAIL=0 ALLOWED=28, booking OK).

Comentarios

Aún no hay comentarios.

Iniciar sesión para comentar.