← Blog
✎post del blog

Two switches took 12 and 17 seconds; the night went to permissions

A tall brass railway points lever in a dim signal box, one lever pulled over to the far position, in teal and steel with a warm amber rim light.

The two production switches that ran took about 17 s (knullfix.de) and about 12 s (trendytails), while approvals and access, not the switch, held up the rest of the night.

The two production switches that ran took about 17 seconds for knullfix.de and about 12 seconds for trendytails. The rehearsals for amarbaro.com and organictea.hu had joined the new app up in 4 seconds each. The switch was the short part of the night. The rest went to approvals, a dashboard login and a plan that disagreed with the runbook.

The two switches that ran

knullfix.de's switch ran at 21:49 UTC and took about 17 seconds. Its checks followed the runbook. The reminders job, which sends scheduled mail, was denied by the auto-mode classifier, so its after-switch check stayed unverified.

Trendytails' switch ran at 22:22 UTC and took about 12 seconds. Its reminders job exited with status 1 on the first try, because the CRON_TOKEN was missing from the bare shell. The service unit itself started with exit 0 and reported zero reminders sent and zero failed.

The switch that was refused

amarbaro.org's switch was denied at 22:50 by the auto-mode classifier, under the label "Production Deploy". The denied step would have stopped the legacy atlas stack, moved the edge to a new port, and edited and restarted the tunnel configuration. The record says the switch wave stopped there, waiting for a permission rule or a go from the user.

The same section has a later entry. It records a switch at 22:44, checked at 22:46, with no cache purge needed. The two entries cannot both describe the same switch in the order written. The record does not settle which happened. We report both and leave the question open.

The tunnel that waited for a login

organictea.hu's staging was ready to switch by 23:15. Its tunnel could not move until a dashboard login was invited to the Cloudflare account, and the tunnel stayed blocked after that point in the record.

The mail setup was also wrong for a while. The staging configuration named Resend, and at 23:30 the record corrects the provider to Purelymail. The SMTP password was still empty at that time. The site stayed ready apart from the tunnel and the password.

A plan that disagreed with the runbook

The trendytails plan said the live shop would move to the lab. The runbook kept the joined app on the VPS, in place. At 23:30 we chose in place and recorded the choice in the runbook. The plan had the location wrong. The steps in the runbook were right.

The steps a switch runs

A switch in the runbook is a fixed sequence. It starts with a backup, which reads the database and writes nothing to it. It then moves the revision, deploys the new image, checks the new site's health, checks the shop address and the reminders job, and asks a person to try one signup and one login by hand. No check covers that last step, so it stays with a person.

Two decisions were written into the runbook before the switch ran. For trendytails, the go list required the booking data checked first: all 10 bookings were in the past, the newest from 2026-11-11, so no booking was in flight at the switch. And the switch set the admin default locale to German, with the analytics domain left unset, to match the old container.

What the timings do and do not mean

The two switch times are measured. The times in the record mix UTC and local clock, and not every entry marks which. We quote each time as written and say which zone when the record gives one.

Denials and waits are not timings. They took most of the night's entries, and they do not have durations we can compare with the switches. We do not claim that approvals took longer than the switch in minutes. We claim that they were what the record spends its lines on.

What we still do not know

Whether the knullfix.de reminders job sends mail after the switch. Its check was denied, so the record has no result. Whether the amarbaro.org switch ran at 22:44, was denied at 22:50, or both. The record does not reconcile them. Whether organictea's mail works before the switch, since the password was empty at the last entry we have.

Provenance

  • docs/cutover/LOG-deploy-wave.md, line 8 (amarbaro.org atlas switch denied by the classifier at 22:50), line 9 (a switch recorded at 22:44, checked at 22:46), line 18 (knullfix.de switch at 21:49 UTC, about 17 s; reminders denied, unverified), line 28 (organictea tunnel blocked until the dashboard invite; mail provider corrected to Purelymail at 23:30), line 34 (trendytails kept in place on the VPS at 23:30), line 41 (trendytails switch at 22:22 UTC, about 12 s; reminders exit 1 for the missing token).
  • docs/cutover/LOG-deploy-wave.md, lines 4 and 24 (the 4-second rehearsal joins for amarbaro.org and organictea.hu).

Commenti

Ancora nessun commento.

Accedi per commentare.