Incident response

A short playbook for when something is on fire.

Solo-dev caveat: there is no on-call rotation today. Incidents reach the owner via Sentry → email/Slack. If you’re an agent or contractor reading this during an incident, the first step is always contact the owner.

Severity quick-reference

Detection

First-five-minutes checklist

1. Confirm scope. One user? One org? All orgs? Check Sentry’s “Users affected” count and event rate.
2. Recent deploy? Railway → Deployments → top of the list. If yes and the regression is clearly correlated, roll back first (Railway one-click). Diagnose after.
3. Recent migration? Check libs/db/drizzle/meta/_journal.json for new entries. If a migration ran and the failure references a column / table, see migrations.md.
4. External dependency? Cardcom, Morning, Clerk, R2, Railway — check their status pages. A common failure mode is upstream timeout cascading to our 5xx.
5. Tell the owner. Even if you’ve already mitigated.

Common failure modes

API boots, then dies with “relation/column does not exist”

Migration didn’t apply (often the monotonic-when bug). Run pnpm db:migrate against prod. If the migration is genuinely missing, fix the journal and redeploy.

CORS errors on web

ALLOWED_ORIGINS doesn’t include the calling origin. Update Railway env, redeploy API.

Clerk session not honored

Webhook for user create/update is failing (apps/api/src/webhooks/clerk*). Symptom: new sign-ups don’t appear in our users table. Check CLERK_WEBHOOK_SECRET matches the Clerk dashboard’s current value, and that Clerk’s webhook endpoint is pointed at ${API_URL}/webhooks/clerk.

Payment provider webhook idempotency failure

A provider may retry the same webhook; our handlers must be idempotent. If you see duplicate charges/refunds in the DB, check apps/api/src/payments/ for the idempotency key and confirm __drizzle_migrations shows the migration that added that index. See features/payments/behavior.md.

Members refused booking with booking_blocked_debt

A plan in debt keeps booking for the gym’s grace days (organizations.billing_grace_days, default 7); on day N the hourly lifecycle job cancels it for non-payment and the member is refused booking on every plan until the whole debt is paid. The 409 body carries billingDebt (amountInCents, since, reason, payBy, cancelsAt, state). Contract: payments › Unpaid payment.

1. One member who says they paid. Read their subscriptions: status, cancellation_reason, debt_amount_in_cents, debt_since, debt_reason, payment_due_at. Another plan may still owe — paying one plan never clears another. Replacing the card clears nothing; the next successful retry, a clear-debt / member pay-debt, or a desk payment for the debt does. If money arrived through a route that left the debt set, that route is missing from the restore list in the behavior doc — fix it in code; do not null columns by hand without owner approval.
2. Many members right after a deploy. The first lifecycle pass acts on everything already overdue (see billing resilience › Deploy). Compare the cancellations’ cancelled_at with the deploy time and escalate to the owner; any data correction needs explicit approval.
3. Members not told. Debt notices are sent at most once per stage (debt_notice_stage) — see scheduled jobs › Subscription lifecycle.

Re-booking a class the member cancelled fails

Symptom. A member books a class, cancels it, then tries to book the same class again and gets an error (500 on POST /organizations/:orgId/sessions/:sessionId/book). Sentry shows Postgres 23505 on bookings_class_session_id_membership_id_unique. Booking any other class works.

Root cause. bookings is unique on (class_session_id, membership_id) across every status, and a cancel keeps its row (status = 'cancelled', cancelled_at set). BookingsService.book() only looked for a confirmed/waitlisted row before it always INSERTed a new one, so the second booking collided with the cancelled row.

Fix (2026-10-09). book() now reads the member’s row for the session whatever its status: an active row is still “Already booked for this session”; a cancelled or soft-deleted row is revived in place (same id) as confirmed or waitlisted, with cancelled_at, checked_in_at, checkin_method, deleted_at cleared and created_at/updated_at set to the re-booking time. No schema change. Consumers keyed on the booking id were made safe for repeated cancel → re-book cycles: the Fizikal outbox (enqueueOccupy settles the previous occupy/release pair; the reconciler re-mirrors a re-booked booking whose seat was given back) and the T-30 push reminder (one BullMQ job per booking, start time and locale). Regression coverage: apps/api/src/bookings/rebook-after-cancel.int.spec.ts, the multi-plan-booking Playwright spec, and the prod write smoke’s re-booking step.

If it comes back. Check for a new code path that INSERTs into bookings for a member who may already hold a row for that session (grep insert(bookings)); it must revive the cancelled row instead, like book() and the Arbox roster import (onConflictDoUpdate). Do not drop or relax the unique constraint without a migration reviewed by the owner.

Join checkout fails with purchase 409: enroll_conflict

Symptom. Three Sentry issues from one join visit: web purchase 409: {"code":"enroll_conflict"…} from join-continue.tsx, and backend Join checkout failed … step=purchase status=409 from POST /join/:token/checkout-failed. The gym gets an urgent payment follow-up task for the member.

Cause. The member already held the link’s plan as a scheduled subscription (a presale sale waiting for opening day, or a dated start) and opened the join link again. PlansService.purchase did not count scheduled holders, so the request fell through to createSubscription’s exclusivity guard, whose generic enroll_conflict the join flow reports as a failed checkout. No money moves on this path.

Fix (TAIKAN-FRONTEND-Y). purchase refuses active and scheduled holders of a recurring plan with 409 plan_already_held (status, subscriptionId in the body), and the join flow treats that as done. The urgent tasks already filed for such members are false alarms: check the member holds a scheduled/active row for the plan, then close the task.

If it comes back. enroll_conflict from purchase means its holder lookup and createSubscription’s guard have drifted apart again; their status sets must match.

Mobile crash whose Sentry stack names the wrong file

Symptom. A taikan-mobile crash points at code the member was not using (TAIKAN-MOBILE-N, 2026-10-09: “undefined is not a function” in WorkoutChat, raised by tapping Cancel booking on the session screen). The React Native frames read as nonsense, e.g. getHighestPriorityLanes → setIsStrictModeForDevtools.

Cause. The device runs an OTA update (ota_updates.is_embedded_launch: false). An OTA keeps the binary’s release and dist (fit.taikan.app@<version>+<build>), so without that update’s own source maps Sentry symbolicates through the binary’s map. Fixed in taikan-mobile eas-update.yml, which uploads dist/ maps after every publish (needs the SENTRY_AUTH_TOKEN repo secret).

Read the real frame meanwhile. sentry api /api/0/organizations/taikan/events/taikan-mobile:<eventId>/ | jq '.entries[] | select(.type=="exception") | .data.values[].rawStacktrace.frames[-3:]'. Hermes keeps function names, so the last raw frame names the throwing function. The breadcrumbs (navigation, touches, xhr) show what the member did.

Mobile throws “undefined is not a function” on a localized string

Cause. A @taikan/shared dictionary value changed from a string to CLDR plural forms ({ one, two, other }), and an app screen read it through a Record<string, string> cast and called .replace on it. TAIKAN-MOBILE-N was memberBooking.cancelPolicy, plural since shared v0.1.38. Every cancel from the session screen crashed, while the schedule list’s inline cancel, which uses the app’s own strings, worked. Fix: read the typed Dictionary (no casts) and format with plural(); tsc then flags every such site. The taikan-mobile nightly Shared drift workflow typechecks and tests against the latest shared release, so a shape change fails there first.

Spotter agent returns 5xx

Common causes (in order of frequency):

1. ANTHROPIC_API_KEY rate-limit or auth — check the response detail in Sentry.
2. Per-org daily $ cap hit — return is structured, not a 5xx; if it IS a 5xx, the cap-enforcement path threw. Look at apps/api/src/ai/agent/.
3. Postgres timeout on tool execution — verify DB pool isn’t saturated.

Forms PDF generation fails

Puppeteer / @sparticuz/chromium issue. Re-deploy the API (Chromium’s tmp files can corrupt). If it persists, fall back to “PDF unavailable; raw HTML signing record stored” while you diagnose. Forms must still be signable; PDF is a downstream artifact.

Analytics report chart raises “Invalid currency code: projected”

The projected chart series key is a label, not an ISO currency. Its tooltip must format the amount with report.presale.currency; cover that mapping with a regression test.

Prod write smoke reports a TLS socket disconnect

The write journey’s preflight GET reached a transient runner-to-API TLS disconnect. It retries that idempotent request once after one second; do not enable Playwright retries for the journey, because repeating a successful purchase could create a second subscription. If the retry also fails, treat the fail heartbeat as a real production-connectivity incident and inspect the API edge and self-hosted runner. First read the smoke job’s “Check IPv6 is off” step: a warning there means the box has IPv6 on again, which is the known cause of this error (see “Self-hosted runner (Hetzner)” in ci-cd.md), not a prod incident.

Owner can’t onboard (OWNER_INVITATION_REQUIRED, or stuck on /invitation-required)

Creating a gym needs a claimable owner grant. Check Admin → Owner Invitations for the person’s email: expired/revoked → issue a new grant; pending but the user still lands on invitation-required → the grant is probably an unbound new-email grant and the user signed up via a different invitation (no ownerInvitationId metadata) or a different email; grant the existing account instead. Full procedure: owner-invitations.md.

Staff (or any staff-sent) invite “held” — invitee stuck on the pending screen

Symptom: invitee has a pending_invitation membership that never activates (or the signed-in join link answers 403 INVITATION_HELD), GET /users/me returns invitationHeld: true, audit shows membership.invitation_held. Cause: their account is a join-link account whose email was never proven (see features/users-auth/behavior.md). Opening the invitation email does not release it (the ticket would sign anyone into a pre-registered account). Fix: confirm out-of-band that the account really belongs to the person who controls the inbox — if in doubt, have them reset the password first — then Admin → Users → Mark email verified; the next /users/me activates the membership.

Join link says “this email already has an account” (or a visitor got “check your email” instead of signing up)

This is expected; see Who receives a join-link sign-up ticket. join.blocked with reason: 'account_exists' means the address has a login, so the person should sign in and the button brings them back to the link. “Check your email” without the flag means the address belongs to a profile a gym created (an import, a staff invitation, a named guardian), so the link went to that inbox. A burst of 503s from request-invite means the Clerk user lookup is failing (logged as Clerk account lookup failed for a join request). Never work around either case by handing out a ticket: a join ticket for an address that already has an account signs the holder straight into it.

Signed-in user stuck on “Confirm it’s your email” (403 EMAIL_OWNERSHIP_REQUIRED)

Symptom: GET /users/me returns 403 { code: 'EMAIL_OWNERSHIP_REQUIRED' }, the web shows email-ownership-required, and the API logs Unproven login refused a gym-created profile. Cause: the login came from a join link handed to the browser (email unproven), and a gym has since filed someone under that address (import, desk registration, staff invitation, another gym). See users-auth. The row is deliberately not linked.

Fix:

1. Confirm out-of-band that the person controls the inbox. If in doubt, have them reset the password first.
2. Open the row in Admin → Users. It has no Clerk id; the page shows the waiting login as Unproven.
3. Click Mark email verified. Their next request links the row.

If it’s not them: leave it, and delete the stray login in Clerk.

Staff told “This email already tried to register” (422 STAFF_REGISTRATION_UNVERIFIED)

Desk registration, lead conversion or an import row was refused because the email belongs to a login that never confirmed it (see memberships). Each attempt is recorded as membership.registration_refused, with the staff member, gym, IP, path and attempted role.

• Import: the import toast lists the refused addresses, and the job’s results.members.unverified has them too. Staff invite each one; the same two cases below apply.
• Someone who already belongs to another gym: their own app shows a banner naming the inviting gym, and that gym sees Awaiting email verification on the pending invitation. Same procedure.
• The person recognises the registration (they used a join link before). Staff send an invitation. When the person opens it they see “Confirm it’s your email”. Confirm out of band that they control the inbox, then click Mark email verified. Their next request links the account and accepts the invitation.
• The person doesn’t recognise it. Treat it as a suspected squatter and investigate below before verifying anything.

Investigating a suspected squatter

On a public join form anyone can type any address. When the sign-up ticket goes back to the browser, the resulting login is unproven, and it may belong to whoever typed the address rather than its owner. Investigate before clicking Mark email verified, and whenever a gym reports a hold or refusal (membership.invitation_held, membership.registration_refused, EMAIL_OWNERSHIP_REQUIRED). Open the account in Admin → Users and read the Email ownership trail card (GET /admin/users/:id/ownership-trail, which takes the DB id or a Clerk id; contract in users-auth).

Likely squatter: a join-link sign-up with the ticket in the browser, no proof, and sessions from an IP or location the address’s owner (or the gym) does not recognise. The join form’s consent IP will match those sessions, not the owner. Holds or refusals at gyms the person never approached point the same way. Confirm with the gym and, out of band, with the owner.

If it is a squatter:

1. Do not click Mark email verified.
2. Delete the login. If the card shows waiting on this row (refused), the row belongs to the real person: delete only the login, in the Clerk Dashboard (Users → the Login id from the card). Otherwise the row is the squatter’s own: Admin → Users → Delete Account. That deletes the Clerk identity (ending its sessions), cancels the memberships it obtained, and frees the address.
3. Revoke anything else it obtained. Ask each gym in Join-form submissions to revoke the squatter’s pending join invitations. Leave held staff invitations alone: they belong to the address’s owner, who can accept them once the squatter’s login is gone.

Follow-up if the card gets slow: the events section scans audit_events across orgs, bounded at 100 rows, because no index leads with target_type. Add (target_type, target_id, created_at).

Import fails with the member-limit message

The whole import was refused up front because it would exceed the tier’s active-member cap (import_jobs.status='failed', localized errorMessage naming limit/current/requested). Nothing was imported. Upgrade the plan or import a smaller file; re-running a file does not count members already on the roster.

Bulk invite reports rate_limited

Clerk allows 100 invitations/hour per instance. The batch stopped at the first 429 and the remaining targets are failed: rate_limited. Wait an hour and re-run; already-invited members are skipped.

Bull jobs stuck

/admin/bull-board UI shows queues. Pause a queue, drain failed jobs, resume. If a job is poisoning the queue, move it to a “graveyard” queue with bullmq and unblock the rest.

Redis “Command timed out” on one replica

ioredis rejects a command after 3 s (commandTimeout, redis/redis.module.ts). A short blip on one replica (2026-10-09: ~25 s on one of two, Redis server logs clean, other replica unaffected) is a network or replica stall, not a Redis outage — check whether the other replica’s requests in the same window were fast, and the Redis service’s own logs, before touching Redis. The rate limiter fails open after 200 ms by design, presigned-URL reads fall back to signing without the cache, and socket presence never touches Redis. If a new Redis caller turns a blip into a 5xx or an unhandled rejection, make it fail open the same way. Sustained timeouts on both replicas are a real Redis incident.

Sentry: “member activation decisions missing from Redis”

The member activation gate’s reconcile sweep found held new-member activations whose BullMQ decision job was gone (Redis flushed, evicted, or down when the hold was placed). The sweep has already decided them from member_activation_holds, so members got their welcome late, not never. Check Redis health and maxmemory-policy (it must not evict BullMQ keys); extra.lost is how many. Nothing to replay by hand. Details: staff-newsletter behavior.

Minisite draft preview is blank or “could not load”

The public site is fine; only the dashboard preview fails. Probe the renderer directly — the method matters, because the two failures look nothing alike:

curl -sS -D- -o /dev/null -X POST https://<minisite-host>/preview --data 'ticket=x'
curl -sS -D- -o /dev/null      https://<minisite-host>/preview

A bodyless 403 on the POST while the GET returns 405 with a full CSP means the request never reached the route. That was the original defect: the middleware refused POST /preview unless MINISITE_PREVIEW_DASHBOARD_ORIGIN was set and matched, and that variable is a local override no deployed environment sets — so the gate could never open, in any environment, ever. The origin is now checked inside the route, after redemption, against the origin the API minted into the snapshot. If this shape returns, look for a new gate that runs before redemption and depends on configuration.

frame-ancestors 'none' in the CSP means the route ran but had no dashboard origin, so the browser refuses to frame the page and every postMessage targets the renderer’s own origin — the minisite-preview-ready handshake is dropped and the editor reports a generic timeout after 30s. The origin is not configured on the renderer: the API mints it from FRONTEND_URL in issuePreviewTicket. Check in order:

1. FRONTEND_URL on the API is the dashboard origin (scheme + host, no path).
2. The API is deployed with the snapshot field. A renderer deployed ahead of the API falls back to MINISITE_PREVIEW_DASHBOARD_ORIGIN, which is normally unset. Deploy the API first.
3. MINISITE_PREVIEW_SERVER_SECRET matches between API and renderer, and NEXT_PUBLIC_MINISITE_PREVIEW_ORIGIN on the web app points at the renderer.

The route answers 503 studio.previewOriginMissing rather than serving an unframeable page, so the symptom names its own cause.

Before blaming Vercel: a bodyless 403 looks exactly like an edge firewall rule, and it is not one. Confirm by comparing cache-control on POST /preview against POST /some-nonexistent-path — our middleware sets private, no-store, Vercel’s own responses do not.

Minisite “join” / pricing buttons lead to a 404

Check what the resolve actually advertises, then try it:

curl -s "https://api.taikan.fit/minisites/resolve?host=<site-host>" | jq -r .platformData.joinUrl
curl -s -o /dev/null -w '%{http_code}\n' "https://api.taikan.fit/join/<token>"

404 Join link not found means the site is advertising a token the join page does not resolve. The CTA is built from join_links — a row that is active, not revoked, and carries no stored plan. organizations.join_token is the superseded pre-join_links capability and is never a source: an org created after that migration holds a value there that no page has ever answered to.

So the live causes are, in order:

1. The org’s only plan-less link was revoked or paused in the dashboard. The site then omits joinUrl entirely and the pricing cards fall back to the contact CTA — correct, but the owner has lost their buy button. Create a general join link.
2. The org has only campaign links (each carrying a plan_id). Those are deliberately not advertised: a link’s stored plan wins over the ?plan= the pricing card appends, so one would send every visitor to that campaign’s offer whichever card they pressed.
3. organizations.join_enabled is off — the org-wide master switch, which 404s every link regardless of its own state.

Brand colours look wrong on the site or in the app

Read the published palette first; most reports are a derivation, not a loss:

curl -s "https://api.taikan.fit/minisites/resolve?host=<site-host>" | jq '.appearance'

• “That is not the colour I picked.” Fills are exact; TEXT roles (links, eyebrows, outline buttons) are walked to 4.5:1 against the derived page background by readableBrandColor. The brand editor shows the adjusted shade next to the chosen one, so compare there before treating it as a bug.
• Dark mode specifically. The scheme-dependent tokens are --brandReadableLight/Dark and --focusLight/Dark; --brandReadable and --focusColor are chosen from them in CSS. Anything scheme-switchable that is passed through Astro’s define:vars lands as an inline style on <html> and can never be overridden by the dark :root block. Keep switchable names out of define:vars.
• Button text differs between the website and the app. Both call onBrandFill in @taikan/shared. If they disagree, the mobile repo’s generated branding.ts copy is stale — resync it.
• The accent does nothing on the website. By design: the site derives its accent roles from the primary, and the accent colour reaches only the focus ring, falling back to the brand colour when it has no chroma left. The accent is applied in full by the native app.
• Colours reverted to the platform default. A failed PostHog evaluation now reuses the last answer that process received, so suspect a cold process plus an outage.

Post-incident

1. Note the date in docs/decisions/ if the incident changes how we work (e.g., the migration outage prompted the _journal.json monotonic-when discipline in CLAUDE.md).
2. Add a regression test under the relevant apps/api/src/<module>/__tests__/ or apps/web/e2e/specs/.
3. Update this runbook with the new failure mode.

What we don’t have yet (gaps)

• No PagerDuty / Opsgenie. Sentry-only.
• No structured incident retro template (see TODO above).
• No status page.
• No automatic rollback on health-check failure. Railway rolls only on manual click.
• Org-scoped audit logging (FIT-20) is shipped — audit_events, see docs/features/audit-logging/ — so “who did what” inside an org is reconstructable from the trail. Platform-admin actions are only in audit_logs for select writes, and migrations leave nothing beyond __drizzle_migrations.created_at; for those, Sentry breadcrumbs + Pino logs remain the only evidence.