CI / CD

GitHub Actions workflows live in .github/workflows/. This document explains what each one does, when it runs, where it runs, and what to do when it’s red.

main is prod: Railway (API) and Vercel (web) deploy on every push to main. There is no pre-deploy gate to hold — the PR checks are the gate, and the full gate is a periodic alarm behind them.

Workflows

“CI_RUNS_ON” means runs-on: ${{ vars.CI_RUNS_ON || 'ubuntu-latest' }}: the job lands on the self-hosted Hetzner box while the repo variable is set to taikan-ci, and falls back to GitHub-hosted minutes when it is blank. Steps that differ between the two (pnpm cache, Playwright browsers) branch on runner.environment.

PR workflow (ci-smoke-tests.yml)

Runs on every PR. Job graph:

The self-hosted taikan-ci pool has three 2-vCPU runners: two on the CX33 taikan-ci-3 and one on the CX23 taikan-ci-2. After changes, typecheck, api, and web are independent and dispatch across the runners; e2e remains terminal so a cheap failure does not spend a browser run. changes uses GitHub-hosted capacity only when CI_CHANGES_RUNS_ON=ubuntu-latest; otherwise it falls back to the pool. Hosted minutes are not the way out: at 6–8 PR pushes a day (peaks of 17), the old 45-minute run would have spent the 3,000 included minutes in under a week. If either runner is offline, the independent jobs serialize and the PR critical path grows by their combined duration.

• changes (~10 s) — dorny/paths-filter decides which suites the PR affects: apps/api/** → api, apps/web/** → web, libs/** or root config → both. It uses GitHub-hosted capacity only when CI_CHANGES_RUNS_ON=ubuntu-latest; otherwise it falls back to the self-hosted pool.
• api (~6 min) — Postgres (pgvector/pg16) + Redis service containers on dynamic host ports (jobs share one box, so fixed 55432/6379 would collide); the “Export service URLs” step writes DATABASE_URL/REDIS_URL to $GITHUB_ENV. Migrates, then unit, integration and e2e as three steps so the summary names the suite that failed. Skipped for web-only PRs.
• typecheck (~2 min) — pnpm exec nx run-many -t typecheck --projects=@taikan/api,@taikan/web plus the brand-asset drift check.
• web (~12 min) — unit → integration. No service containers (the web suites mock the API). Skipped for api-only PRs.
• e2e (~15 min) — Playwright browser suite. needs: [changes, typecheck, api, web] and runs when typecheck passed and api/web each either passed or were skipped — a type error or failing unit test never spends a Playwright run. apps/web/.next/cache is kept under the runner user’s ~/.cache/taikan/next-cache on the box (linked in, since checkout cleans the workspace) so a production build (E2E_WEB_MODE=prod) is incremental; on a hosted runner the Actions cache plays the same role.

With both runners online, typecheck and web overlap api; the browser phase starts once all three succeed. A full gate consumes one runner, leaving the other for PR work. Restore an offline runner before changing test parallelism: the existing job graph already uses the available pool.

CI_NODE_RUNS_ON (repo variable, unset by default) is the emergency valve: set it to ubuntu-latest and typecheck + web run hosted, in parallel with the box, at ~14 hosted minutes per full run. Use it when the box is down or wedged and a review is waiting; unset it after — at this push rate it would spend the month’s 3,000 minutes in about a week if left on.

No job passes --coverage, so PR runs are fast and never flake on thresholds. Nothing on a schedule enforces the floors either — see Coverage below.

Failures annotate the PR: vitest adds its github-actions reporter whenever GITHUB_ACTIONS is set, and apps/web/e2e/playwright.config.ts adds Playwright’s github reporter under CI, so a failing assertion shows on the Files tab at its line. The Playwright HTML report and traces upload as the playwright-report-pr artifact on failure.

Duplicate runs for the same ref are cancelled via concurrency.cancel-in-progress. Budgets (timeout-minutes): changes 5, typecheck 10, api 28, web 25, e2e 60. They are sized above a 2-vCPU runner’s real numbers so a timeout means “hung”, never “slow runner”: a job cancelled mid-run on the box can also wedge the runner’s session (see “jobs queued, runner Idle” below), which costs far more than the minutes a tight budget saves.

Where the suite time goes

Measured locally on 4 cores with vitest’s phase breakdown, before the pool changes described in testing/strategy.md:

“Import” is summed across workers. With the default forks pool and per-file isolation, every spec file re-imports its whole dependency graph from disk (drizzle + schema + NestJS on the API side, react-dom + radix + jsdom on the web side), and on the API unit suite that is 70× the time the tests themselves take. The API unit suite now shares one registry per worker (specs that call vi.mock stay isolated): 4.5 min → 47 s on the box. The same idea was tried on the web suites (vmThreads) and API e2e (isolate: false), passed locally and failed on the box — see the strategy doc for what leaked — so those keep the default pool.

When it’s red

• typecheck: pnpm exec nx run-many -t typecheck --projects=@taikan/api,@taikan/web locally. If the brand step fails, pnpm brand:assets and commit the result.
• api: the failing step names the suite; rerun it locally with the same env (make test-unit-api, make test-integration-api, make test-e2e-api). Most flakes are timing-related — see testing/strategy.md. An API unit spec that passes alone and fails in the suite (or the reverse) after adding a vi.mock, a module-level singleton or a process.env write is a pool-isolation question: see testing/strategy.md.
• web: make test-unit-web, make test-integration-web.
• e2e: make test-e2e-web (E2E_WEB_MODE=prod make test-e2e-web to serve a production build instead of next dev; see testing/strategy.md). Traces, screenshots and video are uploaded as the playwright-report-pr artifact on failure.
• Migration step: the journal-monotonicity bug (migrations.md) sometimes appears as “migration applied successfully” but a follow-up assertion fails because the schema didn’t change. Check libs/db/drizzle/meta/_journal.json for any out-of-order when values.
• Auth bypass surprises: TEST_AUTH_BYPASS=true lets x-test-user-id header replace Clerk. If a test fails because the user isn’t found, the seed step probably didn’t run — check the testApi.seedX() calls in the spec.
• Export service URLs / connection refused: the service container’s health check passed but the mapped port is wrong — check the job’s “Initialize containers” log for the 5432/tcp -> 0.0.0.0:NNNNN line. DATABASE_URL must never be set at job-level env (the job.services context isn’t available there).

Full gate (scheduled + on demand, cd-full-test-gate.yml)

Runs every 2 hours — dispatched by the taikan-ci-clock Worker (see CI clock) — and on manual dispatch (Actions → Main Full Gate → Run workflow). Two questions in order, one job each:

1. full-test-gate on the self-hosted box — migrate, then pnpm test:e2e:api, then pnpm test:e2e:web (a production web build plus Playwright). The second runs even after the first fails (if: ${{ !cancelled() }}), so one red gate lists everything broken on main.
2. prod-smoke — calls cd-prod-smoke.yml against the live deployment. if: ${{ !cancelled() }}, so it runs even when the tree check failed: a broken tree on main says nothing about whether production is up, and “prod is down” is the more urgent answer.

No unit or integration suites run here. They all ran on the PR, on the same code. Re-running them was ~23 of the job’s ~38 minutes and caught nothing: the last five red gates were all the flaky web integration suite on a commit whose tests were fine, and the gate has been green on every run since that flakiness was fixed. The cost of a duplicate alarm is that you stop reading it.

Budget: 35 minutes (expect ~21). The Playwright report uploads as playwright-report-full on every run — if: always(), because a step killed by timeout-minutes leaves the job cancelled, which failure() does not match, and that is exactly the run whose per-spec timings you need. concurrency is cancel-in-progress, so a 2-hourly tick landing on a running gate replaces it instead of queueing. A gate cannot be preempted, but a healthy second runner still accepts PR work; if one runner is offline, the gate occupies the only slot.

When it’s red

A red gate means main — which is already deployed — is broken. It is not tied to one merge, so check what landed since the last green gate. Read the failing job:

• full-test-gate red → a real regression in prod. Same diagnostics as the PR jobs; fix forward or revert the offending merge. Download playwright-report-full for the per-spec detail. For an immediate re-check of a candidate fix, ./scripts/ci-local.sh --with-e2e.
• prod-smoke red while full-test-gate is green → the tree is fine and the deployment is not. See Prod smoke below; the usual causes are Clerk sign-in token minting and a UI change that moved a test id.

Self-hosted runner (Hetzner)

Everything under CI_RUNS_ON executes on the self-hosted taikan-ci pool; only the ubuntu-latest workflows spend the GitHub quota.

The prod smoke has a runner of its own. cd-prod-smoke.yml runs on vars.CI_SMOKE_RUNS_ON (taikan-smoke), which nothing else selects. On the shared pool, a smoke dispatched on time still waited in the queue behind every PR run, so the monitor was late exactly when CI was busiest. With CI_SMOKE_RUNS_ON unset, the smoke falls back to CI_RUNS_ON.

To rebuild the smoke box, run the bootstrap with RUNNER_COUNT=1 RUNNER_LABELS=taikan-smoke.

Every box runs IPv4 only. The bootstrap sets net.ipv6.conf.{all,default}.disable_ipv6=1 (knob DISABLE_IPV6, default 1). The Helsinki box’s IPv6 route to Cloudflare breaks for minutes at a time, mostly between 00:00 and 06:00 UTC: TCP connects, then the TLS handshake stalls. Playwright’s request client and Chromium commit to the first address that connects over TCP, so they never fall back to IPv4. The prod smoke then failed against every dual-stack Cloudflare host (api.taikan.fit, api.clerk.com) with Client network socket disconnected before secure TLS connection was established or ConnectTimeoutError on 2606:4700:… addresses, while app.taikan.fit (Vercel, IPv4 only) passed in the same run and Railway received none of the failed requests. The smoke logs a warning when it runs on a box that still has IPv6 on. Check a box with sysctl net.ipv6.conf.all.disable_ipv6 (want 1). SSH to the boxes over IPv4.

Files: scripts/ci/hetzner-bootstrap.sh (idempotent box setup), scripts/ci/[email protected] (systemd unit template, one instance per runner).

Box layout:

User runner (system user, home /home/runner, in docker group) owns all of it. Node 22 via NodeSource, corepack pnpm pinned to root package.json’s packageManager, Docker CE for the service containers and the semgrep container job.

Setup

1. Hetzner Cloud → New server: image Ubuntu 24.04, type CX33 (shared Intel, 4 vCPU / 8 GB, ~€8.49/mo + VAT + ~€0.60 IPv4 as of the June 2026 repricing; CX32 is the deprecated predecessor) in Falkenstein, Nuremberg or Helsinki — the CX line is only sold there; US locations only offer CPX/CCX at 3–5× the price. api + web + typecheck run concurrently on two runners; if Playwright swaps, step up to CX43. Add your SSH key. Firewall: inbound allow only TCP 22 (from your IP if you have a static one), outbound allow all. No public services run on this box.

   Greyed-out types/locations mean sold out, not disallowed. CX plans go out of stock regularly. If only a CX23 (2 vCPU / 4 GB, ~€4–5) is available: take it, run the bootstrap with RUNNER_COUNT=1, and when CX33 is back in stock use Server → Rescale → “CPU and RAM only” (one click, reversible, no rebuild), then re-run the bootstrap with RUNNER_COUNT=2. Location doesn’t matter for CI. The bootstrap adds a 4 GB swapfile on every size.

   A second CX23 beats waiting for a CX33. The runner needs two vCPU per job, not four per box, so two 2-vCPU servers give the same two runners as one 4-vCPU server — and a CX23 is what is usually in stock, in any location (pick whichever has it; the boxes never talk to each other). Create it, run the same bootstrap with RUNNER_COUNT=1; the runner name is hetzner-<hostname>-1, so it registers alongside the first under the same taikan-ci label and GitHub spreads jobs across both. Each box keeps its own pnpm store, browser cache and ~/.cache/taikan. Do not put two runners on one CX23 (see below) — that is what made two boxes the answer.

2. ssh root@<IP>.

3. The repo is private, so curl raw.githubusercontent.com/... for the script 404s. Copy it from your checkout instead:

   scp scripts/ci/hetzner-bootstrap.sh scripts/ci/[email protected] root@<IP>:/root/

   (both files must sit in the same directory; the script installs the unit from its own directory.)

4. Registration token: repo Settings → Actions → Runners → New self-hosted runner → Linux, copy the value after --token in the “Configure” snippet. It is a one-hour registration token, not a PAT.

5. Clone access for /opt/taikan: either an https URL with a fine-grained PAT (Contents: read) embedded, or a read-only deploy key (Settings → Deploy keys) — the script prints the exact ssh-keygen steps if the clone fails.

6. Run it:

   GH_RUNNER_TOKEN=<token> \
   GIT_CLONE_URL='https://x-access-token:<pat>@github.com/desmotech/taikan' \
   bash /root/hetzner-bootstrap.sh

   Knobs (env): RUNNER_COUNT (2; 1 on a CX23), SWAP_GB (4), DISABLE_IPV6 (1), RUNNER_VERSION (latest release), RUNNER_EPHEMERAL (0), PNPM_VERSION, PLAYWRIGHT_VERSION (from pnpm-lock.yaml). Re-running is safe: configured runners, existing caches and the checkout are skipped/updated, not rebuilt.

   One runner per two vCPU, measured. Two concurrent test jobs on the 2-vCPU CX23 put the box at load 4.0+ (vitest sizes its worker pool from nproc, so each job alone already saturates both cores). Memory was fine — 1.3 GB of 3.7 GB, 89 MB swap — but the contention stretches timing-sensitive specs past their timeouts, which buys throughput with flakes. Deeper queues are the lesser evil: keep RUNNER_COUNT=1 until the box has 4 vCPU.

   On a multi-runner box, each runner is pinned to its own cores. The bootstrap splits nproc evenly across instances with AllowedCPUs (on the CX33: @1 on CPUs 0–1, @2 on 2–3), so each job sees a 2-vCPU machine. Unpinned, both jobs sized their pools to all 4 cores, and the box ran at load 10 on 4 vCPU. Anything that sizes a pool must read nproc or os.availableParallelism(), never os.cpus(), which ignores the pinning. Check a box with systemctl show 'actions-runner@*' -p AllowedCPUs. To take an extra instance out without uninstalling it: systemctl disable --now actions-runner@2 (see the drain note in “Operating it”).

7. Verify: Settings → Actions → Runners shows hetzner-<host>-1 (and -2 with RUNNER_COUNT=2) as Idle with label taikan-ci. On the box: systemctl status 'actions-runner@*'.

8. Flip CI over: Settings → Secrets and variables → Actions → Variables → New repository variable CI_RUNS_ON = taikan-ci.

9. Open a trivial PR (touch a comment under apps/web). Every job in “CI Smoke Tests” should show Runner name: hetzner-… at the top of its “Set up job” log, and pnpm install should take seconds, not minutes.

10. Rollback: delete (or blank) the CI_RUNS_ON variable. Every job falls back to ubuntu-latest on the next run; nothing else changes. Do this before taking the box down for maintenance, otherwise PR checks queue forever.

Persistent vs ephemeral runners

The default is persistent (RUNNER_EPHEMERAL=0): each runner registers once and serves jobs forever; Restart=always in the unit only covers crashes and the runner’s self-update (which exits run.sh). Job workspaces under /opt/actions-runner-N/_work are reused between jobs — actions/checkout cleans the repo dir, pnpm/Playwright are fine with it, and _work growth is bounded by the number of workflows. No credentials live on the box.

Ephemeral (RUNNER_EPHEMERAL=1): the runner de-registers after exactly one job and run.sh exits, so the unit restarts and reconfigure.sh (ExecStartPre) must register it again with a fresh registration token. That needs a fine-grained PAT with repository permission Administration: Read and write (the scope for POST /repos/{owner}/{repo}/actions/runners/registration-token) stored at /etc/taikan/runner-pat:

install -o runner -g runner -m 600 /dev/stdin /etc/taikan/runner-pat <<<'github_pat_…'

Pick ephemeral only if you want a clean workspace per job badly enough to keep an admin-scoped PAT on the box. Switching modes: RUNNER_EPHEMERAL=1 GH_RUNNER_TOKEN=… bash hetzner-bootstrap.sh after rm /opt/actions-runner-*/.runner (the script re-registers whatever is unregistered and rewrites reconfigure.sh).

Operating it

journalctl -u 'actions-runner@*' -f     # live job log lines
systemctl restart actions-runner@1      # after a runner self-update wedges
docker ps                               # leftover service containers = a job that was killed mid-run
docker system prune -af --filter until=168h
du -sh /var/cache/pnpm-store /var/cache/ms-playwright /opt/actions-runner-*/_work

Symptom: jobs queued, runner “Idle”, nothing dispatches. This is the single most disruptive failure this box has; it cost ~80 min of dead queue on 2026-09-04 alone. Settings → Actions → Runners shows the box online, busy=false with the matching taikan-ci label, journalctl shows Listening for Jobs and no errors, nothing in the repo is in_progress, and the queued jobs sit unassigned (runner_id=0, runner_name=""). Both ends look healthy and no work moves.

Cause: the listener’s server-side session still holds a job assignment from a session that died while a job was assigned to it. Two ways to get there, both routine here — a job cancelled mid-run (GitHub cancels a PR’s running jobs when the head branch is deleted on merge), and systemctl restart landing in the same second the listener accepts a job (runsvc.sh escalates SIGINT to SIGKILL immediately, so the assignment is never released).

A restart does not fix it. Observed: restarted 17:46, Listening for Jobs at 17:46:09, then 44 minutes idle with four runs queued. Re-registering took a job in 3 seconds. Escalate in this order:

# 1. Confirm it is actually wedged, not just serialized behind a long job.
#    busy=true, or any run in_progress -> the box is working; a restart would
#    kill a live job. Do nothing.
gh api /repos/desmotech/taikan/actions/runners --jq '.runners[]|"\(.name) busy=\(.busy)"'
gh api "/repos/desmotech/taikan/actions/runs?status=in_progress" --jq '.workflow_runs|length'
 
# 2. Cheap attempt: restart, then wait 60s and re-check for dispatch.
systemctl restart actions-runner@1
 
# 3. The actual cure - re-register, which clears the stale session. Mint both
#    tokens from a machine with `gh` (they are short-lived; nothing is stored
#    on the box).
RM=$(gh api -X POST /repos/desmotech/taikan/actions/runners/remove-token --jq .token)
REG=$(gh api -X POST /repos/desmotech/taikan/actions/runners/registration-token --jq .token)
ssh taikan-ci "systemctl stop actions-runner@1; cd /opt/actions-runner-1; \
  su runner -c './config.sh remove --token $RM'; \
  su runner -c './config.sh --unattended --replace \
    --url https://github.com/desmotech/taikan --token $REG \
    --name hetzner-ubuntu-4gb-hel1-3-1 --labels taikan-ci --work _work'; \
  systemctl start actions-runner@1"

To make step 3 unnecessary, put a fine-grained PAT (repo desmotech/taikan, Administration: read and write) at /etc/taikan/runner-pat, chmod 600. reconfigure.sh then mints its own registration token, so dropping .runner and restarting re-registers unattended - and RUNNER_EPHEMERAL=1 becomes usable, which sidesteps this class of wedge entirely by giving every job a fresh session. Without that file reconfigure.sh exits 1 by design.

Runs whose head branch was deleted keep their remaining jobs in queued forever - they never dispatch and they are not what is blocking you, but they make the queue unreadable. Cancel them: gh api -X POST /repos/desmotech/taikan/actions/runs/<id>/cancel.

Stopping a busy runner drains it. systemctl stop sends SIGTERM, which runsvc.sh turns into the runner’s “finish the current job, then exit” — so the command blocks for as long as the running job needs (TimeoutStopSec is 30 min, above the 28-min api budget). That is correct, not a hang: use --no-block if you don’t want to wait, and confirm afterwards that nothing survived:

systemctl stop --no-block actions-runner@1
pgrep -af 'Runner.Listener|Runner.Worker'   # must be empty for that instance

If a listener or worker is still there with the unit inactive, it is orphaned — it keeps claiming and running jobs with nothing supervising it, and systemctl is-active cannot see it. Kill it by pid. (This is what ExecStart=run.sh + KillMode=process used to cause; the unit now uses bin/runsvc.sh + KillMode=mixed so the cgroup is always cleaned.)

• Upgrading the runner: actions/runner self-updates in place; if GitHub starts refusing an old version, re-run bootstrap with RUNNER_VERSION= unset — it only downloads for directories missing config.sh, so rm -rf /opt/actions-runner-N first (and re-register with a new token).
• Node / pnpm bumps: pnpm follows root package.json packageManager via corepack at bootstrap; re-run bootstrap with the new PNPM_VERSION after bumping it. setup-node in the workflows still selects Node 22 per job.
• Playwright bumps: the job step playwright install chromium downloads the new build into /var/cache/ms-playwright on first use; OS deps were installed once by bootstrap (playwright install-deps chromium) — re-run bootstrap if a new Playwright major needs new libs.
• Disk: the weekly prune handles Docker. pnpm store and browser cache are shared and small; _work holds one checkout per workflow per runner.
• NEVER add a container: job. Container jobs run as uid 0 and bind-mount _work, so they leave root-owned files behind; the next job runs as runner and dies in actions/checkout with could not delete reference refs/heads/main: … packed-refs.lock: Permission denied / EACCES: permission denied, rmdir …. Every job on the box is then red until the workspace is repaired. (This is why smegrep.yml installs semgrep with pip instead of using semgrep/semgrep as a job container.) Service containers under services: are fine — they never touch the workspace. Recovery:

  systemctl stop actions-runner@1
  chown -R runner:runner /opt/actions-runner-1/_work
  rm -rf /opt/actions-runner-1/_work/taikan /opt/actions-runner-1/_work/_temp/_github_home
  systemctl start actions-runner@1

• Security: the runner executes whatever a workflow says, as runner, with Docker access (effectively root). Acceptable for a private single-maintainer repo; never for a repo taking outside PRs. Keep the firewall at SSH-only.

Required status checks (GitHub Pro)

With GitHub Pro, rulesets can require checks on a private repo. Settings → Rules → Rulesets → New branch ruleset:

• Name main, enforcement Active, target branch: default branch.
• Restrict deletions, Block force pushes: on.
• Require a pull request before merging: on, 0 approvals (solo dev). Optionally “Allowed merge methods: squash”.
• Require status checks to pass: add typecheck, api, web, e2e (from “CI Smoke Tests (Web + API)”) and semgrep. Do not tick “Require branches to be up to date before merging” — with serial merges it only forces a rebase + full re-run per PR.
• Leave “Do not require status checks on creation” off.

api and web are path-conditional (if: needs.changes.outputs.api == 'true'). That is fine to require: a job whose if evaluates false still reports to the PR with conclusion skipped, and GitHub counts a skipped check as satisfying a required status check. What would block a merge is a check that never reports at all — which is why every job in ci-smoke-tests.yml is a real job (not a matrix entry) and why the workflow’s on.pull_request.paths filter stays equal to the union of the changes filters: a PR that touches none of those paths never triggers the workflow, and required checks that never run leave the PR unmergeable. If you ever add a path to changes but not to on.paths (or vice versa), PRs will hang on “Expected — Waiting for status to be reported”.

Checks are matched by job name, so renaming a job means editing the ruleset.

Prod smoke monitor

Playwright against the live deployment, from cd-prod-smoke.yml. Two suites, two orgs, never mixed:

The read suite shares an org with Apple’s App Store reviewers, which is why nothing in it may write. The write suite lives in a different org for the same reason. It deletes its session and ends its subscription in afterAll whatever happened, and before starting it sweeps anything a killed run left (sessions titled Smoke session ·…, subscriptions to the smoke plan).

The write journey has no cron of its own. The 15-minute dispatch of cd-prod-smoke.yml carries no inputs, so writes is false and the run is read-only. The journey runs when taikan-ci-clock dispatches cd-full-test-gate.yml (23 */2 * * * UTC: 00:23, 02:23, … 22:23), in its prod-smoke job, which starts once full-test-gate has finished, pass or fail (roughly 20–35 minutes after the tick). A later tick cancels a gate still running, and the write smoke with it. Otherwise it runs only from a manual dispatch: cd-prod-smoke.yml with writes: true, or the full gate.

The re-booking step exists because re-booking a cancelled class once failed in production: the cancelled row stays behind and the second booking collided with its unique key (see incident response › Re-booking a class the member cancelled fails). The journey ends on a cancellation, so it leaves no booking behind.

Without SMOKE_WRITES=1 the config contains no write projects, so a scheduled run or a bare playwright test cannot mutate prod.

Files: apps/web/e2e/playwright.prod.config.ts, prod.setup.ts / prod-write.setup.ts, fixtures/prod-sign-in.ts (shared sign-in), fixtures/prod-write-fixture.ts; pnpm scripts test:smoke:prod and test:smoke:prod:write.

Synthetic traffic is tagged in the User-Agent (TaikanSynthetic/1), never in a custom header: Playwright adds custom headers to cross-origin script loads, the resulting CORS preflight to clerk.taikan.fit is answered with a redirect, and clerk-js never loads.

It used to be prod-smoke.timer on the CI box. That timer never actually ran: there were zero headless sign-in errors in Sentry between its introduction and its removal. The schedule, the secrets and every run’s output were invisible from GitHub.

Setup

1. Seed — scripts/seed-prod-smoke.ts (additive, idempotent, --confirm required):

   railway run --service Postgres -- sh -c \
     'DATABASE_URL="$DATABASE_PUBLIC_URL" pnpm exec tsx scripts/seed-prod-smoke.ts --confirm'

   It creates the free plan “Smoke Free Pass” (in the shop, not on the minisite) and the member [email protected]. It switches off every notification category and channel for that member and the write-side owner. It records each identity’s consent to the latest legal documents its role needs. It prints the write owner’s Clerk id, and the Clerk API call that creates the member — the user.created webhook links that new user to the seeded row by email.

   Re-run it whenever a new legal document version is published. Otherwise every smoke identity is sent to /accept-terms, and the setup fails with a message saying so.

2. Repo secrets:

   • SMOKE_CLERK_SECRET_KEY — prod Clerk Backend API secret (sk_live_…), used only to mint single-use sign-in tokens.

   • SMOKE_CLERK_USER_ID — read suite: the App Store demo org’s owner.

   • SMOKE_WRITE_OWNER_ID, SMOKE_WRITE_MEMBER_ID — write suite, as printed by the seed.

   • SENTRY_DSN — optional; the API project’s DSN, the same value as SENTRY_DSN on the Railway api service. Without it the suites still run and still fail the job; they just can’t page. It must be the DSN, not an auth token: sentry-cli monitors run sends check-ins through the DSN and exits 1 without one, before any test runs.

   • HEARTBEAT_URL_PROD_SMOKE, HEARTBEAT_URL_PROD_SMOKE_WRITE — BetterStack heartbeat URLs, one per suite.

   Optional repo variables SMOKE_BASE_URL / SMOKE_API_URL override https://app.taikan.fit / https://api.taikan.fit.

3. Alerting — BetterStack heartbeats and Sentry cron monitors, both for now. Monitoring is moving to BetterStack, where the scheduler’s crons already report (HEARTBEAT_URL_* on Railway); the Sentry monitors stay until that is done.

   • BetterStack: heartbeats “Taikan prod smoke” (period 15m, grace 30m) and “Taikan prod smoke (write)” (period 2h, grace 2h). A green run pings the URL; a red run pings <url>/fail, which alerts at once instead of after the grace period. A ping that cannot be delivered is a warning, never the job’s result.
   • Sentry: with SENTRY_DSN set, read runs check in to prod-smoke and write runs to prod-smoke-write. The margin, max runtime and two-failure threshold are passed on every check-in, so the UI cannot override them. The org’s plan allows only one active cron monitor, so prod-smoke-write was created disabled — the write journey is alerted on by BetterStack only.

4. Locally: railway run --service @taikan/api -- sh -c 'SMOKE_CLERK_SECRET_KEY="$CLERK_SECRET_KEY" SMOKE_CLERK_USER_ID=… pnpm test:smoke:prod'. Add the write ids and use test:smoke:prod:write for the journey.

Safety rails

• Sign-in stops if an identity lands on /onboarding. Completing onboarding would create a real org in prod; this happened once, with a fresh Clerk user whose email matched no seeded row.
• The write project has retries: 0: a retried journey would buy a second plan before clean-up ran.
• The write fixture retries an idempotent API GET once after a transport failure. It never retries mutations: a disconnected response to a successful purchase is ambiguous and replaying it can create a second subscription.
• A free plan never reaches the payment provider: no Cardcom call, no tax document. Booking emails and pushes respect notification preferences, and the seed switches all of them off. PAYMENT_RECEIPT and EXPIRATION_WARNING ignore preferences; an immediately-ended free plan sends neither.

When it’s red: open the run in Actions and download playwright-report-prod-smoke. A failing read smoke with green Railway/Vercel usually means Clerk or a moved test id. A failing write smoke that leaves rows behind names them in the afterAll error.

CI clock

GitHub’s schedule: trigger is best effort, and on this repo it was unusable. The main gate started 6–104 minutes late and skipped about four ticks in ten. The prod smoke’s schedule didn’t start at all. So neither workflow has a schedule:. A Cloudflare Worker, taikan-ci-clock (deploy/ci-clock/), keeps the time with Cron Triggers and calls GitHub’s workflow_dispatch. A dispatched run starts as soon as a runner is free, and still runs in Actions with its logs and reports.

• Independence. The clock depends on neither GitHub’s scheduler nor the Taikan API, so the smoke still starts while production is down.
• Don’t add schedule: back. A late tick shares the workflow’s concurrency group and would cancel the run the Worker started.
• Changing a schedule means editing WORKFLOWS in worker.js and triggers.crons in wrangler.toml. Keep the Sentry monitor’s --schedule in cd-prod-smoke.yml and the BetterStack heartbeat period in step.
• Deploy: cd deploy/ci-clock && wrangler deploy.
• Token: wrangler secret put GITHUB_TOKEN. It’s a fine-grained token for desmotech/taikan with Actions: Read and write and nothing else. When it expires, dispatches fail with 401 in the Worker’s Cron Events, and the heartbeats go down.
• If a run didn’t start: check the Worker’s Cron Events in the Cloudflare dashboard first. A thrown error there names the workflow and GitHub’s response.

Minutes budget

Expected total: ~150–300 GitHub-hosted minutes/month, against the 3,000 included with Pro. For scale: the PR workflow ran ~45 min per push before the pool changes, at 6–8 pushes a day, so hosted it would have been ~10,000 minutes a month — that is the calculation that put the suites on the box, and it still holds.

Cap the blast radius anyway: Settings → Billing and plans → Spending limits → Actions → $10. If the box goes down with CI_RUNS_ON still set, its jobs queue rather than spend; if you blank the variable to fall back, this limit is the most a runaway month can cost.

Required GitHub secrets

For ci-smoke-tests.yml and cd-full-test-gate.yml:

• CLERK_SECRET_KEY, CLERK_PUBLISHABLE_KEY — a Clerk test instance dedicated to CI.
• E2E_CLERK_USER_ID — the Clerk user the Playwright sign-in mints a ticket for (e2e job and the full gate). The ticket is minted with CLERK_SECRET_KEY, so this id is the whole user-specific credential; there is no email or password.

Repository variables: CI_RUNS_ON and CI_SMOKE_RUNS_ON (see above). Add secrets/variables via Repo Settings → Secrets and variables → Actions.

Prod smoke secrets are repo Actions secrets: SMOKE_CLERK_SECRET_KEY, SMOKE_CLERK_USER_ID, SMOKE_WRITE_OWNER_ID, SMOKE_WRITE_MEMBER_ID and the optional SENTRY_DSN. SENTRY_AUTH_TOKEN is separate: sentry-release.yml uses it to create releases and associate commits, with the same token the Railway api service uses for sourcemaps.

Local “act-like-CI”

To reproduce the PR env locally:

make test-db-up
make test-db-migrate
make test-local-smoke   # unit + integration for API & web (no Playwright)
make test-local-all     # the above + Playwright web e2e
make test-coverage      # the post-merge coverage ratchet

When Actions is unavailable (quota exhausted, billing lapse, outage)

A run that “fails” in 2-4 seconds and whose jobs API 404s never executed — the workflow was rejected before any job was created. That is a billing/quota signal, not a code signal, and re-running it will not help. With the self-hosted runner in place this should only happen to the ubuntu-latest workflows; if the box is the problem, see “Operating it” above.

./scripts/ci-local.sh runs the same gate on your machine. It invokes the same pnpm scripts the workflows invoke, so a pass means what a green check means:

./scripts/ci-local.sh              # api/web suites + semgrep (+ api typecheck, build)
./scripts/ci-local.sh --with-e2e   # + Playwright, the slow `e2e` job
./scripts/ci-local.sh --quick      # skip install/migrate when already current

Every stage runs even after one fails, then a summary lists each stage with its duration — a red gate should tell you everything that is broken, not just the first thing. It refuses to start if DATABASE_URL points at taikan_dev, because the e2e reset truncates its target.

It loads .env.test and fills in the same defaults the workflow env: blocks set (encryption keys, TEST_AUTH_BYPASS, Morning/R2 dummies), so the suites see what they see in CI. A whole local gate is ~3 min on an M-series laptop against ~25 min on the box — when you need an answer now, run it here.

Stages that cannot run locally are reported as SKIP, never as pass, and the footer says the run is not equivalent to a green PR. Today that is build and web e2e when NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY is missing or still the .env.test.example placeholder: next build prerenders Clerk-wrapped routes and dies with “The publishableKey passed to Clerk is invalid”. Put your Clerk dev publishable key in .env.test to enable both.

What it does NOT cover:

• Linux. The runners are Ubuntu; you are on macOS/arm64. Native modules and case-sensitive imports can pass here and fail there.
• A clean checkout. It runs against your working tree, including untracked files. git stash -u first if you want the real thing.
• Managed Semgrep rules. Without SEMGREP_APP_TOKEN it falls back to semgrep scan --config=auto, the open ruleset only — the managed supply-chain policy that blocks on this repo is not evaluated.

If you need the real Linux environment, act (brew install act) runs the workflow YAML in Docker. Slower, and secrets have to be wired by hand.

Adding a workflow

Pre-flight:

• runs-on: ${{ vars.CI_RUNS_ON || 'ubuntu-latest' }} for anything that is a test or check; keep deploys on ubuntu-latest (they need nothing from the box and shouldn’t wait behind test jobs). The box has one runner, so a new check there adds its whole duration to every PR — measure it before adding it.
• Service containers on the box must publish dynamic ports (ports: ['5432']) and export URLs in a step — see the api job.
• No container: jobs — they poison the self-hosted workspace with root-owned files (see “Operating it”). Install the tool in a step instead.
• Use concurrency.cancel-in-progress: true for everything that doesn’t deploy.
• Set timeout-minutes so a hung step doesn’t tie up a runner.
• Pin actions to a full commit SHA with the version in a trailing comment.
• If the workflow needs secrets, document them in this file. If it should gate merges, add its job name to the main ruleset.