Testing strategy

Three test types, three commands, two databases, one driver pattern. CI runs the fast gate on every PR (typecheck + path-scoped suites + Playwright, split into parallel jobs on the self-hosted runner) and, every two hours against main, an end-to-end gate followed by a read-only smoke against the live deployment. That smoke also runs on its own schedule every 15 minutes.

The pyramid

The goal: most coverage in unit, behaviour in integration, the user journeys in E2E.

Commands

The canonical entry points are Makefile targets (see make help).

make test-unit-api           # API unit tests
make test-integration-api    # API integration tests (need test DB up)
make test-e2e-api            # API E2E tests (test DB + redis)
 
make test-unit-web           # Web unit tests
make test-integration-web    # Web integration tests (E2E_TEST_MODE on)
make test-e2e-web            # Playwright — boots both api + web
 
make test-local-smoke        # All non-Playwright tests
make test-local-all          # Smoke + Playwright
make test-coverage           # Vitest suites with --coverage, fails below the thresholds
make update-screenshots      # Update Playwright visual baselines
 
pnpm test:smoke:prod         # Prod smoke (needs SMOKE_CLERK_SECRET_KEY / SMOKE_CLERK_USER_ID)

Infrastructure helpers:

make test-db-up              # Postgres on 55432, Redis on 56379
make test-db-down            # Tear down
make test-db-migrate         # Run drizzle migrations against the test DB

The test stack runs on docker-compose.test.yml with ports offset (55432 / 56379) so it doesn’t collide with make dev-up (5432 / 6379). You can keep both running simultaneously.

Databases

The Makefile’s EXPORT_API_ENV macro injects safe defaults (test DB URL, dummy encryption keys, dummy R2 creds) so individual targets don’t need an .env.test to function.

Per-worker isolation (API integration + API e2e)

taikan_test is the migration target and the template, not what the specs run against. apps/api/test/setup/global-db.ts (vitest globalSetup) creates one database per worker — taikan_int_w<N>_test / taikan_e2e_w<N>_test — with CREATE DATABASE … TEMPLATE taikan_test, and drops them on teardown. apps/api/test/setup/worker-db.ts (setupFiles) points each worker at its own database and its own Redis logical database (redis://…/<N>), where the throttler counters and Bull queues live.

Why, and why not fileParallelism: false (which is what stood here before): every spec calls POST /testing/reset → TRUNCATE TABLE <every table>, so the only thing serializing files bought was “no two specs truncate at the same instant”. Work that outlives a test still landed in the next spec’s rows — CI logs show bulk-email Bull retries firing during the two following tests, and admin-platform-billing.e2e.spec.ts intermittently failing expected 201, got 404 in a full run while passing 7/7 alone. Separate databases make that impossible rather than merely rarer, and the suites got ~2-3x faster as a side effect (integration 44s → 14s, e2e 73s → 37s).

Worker count is TEST_DB_WORKERS, else min(cpus, 4) — so the 2-vCPU CI box uses 2 automatically. The suite tag (int / e2e, set by each vitest config) is what lets both suites run at the same time without owning the same database. Names must end in _test: TestingService.assertTestDatabase() refuses the destructive reset otherwise.

Playwright is deliberately not sharded this way: its specs drive one shared API server, so isolating them needs a full API + web + Postgres stack per worker. It stays workers: 1.

Playwright and the dev server (E2E_WEB_MODE)

scripts/test-e2e-web.sh serves next dev by default and a production build under E2E_WEB_MODE=prod. The dev server is the wrong target and prod should become the default — it is not yet, and the reason is worth knowing before you debug a Playwright flake here.

The dev server edits the page while the test is using it. A Playwright failure trace caught it —

[HMR] connected
[Fast Refresh] rebuilding
[Fast Refresh] done in 185ms

— fired mid-test, React remounted, every handle Playwright held became “detached from the DOM”, and the leads board re-rendered as “No leads” with its state gone. It surfaced as “element is not stable” and “<html> intercepts pointer events” on whichever click was running, so it read as a click race for four CI runs while each click got hardened in turn. If you see a detached element or a vanished list in an e2e failure, suspect this before the click.

E2E_WEB_MODE=prod also cut the suite from 14.9 to 10.4 min by removing on-demand route compilation. It is not the default because it changed the timing the specs were tuned against: 1 failure became 6 (leads composer, pipeline create, three session-sheets). The three session-sheets failures were a clock bug in the seed (sessions at now + 24h fall outside the calendar’s rendered hours for a third of the day) and the leads two now retry until the page settles, but the switch has not been run green under prod. The prod path builds with NODE_ENV=production (a test value there makes Next emit a development bundle) and is incremental against apps/web/.next/cache, which CI keeps across runs.

Finishing the switch needs Playwright runnable locally, which needs Clerk dev keys in .env.test (with the .env.test.example placeholder, build and web e2e report SKIP in ci-local.sh). Then: E2E_WEB_MODE=prod make test-e2e-web until green, then flip the default in the script.

Vitest pools and isolation

Vitest isolates every spec file by default: a fresh process (forks) per file, so each file re-imports its whole dependency graph from disk. Measured on this repo that is where the time goes — API unit: import 353 s, tests 5 s; web integration: import 278 s, tests 175 s (4 cores, summed over workers). The API unit suite is the one place a shared registry has proven safe; the others were tried and reverted:

The project membership for the API unit split is computed at config load by tools/vitest/partition-specs.ts, which scans spec files for vi.mock( / vi.doMock(, so a new spec lands on the right side without a list to maintain.

What this means when writing an API unit spec: it shares a module registry with the other non-mocking specs in its worker. Module-level state you mutate — a process.env write, a singleton’s field, fake timers — must be restored in afterEach/afterAll, or the next file in the worker sees it. A spec that passes alone and fails in the suite is usually this. vi.mock is fine: its presence moves the file to the isolated project.

The driver pattern

Every UI/component test uses a co-located driver (*.driver.tsx / *.driver.ts) that encapsulates every screen.* call. Specs never import screen directly. The driver exposes a given / render / get / has / find / click API.

See driver-pattern.md for the full contract and examples.

Why: when a class name, role, or test id changes, you fix it in one place. Specs stay business-readable.

Mocking rules

• vi.mock() calls live in spec files, not drivers. Vitest hoists per-file; mocks declared in drivers don’t apply where they’re imported.
• vi.advanceTimersByTime() wraps in act() to avoid React warnings.
• No real Clerk calls in unit/integration — Clerk hooks are mocked via the helpers in apps/web/src/test-utils/.
• No real API calls in unit/integration — useApi and serverFetch are mocked per-test.

Selectors — hard rule

Use data-testid only for element selection. Never getByText, queryByText, CSS classes, tag names, or structural selectors.

Why: translated strings (en/he/ru) make text-based selectors flaky; structural selectors break on layout refactors.

Exceptions are narrow — getByRole is acceptable for accessibility-affirming assertions (e.g. asserting a button has the right name) and is used in some E2E drivers, but the default is testid.

Async

• Use waitFor and findBy* for anything async.
• No setTimeout-based waits; no arbitrary await new Promise(...) sleeps.
• Use vi.useFakeTimers() + vi.advanceTimersByTime() inside act() when testing time-based behaviour.

Radix overlays

Every Drawer | Sheet | Dialog | AlertDialog must include a visually hidden description component (DrawerDescription, SheetDescription, DialogDescription, AlertDialogDescription). Required for screen-reader compliance — Radix warns in console otherwise, and tests assert no React/accessibility warnings.

E2E specifics (Playwright)

Auth caching

apps/web/e2e/global.setup.ts signs in once per CI run: it mints a single-use sign-in token for E2E_CLERK_USER_ID from the Clerk Backend API and hands it to clerk.signIn({ strategy: 'ticket' }). Result is stored at e2e/.auth/signed-in-state.json via Playwright’s storageState. Specs inherit it through fixtures.

Persona swapping

Specs use a usePersona(role) fixture to flip identity. Mechanism: the x-test-user-id header is set per-request (no Clerk re-sign-in). The API’s AuthGuard test-bypass branch (only on when TEST_AUTH_BYPASS=true) honors the header. The web middleware skips auth.protect() when NEXT_PUBLIC_E2E_TEST_MODE=true.

Data setup

Specs use testApi.seedX() calls that hit /testing/* endpoints exposed by TestingModule (apps/api/src/testing/). The module is only mounted when NODE_ENV !== 'production' && TEST_HOOKS_ENABLED === 'true'. Endpoints seed memberships, plans, sessions, etc., with idempotent SQL.

Feature-flag overrides

Flag-gated features (PostHog, fail-closed) can’t be evaluated in tests — PostHog isn’t wired when NODE_ENV=test. The e2e runner (scripts/test-e2e-web.sh, used by make test-e2e-web locally and by CI’s pnpm test:e2e:web) therefore exports the local override on both sides, same "key:true,key2:false" CSV format:

• FEATURE_FLAGS — read by the API (EventTrackingService.envFlagOverride) when no PostHog client exists.
• NEXT_PUBLIC_FEATURE_FLAGS — bootstraps the web’s posthog-js flags in any build without a PostHog key: next dev, and the production build E2E_WEB_MODE=prod serves, which the script builds with NEXT_PUBLIC_POSTHOG_KEY empty (posthog-provider.tsx; see web architecture).

Every pass exports forms-outstanding-followup:true (required by forms-outstanding-follow-up.spec.ts). A flag whose OFF path the rest of the suite must keep covering (money, access) is forced only for the specs that assert it ON: they are their own Playwright project in e2e/playwright.config.ts, and the script runs one pass per project, against an API and a web server restarted with that project’s flags (and, under E2E_WEB_MODE=prod, a web build of its own; the Nx cache keys the web build on NEXT_PUBLIC_FEATURE_FLAGS). Today that is booking-policy-rules:true for booking-policy.spec.ts (project booking-policy-rules); every other spec runs with the flag off. A spec path or filter starts only the passes it selects, and a later pass writes its report and traces under playwright-report/<project>/ and test-results/<project>/. Flags from .env.test or the caller’s environment are added to every pass; a value the script forces wins on both sides.

Drivers (E2E)

E2E drivers live in apps/web/e2e/drivers/, one per feature: onboarding-driver.ts, coaching-grid-driver.ts, purchase-driver.ts, etc.

Pattern (see driver-pattern.md):

• class FeatureDriver { constructor(private page: Page) {} }
• given.opened() — navigate + wait for the surface to appear.
• when.someAction() — user action, returns this for chaining.
• get.someLocator() — returns a Playwright Locator (assertions live in the spec).

Constants — routes, test ids — come from apps/web/e2e/constants.ts.

What to write when

When in doubt, write the lower-cost test (unit > integration > E2E).

CI gates

Full detail, including the self-hosted runner and how to get a check green again, lives in runbooks/ci-cd.md.

PR (.github/workflows/ci-smoke-tests.yml)

Triggers on every PR; the changes job scopes the suites to what the PR touches (apps/api, apps/web, libs/**, root config files). Five jobs. The self-hosted taikan-ci pool has two 2-vCPU runners, so typecheck, api, and web can overlap after changes; e2e waits for all three to pass. changes falls back to that pool unless CI_CHANGES_RUNS_ON=ubuntu-latest is set.

CI_NODE_RUNS_ON (repo variable, normally unset) lifts typecheck and web onto hosted runners for the day the pool is down; CI_CHANGES_RUNS_ON=ubuntu-latest moves only the fast path dispatcher. See the runbook’s minutes budget before leaving either override on.

Env for the DB-backed jobs: TEST_AUTH_BYPASS=true, TEST_HOOKS_ENABLED=true, CRONS_ENABLED=true, dummy encryption keys. Service containers bind dynamic host ports; DATABASE_URL / REDIS_URL are exported by a step, not job env.

typecheck, api, web, e2e (plus semgrep) are required status checks on main. Skipped api/web still report and satisfy the requirement.

No --coverage on PRs. Coverage collection roughly doubles vitest wall time and threshold failures on a PR are noise about the base, not the change.

Timeouts: changes 5, typecheck 10, api 28, web 25, e2e 60 minutes — sized above a 2-vCPU runner’s real numbers so a timeout means “hung”, never “slow”.

Full (.github/workflows/cd-full-test-gate.yml)

Runs every 2 hours and on workflow_dispatch. main is prod — Railway and Vercel deploy on push — so this is an alarm, not a gate. Two jobs: full-test-gate runs the API e2e suite then Playwright against a throwaway database (both run even after one fails), and prod-smoke then calls cd-prod-smoke.yml against the live deployment. Timeout 35 minutes; expect ~21. Requires E2E_CLERK_USER_* secrets for the cached sign-in.

It runs no unit or integration suites. Those already ran on the PR, on the same code. Re-running them cost roughly 23 of the job’s 38 minutes and, because the web integration suite was the flaky one, the duplicate’s only recent contribution was five consecutive red gates on a commit whose tests were fine. A gate you learn to ignore protects nothing.

It is deliberately not per-merge: it occupies one self-hosted runner, leaving the other for PR checks when both are healthy. Dispatch it by hand after shipping something you care about, or run ./scripts/ci-local.sh --with-e2e locally.

Prod smoke (.github/workflows/cd-prod-smoke.yml)

Runs every 15 minutes, on workflow_dispatch, and as the second job of the full gate — one definition, three ways in. pnpm test:smoke:prod drives apps/web/e2e/playwright.prod.config.ts against SMOKE_BASE_URL, wrapped in a Sentry cron monitor check-in (prod-smoke, or prod-smoke-write for write runs) when SENTRY_DSN is set, so a failed or missing run pages.

Two suites in two orgs. apps/web/e2e/prod/ only reads, signed in as the App Store demo org’s owner (SMOKE_CLERK_USER_ID). apps/web/e2e/prod-write/ runs a real member journey in the Taikan Demo org — an owner publishes a session, then a member buys the free plan, books and cancels — and removes what it created. The write suite runs only inside the full gate: without SMOKE_WRITES=1 the config has no write projects at all. scripts/seed-prod-smoke.ts provisions the plan, the member, notification opt-outs and legal consents; setup, secrets and safety rails are in runbooks/ci-cd.md. Every identity belongs to one test org only — never point one at a customer.

This was a systemd timer on the CI box until it moved here. On the box the schedule, the secrets and every run’s output were invisible from GitHub; now each run is in the Actions tab with its report attached. The interval went 10 → 15 minutes in the move: a run holds a runner for ~3 minutes, and at 10-minute spacing that reserves a third of one of the two boxes, which every PR pays for. Scheduled workflows are also best-effort — GitHub delays or drops them under load — so give the Sentry monitor a check-in margin wide enough to absorb a late tick.

Coverage

Coverage is a ratchet in the full gate, not a PR check.

• make test-unit-api, make test-unit-web, make test-integration-web (and the test:unit:* / test:integration:* pnpm scripts) run without --coverage.
• make test-coverage / pnpm test:coverage runs the API unit, web unit and web integration suites with --coverage (test:coverage:api, test:coverage:web). Reports land in apps/*/coverage/.
• Thresholds live in coverage.thresholds (lines / functions / branches / statements) of apps/api/vitest.unit.config.ts, apps/web/vitest.unit.config.ts, apps/web/vitest.integration.config.ts. They are floors set at the measured value rounded down to a whole percent; vitest fails the run when a metric drops below.
• Nothing in CI enforces them today. The full gate used to, as a side effect of re-running the unit and integration suites it no longer runs. The floors still fail make test-coverage locally, so they are real when you ask; they are not a ratchet until something asks on a schedule. If that matters, give coverage its own workflow — a day of latency costs nothing on a measurement that only moves when someone merges tests, and it keeps a threshold undercut off the path that tells you whether main is broken.
• coverage.include covers all of src/** for the API, src/lib + src/hooks for web unit, and src/components + src/app + src/hooks for web integration; src/testing/** and *.driver.ts* are excluded, DTOs are not.

Common gotchas

• Mock placement — vi.mock() in a driver does nothing. Put it in the spec.
• Timer + act — vi.advanceTimersByTime outside act() produces noisy React warnings that fail strict tests.
• Stale storage state — when clerk auth changes upstream, delete apps/web/e2e/.auth/signed-in-state.json to force a fresh sign-in on next E2E run.
• Forgetting TEST_HOOKS_ENABLED — without it, /testing/* endpoints 404, and testApi.seedX() calls fail.
• Scheduled-job handlers in tests — billing-retry tests call the work methods directly via /testing/*. Handlers carry no CRONS_ENABLED check of their own: that switch only decides whether a process schedules jobs.

Where to read next

• driver-pattern.md — the canonical driver shape.
• qa-contractor-onboarding.md — how external QA picks up per-feature test plans.