Observability

Three tools, plus two audit tables.

Org-scoped audit logging (FIT-20) is shipped: audit_events written by AuditService, with an owner-facing viewer — see audit-logging and ADR-0018. The platform-admin audit_logs table is a separate, narrower ledger (see Audit logging below).

Sentry

API

Init in apps/api/src/instrument.ts. Imported at the top of main.ts so it loads before NestFactory.

Key behaviour:

Per-request scope is set in LoggingInterceptor (apps/api/src/app/logging.interceptor.ts):

• Sentry.getCurrentScope().setTag('requestId', requestId)
• Sentry.getCurrentScope().setUser({ id: userId }) when authenticated
• orgId when present, via attributeRequestToOrg (apps/api/src/common/sentry-org-attribution.ts): the error tag, plus a span attribute of the same name. The request’s root span gets it directly; every span started afterwards in the request (db, http.client, gen_ai.*, …) gets it from a spanStart hook registered in main.ts. McpAuthService does the same for /mcp, whose org comes from the token, not the URL. Explore: filter or group spans by orgId.

The AllExceptionsFilter extends SentryGlobalFilter so unhandled exceptions are reported automatically.

Entitlement ledger checks (ADR-0024; the ledger is the sole credit authority). All carry subscription ids only, no PII, and a check tag starting entitlement_ledger, which the production Sentry alert rule “Entitlement ledger: drift / unopened / negative counter” routes to email and Linear. What to do about each: Ledger operations.

The scan also pings the HEARTBEAT_URL_ENTITLEMENT_DRIFT Better Stack heartbeat after every completed run (monitoring.md).

Booking-policy rollout signals (ADR-0024, flag booking-policy-rules). Each decision is one structured log line with a check tag (Better Stack) and one server PostHog event (BookingEvents in libs/shared/src/lib/analytics/booking-events.ts, organization group, ids and enums only). Helpers: apps/api/src/booking-policy/booking-policy-reports.ts.

Rows that do not parse never fail a request; each read that leaves one out logs one warn with a check tag, no event: policy_snapshot_invalid (bookingId; the cancel reads the org columns), booking_policy_rule_invalid (orgId, ruleIds; the rules are left out, as if archived), plan_quota_invalid (bucketIds; the quotas are left out). They should never fire: every writer validates. If one does, a row was edited by hand or holds a retired shape.

Waitlist signals (always on, no flag). Same shape: a structured log line with a check tag and a server PostHog event from BookingEvents, organization group, ids and enums only. Every promotion signal is sent after the promoting transaction commits (publishPromotions in apps/api/src/bookings/waitlist-promotion-policy.ts), so a serializable retry or a rollback never reports a seat nobody got. Rules: scheduling-bookings › Waitlist.

Web

@sentry/nextjs configured via the standard sentry.client.config.ts / sentry.server.config.ts / sentry.edge.config.ts pattern (Next.js Sentry SDK). Source maps uploaded at build time when SENTRY_AUTH_TOKEN is set; build still succeeds without it.

AnalyticsUserSync in apps/web/src/providers.tsx ties Clerk user identity into Sentry:

Sentry.setUser({ id: user.id, email: user.primaryEmailAddress?.emailAddress });
Sentry.setTag('posthog_session_id', sessionId);

posthog_session_id lets you cross-link a Sentry issue to the PostHog session replay.

Marketing + minisites

Each has its own Sentry project key in env. Same SDK, smaller footprint.

Env vars

SENTRY_DSN                   API
SENTRY_RELEASE               optional, used for release tracking
NEXT_PUBLIC_SENTRY_DSN       Web (client + server)
EXPO_PUBLIC_SENTRY_DSN       Mobile (taikan-mobile) — distinct project
SENTRY_URL                   default https://de.sentry.io
SENTRY_ORG                   taikan (or taikan1 for mobile)
SENTRY_API_PROJECT           taikan-backend
SENTRY_WEB_PROJECT           taikan-frontend
SENTRY_MOBILE_PROJECT        taikan-mobile
SENTRY_AUTH_TOKEN            CI/Railway/EAS — enables source map upload

Mobile (taikan-mobile)

Separate Sentry project (taikan1/taikan-mobile). Init happens at module top of app/_layout.tsx, before any RN screen mounts. The @sentry/react-native/expo config plugin uploads source maps + debug IDs during EAS Build (needs SENTRY_AUTH_TOKEN as an EAS secret).

Sample rates:

The mobile-replay integration ships UI frames; combined with replaysOnErrorSampleRate: 1.0 you get a visual recording of every crash. iOS dSYMs and Android proguard mapping are auto-uploaded by EAS.

PostHog

Web client

Initialised in apps/web/src/providers/posthog-provider.tsx. Env: NEXT_PUBLIC_POSTHOG_KEY, NEXT_PUBLIC_POSTHOG_HOST (default https://eu.i.posthog.com).

PostHogPageView (apps/web/src/components/posthog-pageview.tsx) sends a $pageview per navigation. AnalyticsUserSync calls posthog.identify(user.id, { email, name }) on Clerk-user changes and posthog.reset() on sign-out.

Mobile client

posthog-react-native. Env: EXPO_PUBLIC_POSTHOG_KEY, EXPO_PUBLIC_POSTHOG_HOST (default EU). Initialised inside useAnalyticsIdentify() (src/hooks/use-analytics-identify.ts) once Clerk loads — distinct id = Clerk user id, mirrors web so the same identified user shows up across both surfaces. Screen views are tracked via Expo Router’s navigation events.

API server-side

apps/api/src/event-tracking/event-tracking.service.ts. Env: POSTHOG_API_KEY, POSTHOG_PROJECT_ID, POSTHOG_HOST (default EU). Used to fire server-authoritative events:

• user_signed_up (from the Clerk webhook handler)
• Payment funnel: checkout_started, form_created, webhook_received, transaction-stage events
• Spotter telemetry: per-turn cost, cache hit ratio, tool counts (apps/api/src/ai/agent/observability/)
• Booking and waitlist decisions (BookingEvents, see the Sentry › API tables above)

Why server-side

The web client can’t be trusted to fire a “payment succeeded” event — it might be a different browser, an offline reconnect, or a payment-provider callback hitting only the API. Critical funnel events fire from the API.

Data warehouse source (Railway Postgres)

PostHog pulls production Postgres into the warehouse so product data can be joined against events in HogQL. Source 01a0e18c-777e-0000-464c-2ef5399c10f3, created 2026-09-27, replacing a dead Neon source that had been failing silently since 2026-02-12.

• Connection: yamanote.proxy.rlwy.net:10911, database railway, schema public, over the Railway public TCP proxy (TLS 1.3). No SSH tunnel, no CDC (wal_level = replica), no direct query — import mode only.
• Credentials: dedicated Postgres role posthog_ro. Login-only, no superuser, default_transaction_read_only = on, statement_timeout = 120s. It holds no INSERT/UPDATE/DELETE and no CREATE on public.
• Tables (17, HogQL prefix prodpostgres_): bookings, class_sessions, class_types, locations, memberships, organizations, payment_transactions, plans, programs, subscriptions, users, workouts, workout_sections, workout_movements, workout_assignments, workout_results, workout_set_results. All incremental on updated_at, primary key id, every 24h at 03:00 UTC. auto_sync_new_schemas is off.
• Deliberately excluded: israel_streets (63,571 static rows — syncing it on full refresh alone would blow the 1M rows/month free allotment), israel_cities, audit_events, ai_*, leads*, member_profiles, form_*, legal_consents, device_tokens.
• PII policy — enforced in Postgres, not in PostHog. The eight tables carrying identifiers or large embeddings are granted with column-level GRANT SELECT (col, …), so the excluded columns are invisible to schema discovery and can never be selected back on by a UI click. prodpostgres_users therefore holds only id, clerk_id, created_at, updated_at, deleted_at, analytics_consent, analytics_consent_at, guided_tour_completed_at — no email, name, phone, national id, or embeddings. Same pattern drops free text and provider payloads from payment_transactions, contact details and join_token from organizations, and pgvector columns from programs / workouts. Any migration that adds a column to one of those eight tables must also add an explicit grant, or the column silently never syncs.
• Rows only ever soft-delete (deleted_at), so incremental merge is safe; filter deleted_at IS NULL at query time.
• Health-alert caveat: PostHog sets should_sync = false and stops scheduling a source after a hard error — that is how the Neon source stayed broken for 5.5 months unnoticed. Nothing in docs/runbooks/monitoring.md watches PostHog Health, so a failure here is still silent. Check https://eu.posthog.com/project/125602/health  when warehouse tables look stale.
• Spaces columns: posthog_ro reads locations and class_types with a table-level SELECT, so locations.concurrency, sort_order, is_default, auto_created, kind, parent_id and class_types.default_location_id / default_area_id need no grant. class_sessions is column-granted: location_id is in the grant, area_id is not, so the owner runs GRANT SELECT (area_id) ON class_sessions TO posthog_ro once after the TKN-479 migrations deploy (not part of a migration). New columns appear in the warehouse only after a PostHog schema refresh of the source.
• class_sessions.location_id means the hall since TKN-479 (ADR-0024 D13): a business’s pre-hall “locations” became areas of one hidden hall, and a session’s old location is its area_id. A warehouse query that grouped sessions by location must group by coalesce(area_id, location_id) to keep its meaning; locations.kind tells halls from areas.

Space context

• API: when SpaceScopeService.resolve accepts a spaceId, it tags the request’s Sentry scope with space.id and space.context=on. A spaceId from a caller who is not a member, or one that belongs to another organization, logs a structured pino warn { orgId, userId | clerkId, spaceId, reason: not_a_member | other_org } (a probing signal; the response is the same 403/404 as always). Malformed, unknown and inactive ids log at debug.
• Splits by space (TKN-467, TKN-464): an accepted groupBy=space tags the scope space.group_by=space (the Overview reads, the Reports snapshot, attendance report, retention, attendance KPIs and exports); an unknown groupBy (400 invalid_group_by) and a “Spaces this week” or Reports “Spaces” (/analytics/spaces/range) read without the context (404 space_context_off) log at debug. Neither is a probing signal: the caller is already a member.
• Every session has a space: DefaultSpace logs info { orgId, spaceId?, reason: no_default | no_active_space } when a business’s default is reassigned or its hidden “Main space” is created at write time. After the backfill both should be rare; a steady stream means a writer is racing a removed default. A class type’s space change logs info { orgId, classTypeId, fromLocationId, toLocationId, moved, overridden, kept }.
• Web: WorkingContextProvider sets the Sentry tag space_context (all | single) on load and on every switch. The space’s name is never sent to Sentry or PostHog.
• Kiosk door: a door request the API accepts tags the Sentry scope space.kiosk=door; the web sets the same tag while a door is on screen and clears it when the door closes.
• Events: see docs/features/space-context/README.md#analytics.

Pino structured logs

Set up in apps/api/src/app/app.module.ts via LoggerModule.forRoot:

• pino-pretty transport in non-prod (colorized, human-readable).
• JSON logs in prod (Railway captures and forwards).
• autoLogging: true — request/response auto-logged.
• genReqId — x-request-id header passthrough, else generates req_<ts36>_<rand>.
• customLogLevel:
  • err || statusCode >= 500 → error
  • statusCode >= 400 → warn
  • else → info
• Serializers strip request to { id, method, url, remoteAddress } and response to { statusCode }.

Slow-request warnings

LoggingInterceptor emits a SLOW warn whenever a request exceeds 800ms (lowered from 3s so the 500–2000ms tail surfaces).

Error log shape

{
  "level": "error",
  "requestId": "req_l2x...",
  "userId": "user_2gT...",
  "orgId": "0193...",
  "duration": 1241,
  "statusCode": 500,
  "error": "...",
  "pgError": "duplicate key value violates unique constraint",  // when err.cause is a pg error
  "stack": "..."  // only for 5xx
}

Health endpoint

GET /health — apps/api/src/app/health.controller.ts. Public + skip-throttle. Returns:

{
  "status": "ok" | "degraded",
  "timestamp": "2026-05-28T...",
  "checks": { "database": "up" | "down", "redis": "up" | "down" }
}

Used by Railway for liveness probing and by uptime monitors.

Bull-Board

apps/api/src/bull-board/bull-board.module.ts mounts the BullMQ admin UI (route name lives in that module). Access is gated to Taikan-admin users.

Use it to:

• Inspect queue depth and failed jobs.
• Retry failures.
• Browse job payloads.

In a production incident, Bull-Board is the fastest way to see whether the push-notification, embedding, or import queues are backed up.

Spotter observability

Spotter ships a dedicated observability service: apps/api/src/ai/agent/observability/agent-observability.service.ts. Tracks:

• Per-turn cost (USD) — derived from token counts × model price.
• Cache hit ratio (RAG + tool memoization).
• Tool counts per conversation.
• Storage growth (snapshot service: storage-snapshot.service.ts).

These flow into PostHog so the admin cost dashboards can show per-org Spotter spend and watch the cache-hit ratio drift.

Audit logging

Two tables, deliberately not merged (see docs/features/audit-logging/data-model.md › Relationship to audit_logs):

• audit_events (libs/db/src/lib/schema/audit-events.ts) — the org-scoped trail: every consequential change inside one gym, written by AuditService, read by the owner through a filterable viewer with CSV export, aged to R2 after 24 months. This is the “who did what” record for member-facing incidents. Behavior, categories and code map: docs/features/audit-logging/, decision: ADR-0018.
• audit_logs (libs/db/src/lib/schema/admin.ts) — the platform-admin ledger: Taikan staff acting across orgs, keyed on actor_clerk_id. Only select admin paths write it (tier changes, user resets, bulk scheduling ops); there is no global interceptor and no org-facing UI for it.

For incident forensics inside an org, start from audit_events; for platform-admin actions, audit_logs; for anything neither captures (migrations, infra), Pino logs (Railway log retention) + Sentry breadcrumbs.

Per-request id propagation

Every request that arrives at the API:

1. Reads x-request-id if present, else generates one (Pino).
2. Sets requestId on Pino’s per-request log context.
3. Tags Sentry scope with the same id.
4. Sets the response header X-Request-Id (LoggingInterceptor).

serverFetch on the web side surfaces it back into Sentry context on failed API calls:

scope.setContext('api_request', { path, status, requestId: res.headers.get('x-request-id') });

When a customer reports a bug, ask for the X-Request-Id from a network tab — it pivots into both Sentry and Pino quickly.