Workout Parse — Operations Runbook
Operational guide for watching the transformer and responding when it misbehaves.
Gating
The only gate beyond auth + staff role is the workout_builder tier feature (Pro/Elite): a Lite org gets the upgrade card in the web and a 403 from the API. Every Pro/Elite org has all four endpoints (parse, get, commit, discard) and every web entry point.
Paste is first everywhere a coach adds a workout, and commit can save to the library (isLibrary) and publish directly (publishTo, for the board/session sheet’s “Publish”). The structured builder itself stays reachable via “Advanced builder”/“Adjust in builder”.
Prereqs to verify in the deploy env: ANTHROPIC_API_KEY (extraction) and VOYAGE_API_KEY (resolution embeddings) present with budget headroom; NODE_ENV=production on the API (otherwise PostHog capture is off).
Budget interplay (D9)
Parse spends from the same meter as Spotter: ai_usage_daily, keyed (organization_id, user_id, day).
- Before the LLM call:
AgentRateLimitService.preCheck(orgId)— checks the monthly backstop first, then the daily burst cap (both fromPLATFORM_TIER_MAP.aiDailyBudgetUsdMicros/aiMonthlyBudgetUsdMicros). Breach ⇒ HTTP 429, codeai_budget_exceeded, body carries{ period: 'day' | 'month' }. - After the call:
AgentCostTracker.recordTurnupserts tokens +cost_usd_micros(samecomputeCostUsdMicrosmath as the agent,apps/api/src/ai/cost.util.ts). - A parse rejected after extraction (e.g.
parse_multiple_sessions) is still metered — the extraction call happened. - Consequences to expect: heavy parse use eats the org’s Spotter budget (and vice versa) — one meter by design. Commit/discard never touch the budget (no LLM call). One parse ≈ one Sonnet call with a ~1h-cached system prompt, so cost/parse should sit well under a chat turn once the cache is warm.
PostHog dashboard (manual build — panel list)
All events fan out from parse-observability.service.ts (production only), distinctId = internal user id, $groups.organization = org id — slice every panel by the organization group. Properties are scrubbed: counts and enums only, never text.
| Panel | Event(s) | Property / formula |
|---|---|---|
| Parse volume | parse.requested (trend, count) | Break down by source (paste | freeform_form); compare with parse.completed count for the success rate |
| Block reasons | parse.blocked (trend, count) | Break down by reason: too_long | budget. (budget carries period.) Note: tier rejections happen in the platform-tier guard and do not emit this event; reason: 'flag', reason: 'tier' and reason: 'non_english' exist in the type but have no emit site. |
| Cost per parse | parse.completed | Average of costUsdMicros (÷ 1,000,000 for USD); secondary series: inputTokens, outputTokens; break down by model |
| Latency | parse.completed | p50/p95 of durationMs; stageTimings.extractionMs vs stageTimings.resolutionMs for stage attribution |
| Unresolved rate | parse.completed | unresolvedCount / movementCount (and suggestedCount / movementCount as the softer series); autoCount for the win rate |
| Shape confidence mix | parse.completed | Break down by shapeSources buckets (grammar / llm / none counts per parse) |
| Multi-workout pastes | parse.completed | Filter multiWorkoutDetected = true; workoutCount distribution (demand signal for FIT-38 import) |
| Zero-edit rate | parse.committed | Share with zeroEdit = true; mean editDistance as the companion number — the headline quality metric |
| Correction anatomy | parse.committed | Averages of movementsSwapped, movementsAdded, movementsRemoved, sectionsShapeChanged, exercisesCreatedCount |
| Commit conversion | funnel parse.completed → parse.committed | Add parse.discarded as the explicit drop-off; timeToCommitMs p50 for adjustment effort |
| Failures | parse.failed (trend, count) | Break down by errorCode / stage; alert-worthy if it trends above a few % of parse.requested |
(parse.exercise_created is declared in the taxonomy but currently has no emit site — exercise creations are counted via exercisesCreatedCount on parse.committed.)
What to watch
parse.failedrate — extraction schema-validation failures retry once then fail; a spike means the prompt/schema drifted or Anthropic is degraded (also visible as Sentry breadcrumbs, categoryparse, tagsparse.org_id/parse.job_id).unresolvedCount— high values mean the exercise library (or org-local embeddings) doesn’t cover the org’s vocabulary; check theexercise-enrichmentqueue is draining.ai_budget_exceededblocks — parse sharing Spotter’s meter means a paste-heavy day can starve chat; if it happens in practice, that’s the signal to revisit a separate budget line.- Zero-edit + editDistance on real content — the go/no-go input for graduating to multi-workout import (FIT-38), measured by the eval harness on real pastes (
apps/api/src/ai/parse/eval/README.md), not the synthetic corpus.
Incident response
- Feature misbehaving (bad commits, errors, runaway cost) → there is no runtime kill switch; the fix ships as a deploy. Existing
ai_parse_jobsrows are never touched by a revert. - Post-hoc forensics: every attempt has an
ai_parse_jobsrow (input text, draft, payloads, delta, cost, timings,error_code) — see data-model.md. - General escalation: docs/runbooks/incident-response.md; AI budget/monitoring context: docs/runbooks/monitoring.md.