Workout Parse — Rollout Runbook

Operational guide for turning the transformer on, watching it, and turning it off. Owner does all of this from the PostHog UI + Railway; no deploy is needed to change exposure.

The flag


Key	`workout-parse-transformer` (source of truth: `PARSE_FLAG_KEY` in `libs/shared/src/lib/parse-schemas/constants.ts`; web catalog mirror: `FeatureFlags.WORKOUT_PARSE`)
Default	OFF
Targeting	Per-org via the `organization` group (the API evaluates `isFeatureEnabled(key, orgId, { organization: orgId })`; the web reads the same flag via `useFeatureFlag`)
Failure posture	Fail-closed — only an explicit `=== true` enables anything. PostHog unreachable, flag unset, or non-prod without an override ⇒ feature hidden and API 403 `parse_feature_disabled`.
Dev / e2e override	`FEATURE_FLAGS=workout-parse-transformer:true` (API) + `NEXT_PUBLIC_FEATURE_FLAGS=workout-parse-transformer:true` (web). `scripts/test-e2e-web.sh` sets both by default.

The flag gates all four endpoints (parse, get, commit, discard) and every web entry point. Turning it off mid-flight is safe: existing ai_parse_jobs rows are kept, in-progress drafts simply become unreachable (403) until re-enabled — no data loss, no partial writes.

Tier is a second, independent gate: workout_builder (Pro/Elite). Flag ON for a Lite org still yields the upgrade card / 403.

Rollout sequence

Migration applied — ai_parse_jobs must exist before the flag goes on (CI/deploy applies migrations; verify the table exists in prod before step 3).
Create the flag in PostHog (project VibraFit): key workout-parse-transformer, group-based release on organization, default off.
Enable for the internal test org only. Run the QA plan’s golden path + non-English + commit scenarios against prod.
Enable for the first design partner (content is Hebrew — expect parse.blocked { reason: 'non_english' } volume; that’s the measured gap, not a bug).
Widen per-org as confidence grows. There is no percentage rollout — org-by-org only, matching the flag’s group targeting.
Rollback at any point = toggle the flag off (instant, fail-closed, no deploy).

Prereqs to verify in the deploy env: ANTHROPIC_API_KEY (extraction) and VOYAGE_API_KEY (resolution embeddings) present with budget headroom; NODE_ENV=production on the API (otherwise PostHog capture and flag evaluation are off ⇒ feature stays closed).

Budget interplay (D9)

Parse spends from the same meter as Spotter: ai_usage_daily, keyed (organization_id, user_id, day).

Before the LLM call: AgentRateLimitService.preCheck(orgId) — checks the monthly backstop first, then the daily burst cap (both from PLATFORM_TIER_MAP.aiDailyBudgetUsdMicros / aiMonthlyBudgetUsdMicros). Breach ⇒ HTTP 429, code ai_budget_exceeded, body carries { period: 'day' | 'month' }.
After the call: AgentCostTracker.recordTurn upserts tokens + cost_usd_micros (same computeCostUsdMicros math as the agent, apps/api/src/ai/cost.util.ts).
A rejected non-English parse is still metered — the extraction call happened.
Consequences to expect: heavy parse use eats the org’s Spotter budget (and vice versa) — one meter by design. Commit/discard never touch the budget (no LLM call). One parse ≈ one Sonnet call with a ~1h-cached system prompt, so cost/parse should sit well under a chat turn once the cache is warm.

PostHog dashboard (manual build — panel list)

All events fan out from parse-observability.service.ts (production only), distinctId = internal user id, $groups.organization = org id — slice every panel by the organization group. Properties are scrubbed: counts and enums only, never text.

Panel	Event(s)	Property / formula
Parse volume	`parse.requested` (trend, count)	Break down by `source` (`paste` \| `freeform_form`); compare with `parse.completed` count for the success rate
Block reasons	`parse.blocked` (trend, count)	Break down by `reason`: `flag` \| `too_long` \| `budget` \| `non_english`. (`non_english` also carries `language`; `budget` carries `period`.) Note: tier rejections happen in the platform-tier guard and do not emit this event; `reason: 'tier'` exists in the type but has no emit site.
Cost per parse	`parse.completed`	Average of `costUsdMicros` (÷ 1,000,000 for USD); secondary series: `inputTokens`, `outputTokens`; break down by `model`
Latency	`parse.completed`	p50/p95 of `durationMs`; `stageTimings.extractionMs` vs `stageTimings.resolutionMs` for stage attribution
Unresolved rate	`parse.completed`	`unresolvedCount / movementCount` (and `suggestedCount / movementCount` as the softer series); `autoCount` for the win rate
Shape confidence mix	`parse.completed`	Break down by `shapeSources` buckets (`grammar` / `llm` / `none` counts per parse)
Multi-workout pastes	`parse.completed`	Filter `multiWorkoutDetected = true`; `workoutCount` distribution (demand signal for FIT-38 import)
Zero-edit rate	`parse.committed`	Share with `zeroEdit = true`; mean `editDistance` as the companion number — the headline quality metric
Correction anatomy	`parse.committed`	Averages of `movementsSwapped`, `movementsAdded`, `movementsRemoved`, `sectionsShapeChanged`, `exercisesCreatedCount`
Commit conversion	funnel `parse.completed` → `parse.committed`	Add `parse.discarded` as the explicit drop-off; `timeToCommitMs` p50 for adjustment effort
Failures	`parse.failed` (trend, count)	Break down by `errorCode` / `stage`; alert-worthy if it trends above a few % of `parse.requested`

(parse.exercise_created is declared in the taxonomy but currently has no emit site — exercise creations are counted via exercisesCreatedCount on parse.committed.)

What to watch in week one

parse.failed rate — extraction schema-validation failures retry once then fail; a spike means the prompt/schema drifted or Anthropic is degraded (also visible as Sentry breadcrumbs, category parse, tags parse.org_id / parse.job_id).
unresolvedCount — high values mean the exercise library (or org-local embeddings) doesn’t cover the org’s vocabulary; check the exercise-enrichment queue is draining.
ai_budget_exceeded blocks — parse sharing Spotter’s meter means a paste-heavy day can starve chat; if it happens in practice, that’s the signal to revisit a separate budget line.
Zero-edit + editDistance on real content — the go/no-go input for graduating to multi-workout import (FIT-38), measured by the eval harness on real pastes (apps/api/src/ai/parse/eval/README.md), not the synthetic corpus.

Incident response

Feature misbehaving (bad commits, errors, runaway cost) → turn the flag off in PostHog. Fail-closed takes effect on the next request; no deploy, no data loss.
Post-hoc forensics: every attempt has an ai_parse_jobs row (input text, draft, payloads, delta, cost, timings, error_code) — see data-model.md.
General escalation: docs/runbooks/incident-response.md; AI budget/monitoring context: docs/runbooks/monitoring.md.