Testing strategy

Three test types, three commands, two databases, one driver pattern. CI runs the fast gate on every PR (typecheck + path-scoped suites + Playwright, split into parallel jobs on the self-hosted runner) and the full gate with the coverage ratchet every two hours against main. A separate Playwright smoke runs against production every 10 minutes.

The pyramid

The goal: most coverage in unit, behaviour in integration, the user journeys in E2E.

Commands

The canonical entry points are Makefile targets (see make help).

make test-unit-api           # API unit tests
make test-integration-api    # API integration tests (need test DB up)
make test-e2e-api            # API E2E tests (test DB + redis)
 
make test-unit-web           # Web unit tests
make test-integration-web    # Web integration tests (E2E_TEST_MODE on)
make test-e2e-web            # Playwright — boots both api + web
 
make test-local-smoke        # All non-Playwright tests
make test-local-all          # Smoke + Playwright
make test-coverage           # Vitest suites with --coverage, fails below the thresholds
make update-screenshots      # Update Playwright visual baselines
 
pnpm test:smoke:prod         # Prod smoke (needs SMOKE_CLERK_SECRET_KEY / SMOKE_CLERK_USER_ID)

Infrastructure helpers:

make test-db-up              # Postgres on 55432, Redis on 56379
make test-db-down            # Tear down
make test-db-migrate         # Run drizzle migrations against the test DB

The test stack runs on docker-compose.test.yml with ports offset (55432 / 56379) so it doesn’t collide with make dev-up (5432 / 6379). You can keep both running simultaneously.

Databases

The Makefile’s EXPORT_API_ENV macro injects safe defaults (test DB URL, dummy encryption keys, dummy R2 creds) so individual targets don’t need an .env.test to function.

Per-worker isolation (API integration + API e2e)

taikan_test is the migration target and the template, not what the specs run against. apps/api/test/setup/global-db.ts (vitest globalSetup) creates one database per worker — taikan_int_w<N>_test / taikan_e2e_w<N>_test — with CREATE DATABASE … TEMPLATE taikan_test, and drops them on teardown. apps/api/test/setup/worker-db.ts (setupFiles) points each worker at its own database and its own Redis logical database (redis://…/<N>), where the throttler counters and Bull queues live.

Why, and why not fileParallelism: false (which is what stood here before): every spec calls POST /testing/reset → TRUNCATE TABLE <every table>, so the only thing serializing files bought was “no two specs truncate at the same instant”. Work that outlives a test still landed in the next spec’s rows — CI logs show owner-leads bulk-email Bull retries firing during the two following tests, and admin-platform-billing.e2e.spec.ts intermittently failing expected 201, got 404 in a full run while passing 7/7 alone. Separate databases make that impossible rather than merely rarer, and the suites got ~2-3x faster as a side effect (integration 44s → 14s, e2e 73s → 37s).

Worker count is TEST_DB_WORKERS, else min(cpus, 4) — so the 2-vCPU CI box uses 2 automatically. The suite tag (int / e2e, set by each vitest config) is what lets both suites run at the same time without owning the same database. Names must end in _test: TestingService.assertTestDatabase() refuses the destructive reset otherwise.

Playwright is deliberately not sharded this way: its specs drive one shared API server, so isolating them needs a full API + web + Postgres stack per worker. It stays workers: 1.

Playwright and the dev server (E2E_WEB_MODE)

scripts/test-e2e-web.sh serves next dev by default and a production build under E2E_WEB_MODE=prod. The dev server is the wrong target and prod should become the default — it is not yet, and the reason is worth knowing before you debug a Playwright flake here.

The dev server edits the page while the test is using it. A failure trace for owner-leads-follow-up caught it —

[HMR] connected
[Fast Refresh] rebuilding
[Fast Refresh] done in 185ms

— fired mid-test, React remounted, every handle Playwright held became “detached from the DOM”, and the leads board re-rendered as “No leads” with its state gone. It surfaced as “element is not stable” and “<html> intercepts pointer events” on whichever click was running, so it read as a click race for four CI runs while each click got hardened in turn. If you see a detached element or a vanished list in an e2e failure, suspect this before the click.

E2E_WEB_MODE=prod also cut the suite from 14.9 to 10.4 min by removing on-demand route compilation. It is not the default because it changed the timing the specs were tuned against: 1 failure became 6 (leads composer, pipeline create, three session-sheets). The three session-sheets failures were a clock bug in the seed (sessions at now + 24h fall outside the calendar’s rendered hours for a third of the day) and the leads two now retry until the page settles, but the switch has not been run green under prod. The prod path builds with NODE_ENV=production (a test value there makes Next emit a development bundle) and is incremental against apps/web/.next/cache, which CI keeps across runs.

Finishing the switch needs Playwright runnable locally, which needs Clerk dev keys in .env.test (with the .env.test.example placeholder, build and web e2e report SKIP in ci-local.sh). Then: E2E_WEB_MODE=prod make test-e2e-web until green, flip the default in the script, un-quarantine owner-leads-follow-up.spec.ts.

Vitest pools and isolation

Vitest isolates every spec file by default: a fresh process (forks) per file, so each file re-imports its whole dependency graph from disk. Measured on this repo that is where the time goes — API unit: import 353 s, tests 5 s; web integration: import 278 s, tests 175 s (4 cores, summed over workers). The API unit suite is the one place a shared registry has proven safe; the others were tried and reverted:

The project membership for the API unit split is computed at config load by tools/vitest/partition-specs.ts, which scans spec files for vi.mock( / vi.doMock(, so a new spec lands on the right side without a list to maintain.

What this means when writing an API unit spec: it shares a module registry with the other non-mocking specs in its worker. Module-level state you mutate — a process.env write, a singleton’s field, fake timers — must be restored in afterEach/afterAll, or the next file in the worker sees it. A spec that passes alone and fails in the suite is usually this. vi.mock is fine: its presence moves the file to the isolated project.

The driver pattern

Every UI/component test uses a co-located driver (*.driver.tsx / *.driver.ts) that encapsulates every screen.* call. Specs never import screen directly. The driver exposes a given / render / get / has / find / click API.

See driver-pattern.md for the full contract and examples.

Why: when a class name, role, or test id changes, you fix it in one place. Specs stay business-readable.

Mocking rules

• vi.mock() calls live in spec files, not drivers. Vitest hoists per-file; mocks declared in drivers don’t apply where they’re imported.
• vi.advanceTimersByTime() wraps in act() to avoid React warnings.
• No real Clerk calls in unit/integration — Clerk hooks are mocked via the helpers in apps/web/src/test-utils/.
• No real API calls in unit/integration — useApi and serverFetch are mocked per-test.

Selectors — hard rule

Use data-testid only for element selection. Never getByText, queryByText, CSS classes, tag names, or structural selectors.

Why: translated strings (en/he/ru) make text-based selectors flaky; structural selectors break on layout refactors.

Exceptions are narrow — getByRole is acceptable for accessibility-affirming assertions (e.g. asserting a button has the right name) and is used in some E2E drivers, but the default is testid.

Async

• Use waitFor and findBy* for anything async.
• No setTimeout-based waits; no arbitrary await new Promise(...) sleeps.
• Use vi.useFakeTimers() + vi.advanceTimersByTime() inside act() when testing time-based behaviour.

Radix overlays

Every Drawer | Sheet | Dialog | AlertDialog must include a visually hidden description component (DrawerDescription, SheetDescription, DialogDescription, AlertDialogDescription). Required for screen-reader compliance — Radix warns in console otherwise, and tests assert no React/accessibility warnings.

E2E specifics (Playwright)

Auth caching

apps/web/e2e/global.setup.ts runs a single real Clerk sign-in per CI run using E2E_CLERK_USER_EMAIL / E2E_CLERK_USER_PASSWORD. Result is stored at e2e/.auth/signed-in-state.json via Playwright’s storageState. Specs inherit it through fixtures.

Persona swapping

Specs use a usePersona(role) fixture to flip identity. Mechanism: the x-test-user-id header is set per-request (no Clerk re-sign-in). The API’s AuthGuard test-bypass branch (only on when TEST_AUTH_BYPASS=true) honors the header. The web middleware skips auth.protect() when NEXT_PUBLIC_E2E_TEST_MODE=true.

Data setup

Specs use testApi.seedX() calls that hit /testing/* endpoints exposed by TestingModule (apps/api/src/testing/). The module is only mounted when NODE_ENV !== 'production' && TEST_HOOKS_ENABLED === 'true'. Endpoints seed memberships, plans, sessions, etc., with idempotent SQL.

Feature-flag overrides

Flag-gated features (PostHog, fail-closed) can’t be evaluated in tests — PostHog isn’t wired when NODE_ENV=test. The e2e runner (scripts/test-e2e-web.sh, used by make test-e2e-web locally and by CI’s pnpm test:e2e:web) therefore exports the local override on both sides, same "key:true,key2:false" CSV format:

• FEATURE_FLAGS — read by the API (EventTrackingService.envFlagOverride) when no PostHog client exists.
• NEXT_PUBLIC_FEATURE_FLAGS — bootstraps the web’s posthog-js flags in dev (posthog-provider.tsx).

Currently exported: workout-parse-transformer:true (required by paste-workout.spec.ts). Values set in .env.test take precedence over the script defaults.

Drivers (E2E)

E2E drivers live in apps/web/e2e/drivers/, one per feature: onboarding-driver.ts, coaching-grid-driver.ts, purchase-driver.ts, etc.

Pattern (see driver-pattern.md):

• class FeatureDriver { constructor(private page: Page) {} }
• given.opened() — navigate + wait for the surface to appear.
• when.someAction() — user action, returns this for chaining.
• get.someLocator() — returns a Playwright Locator (assertions live in the spec).

Constants — routes, test ids — come from apps/web/e2e/constants.ts.

What to write when

When in doubt, write the lower-cost test (unit > integration > E2E).

CI gates

Full detail, including the self-hosted runner and how to get a check green again, lives in runbooks/ci-cd.md.

PR (.github/workflows/ci-smoke-tests.yml)

Triggers on every PR; the changes job scopes the suites to what the PR touches (apps/api, apps/web, libs/**, root config files). Five jobs. changes is GitHub-hosted; the rest run on the self-hosted box, which has one runner, so they run one after another and a full run is their sum (~35 min):

CI_NODE_RUNS_ON (repo variable, normally unset) lifts typecheck and web onto hosted runners for the day the box is down; see the runbook’s minutes budget before leaving it on.

Env for the DB-backed jobs: TEST_AUTH_BYPASS=true, TEST_HOOKS_ENABLED=true, CRONS_ENABLED=true, dummy encryption keys. Service containers bind dynamic host ports; DATABASE_URL / REDIS_URL are exported by a step, not job env.

typecheck, api, web, e2e (plus semgrep) are required status checks on main. Skipped api/web still report and satisfy the requirement.

No --coverage on PRs. Coverage collection roughly doubles vitest wall time and threshold failures on a PR are noise about the base, not the change.

Timeouts: changes 5, typecheck 10, api 28, web 25, e2e 30 minutes — sized above a 2-vCPU runner’s real numbers so a timeout means “hung”, never “slow”.

Full (.github/workflows/cd-full-test-gate.yml)

Runs every 2 hours and on workflow_dispatch. main is prod — Railway and Vercel deploy on push — so this is an alarm, not a gate: one step per suite in a single job, with --coverage on the suites that carry a threshold (pnpm test:coverage:api, pnpm test:coverage:web), then Playwright; every step runs even after one fails. Timeout 65 minutes. Requires E2E_CLERK_USER_* secrets for the cached sign-in. It is deliberately not per-merge: it occupies the single self-hosted runner for ~20-30 min and PR checks cannot preempt it, so a per-push gate starved the checks that actually gate merges. Dispatch it by hand after shipping something you care about, or run ./scripts/ci-local.sh --with-e2e for the same suites in ~3 min.

Prod smoke (scripts/prod-smoke/)

Not a GitHub workflow. prod-smoke.timer on the CI box runs scripts/prod-smoke/run.sh every 10 minutes: sync /opt/taikan to origin/main, pnpm test:smoke:prod (apps/web/e2e/playwright.prod.config.ts), wrapped in a Sentry cron monitor check-in (prod-smoke). apps/web/e2e/prod.setup.ts signs in as SMOKE_CLERK_USER_ID via a Clerk sign-in token and stores e2e/.auth/prod-state.json; specs under apps/web/e2e/prod/ only read. The user belongs to a dedicated synthetic org — never point it at a customer.

PR preview deploy

.github/workflows/deploy-pr-preview.yml deploys the API to a shared Railway “preview” environment whenever Vercel finishes its preview build. Not a test gate per se, but used for manual smoke-testing of in-progress branches.

Coverage

Coverage is a ratchet in the full gate, not a PR check.

• make test-unit-api, make test-unit-web, make test-integration-web (and the test:unit:* / test:integration:* pnpm scripts) run without --coverage.
• make test-coverage / pnpm test:coverage runs the API unit, web unit and web integration suites with --coverage (test:coverage:api, test:coverage:web). Reports land in apps/*/coverage/.
• Thresholds live in coverage.thresholds (lines / functions / branches / statements) of apps/api/vitest.unit.config.ts, apps/web/vitest.unit.config.ts, apps/web/vitest.integration.config.ts. They are floors set at the measured value rounded down to a whole percent; vitest fails the run when a metric drops below.
• The full gate enforces them. A red “Coverage ratchet” step means coverage fell below a floor: add tests, or lower the floor deliberately in a PR that says why. When coverage climbs, raise the floors in the same PR that added the tests.
• coverage.include covers all of src/** for the API, src/lib + src/hooks for web unit, and src/components + src/app + src/hooks for web integration; src/testing/** and *.driver.ts* are excluded, DTOs are not.

Common gotchas

• Mock placement — vi.mock() in a driver does nothing. Put it in the spec.
• Timer + act — vi.advanceTimersByTime outside act() produces noisy React warnings that fail strict tests.
• Stale storage state — when clerk auth changes upstream, delete apps/web/e2e/.auth/signed-in-state.json to force a fresh sign-in on next E2E run.
• Forgetting TEST_HOOKS_ENABLED — without it, /testing/* endpoints 404, and testApi.seedX() calls fail.
• Cron handlers in tests — billing-retry tests call cron handlers directly via /testing/*; they require CRONS_ENABLED=true to do real work.

Where to read next

• driver-pattern.md — the canonical driver shape.
• qa-contractor-onboarding.md — how external QA picks up per-feature test plans.