Skip to Content
Living documentation — last reviewed 2026-05-28
RunbooksAPI web/scheduler split

API web/scheduler split

The API runs as two Railway services from one codebase: a web service that serves HTTP with multiple replicas, and a scheduler service that runs the cron jobs with exactly one replica.

Why

@nestjs/schedule registers cron jobs per process. Every replica of a cron-enabled service runs every @Cron on every tick. Two web replicas would mean two executions of all nineteen scheduled jobs — the distributed twin of the duplicate-ScheduleModule bug found on 2026-08-10, except spread across containers where no in-process guard can detect it.

assertSingleCronRegistration (apps/api/src/common/assert-single-cron-registration.ts) only counts registrations inside one process. It cannot see a second container. The split is the only thing preventing cross-container duplication.

Some jobs would survive duplication — the two charge sweeps hold Postgres advisory locks, and check-in dispatch and auto-no-show claim rows before acting. Roughly fifteen others have no such guard.

Topology

ServiceConfig fileCRONS_ENABLEDReplicasPublic domainMigrations
@taikan/api (web)/apps/api/railway.tomlunset2 (see below)api.taikan.fityes (preDeployCommand)
@taikan/api-scheduler/apps/api/railway.scheduler.tomltrue1noneno

Web ships at 1 replica and is raised to 2 only once the cutover is done. Until CRONS_ENABLED is removed from it, a second web replica would fire every cron twice per tick — the same duplication this split exists to prevent, only spread across containers where the boot guard cannot detect it.

Both build from the same commit and run the same image. Everything that differs is in the two config files plus one environment variable.

Invariants

Breaking any of these reintroduces a bug that is silent in production:

  1. The web service must not have CRONS_ENABLED set. If it does, every cron runs on every web replica.

  2. The scheduler must stay at one replica. numReplicas = 1 in railway.scheduler.toml is not a default to tune.

  3. Only the web service runs migrations. drizzle-kit migrate takes no advisory lock, so two services migrating concurrently can double-apply.

  4. The scheduler gets no public domain. It is not load-balanced and is not sized for request traffic.

  5. CRONS_ENABLED is the only variable that may differ between the two services. They run the same image, so any other difference means one of them is running on stale configuration. Verify with bash scripts/check-env-parity.sh.

One-time setup

Steps 1–3 are dashboard-only — the CLI cannot create a service or set production variables.

  1. Create the service. Railway → Taikan project → New → GitHub Repo → desmotech/taikan. Name it @taikan/api-scheduler.

  2. Point it at the scheduler config. Service → Settings → Config-as-code → set the path to:

    /apps/api/railway.scheduler.toml

    Leave Root Directory empty, matching the web service.

  3. Copy the environment. The scheduler needs the same variables as the web service. Fastest path: web service → Variables → Raw Editor → copy all → paste into the scheduler’s Raw Editor. Then add:

    CRONS_ENABLED=true

    ⚠️ This copy is a point-in-time snapshot with no link back to the source. From here on the two services drift with every one-sided edit, and drift on the scheduler is invisible: nothing serves traffic there, so only a cron can surface it. That is exactly how TAIKAN-BACKEND-46 happened — the rebrand flipped RESEND_FROM_ADDRESS on the web service, the scheduler kept the pre-rebrand sender, and every cron-sent email began failing ten days later when the old domain was pulled from Resend. Nobody had touched either service.

    Any variable change from now on must be applied to both services. scripts/check-env-parity.sh diffs them and fails on anything outside the allowlist; the Railway Env Parity workflow runs it daily.

  4. Remove CRONS_ENABLED from the web service. This is the cutover. Until it is removed, crons run in both places.

  5. Do not generate a domain for the scheduler.

  6. Deploy web first, then the scheduler — web owns migrations, so it should apply schema changes before the scheduler boots against them.

  7. Only now raise web to 2 replicas. Set numReplicas = 2 under [deploy.multiRegionConfig.us-west2] in apps/api/railway.toml and deploy. Doing this before step 4 double-fires every cron.

Verifying the cutover

After both services are live, confirm each cron fires exactly once per tick, from the scheduler only:

# Should show cron activity — one line per job per tick
railway logs --service "@taikan/api-scheduler" --environment production \
  --since 30m --json | grep -c "Check-in dispatch"
 
# Should show NONE
railway logs --service "@taikan/api" --environment production \
  --since 30m --json | grep -c "Check-in dispatch"

The */15 jobs (check-in dispatch, payment monitoring) are the quickest signal — at most a 15-minute wait. Boot logs should also carry, once per process:

[CronRegistration] ScheduleModule registered once — cron jobs will fire once per tick

Rolling back

Set CRONS_ENABLED=true on the web service and scale the scheduler to zero. Crons resume on web. If web is still at two replicas, drop it to one first, or every job double-fires.

  • scripts/check-env-parity.sh — diffs the two services’ variables; run it after any variable edit, and see .github/workflows/env-parity.yml for the daily run
  • docs/runbooks/deploy.md — overall promotion path
  • docs/runbooks/migrations.md — migration policy
  • docs/runbooks/monitoring.md — Better Stack probes; readiness lives at /health/ready, liveness at /health/live
AI incident agentsCI / CD