Render standby and cutover

A second home for the API, ready to take production from Railway in about 30 minutes of operator time and ~5–10 minutes of downtime, and a tested way back.

Status: the cutover has been executed — production runs on Render (api.taikan.fit → taikan-api.onrender.com; the Railway production services hold no deployment). The “Standby state” column below describes the pre-cutover standby; deploy.md is the current topology. The Revert section applies only while Railway is kept intact.

Why it exists: on 2026-10-21/22 Railway paused every build and deploy for about 30 hours (security incident, see the disclosure ). The running API never went down, but no fix could ship. The standby removes that single point of failure for deploys.

All commands below are scripts/render-cutover/cutover.sh <phase>. It prints no secret values: credentials are fetched at run time from Render (CLI login, ~/.render/cli.yaml) and Railway (railway login), and never written to disk except the rotated Railway Postgres password in ~/.taikan-cutover/ (chmod 600), which rollback needs.

What exists on Render

Workspace Taikan, region Frankfurt.

• Build: apps/api/Dockerfile, context = repo root, runs as the unprivileged node user. Pre-deploy command pnpm db:migrate; health check /health/live.
• The Dockerfile must never sit at the repo root. Railway auto-detects a root Dockerfile despite builder = "RAILPACK" in apps/api/railway.toml, and its builder rejects the BuildKit secret mount, which failed every Railway deploy until it moved (#624).
• Key Value needs the 1g plan. The API holds ~100 Redis connections per instance (one BullMQ connection set per queue); a rolling deploy briefly runs 4 instances. The 256mb plan caps at 250 connections and failed the rollout.
• Env: a copy of Railway’s @taikan/api variables, with Render’s own DATABASE_URL/DIRECT_DATABASE_URL/REDIS_URL, and CRONS_ENABLED/AUTOMATIONS_ENABLED/FIZIKAL_ENABLED off until cutover. Railway stays the source of truth: run sync-env on every wake-up, because keys rotated on Railway since the copy are stale on Render.
• External DB/Key Value access is closed (empty IP allowlist). allow-me opens it to the operator’s current IP; the IP changes often (mobile hotspot, home), so re-run it before restore.
• Deploy times measured: Render 3–5 min from merge to live. Railway during the incident: 17 min to 1.5 h, two builds failed on Railway’s side.

Wake up (standby → ready, no production impact)

1. allow-me
2. wake — creates the Key Value (1g, noeviction), resumes Postgres and the API, turns auto-deploy back on. Run allow-me again afterwards so the new Key Value accepts your IP.
3. sync-env, then render deploys create srv-dbcpbsi9lscs73em0150 --confirm.
4. preflight — row counts on both sides, Render health, Redis key counts.
5. Optional rehearsal with fresh data: take a read-only dump of Railway and run restore against it (set ~/.taikan-cutover/final-dump-path to the dump), then check /health/ready on https://taikan-api.onrender.com.
6. Optional canary: route one Clerk user to Render through the Cloudflare Worker in scripts/render-cutover/canary-worker/ (wrangler deploy --route "api.taikan.fit/*"). Every write that user makes lands in Render’s copy and is lost at cutover; payments and messages are real. Remove the route to end it.

Cutover

Pick a quiet window. PostHog traffic by Israel hour (launch week): 02:00–05:00 lowest; 23:00 is the latest hour an operator is usually awake. Avoid 04:00–06:15 Israel time (daily charge run at 02:00 UTC and the daily jobs after it), and never cut over in the days right before a large charge run. Announce a ~10 minute window to gym owners. Do not merge PRs with migrations until the new host has been stable for a week.

After it is stable: Cloudflare back to proxied with SSL mode Full; update docs/runbooks/deploy.md, incident-response.md, monitoring.md, AGENTS.md’s Railway references.

Revert

Before real users write to Render: rollback (restores Railway’s original Postgres password; the Railway API reconnects by itself, crons resume) and point Cloudflare api back to the Railway domain, proxied. 2–5 minutes, nothing lost.

After real writes on Render, at any time while Railway is kept intact: revert-suspend → revert-dump → revert-restore → revert-redis → rollback → Cloudflare back to Railway. ~10 minutes of downtime, nothing lost. None of these steps needs Railway’s deploy pipeline.

Keep every Railway service untouched for 1–2 weeks after a cutover.

Back to standby

standby — crons off, auto-deploy off, API and Postgres suspended, external access closed, Key Value deleted.

Cost

Measured on Render’s billing page: about $3.7/day ≈ $110/month with two Standard instances, Postgres 1c-2g and Key Value 1g (Railway Pro was ~$20/month). One instance is not acceptable for production resilience. Possible reductions: smaller Postgres (the DB is ~110 MB), fewer Redis connections per instance (would allow a smaller Key Value), and Render’s migration credits. On standby: suspended services do not bill compute, and the Key Value is deleted.

Security notes from the Railway incident

• Railway variables include the production DB password and public URL, Redis, the pg-s3-backups bucket keys (every historical dump), and every API key. The cutover’s freeze rotates the DB password; everything else follows credentials-rotation.md.
• Cardcom v11 authenticates charges with TerminalNumber + ApiName (+ token); the API password is only needed for refunds and cancellations. The portal’s “IP מורשה” list did not block ApiName calls in testing, does not honor CIDR ranges, and applied to “all users” it locks humans out of the portal. Protection that works: refunds limited to the original deal (no free-standing credit), a new API key (new ApiName) when rotating, and scripts/ops/cardcom-deal-watch.cjs, which flags any Cardcom deal Taikan never recorded.