AI incident agents

How Taikan uses AI agents to triage and fix production incidents, and the boundaries they operate within. This is a solo-dev setup: the goal is to compress mean-time-to-resolution without standing up servers or handing autonomous write access to production.

Read first: incident-response.md is the human playbook. This runbook describes the agent layer that sits on top of it. The severity tiers, detection channels, and first-five-minutes checklist there still apply.

Status: what’s active vs. planned

Everything below the “Active setup” section is documented so the upgrade path is recorded — none of it is wired up today. Don’t read it as current state.

Active setup: immediate bug-solving (no Seer, no bot, $0)

This is the whole setup right now. It needs nothing built or hosted — just the tools already in .mcp.json and Claude Code on web or mobile.

Sentry email ─▶ open Claude Code (web or phone) ─▶ "fix Sentry issue TAIKAN-123"
                   ├─ pulls issue + stack trace + breadcrumbs via the Sentry MCP
                   ├─ reads the repo + AGENTS.md + these runbooks
                   ├─ writes the fix, runs `make test-local-smoke`
                   └─ opens a PR on a branch (you review + merge)

Why this is the right setup for a solo dev — and why we did not build a Discord bot or buy Seer:

• No webhooks. The Sentry MCP reads issue context on demand; you are the trigger when the email lands. Unattended automation isn’t wanted — agents shouldn’t touch prod on their own.
• No service to maintain. A Discord bot wired to the Claude Agent SDK re-implements a weaker Claude Code and adds a 24/7 service to own. Claude Code already has the Sentry MCP, the repo, the test loop, the PR loop, and phone reachability.
• No Seer required. The MCP supplies the read-context Seer would have; the reasoning is the agent’s.

Keep Discord for human notifications only — free Sentry email, or a free community forwarder (e.g. sentrydiscord.dev) if you want the ping in a channel.

The human-approval boundary (applies now)

This is the one rule that does not bend, regardless of which path you’re on:

• Agents read production freely — Sentry, PostHog, Railway metrics/logs/variables — to diagnose.
• Agents never mutate production autonomously. No railway variables set on prod, no DDL/db:migrate, no scaling, no restart, no rollback without the owner’s explicit go-ahead.
• This mirrors the confirm-on-write model the Spotter agent already uses (ADR-0010) and the project’s standing rule: never run db:migrate without approval.

A fix from an agent is a PR, not a deploy. The deploy path stays human + CI-gated (deploy.md).

Infra incidents (Railway): what an agent can and can’t fix (applies now)

Claude fixes code. Most infra incidents are not code. They split:

• Config-as-code → agent can PR. Connection-pool exhaustion → DB_POOL_MAX; healthcheck timeout → apps/api/railway.toml; a missing retry/backoff; a memory limit in the service config. These live in the repo and are normal PRs.
• Capacity / network / stuck deploy → not a code fix. OOM under load, needing more replicas, a Railway network blip, a wedged deployment. The agent diagnoses via the railway MCP (deployments, logs, metrics, variables) and reports back; the owner (or Railway autoscaling / one-click rollback) acts. Prefer Railway’s native autoscaling + healthcheck-restart as the first line of defense for capacity, not an agent.

What an agent-authored fix MUST satisfy (applies now)

Any PR opened by an agent is held to the same bar as a human PR. The agents enforce this on themselves via AGENTS.md:

• Tests pass — make test-local-smoke at minimum; add a regression test for the failure mode.
• Localization — every user-facing string via the i18n dictionary (en.json, he.json, ru.json). No hardcoded English.
• No console.* statements left in the diff.
• Migration discipline — if the schema changed, _journal.json when values stay strictly monotonic (the 2026-04-18 outage cause). Never db:push.
• Conventional Commits with the right scope (api/web/db/shared).
• Feature flag for behavioral changes — PostHog flag, default OFF, fail toward current behavior.

Future upgrade (NOT set up): unattended Seer automation

Everything in this section is deliberately not built. It’s recorded so the upgrade is a known quantity if the manual loop ever proves too slow for off-hours incidents.

The upgrade swaps “you trigger the fix from Claude Code” for “Sentry triggers it automatically”:

Sentry alert ─▶ Seer (auto RCA, ~6 min)
                 └─▶ hands off to Claude Code (runs in this repo → PR per our conventions)
       └─▶ (optional) Sentry Agent in Linear ─▶ auto-creates the incident issue

What it would take, and the catches:

• Cost: Seer is not on the free Developer plan — it needs Team (~$26/mo) + Seer ($40/active contributor) ≈ $66–70/mo.
• Setup: enable Seer + Autofix on desmotech/taikan, set the code-generation step to hand off to Claude Code, and add a Sentry Automation to run Autofix on alert. Optionally connect the Sentry Agent in Linear.
• Discord catch: Sentry’s native Discord integration only does alerts + issue management (assign/resolve/archive). The conversational “Fix with Seer” trigger is Slack-only. The workaround is to auto-run Seer on alert so there’s nothing to trigger from chat; if interactive chat matters, switch the ops channel to Slack.

What we don’t have yet (gaps)

• No unattended automation — by design (see above); triggering is manual via Claude Code.
• No autonomous prod remediation — by design. Infra mutations are human-approved.
• No agent-driven rollback — Railway rollback stays a manual one-click.
• Carries over the incident-response.md gaps: no PagerDuty, no status page, audit logging (FIT-20) unimplemented.