Incident 2026-10-26: Cardcom outage holds a member charge


Status	Resolved. Remediation merged and live the same day (#643 ).
Severity	Low as it happened: one 2.00 ILS test renewal on Taikan’s own test organization. The same event six days later, on Kinetics’ first live collection of 100+ members, would have been Sev-1.
Duration	02:01–07:18 UTC (04:01–09:18 Israel). Detection to resolution 5 h 17 min, of which about 3.5 h was Cardcom down and the rest was waiting for a human holding a vendor confirmation.
Money	None moved incorrectly. No double debit was possible at any point.
Vendor	Cardcom, ticket #649495. Root cause and the operational questions below are still unanswered by Cardcom at the time of writing.
Runbook	Cardcom outage / unknown charge

Summary

Cardcom’s v11 API and merchant portal returned HTTP 500 to everyone for roughly three and a half hours overnight. Taikan’s member-billing catch-up sent one charge into that window: a deliberate 2.00 ILS test renewal, placed six days before Kinetics’ first live collection to rehearse exactly this. The charge was recorded as unknown, the test organization’s terminal was held, and every five-minute recovery lookup got the same 500 until Cardcom came back. Cardcom then answered “not found” (code 9998), which by design never releases a hold on its own. The owner released it through Review billing at 07:12 UTC with Cardcom’s written confirmation in hand, authorized one technical retry under the same external id, and the retry settled at 07:18:01 (Cardcom transaction 267721373, receipt #4).

The money protocol worked as designed. What the rehearsal exposed was everything around it: no log said why the charge was unknown, the gym was not told, the only release path was one owner per gym with a vendor ticket, every organization would have sent one charge into the outage before its own terminal blocked, and one hold would have produced one Sentry issue per member queued behind it. All of that was addressed the same day.

Impact

One 2.00 ILS test charge settled five hours late. No member was affected; the card belongs to the test user.
One organization held: Taikan’s test org. Kinetics was not yet collecting.
Alerts fired on both channels: the Better Stack heartbeat Taikan member billing catch-up in #alerts at 02:15 UTC, and Sentry issues TAIKAN-BACKEND-7V through 80 (several of them artifacts of the pre-#641 reporting, see below).
Owner time: about five hours across the night and morning, including the vendor ticket.

Timeline

Times UTC; Israel is UTC+2.

Time	Event
02:01:00	Catch-up charges the test renewal. Cardcom answers HTTP 500 in about 200 ms. Recorded `unknown`; terminal held. Sentry `7V`/`7W`.
02:15	Better Stack heartbeat alert in `#alerts`.
02:15–05:30	Every recovery lookup gets HTTP 500. Until 04:11 the adapter swallowed the error body, so nothing said why.
04:06–05:04	Observability shipped during the incident, each merged without waiting for CI: #640 (log every Cardcom call and every hold decision, keep the cause), #641 (one Sentry report per endpoint and failure kind per ten minutes), #642 (no unlabeled Sentry events).
04:15	First lookup after #640 shows Cardcom’s IIS 500 page: cause confirmed from our side. Cardcom’s portal returns the same page.
04:21:50	Ticket #649495 opened by email with a portal screenshot. Auto-acknowledged 21 seconds later with support hours: Sunday to Thursday 08:30–17:30, Friday 08:00–12:00 Israel.
05:30:56	Sentry `7Y` (the throttled `Cardcom API call failed`) escalates.
05:35	Cardcom answers again: `9998 not found`. No money had moved.
06:43:57	First human reply from Cardcom support (08:43 Israel, thirteen minutes after support hours open): there was a sweeping failure for all Cardcom customers, it has been handled, please refresh and log in again. No cause, no window, no guidance on charges.
06:45:49	Owner replies: charges were due and failed; will Cardcom re-run them, and what should the merchant do now.
07:12:14	Owner releases the hold (Review billing, confirmation reference: the ticket). The server’s final lookup still says not found.
07:12:28	Owner authorizes the retry. Until 07:17 the log reads `collection_paused_by_owner`, a mislabel for “retry scheduled”.
07:18:01	Retry charges 2.00 ILS under the original external id. Cardcom transaction 267721373; receipt #4 emailed. Catch-up reports nothing due at 07:19.
07:53	PR #643 opened with the scale remediation.
10:43	#643 merged. Live on Render at 10:53 after a no-op migration step.
11:52	First hour live is clean: no circuit trips, no holds, no withheld heartbeat, no Cardcom call failures.
12:04:50	Owner’s follow-up to Cardcom: no reply in five hours on a critical matter; asks for the cause, why it was not on their status page, and what to do with failed charges. Unanswered at the time of writing.

Root cause

A Cardcom-side platform outage. Their v11 API and merchant portal returned an IIS HTTP 500 page for every request from before 02:01 UTC until about 05:30 UTC. Cardcom confirmed in the ticket that it affected all their customers and gave no cause. We found no notice of it on a Cardcom status page, before or after.

Contributing factors on our side, none of which caused the hold but all of which made it cost more:

The nightly collection window (04:00 Israel) falls outside Cardcom’s support hours, so a held charge waits for the morning no matter how fast we are.
There was no platform-wide breaker: each organization’s terminal stops only after its own first unanswered charge.
The only release authority was the organization owner, with a vendor confirmation: a product decision made for one gym that does not survive thirty.
The adapter discarded the provider’s error body, so the first two hours of diagnosis ran blind.

What worked

The durable claim made a double charge impossible: the retry reused the original external id and Cardcom was asked to return the original if it existed.
One unanswered charge held the whole terminal; nothing else on that terminal was attempted.
9998 not found right after an outage did not release the hold, which is correct: “not found” is not “not debited”.
Both alert channels fired within fifteen minutes.
The owner flow (check provider, release on confirmation, authorize retry) worked on the first attempt.
Observability could be shipped and deployed mid-incident, three times, with Render rolling each one out in under five minutes.

What failed and what changed

Gap	Change (all in #643 unless noted)
Only the gym owner could release, with a vendor ticket in hand. At scale that is one ticket and one owner per gym.	Platform admins run the same decisions, singly or in bulk on one confirmation, from the admin app’s Held Charges page (ADR-0026).
The gym was told nothing. Launch gate 4 of the billing-resilience runbook (owner notification during a terminal-wide outage) had never been exercised.	Action Center row `finance.billing_attention`, urgent, linking each member’s Review billing.
No log said why a charge was unknown, why a row stayed held, or what the lookup returned.	#640–#642, live during the incident.
Every organization sends one charge into an outage before its own terminal blocks.	Platform-wide Cardcom circuit: a transport failure on any Cardcom call pauses every protected collector for two minutes; the first collector afterwards proves recovery with a read-only lookup before any debit. Desk and debt charges are refused while it is open.
One hold alerts once per subscription waiting behind it: about 99 Sentry issues for Kinetics.	Organization-scoped holds alert once per organization and reason.
A scheduled retry inside its wait paged as `collection_paused_by_owner`.	`retry_scheduled`, logged at info, never alerted.
Cardcom’s coded “not found” counted as an API failure after Cardcom had recovered.	A non-2xx carrying Cardcom’s JSON envelope is an answer: logged, never reported as an outage, never trips the circuit.
Uneven handling of Cardcom’s other answers across endpoints: a 429 passed as an empty answer on some paths, the legacy charge path reported a timeout as a failed charge (which callers retry), early renewal answered a generic 500, refunds reported a coded refusal as unknown, terminal provisioning logged the new terminal’s secret in the clear.	Every answer classified the same way on every endpoint; the runbook carries the matrix.
No runbook.	Cardcom outage / unknown charge.

No feature flags were used: protected collection had exactly one organization with due rows in production, and the one money-path change (the circuit) trips only on transport failures, which already produced a hold, and fails toward current behavior without Redis.

Vendor response

Ticket opened 04:21:50 UTC. First human reply 2 h 22 min later, thirteen minutes after their support hours opened. The reply confirmed a platform-wide failure and that it was fixed, nothing more.
The merchant’s question at 06:45 UTC (will you re-run charges, what should we do) had no answer after five hours of business time. A follow-up asking for the cause and the status-page omission went out at 12:04 UTC.
The questions that decide our launch posture remain open, all listed in the runbook’s support template: whether a request that got a 500 can still have debited the card; whether 9998 is final right after an incident; whether resending the same external id is guaranteed never to debit twice; maintenance windows; a status page; rate limits for 100+ charges in twenty minutes; and whether a 4xx that carries a ResponseCode can ever mean the transaction was processed.

An outage of three and a half hours inside the nightly billing window, with no status-page notice, no proactive communication, and no answer within business hours to “what happened to my charges”, is a vendor risk in its own right, separate from anything in our code.

Open items

Cardcom answers (owner: Saar, needed before Oct 31). Until they arrive, the durable path keeps treating a coded non-2xx as unknown, and Kinetics’ charge time stays as scheduled.
Better Stack heartbeat grace for Taikan member billing catch-up: confirm it is at least five minutes, so a two-minute circuit pause on a transient failure does not page by itself. A real outage still pages through the withheld heartbeat and the Cardcom circuit opened Sentry warning.
Admin app check: open Held Charges in production with a platform-admin user that has a Taikan user row. Expect an empty table.
Sentry hygiene: resolve TAIKAN-BACKEND-7V through 80. Future outages report as one Cardcom API call failed per endpoint and failure kind, one Cardcom circuit opened, and one Member billing held for an organization per organization.
Provider strategy. An alternative provider (Payme was raised) is a strategic question, not a pre-launch one. The adapter boundary exists (PaymentProviderAdapter, with iCredit, Meshulam, Tranzila and Morning adapters already present, none of them production-grade for recurring billing), but the parts that made this incident safe are built against Cardcom’s API specifically: the external-id claim and lookup that reconcile an unanswered charge, the circuit probe, tax documents, webhook verification and terminal provisioning. A switch means a new adapter with an equivalent reconciliation primitive, re-tokenizing every saved card, and new receipts. For Kinetics’ launch the choice is Cardcom with the safeguards above; the evaluation should start with the reconciliation question (can the new provider look up a charge by our id after a timeout), because without it none of the protection transfers.

Lessons

A provider hold is a platform event, not a gym event. Authority, alerting and tooling have to be sized for every organization at once.
Observability has to exist before the incident. The first two hours were spent not knowing what Cardcom had answered, because the adapter threw the body away.
The nightly window sits outside the vendor’s support hours. Either the window moves, or the system must be able to hold safely until morning without a human, which it now does.
Rehearse with real money before real customers. The 2.00 ILS charge paid for itself many times over.