Cardcom outage / unknown charge

What a Cardcom outage looks like from Taikan, what the system does on its own, what a human has to do, and the 2026-10-26 rehearsal that produced this page.

How it presents

What the system does on its own

• Platform-wide circuit (CardcomCircuitService, Redis, fail-open). A transport failure on any Cardcom call — network error, timeout, 5xx, unreadable body, or an HTTP 429 whatever its body; never another coded Cardcom answer — pauses every protected collector for 2 minutes (Cardcom circuit opened in the logs and one Sentry warning). The catch-up logs Billing catch-up heartbeat withheld: Cardcom circuit open and withholds its heartbeat, so the outage still pages. When the pause lapses, the first collector runs a read-only lookup for an id that cannot exist; a coded answer closes the circuit, a transport failure re-opens it. Recovery is proven by a lookup, never by a debit. Without Redis the circuit reads closed and the per-terminal hold below is the only guard.

• The charge that hit the outage is recorded as outcome unknown: money may or may not have moved, so it is held, never declined, never retried blindly. See billing resilience.

• One unknown attempt blocks the whole terminal: nothing else at that organization is charged until it is resolved. Charges go out one at a time per terminal, so an outage exposes at most one member per organization.

• Every 5 minutes recoverBillingAttempts asks Cardcom for the attempt by its external id (GetTransactionByExternalUniqTran). A settled debit is applied automatically and the hold clears. Any other answer leaves the hold in place: a 9998 ExternalUniqTranId not found is not read as “not charged”.

• Every minute the catch-up revisits the held row, logs the hold and moves on. No new debit is made.

Send rate

Collection is sequential by construction: the catch-up runs under one advisory lock, visits at most 20 organizations × 5 rows per minute pass, and one claim is in flight per terminal at a time; the daily sweep is also advisory-locked and sequential. One organization is therefore charged at most 5 members per minute (Kinetics’ ~100 presales take about 20 minutes), and the platform sends roughly one charge at a time. There is no retry of a charge on a transport failure or 429: a resend without Cardcom’s duplicate-protection guarantee could debit twice, so the attempt is held instead. Cardcom’s actual limits are question 8 in the support template below.

What each Cardcom answer means to Taikan

Every call goes through one client (CardcomProvider.post), so the rules below hold for all endpoints. The money column is what a charge becomes; other endpoints map the same way (a refund becomes unknown with a manual-refund task, a webhook lookup becomes a 503 so Cardcom redelivers, a page or document becomes a plain failure the caller reports).

The legacy (non-durable) charge path is used by staff enrollment, plan change, early renewal, the cancellation tail and platform billing (Taikan charging studios). It has no external id to reconcile by, so a transport failure is thrown to the caller and the row stays pending with charge_started_at set — the in-flight shape of in-flight charges, settled by hand against the Cardcom portal. It is never reported as a declined card: that is the path to a second debit. The request-driven writers (member renewal, early renewal, staff enrollment, plan change) answer 409 charge_outcome_unknown with the pending transaction id; the daily sweep counts the row as an error and moves on.

A desk or debt charge while the circuit is open is refused before any claim exists (409 billing_provider_unavailable): staff see “provider unavailable”, not money to reconcile.

Reading the answers

Resolving a hold

Two surfaces run the same service, locks, provider veto and audit trail:

• Admin app → Held Charges (/held-charges, platform admins). Lists holds across organizations with the member, amount, Cardcom id (ExternalUniqTranId), terminal and the last provider answer. Per row: Check provider, Release…, Authorize retry. Select several rows to release them on one Cardcom confirmation (an incident ticket covers every charge it swallowed) and authorize their retries together. API: /admin/billing/held-charges (GET, GET :id, POST :id/reconcile, POST :id/confirm-no-debit, POST :id/authorize-retry, bulk POST confirm-no-debit and POST authorize-retry). Audited in audit_logs (the admin interceptor) and as an org audit_events row attributed to the operator’s Taikan user with authority: platform_admin.
• Gym dashboard → member finance tab → Review billing (organization owner). Same form, one attempt at a time. See billing resilience › Operator reconciliation.

The release form needs Cardcom’s confirmation reference (6–120 characters: letters, digits, . _ / -; a ticket number such as 649495 qualifies), the confirmation time (after the charge started, not in the future), and the three attestations. The server asks Cardcom once more before releasing: a settled debit vetoes the release and is recorded as paid. Authorize retry schedules one technical retry 5 minutes later (then 15, then 60) that reuses the original external id, so a charge that did settle after all comes back as the original instead of a second debit.

Message to Cardcom support

Include: terminal number, ExternalUniqTranId, amount and currency, the time in Israel time, the endpoint (/api/v11/Transactions/Transaction) and what it answered. Ask: (1) was there an incident and from when to when; (2) confirm in writing no debit and that it cannot settle later; (3) can a Transaction request that returned 500 still have debited the card; (4) is 9998 final right after an incident; (5) resending the same ExternalUniqTranId with ExternalUniqUniqTranIdResponse: true — is it guaranteed to return the original and never debit twice; (6) maintenance windows; (7) status page; (8) rate limits for 100+ token charges in 20 minutes; (9) which HTTP statuses the v11 API uses — in particular, can a 4xx that carries a ResponseCode ever mean the transaction was processed, and does a 429 guarantee it was not.

The external id is mb_ + the transaction id’s 128 bits, base64url (BillingAttemptService.collect); it is shown on the admin page and in the Review billing dialog.

Rehearsal: 2026-10-26

Full postmortem, including the vendor response and open items: Incident 2026-10-26: Cardcom outage.

The gym was Taikan’s own test organization with one 2.00 ILS test renewal, on purpose, six days before Kinetics’ first live collection. Times UTC (Israel = UTC+2).

Detection to resolution: 5 h 17 min, of which ~3.5 h was Cardcom down and the rest was waiting for a human with a vendor confirmation.

What held: no double charge was possible, the terminal stopped after one attempt, the retry reused the original id, both alert channels fired, the owner flow worked first time.

What failed, and what changed: