Billing resilience — implementation and release gates

Status: the only collection path for Cardcom organizations. It ran behind the per-org flag payment-billing-resilience for Kinetics and Taikan Gym from 2026-09-28; the flag was retired and the provider-direct branches it guarded were deleted. Organizations on other providers collect provider-direct because the collector reconciles by Cardcom external attempt id only. History: TKN-353 .

Safety rules

• A pending billing attempt means the local outcome is unresolved, not that the card was uncharged. Never clear its claim, cancel its transaction, or issue a manual replacement debit solely because the request timed out or a lookup found nothing.
• Reconciliation uses the stored external attempt ID, exact amount/currency and original merchant terminal. Only authoritative successful debit evidence can automatically repair a lost approval. Replacing credentials, providers or cards does not resolve an older payment outcome.
• Settlement updates the financial ledger, membership and subscription in one transaction. A failed fulfillment remains pending for recovery; do not create a second payment to repair access.
• Receipts and automation delivery are separate durable work. A notification failure does not undo successful collection or authorize another debit. Email retries stop before the provider’s deduplication window expires and then require review.
• Switching an organization’s provider does not erase persisted protocol obligations: a first charge already on the protocol finishes on it.

Readiness inspection (admin app only)

Readiness is a Taikan operator tool. Gyms never see it: there is no gym-dashboard panel and no org-scoped route. It lives in the admin app on the organization page (Organizations → show → Presale collection readiness) and is served by platform-admin routes:

GET /admin/organizations/:orgId/billing-readiness?cursor=<last-subscription-id>

Access is the admin app’s PlatformAdminGuard (Clerk privateMetadata.platformRole === 'admin'); there is no feature flag.

Responses include:

• checkState: blocked, review, or no_known_blockers. The last value is not a promise of successful authorization.
• Machine-readable issue codes, severity and next-action categories; no card tokens, ciphertext, API credentials or recipient emails.
• The expected first charge — the plan as it stands on the charge date (live_estimate), or the amount of a first charge already on the protocol (declined_attempt_snapshot) — plus charge/opening floors, expiry/account checks and retry counts.
• Per-subscription member name, user id and payment-method id so a blocked row can be acted on without a second lookup; never an email or a card token.
• unresolvedProviderTransactions: pending money that carries a v1 protocol — the rows the recovery jobs and the resolution API can address.
• legacyPendingTransactions: pending money with no v1 protocol, including old checkout rows. Nothing reconciles these automatically and the resolution API refuses them; they need a Cardcom-side answer. Affected subscriptions carry the legacy_pending_transaction issue. Both lists are capped at 100 and truncation is marked explicitly.
• Non-presale renewals an owner released without authorizing a retry (collection_paused_by_owner) or whose price changed afterwards (retry_terms_changed). Nothing else reports these — no collector raises them and no dunning runs.
• pageTotals split by currency and nextCursor. These are page totals, never represented as complete organization totals. Follow pagination to cover the full cohort.

The report reads local records only. It makes no charge, hold, refund, email or provider request. Changes can race a read-only report; rerun it after repairs and immediately before collection.

Readiness repairs (operator actions)

Two platform-admin write actions share the readiness path. Each is recorded twice: in the admin audit log (with the request body) and as an organization audit event whose actor is the operator’s Taikan user (an operator who has never signed in to the app has no user row and is refused with 403). Neither moves money, and each takes the collection locks in the same order as a collector (organization → membership → subscription), so no repair can interleave with a charge.

POST /admin/organizations/:orgId/billing-readiness/subscriptions/:subscriptionId/verify-account — body { "lowProfileId"?: string }. Proves a historical card belongs to the organization’s current Cardcom terminal by reading that deal back with LowProfile/GetLpResult on that terminal and comparing the returned token with the stored one. Only an exact match (and matching card expiry, when the deal carries one) writes terminalNumber into the encrypted payload, with an audit row naming the deal, the terminal and the actor — never the token. Outcomes: verified, already_verified (the payload already names today’s terminal, or the terminal history below places the card on it; no provider call), not_verified with a reason (unknown deal, no token on the deal, token belongs to another account, expiry differs). A transport failure returns 409 billing_provider_unavailable and changes nothing — retry the identical request. The deal reference is derived from the subscription’s own token-only checkout row: its billingCheckout.pageId, else metadata.lowProfileId, else providerTransactionId. Settlement overwrites providerTransactionId with the acquirer’s TranzactionId, so on an older checkout that last fallback is a deal number, not a page id, and Cardcom answers it as an unknown page: the action returns not_verified / deal_not_found_on_current_terminal, not a refusal. For older checkouts supply the page id in the body: it is in the API log line Cardcom LowProfile/Create ok lowProfileId= and on the PostHog payment.form_created event as providerTransactionId. Only when the subscription has no token-only checkout row at all does the action refuse with 409 billing_account_evidence_missing. Cardcom keeps answering GetLpResult for old pages: a presale page from 2026-09-07 still returned its full deal and token 21 days later. The deal_not_found_on_current_terminal results for the September 1 and 3 cards on 2026-09-28 were the fallback above: both runs sent no lowProfileId, and both checkout rows name only the TranzactionId (261014155, 261440028). Most historical cards never need this action — see the terminal history rule under the launch gates.

POST …/billing-readiness/subscriptions/:subscriptionId/repair-schedule — body { "expectedNextChargeAt": string | null }. Moves a scheduled presale’s first charge to its own floor — max(opening day, subscription start) — and clears the period, quota and credit fields a presale must not hold before it is paid. It only repairs a missing or too-early date: a date already at or after the floor is a deliberate choice and is refused with 409 billing_schedule_not_anomalous, unchanged. expectedNextChargeAt must equal the date currently on file (409 billing_readiness_state_changed otherwise), and any settled or unresolved first charge for that member blocks it (409 billing_first_charge_unresolved). This is the route for an early presale charge date; it is not a general charge-date editor (that remains the schedule endpoint).

Every action returns { "data": { "item": … } } — the freshly recomputed readiness row — so the panel reflects committed state rather than an optimistic guess. verify-account also returns outcome and, when refused, reason. Unresolved provider payments listed on the panel are read-only there; the gym resolves them through its existing Review billing flow.

Owner-only launch gates

Do not enable collection until the remaining implementation checklist is completed and reviewed. Then obtain explicit owner approval for each operational action:

1. Historical cards must belong to the current terminal. Cards saved since TKN-353 carry their terminal. Older cards are placed automatically by the organization’s terminal history: payment_provider_configs.terminal_since moves whenever the active terminal could have changed (insert, a credential change naming another terminal, reactivation, KYC approval), and a card saved at least 45 minutes (one Cardcom page lifetime plus margin) after it was created on today’s terminal. Only a card saved before that instant shows payment_method_account_unverified; verify-account or a replacement card clears it. On 2026-09-28 the rule placed all 48 active Kinetics cards and 6 of 7 Taikan Gym cards (the seventh was saved 12 minutes after that terminal was configured).
2. Resolve the known Kinetics expiry, anomalous early charge date and old pending checkout outcomes from the audit. repair-schedule moves an early presale date to its floor without enabling collection; the old pending checkout rows appear in legacyPendingTransactions and still need a Cardcom-side answer. Never charge, mutate production data or cancel pending money as part of a diagnostic check.
3. Validate Cardcom’s off-session/token permissions, external-ID behavior, lookup evidence and invoice/document behavior on the intended terminal. Unit mocks are not a live vendor certification.
4. Verify alert routing and actual owner notification delivery, including a terminal-wide outage and paid-but-unfulfilled attempt.
5. Approve a tightly scoped canary with an explicit amount and consenting cardholder; inspect both the provider and local financial/access records. No canary is authorized by this document.
6. Roll out by organization only after observing expected versus settled amounts, no duplicate debits, correct access and successful follow-up delivery.

Kinetics sequence (dated, each step needs its own owner approval)

1. Before deploy: apply migrations 0153 and 0154 (six payment_transactions indexes) through the normal reviewed db:migrate path. Drizzle builds them with plain CREATE INDEX, which blocks writes to payment_transactions for the build duration; run it at a quiet hour or hand-convert to CREATE INDEX CONCURRENTLY if the table is large.
2. Before October 8, 02:00 UTC: in the admin app, run Repair first-charge date on subscription 9b5c7709-… (the October 7 anomaly). No flag is needed. With collection still OFF, the legacy daily sweep has no opening-day floor and would otherwise debit it on October 8.
3. Confirm readiness shows no payment_method_account_unverified row for the cohort (the terminal history places every card saved after August 17, 09:49 UTC plus 45 minutes). Replace the September-expiring card through the member flow. Obtain Cardcom answers for the two legacy pending checkout rows.
4. Rehearse on a member-less organization: a small presale bought with an owner’s card on the current production code, and let its opening collect once. Inspect Cardcom and Taikan, then refund.
5. Only when readiness shows no blocking issue for the cohort: the launch gates above, then let the opening collect (Oct 31, 22:00 UTC). Presales are charged the plan as it stands on the charge date; do not edit the presale plans before opening unless the change is meant for every buyer.

The operator resolution routes and ledger recovery dialog below are implemented for protected member billing. Browser-level verification and platform-billing coverage are still incomplete. Do not work around those gaps with direct SQL or a provider-direct collection route.

Operator reconciliation and release

Using the payments screen

In the organization transaction ledger or a member’s payment history, protected pending or cancelled attempts show Review billing, not Dismiss. Recovery is driven by the persisted protocol and remains accessible after flag rollback. Older, unprotected transactions do not acquire these controls automatically. Completed charges retain their existing refund action.

1. Open Review billing and compare the member, amount/currency, original terminal and external attempt ID. Opening or refreshing this dialog only reads the local record.
2. Check provider / repair settlement performs the original-account lookup and repairs a verified result without sending a new debit. A lookup that finds nothing leaves the hold intact. An in-flight attempt requires waiting and refreshing; contradictory evidence requires support review.
3. Only the organization owner sees the release form. Obtain written Cardcom confirmation of both no debit and no later settlement. Enter its case/reference ID and actual confirmation time (browser-local time, sent as an explicit UTC timestamp), then attest to both statements and matching attempt/account/money details. None of the attestations is preselected.
4. Record confirmation and release hold leaves recurring retry unauthorized. The response determines the displayed outcome: a verified approval may veto the release and complete the payment instead. Never interpret an error or lost HTTP response as proof of release. Refresh; if resubmitting an unchanged form, the same request ID is retained. Reopening reads the current server record before showing actions.
5. For a released recurring obligation that remains eligible, the owner may separately check the new-debit consent and choose Authorize one recurring retry. The eligible-after timestamp is not a receipt or a guarantee that collection happened. Manual/debt collections do not expose this automatic retry action.

The dialog shows recorded reference, confirmation time, release time and actor ID to authorized finance staff. It does not store this evidence in browser storage or expose it in general/member transaction metadata. Switching organizations or users discards the form. A replayed closed protected desk request shows a closed state rather than a decline or an invitation to retry.

API contract

Base path: /organizations/:orgId/payments/billing/transactions/:transactionId. Only persisted v1 recurring/manual/debt attempts are accepted; this cannot upgrade or dismiss a legacy payment. Persisted recovery remains available after flag rollback.

The inspection response includes a typed state and transaction-state actions (reconcile, confirmNoDebit, authorizeRetry). These describe available operations, not authorization: the controller/service still enforce permissions and revalidate the financial state at mutation time. API consumers must not infer a retry permission from a generic transaction status.

The owner-only authority is the owner’s explicit product decision for TKN-353. Release requires a UUIDv4 request ID, provider case/reference ID, confirmation timestamp, exact external attempt ID/terminal/amount/currency, and explicit attestations that no debit occurred and the old request cannot settle later. Keep the written provider confirmation in the referenced case. The application records an attestation; it cannot independently verify what a support representative said. A dashboard search, timeout or not-found lookup is not that confirmation.

The release route first checks the provider. A verified approval vetoes the requested non-debit resolution and goes through normal atomic settlement. Known success, confirmed decline, an executing attempt (90-second grace), mismatched evidence or an unavailable original account cannot be dismissed. The normal pending-payment dismiss route also refuses protected attempts, including zero-value fulfillment.

The original transaction, external ID and idempotency key remain intact. The local transaction becomes cancelled with billingResolution.decision = not_charged; it is not a customer decline. Subscription dates, paid access and decline counts do not change. Resolution and the owner audit row commit together, or neither changes. General/member transaction metadata omits the internal evidence; the permission-gated inspection route exposes its case reference for operators.

Recurring retry requires an explicit authorizeRetry choice. Authorized retries wait 5, 15 and 60 minutes, with at most three technical retries for the same billing obligation. Each is a separate durable claim that reuses the original attempt’s external ID, so if the first request did settle at Cardcom, the duplicate-return flag makes Cardcom answer with that original transaction instead of debiting again. None is created while the prior outcome is uncertain. The exact amount and currency must still match. An uncertain retry needs its own provider confirmation before another is allowed. The issuer-decline counter is never used to mint technical retry keys. Exhaustion stops automatic collection without cancelling the presale and appears in readiness reporting.

Releasing without authorizing leaves the obligation paused; the separate retry-authorization action can resume it later, with its own atomic audit and backoff. Desk charges and debt clearance are never automatically retried by this endpoint. A resolved desk request remains replayable under its original key; a subsequent deliberate desk charge needs a new request key. Explicit debt clearance can claim a new attempt after confirmed non-settlement.

If late positive evidence contradicts an owner-recorded non-debit confirmation, the original record returns to pending, an audit and alert record the contradiction, and member/terminal collection is held again. It does not automatically grant a second period or refund money. Escalate the conflicting evidence for engineering/provider review. No local protocol can make a false provider confirmation true, or undo a debit already started on the strength of that confirmation.

Collection catch-up

BillingCatchupService.collectDue runs every minute on the dedicated billing-collection queue. It never uses the provider-direct collection path: a Cardcom organization or an existing protected obligation is required. The existing daily sweep remains; both protected paths claim the same obligation before calling the provider.

Protected collection covers organizations whose active provider the collector can reconcile — Cardcom. An organization on any other provider keeps provider-direct recurring collection (as its desk and debt charges do): the catch-up loads only rows whose protocol is already persisted (a v1 billing attempt), filtered in SQL, and the daily sweep bills the rest, instead of holding every renewal as provider_not_reconcilable. Both collectors answer that question once per organization per pass, never per row.

A pass visits at most 20 organizations and 5 subscriptions per organization, and stops starting new work after a 30-second monotonic-time budget. The budget is checked once per row, before that row’s fairness checkpoint moves, so a row that was checkpointed is always attempted rather than being skipped until the organization rotates again. It finishes an in-flight provider call rather than abandoning its response. This is a start budget, not a hard maximum job duration or a guaranteed one-minute collection SLA. Provider/DB limits and backlog still affect latency.

Redis holds expiring, per-organization keyset cursors for fair rotation; checkpoints are saved before processing. A restart skips past a poison row on the next tick and returns to it on a later rotation. Cursor loss can repeat selection, never authorize another debit: PostgreSQL is authoritative for claims and settlement. A Redis checkpoint failure prevents the next debit. A row error does not stop the remaining batch, but the tick then fails and reaches the existing worker failure reporting. Unknown outcomes remain pending and do not consume issuer-decline retries.

Inspect billing-collection in the queue UI and the structured Billing catch-up completed summaries (organizations, unevaluable, selected, charged, declined, busy, blocked, skipped, errors). busy is a claim another writer holds right now: it is neither a failure nor a reason to stop the organization. The separate memberBillingCatchup heartbeat is pinged only when the pass moved something or had nothing to select; failed, advisory-lock-skipped and all-held ticks do not ping, so a cohort stuck behind unverified accounts or an open circuit shows up as a missed heartbeat instead of a healthy-looking run. An owner must provision and verify HEARTBEAT_URL_MEMBER_BILLING_CATCHUP before rollout. A heartbeat proves the collector is running, not that all outstanding money settled. Backlog/lag alerting and production-scale query/index review remain implementation gates. Existing reconciliation jobs still use the shared scheduler lane.

Settlement review, follow-up and side effects

A settled provider outcome is never held back because the membership moved underneath it. If the subscription changed between the claim and settlement (notice given, plan/status/decline-count changed, or the row was deleted), the ledger still settles — completed for a collected charge, failed for a confirmed decline — and billingAttempt.settlementReview records why nothing was granted or withdrawn. Settlement then opens an urgent payment_follow_up task, assigned to an active owner (or admin), naming the amount and the transaction id: refunding, fulfilling by hand, or deciding that nothing is owed is a judgement only staff can make. The money is not left pending, so the member is billable again and the desk is not frozen. A drifted debt collection behaves the same way (settlementReview.reason = debt_changed): the collected payment settles, the new balance is left exactly as it stands.

A retry date that staff moved after the claim outranks the debt’s retry schedule: the decline count advances, their date survives, and the settlement is flagged retry_date_changed. A presale decline in an organization with no active owner still enters debt — the retention task is skipped and logged, never the debt, and the member is still notified. Staff copies of that notice are added only when a case owner with an email exists; the member’s own notice never waits on staff.

Everything the charge does around the money — booking-sweep mail, subscription.debt_opened, payment.charge_failed, and the renewal/presale telemetry — is delivered from the settled row by recoverBillingAttempts. Each effect is recorded under billingSideEffects.done as soon as it is handed off, and billingSideEffects.at closes the row once all are done: a failure in one effect retries only what is still owed, never an email or staff event that already went out. (A crash in the instant between a send and its record can repeat that one effect.) A settlement held for review emits only the gateway outcome: it never claims a promotion or a debt entry that did not happen.

Alert reasons to expect on the collector, beyond the provider ones: collection_paused_by_owner (money the owner released without authorizing a retry — an explicit authorization is the only way forward), retry_terms_changed (the released key’s amount/currency no longer matches what is due), technical_retries_exhausted, settlement_review:<reason>, and settlement_side_effects_pending. None of them is permission to debit again.

The daily 02:00 sweep repairs settlements only; notification and lifecycle delivery stay on recoverBillingAttempts’ own five-minute schedule, so a hung email provider can no longer delay the day’s collection. Its memberRecurringCharge heartbeat is pinged only when the sweep actually ran to completion — an advisory-lock-skipped or aborted tick shows up as a missed heartbeat.

Manual charges and debt recovery

On a Cardcom organization, desk charges require a UUIDv4 idempotencyKey. Reuse the exact key and request after a lost HTTP response. A different amount/member/description with the same key is rejected; do not bypass that rejection with a new key until the original payment is understood. The web dialog generates its key on confirmation.

A pending response, or billing_reconciliation_required, is not a decline. The staff dialogs withhold the retry action for that state. A confirmed issuer decline permits an explicit new attempt. Debt balances remain unchanged until successful settlement commits atomically; recovery uses the original provider external ID and account, even after a provider or card change.

The manual/debt recovery job runs every five minutes, with bounded batches. Receipt failures are retried separately and cannot trigger another debit. The same pending-payment guard also excludes recurring charges for that membership. Legacy ambiguous manual/debt records without a start marker are held for review.

Legacy pending-row-first desk, debt, plan-change and staff-enrollment claims, and keyed early-renewal/daily-renewal/cancellation-tail intents, participate in the same first organization lock before inserting a positive pending charge. Admission rechecks persisted protected holds inside that transaction, including after flag rollback; an earlier permission or pending-payment read cannot authorize bypassing a newer claim. The provider request runs only after the lock is released. This does not upgrade the legacy path’s settlement or outcome classification.

A pending protected obligation blocks another covered legacy claim for that member. An uncertain protected Cardcom attempt also blocks legacy Cardcom claims for other members in the organization: legacy writers cannot prove their original terminal binding. Conversely, protected Cardcom collection holds on older positive pending Cardcom pre-debit records even without a start timestamp when their source identifies a pre-debit claim. Review those historical records before rollout; do not delete them or toggle the flag to clear the hold. Known approval awaiting local fulfillment blocks its member, not unrelated members on the terminal. Non-debit ledger recording and legacy-only admission with no protected hold retain their existing behavior.

A claim another worker is still executing (inside the 90-second grace) is not a hold: legacy admission for other members proceeds while a collector is mid-call, and is refused only once the claim is past the grace with no recorded outcome.

The browser currently retains an uncertain manual request only while the dialog remains mounted. After a full reload or navigation, check the transaction history before starting another desk charge. Durable client-intent recovery, remaining hosted/single-shot paths and complete legacy settlement migration are still implementation gates, not operator workarounds.

Opening and charge-date edits

For protected organizations, changing opening day is serialized with collection. An unresolved first charge blocks the entire opening edit: neither the organization date nor part of the cohort is allowed to move underneath it. Reconcile the payment first.

Postponement pushes deferred charges/starts to at least the new opening instant. A later member-specific date remains later. Moving opening earlier does not bring an existing charge forward; use the explicit staff charge-date action to request a different date. That action cannot precede opening or the member’s agreed start and moves active access by the same delta without creating new credits.

Deferred cash, gifts and unpaid grace retain their purchased/granted duration. Follow-up tasks move with them, keeping the cash renewal’s seven-day lead. Original cash sale rows and priced periods are not rewritten; append-only cashPauses entries marked opening_delay preserve the refund basis. The promotion worker repairs a deferred term inserted with a stale opening date before granting access.

Date changes have atomic subscription audit records. Persisted protocols remain protected after a provider change. Enrollment, freeze/resume and the remaining legacy money writers still need the shared guard before rollout; these protections are not permission to bypass an unresolved payment through another route.

An opening postponement or a staff charge-date edit rewrites the presale follow-up task’s due date and copy to the date staff will actually see. The subscription stays on the protected protocol through its persisted v1 attempt.

Card replacement and re-registration

A replacement re-points a scheduled presale at the new card. Legacy collection reads the membership’s active card and ignores the subscription’s binding, so this changes nothing for it; leaving it behind stranded the sale on an inactive card the protected collector then refused to charge.

A registration is identified by its Cardcom LowProfileId (stored as provider_payment_method_id). A redelivered or re-run registration for a card already held is a no-op — no new row, no re-binding, no audit, no recovery acceleration — so a redelivery cannot resurrect a card that has since been superseded. Acceleration additionally requires a card no declined attempt of that subscription has ever used (decrypted-token comparison); an unreadable stored payload counts as “not proven new” and blocks acceleration.

A verified hosted-checkout or token-registration completion that merely coincides with a held first charge is no longer acknowledged and dropped: presale fulfillment handles it when the page identity matches, otherwise the webhook is answered 503 (with an alert) so Cardcom redelivers. A late verified completion for a checkout the abandoned sweep released is revived exactly as the legacy path revives it — status back to scheduled, token stored, no paid period, membership restored only for the join-link demotion signature. Any other cancelled sale is alerted and left cancelled.

Cardcom reports a failed attempt inside the payment form the same way as a page timeout (a payment.failed with no deal id), so a checkout row can be closed as “page expired” while the member is still typing. That closure now records checkoutPageExpiredAt on the row. A later verified completion for that exact page reopens it and lands the presale once — card saved, first charge at the floor, no paid period — and any redelivery of the same success changes nothing. A completion for an attempt closed for any other reason is not applied, and raises presale_checkout_attempt_cancelled (or presale_checkout_transaction_failed) instead of being acknowledged silently. This is the sequence that moved Kinetics subscription 9b5c7709-1003-4197-a380-3bce462cdd33 to October 7 on the legacy path (2026-09-07: closed 34 s after the page opened, completed 17 s later, the same success redelivered 50 s after that).

Cardcom webhook verification

Cardcom signs nothing, so every Cardcom webhook is read back with LowProfile/GetLpResult on the organization’s terminal and Cardcom’s answer, not the posted body, is what gets processed. An unverified event never reaches a handler: WebhookProcessingService drops anything without providerVerified and emits payment.webhook_ignored with reason: unverified (Sentry unverified payment webhook refused). For Cardcom the HTTP answer is always 400 (payment.webhook_rejected, outcome: failure): Cardcom’s own API is the verifier, so a page it cannot place is never a secret the owner has yet to paste. For Morning and Tranzila payment-webhook-signature-enforcement decides between 400 and a 200 logged with a reason ending _unenforced. A transient failure (provider_unreachable, deal_not_finalized) is answered 503 in every case so Cardcom redelivers.

Turning payment-webhook-signature-enforcement on (Morning, Tranzila)

The flag is read per request and fails open (anything but a definite true means off), so it can be flipped either way in PostHog with no deploy. Cardcom does not consult it.

1. Dry run, one week, flag off. The PostHog alert Webhook rejected or accepted unverified fires on any payment.webhook_rejected; during the dry run a hit with a reason ending _unenforced is a webhook that enforcement would answer 400. Investigate every hit before going on.
2. Per org. Target the organization group by $group_key (never id), one Morning or Tranzila org at a time. The same alert now reports outcome: failure rejections; the Sentry issue unverified payment webhook refused reports bodies the processor dropped.
3. Rollback is the flag off for that org, effective on the next webhook. Nothing to redeploy. Cardcom redelivers a non-200 up to seven times over about 25 hours, so a brief mistaken 400 heals once the flag is off, provided it is caught within that window; a redelivery that has run out is recovered by the member’s return page or by POST /verify-return.
4. Retire. Once every provider org is on and the _unenforced alert has been quiet for a month, delete the flag and the unenforced branch together.

GetLpResult’s root ResponseCode is the outcome of the page, not of the lookup. Answers observed on the Kinetics terminal on 2026-09-28:

Cardcom resends every webhook that is not answered 200, seven more times: after 1, 2, 2, 16 and 60 minutes, then twice 12 hours apart. A 503 therefore gets about 25 hours to resolve on its own. A 400 is resent the same way, so a burst of payment.webhook_rejected with unverifiable_deal_not_found repeating on this schedule for one page means Cardcom does not recognise that page on the organization’s current terminal. Check the terminal on the provider config before anything else.

Before 2026-09-28 every failed page read as deal_not_found. That is the source of the 40 unverifiable_deal_not_found_unenforced events between 2026-08-15 and 2026-09-23; all of them were failed or timed-out checkouts, and none was a success.

Cash credit notes, refunds and unpaid-payment rollout

TKN-425 (cash credit notes, refund cap and task close), TKN-426 (pay-later dates) and TKN-427 (unpaid payment: grace, then cancellation for non-payment, ADR-0021) ship together with no feature flag. Behavior contract: payments › Cash credit notes, Refund flows, Pay-later terms, Unpaid payment.

Deploy

1. Apply migration 0164_special_meteorite through the normal reviewed db:migrate path (owner approval), right before the API deploy. Six additive columns (organizations.billing_grace_days, plan_required_to_book, billing_lifecycle_day; subscriptions.payment_last_day, debt_reason, debt_notice_stage), the partial index subscriptions_debt_since_idx, then two backfill updates: subscriptions.payment_last_day for every row with a payment_due_at, and open collection tasks re-dated to it. Nothing in the migration changes a subscription’s status or starts a clock; applying it ahead of the code is harmless.
2. The first hourly SubscriptionLifecycleService.tick after deploy runs one pass per org and acts on everything already due, by design:
   • packs/drop-ins past validity end expired; set-period gifts past coverage end gift_ended; paid card plans with no card and no renewal end term_ended; ₪0 open-ended card plans roll instead of being cancelled;
   • lapsed cash plans enter debt (cash_lapsed, day 0 = cash_prepaid_through) and pay-later terms past payment_due_at enter debt (payment_due, day 0 = the due instant), each with a debt-opened notice;
   • a debt row whose day N (counted from its existing debt_since, reason read as charge_failed when null) has already passed is cancelled for non-payment, with a cancellation notice, on the next pass that sees it — a debt entered on the first pass is cancelled at the earliest on the next org day.
   • Run the live check below before deploy; the ADR records that no member was behind in production, so the expected burst is zero or a handful of rows.
3. Legacy past_due rows are not touched by the lifecycle; the renewal sweep still retries them and their next decline enters debt with day 0 = that decline. What changes for members otherwise: a member whose only live plans are presales that have not started is refused booking (plan_not_started); a member with a plan cancelled for non-payment and open debt is refused even when another plan is healthy (booking_blocked_debt); a plan in debt keeps booking until its cancellation instant.

Pre-deploy live check (read-only)

What the first lifecycle pass would act on. Run against production read-only before deploying; every count is expected to be zero or very small.

-- Debts that day N would cancel at once (default 7 days; the column does not exist yet before migrating)
select s.id, s.status, s.debt_since, s.debt_amount_in_cents, o.timezone
from subscriptions s join memberships m on m.id = s.membership_id join organizations o on o.id = m.organization_id
where s.deleted_at is null and s.status = 'debt' and s.debt_since < now() - interval '7 days';
 
-- Legacy past_due rows (retried by the renewal sweep; next decline enters debt)
select count(*) from subscriptions where deleted_at is null and status = 'past_due';
 
-- Active rows whose period already ended (natural ends, cash lapses, card-less terms)
select p.type, s.payment_collection_method, count(*)
from subscriptions s join plans p on p.id = s.plan_id
where s.deleted_at is null and s.status = 'active' and s.current_period_end <= now()
  and s.next_charge_date is null and s.cancel_at_period_end = false and p.type <> 'course'
group by 1, 2;
 
-- Pay-later terms already past their due date
select count(*) from subscriptions
where deleted_at is null and status = 'active' and payment_due_at <= now() and unpaid_amount_in_cents > 0;

Read-only backfill queries

Before this change every cash credit was filed as Cardcom ReceiptRefund (type 4) whatever the original, and a matured cash notice could be closed through its manual-refund task with no credit document at all. List both; correction is manual with the accountant. Read-only — never update these rows by hand.

-- Cash credits filed as ReceiptRefund (type 4) against a TaxInvoiceAndReceipt (type 1) original
select r.id, r.organization_id, r.document_number, o.document_number as original_document,
       r.amount_in_cents, r.completed_at
from payment_transactions r
join payment_transactions o on o.id = (r.metadata->>'refundedTransactionId')::uuid
where r.type = 'refund' and r.provider = 'manual' and r.document_provider = 'cardcom'
  and r.document_type = 'ReceiptRefund' and o.document_type = 'TaxInvoiceAndReceipt';
 
-- Cash-notice refunds settled with no credit document. Closing the task by hand
-- set the reservation row to `completed` (and its sale to `refunded`) without
-- filing anything.
select r.id, r.organization_id, r.subscription_id, r.metadata->>'refundedTransactionId' as original_id,
       r.amount_in_cents, r.status, r.refund_external_reference, r.updated_at
from payment_transactions r
where r.metadata->>'source' = 'cash_notice_settlement'
  and r.status = 'completed'
  and r.document_number is null;

Before issuing a corrective Credit note on a sale from either list, check its existing refund rows:

• A sale already fully credited with the wrong type has a cap of 0; the button refuses it (credit_note_amount_exceeds_cap). Correct it in the provider portal with the accountant.
• A reservation closed by hand is a completed refund row, so it already counts toward the cap: a sale whose reservation covered all of it has a cap of 0 and the button refuses it. The missing credit document for that reserved cash is filed in the provider portal with the accountant — the button never re-credits money already on the ledger.
• A legacy partial cash refund flipped its sale to refunded; the remainder is still creditable through the button (its earlier row is completed and counted).

Erez’s ₪650 case is to be corrected with Credit note after deploy (the sale was cancelled without a refund, so it has no reservation and its full amount is creditable).

One-time live checks (each needs owner approval)

Unit tests pin the request shapes; nobody has yet read the resulting documents from a live terminal.

1. Cardcom DocumentDate format. The swagger types it as a plain string and the vendor article’s only example is 05/03/2025; Taikan sends MM/dd/yyyy, and only for a back-dated document (an org-local day other than today in Asia/Jerusalem). On a test member, settle a ₪1 unpaid term with paidOn = yesterday (or record a ₪1 payment dated yesterday) and open the PDF: the document date must be yesterday. If day and month are swapped, change only toCardcomDocumentDate (cardcom.provider.ts) and credit the test sale.
2. ₪1 cash sale → credit on the Kinetics terminal. Record a ₪1 cash sale for a consenting test member (document TaxInvoiceAndReceipt), then issue a Credit note for ₪1 with Cash returned on. Confirm in the Cardcom dashboard and the PDF: type TaxInvoiceAndReceiptRefund (2), payment form cash, positive amount, Comments reading זיכוי למסמך מס' <sale number>, the member’s email received it, and Taikan’s refund row carries the document number and URL with no document_error. Re-posting the same request (same idempotency key) must return the same refund row with replayed: true and file nothing; Cardcom’s own ExternalId dedupe only matters for a timed-out document call and cannot be triggered from the UI.
3. Optionally repeat step 2 on a new ₪1 sale with Cash returned off and confirm the same type-2 document; the refund row carries metadata.cashReturned = false.

Until step 2 passes, a failure surfaces as document_error on the refund row (owed-documents queue), never as lost money.

Local verification

Run only in worktrees/tkn-353-billing-resilience. The fault-injection driver refuses any database except the local per-worker test database the integration harness created for the current vitest worker (taikan_<tag>_w<N>_test), so it runs inside the normal vitest.integration.config.ts suite in CI as well as the dedicated billing config. Queue isolation tests use local Redis with a unique random key namespace and remove only that namespace. These tests must never run against development or production data.

pnpm -C apps/api exec tsc --noEmit --project tsconfig.app.json
pnpm -C apps/api exec vitest run --config vitest.unit.config.ts src/payments --silent
pnpm -C apps/web exec tsc --noEmit --incremental false
pnpm -C apps/web exec vitest run --config vitest.integration.config.ts src/components/payments src/components/members/manual-charge-dialog.int.spec.tsx src/components/members/collect-debt-dialog.int.spec.tsx --silent
# BILLING_TEST_DB_TAG + a matching template let several copies run side by side.
TEST_BASE_DATABASE_URL=postgresql://postgres:postgres@localhost:55432/taikan_test DATABASE_URL=postgresql://postgres:postgres@localhost:55432/taikan_test pnpm -C apps/api exec vitest run --config vitest.billing-resilience.config.ts --silent

The full API unit suite additionally includes an existing module-boot test that reads encryption and storage configuration. Supply throwaway test settings; do not copy production environment files or secrets into this worktree.