Scheduled jobs

Every scheduled job is a @ScheduledJob(name, pattern) method, normally run from the BullMQ queue scheduled-jobs. The protected member-billing catch-up handler uses the explicit billing-collection lane so slow shared cron jobs cannot consume its worker capacity. Each API process with CRONS_ENABLED=true registers schedules and runs one worker per lane. BullMQ hands a tick to one worker; handlers still need application-level idempotency for retries and overlapping ticks. The original shared-queue decision is in ADR-0020; billing rollout gates are in the billing runbook.

In production, every @taikan/api replica is a worker.

Where to look

bull-board, queue scheduled-jobs: every run, its duration, and the error when one failed. A failed tick also reaches Sentry once, through ObservableWorkerHost.
bull-board, queue billing-collection: BillingCatchupService.collectDue runs every minute. Each pass is bounded and rotates through organizations/subscriptions. Financial completion is held in PostgreSQL, not a Redis job status or cursor.
Billing catch-up heartbeat (HEARTBEAT_URL_MEMBER_BILLING_CATCHUP): isolated passes that collected something or had nothing to select. A pass whose every selected obligation was held does not ping, so a stuck cohort is a missed heartbeat rather than a healthy-looking run. Provisioning and alert-delivery verification require owner approval; the shared-scheduler heartbeat does not monitor this worker lane.
BetterStack heartbeats: “Taikan scheduler alive” is HeartbeatService.schedulerHeartbeat, every 5 minutes. If it stops, nothing is consuming the queue. The money jobs each have their own heartbeat, pinged only on a run that actually did its work with no row failures: billing reconciler, member recurring charge, member billing catch-up, platform billing recurring, subscription lifecycle (hourly), cancellation sweep (daily 02:30 UTC). The full inventory, with cadences and what each ping means, is in monitoring.md.

Adding a job


@ScheduledJob('LeadSweepService.releaseStale', '15 * * * *')
async releaseStale() {
  // …
}

Name — Class.method, written literally. It is the scheduler’s key in Redis. Renaming it removes the old schedule and creates a new one on the next boot.
Pattern — standard cron, evaluated in UTC. A six-field pattern adds seconds.
No guard. Don’t start the method with if (!cronsEnabled()) return;. The processor already decides whether this process runs jobs at all.
Lane isolation. The optional third argument is a declared queue constant. Only the processor for that lane registers, handles and removes its schedules. Ordinary jobs stay on the default shared lane; do not move jobs without reviewing idempotency and capacity. Both lanes obey CRONS_ENABLED.
Make it safe to overlap. A run that outlasts its interval is still going on one worker when the next tick starts on another. Anything that must not run twice at once needs its own guard, like the advisory lock on the charge sweeps or the claim-before-send update in check-ins.
A failure is not retried. The next tick runs as normal. A job whose container dies mid-run (a deploy) also fails rather than re-running elsewhere.

Removing a method removes its schedule on the next boot.

`CRONS_ENABLED`

where	value	effect
production API	`true`	registers every schedule, runs the worker
dev, preview	unset	nothing is scheduled or consumed; schedules already in Redis are left alone
CI	`true`	the suites exercise real handlers

A process without the flag never touches the queue’s schedules. If it did, booting it would unschedule the jobs another process runs.

Stopping all scheduled jobs

Changing CRONS_ENABLED is an owner-approved operational action, never an autonomous code-fix step. Unsetting it on @taikan/api and redeploying stops both lanes. The “Taikan scheduler alive” heartbeat goes red. Each schedule keeps one pending job in Redis, so setting the flag again resumes the schedules; missed periods are not individually replayed. The billing collector rediscovers outstanding obligations from PostgreSQL.

Subscription lifecycle

SubscriptionLifecycleService (apps/api/src/subscriptions/subscription-lifecycle.service.ts) owns every calendar-driven subscription transition; it replaced NotificationSchedulerService’s daily expiry sweep and the old grace-phase notice job. BillingDebtNotificationsService (apps/api/src/notifications/billing-debt-notifications.service.ts) sends the member billing notices by email and push, under the member’s billing notification preference.

job	pattern (UTC)	what it does	why it does not double-act
`SubscriptionLifecycleService.tick`	`5 * * * *`	For each org whose local day has moved past `organizations.billing_lifecycle_day`: ends packs/drop-ins (`expired`), set-period gifts (`gift_ended`) and paid card-less terms (`term_ended`); rolls ₪0 card plans; puts lapsed cash plans (`cash_lapsed`) and passed pay-later due dates (`payment_due`) in debt; sends the last-day notice on the pay-by day; cancels debts past day N for non-payment and sends the cancellation notice. Rows with a pending charge, a retry still due or a renewal the charge jobs own are skipped.	Per-org advisory lock `taikan:lifecycle:<orgId>`; every transition is a guarded `UPDATE … WHERE status = <expected>`; the day is stamped only when the whole pass succeeded, so a failed row is retried on the next tick and a re-run is a no-op. Notices claim `subscriptions.debt_notice_stage` by compare-and-set first.
`BillingDebtNotificationsService.runCardExpiryReminders`	`0 8 * * *`	Saved cards that a renewing subscription still uses (live, `next_charge_date` set, not cancelling, card rail) and that stop working in 14, 7 or 1 org calendar days. A card stops working at the start of the UTC month after its expiry month (`cardUsableAt`).	Redis `SET card-expiry:<paymentMethodId>:<daysLeft> NX EX 172800` before sending. If Redis cannot answer, the reminder is sent anyway.

Day-0 notices are event-driven: SUBSCRIPTION_DEBT_OPENED (every debt entry, from the charge jobs, the webhook, the cancellation tail or the lifecycle) sends the debt-opened notice, and billing.card_unusable (a renewal found the card expired or missing) sends the card notice; both claim the opened stage.

Each notice is sent at most once per debt. A notice whose claim was written but whose delivery failed is not retried. Look for Subscription lifecycle: (passes, transitions, failures) and Card-expiry reminders: in the API logs. To re-run an org’s pass the same day, set its billing_lifecycle_day back to yesterday (or NULL); to send a debt notice again, set debt_notice_stage back to its previous value.

Bookings

job	pattern (UTC)	what it does	why it does not double-act
`WaitlistExpiryService.expireStarted`	`/5 * * *`	Cancels every `waitlisted` booking whose class has started (`cancellation_reason='waitlist_expired'`), so it stops counting against the member’s caps and quotas. No notification. Logs `check=waitlist_expired` and tracks `waitlist_expired { count }` per org when it closed any. Then refills free seats from the waitlist on published sessions starting 15 minutes to 14 days out (`refillFreeSeats`, trigger `scheduler`): the backstop for a fill that failed.	The expiry update matches only rows still `waitlisted`; each refill runs `fillFreeSeats` in its own serializable transaction per session, so two overlapping runs cannot promote past capacity.

Invariants

Job names are unique. Two classes claiming one name fail boot. A class provided by two modules is scheduled once.
Every worker’s egress IP is on Fizikal’s allowlist. Fizikal jobs run on whichever replica takes the tick. @taikan/api has Static Outbound IPs enabled for this, so any other service that runs the worker needs them too.