ADR-0020: Scheduled jobs run from a BullMQ queue

Status: Accepted Date: 2026-09-12 Context owner: Saar Related: ADR-0009 (BullMQ + Redis) · runbook: scheduled jobs

TL;DR

Every scheduled job is a @ScheduledJob(name, pattern) method. Each API process with CRONS_ENABLED=true registers every schedule in one BullMQ queue (scheduled-jobs) and runs a worker on it. BullMQ produces one job per tick and one worker takes it, so the API scales to any number of replicas without a job running twice. This replaces @nestjs/schedule and the single-replica @taikan/api-scheduler service that existed only to keep crons off the web replicas.

Context

@nestjs/schedule runs every @Cron in every process that loads it. Keeping 37 jobs from running once per replica took three separate mechanisms, each with its own failure:

• A second Railway service. @taikan/api-scheduler ran the same image with CRONS_ENABLED=true and exactly one replica, while @taikan/api scaled with crons off. Two services meant two copies of the environment, bootstrapped by copy-paste with no single source. They drifted. The check written to catch that drift (env-parity.yml) was pinned to GitHub-hosted runners and had not completed a run in days.
• A guard in every handler. ScheduleModule still scheduled all 37 jobs on the web replicas, so each handler began with if (!cronsEnabled()) return;. One handler, sweepAbandonedJoinShells, never got it, and ran on all three processes every hour.
• A boot-time registration check. A second ScheduleModule.forRoot() anywhere doubles every job in the process. That happened once, and every cron ran three times per tick. assertSingleCronRegistration existed to catch a recurrence.

Decision

• @ScheduledJob(name, pattern) replaces @Cron. The name is written out literally: it is the BullMQ scheduler key and what bull-board shows. A bundler may rename classes, so the name can’t be derived from one. Patterns are evaluated in UTC, as before.

• ScheduledJobsProcessor discovers every decorated method at boot. Where CRONS_ENABLED=true, it:

  • upserts each schedule (upsertJobScheduler, idempotent by name),
  • removes schedules that no method declares any more,
  • starts its worker.

  Everywhere else it neither schedules, unschedules, nor consumes. That way a process without the flag can’t unschedule the jobs another one runs.

• Worker settings:

  • concurrency: 1.
  • attempts: 1: a failed tick is reported to Sentry, and the next tick runs normally.
  • maxStalledCount: 0: a job whose container dies mid-run, e.g. in a deploy, fails instead of being re-run elsewhere. The next tick picks the work up, and nothing that charges or emails runs twice for a restart.

Alternatives considered

• Railway cron services. One schedule per service: about 20 services for the current schedules, or a single dispatcher that rebuilds a scheduler. Railway’s 5-minute minimum rules out the every-minute Fizikal drain. And it multiplies the duplicated-environment problem instead of removing it.
• Cloudflare Cron Triggers calling the API. Adds a Worker to deploy plus an authenticated internal endpoint. A tick that fires while the API is restarting is lost. Jobs longer than Cloudflare’s ~100 s proxy timeout would still need a queue behind the endpoint.
• A lock around each @Cron. Hand-built leader election, written 37 times. BullMQ’s job schedulers already provide exactly this, on infrastructure the API already runs.

Consequences

• The API scales freely. Once @taikan/api has CRONS_ENABLED=true, these go:
  • the scheduler service and railway.scheduler.toml,
  • env-parity.yml,
  • assertSingleCronRegistration,
  • the 37 handler guards.
• Every run, failure and duration is visible in bull-board under scheduled-jobs.
• Scheduled work shares CPU with web traffic. At concurrency: 1 per process that is bounded, and heavy jobs are already batched.
• Runs of one job can still overlap: a tick that outlasts its interval is still going on one worker when the next starts on another. Jobs that must never overlap keep their own guards: the Postgres advisory locks on the charge sweeps, and the claim-before-send updates in check-ins.
• Redis is now required for scheduling, not just queues. It already backs every queue and the rate limiter.
• Every process that runs the API can run Fizikal jobs, so each needs its egress IP on Fizikal’s allowlist. This was already true for the Fizikal queue processor.