BullMQ Essentials: Background Jobs and Queues for Backend Developers

BullMQ retries a failed job only when you ask it to. Set attempts above 1 (the default is 1, which means no retries) and a backoff, fixed or exponential, and BullMQ re-runs the job each time its processor throws, until it succeeds, runs out of attempts, or throws UnrecoverableError. The retry section below shows each option with code I ran against BullMQ 6.

Why you'd want a queue at all: a signup should respond in milliseconds, but the welcome email behind it can take seconds, or fail because the email provider is down. That work doesn't belong in the HTTP request. BullMQ moves it out: the request records "send this email" in a Redis-backed queue and returns right away, and a separate worker sends it, retrying if it fails.

Why background queues exist

Plenty of backend work is simply too slow or too unreliable to run inside a request-response cycle: SMTP servers that hang, CPU-heavy PDF generation, third-party APIs that rate-limit you, fanning push notifications out to thousands of devices, resizing an uploaded image. The fix is always the same shape — the HTTP handler does the fast part itself (save to the database), adds a job describing the slow part to a queue (which is really just a fast write to Redis), and returns a response immediately. A completely separate worker process picks the job up afterward and actually does the slow work, retrying automatically if it fails.

Three concepts: Queue, Job, Worker

A Queue is a named list of jobs stored in Redis. It's the producer side: it doesn't process anything, it just holds work. A Job is one unit of work inside that queue: a name for the kind of work ("send-welcome"), a data payload the worker needs, and options like retry count or delay. A Worker does the work. It listens to a queue, pulls jobs off it, runs your processor function on each one, and marks it completed or failed. You can run many workers across many processes or machines, all pulling from the same queue, which is why scaling background work is often as simple as starting more workers.

One connection setting to get right

Workers need maxRetriesPerRequest: null on their Redis connection. BullMQ uses blocking Redis commands to wait for new jobs, and ioredis's default retry limit on those commands breaks the worker. If you pass connection options, as below, BullMQ sets it on the Worker's connections for you. If you pass your own ioredis instance to a Worker, you have to set it yourself, or BullMQ warns or throws. Producers are the opposite: leave the default (20) on the Queue's connection, so adding a job fails fast when Redis is down instead of hanging the HTTP request. Redis itself must run with maxmemory-policy noeviction. BullMQ's docs call it the only setting that keeps queues correct, because any other policy can delete job data when memory runs low.

import { Queue } from 'bullmq';

const connection = {
  host: process.env.REDIS_HOST ?? '127.0.0.1',
  port: Number(process.env.REDIS_PORT ?? 6379),
  // No maxRetriesPerRequest here. Workers get null automatically,
  // and the Queue keeps the default (20), so add() fails fast if Redis is down.
};

export const emailQueue = new Queue('email', { connection });

Adding jobs

emailQueue.add('send-welcome', { userId: 42, email, name }) adds one job. Job names matter because a single worker can handle several different kinds of work on the same queue — send-welcome, send-receipt, and send-newsletter can all live in one "email" queue, with the worker's processor function switching on job.name to decide what to actually do. addBulk() adds many jobs in one call, which is far more efficient than looping and calling add() individually when you're enqueueing, say, a notification for every user in a campaign.

// One job
await emailQueue.add('send-welcome', {
  userId: 42,
  email: 'jane@example.com',
  name: 'Jane',
});

// Many jobs in a single round trip to Redis
await emailQueue.addBulk(
  users.map((user) => ({
    name: 'send-newsletter',
    data: { userId: user.id, email: user.email },
  }))
);

The job lifecycle

Every job moves through a defined set of states. It starts waiting in the queue until a worker is free. Once picked up, it becomes active while the processor function runs. If the processor resolves, the job becomes completed. If it throws and attempts remain, it moves to delayed to wait out its backoff period before being retried (or straight back to waiting if there's no backoff). Only when the last attempt throws does it become failed. There's also stalled — if a worker crashes mid-job, its lock on the job expires, and BullMQ detects the abandoned job and moves it back to waiting automatically, so work doesn't just vanish because a process died. A job that stalls more often than maxStalledCount allows (1 by default) is moved to failed instead.

BullMQ retry: attempts and backoff

{ attempts: 5, backoff: { type: 'exponential', delay: 1000 } } tries a job up to 5 times, waiting 1s, 2s, 4s, 8s between attempts rather than hammering a struggling service at a fixed interval. Exponential backoff is a sensible default for almost everything, because it gives a service that's temporarily down room to recover. removeOnComplete and removeOnFail cap how many finished jobs Redis keeps around. Delayed jobs use the same delay option to schedule work in the future: a follow-up email 3 days after signup is just { delay: 3 * 24 * 60 * 60 * 1000 }.

await emailQueue.add(
  'send-welcome',
  { userId: 42, email },
  {
    attempts: 5,
    backoff: { type: 'exponential', delay: 1000 }, // 1s, 2s, 4s, 8s between tries
    removeOnComplete: 1000, // keep only the last 1000 completed jobs
    removeOnFail: 5000, // keep more failures around for debugging
  }
);

// Scheduled for later — a follow-up email 3 days after signup
await emailQueue.add(
  'signup-followup',
  { userId: 42, email },
  { delay: 3 * 24 * 60 * 60 * 1000 }
);

Two details worth knowing. The exponential delay is 2^(attemptsMade - 1) × delay, so with delay: 1000 the waits are 1 s, 2 s, 4 s and 8 s. And attempts counts the first run, so attempts: 5 is one try plus up to four retries. A job added without attempts fails on its first error and is never retried. To avoid repeating options on every add(), set them once in the Queue's defaultJobOptions.

Fixed vs exponential backoff, and jitter

Fixed waits the same delay before every retry, and a plain number (backoff: 5000) is shorthand for it. Exponential doubles the wait each time, which suits a service that is down and needs room to recover. Both accept jitter, a number from 0 to 1: with jitter: 0.5, each wait is a random value between half and all of the calculated delay. When hundreds of jobs fail in the same second because a provider had a short outage, jitter stops them all retrying in the same second too. Leave backoff out entirely and failed jobs retry immediately, which is rarely what you want.

// Fixed: wait 5 seconds before every retry
await emailQueue.add('send-receipt', { orderId: 1001 }, {
  attempts: 3,
  backoff: { type: 'fixed', delay: 5000 }, // same as backoff: 5000
});

// Exponential with jitter: up to 1s, 2s, 4s, 8s, each a random 50-100% of that
await emailQueue.add('send-welcome', { userId: 42, email }, {
  attempts: 5,
  backoff: { type: 'exponential', delay: 1000, jitter: 0.5 },
});

Custom backoff: respect the provider's Retry-After

For anything smarter, define backoffStrategy in the worker's settings and give jobs a backoff type that isn't fixed or exponential. BullMQ calls it with the attempt number, the backoff type, the error and the job. Return a delay in milliseconds, 0 to put the job straight back in the waiting list, or -1 to stop retrying and fail the job now. The BullMQ guide to retrying failing jobs documents these return values. Here a rate-limited send waits exactly as long as the provider asked, and everything else backs off exponentially:

import { Worker, UnrecoverableError } from 'bullmq';

// An error type your email client throws on HTTP 429
class RateLimitedError extends Error {
  retryAfterMs: number;
  constructor(retryAfterMs: number) {
    super('Rate limited by the email provider');
    this.name = 'RateLimitedError';
    this.retryAfterMs = retryAfterMs;
  }
}

const worker = new Worker(
  'email',
  async (job) => {
    if (!job.data.email?.includes('@')) {
      // Bad input won't fix itself: fail now and skip the remaining attempts.
      throw new UnrecoverableError(`Invalid email for user ${job.data.userId}`);
    }
    await sendEmail(job.data); // your email client; throws RateLimitedError on a 429
  },
  {
    connection,
    settings: {
      // Used for any job whose backoff type isn't 'fixed' or 'exponential'
      backoffStrategy: (attemptsMade, type, err) => {
        if (type !== 'provider-aware') return -1; // unknown type: fail now
        if (err instanceof RateLimitedError) return err.retryAfterMs;
        return 2 ** (attemptsMade - 1) * 1000; // otherwise 1s, 2s, 4s...
      },
    },
  }
);

await emailQueue.add('send-welcome', { userId: 42, email }, {
  attempts: 4,
  backoff: { type: 'provider-aware' },
});

Stop retrying with UnrecoverableError

Some failures will happen the same way every time: a malformed address, a deleted user, a 400 from an API. Retrying them burns attempts and delays real work. Throw UnrecoverableError (exported by bullmq) and the job goes straight to failed, however many attempts are left. When I ran the worker above, the job with an invalid address ran once and failed with attemptsMade: 1, while a job whose first send hit a 1.5-second rate limit waited about 1.5 seconds and completed on its second run.

Retrying failed jobs manually

Once the cause is fixed (the provider is back, the bug fix is deployed), push failed jobs back into the queue. job.retry() moves one failed job back to waiting. By default it keeps attemptsMade, so a job that already used all its attempts gets one more run and fails for good on the next error. Pass resetAttemptsMade: true to give it its full set of retries back. queue.retryJobs() moves every failed job back to waiting, in batches of count. Both only work on jobs that still exist, which is one more reason to keep failures around with a generous removeOnFail.

// One job, with a fresh set of attempts
const job = await emailQueue.getJob(jobId);
if (job && (await job.isFailed())) {
  await job.retry('failed', { resetAttemptsMade: true });
}

// Every failed job in the queue, moved back 1000 at a time
await emailQueue.retryJobs({ state: 'failed', count: 1000 });

Make jobs idempotent before you rely on retries

Retries mean a job can run more than once, and so can a stalled job that BullMQ recovers after a worker crash. If the processor sends an email or charges a card, a second run is a real bug, so design it so that running twice ends in the same state as running once. BullMQ's docs call this idempotent jobs. Two tools help:

• –A custom jobId when you add the job. Job ids are unique per queue, so adding a job whose id already exists is ignored, and welcome-42 can only be queued once. The catch: once removeOnComplete or removeOnFail deletes the job, the same id can be added again. Custom ids must not contain a colon.
• –A check inside the processor. Record what you've done, for example an email_log row keyed by job id with a unique constraint, and skip work that's already recorded. If your email or payment provider accepts an idempotency key, pass the job id as that key; it also covers a crash between sending and recording.

// jobId makes the add idempotent: a second add with the same id is ignored
await emailQueue.add(
  'send-welcome',
  { userId: user.id, email: user.email },
  { jobId: `welcome-${user.id}`, attempts: 5, backoff: { type: 'exponential', delay: 1000 } }
);

Concurrency, rate limiting and cron jobs with Job Schedulers

A worker's concurrency option controls how many jobs it processes in parallel: concurrency: 5 handles five at once. But Node.js is single-threaded, so this only helps for I/O-bound work (waiting on network calls). CPU-heavy work like PDF generation blocks the event loop and stalls every job on that worker. If it blocks for longer than the job lock (30 seconds by default), BullMQ assumes the worker died and runs the same job again elsewhere. Keep CPU-heavy processors at concurrency 1–2 or move the work to a worker thread. A limiter option (max: 10, duration: 1000) rate-limits the whole queue, across every worker attached to it, which helps when you're calling a third-party API with its own rate limit. For recurring work, upsertJobScheduler() turns a job into a cron-scheduled task, like a weekly newsletter every Monday at 9am, with no separate cron server. Use it instead of the older repeat option, which BullMQ v6 removed. It creates or updates one schedule by id, so a redeploy doesn't add a second copy of the job.

import { Worker } from 'bullmq';

const worker = new Worker(
  'email',
  async (job) => {
    switch (job.name) {
      case 'send-welcome':
        return sendWelcomeEmail(job.data);
      case 'send-newsletter':
        return sendNewsletter(job.data);
      default:
        throw new Error(`Unknown job name: ${job.name}`);
    }
  },
  {
    connection,
    concurrency: 5, // fine for I/O-bound work like sending emails
    limiter: { max: 10, duration: 1000 }, // 10 jobs/second for the whole queue, across all workers
  }
);

// Cron-style recurring job: every Monday at 9am, no separate cron server needed.
// upsertJobScheduler (BullMQ 5.16+, and the only way in v6) creates or updates
// one schedule by id, so redeploying doesn't add a second copy of the job.
await emailQueue.upsertJobScheduler(
  'weekly-newsletter',
  { pattern: '0 9 * * 1' },
  { name: 'send-newsletter', data: {} }
);

The dead letter queue pattern

BullMQ doesn't have a built-in dead letter queue, but the pattern is simple to add: listen for the worker's failed event, check if job.attemptsMade has reached the configured limit, and if so, push the job's data and failure reason into a separate email-dead queue instead of letting it disappear. That second queue becomes a place to inspect what's permanently failing, understand why, and manually replay jobs once the underlying issue is fixed — without a dead letter queue, jobs that exhaust every retry are simply gone, with only a log line to show for it.

import { Queue } from 'bullmq';

const emailDeadQueue = new Queue('email-dead', { connection });

worker.on('failed', async (job, error) => {
  if (!job) return;

  const attemptsAllowed = job.opts.attempts ?? 1;
  const noMoreRetries =
    job.attemptsMade >= attemptsAllowed ||
    error.name === 'UnrecoverableError'; // thrown to skip retries
  if (!noMoreRetries) return;

  try {
    await emailDeadQueue.add('failed-job', {
      originalName: job.name,
      data: job.data,
      failedReason: error.message,
    });
  } catch (err) {
    console.error('Could not move job to the dead letter queue', job.id, err);
  }
});

Using BullMQ inside NestJS

@nestjs/bullmq gives first-class integration. BullModule.forRoot({ connection: {...} }) goes in AppModule and configures the shared Redis connection once. BullModule.registerQueue({ name: 'email' }) registers each queue in the module that uses it. From there, @InjectQueue('email') private emailQueue: Queue makes the queue injectable into any service, so a UsersService can call this.emailQueue.add('send-welcome', {...}) right after creating a user, without blocking the response on sending that email. The worker side is a class decorated with @Processor('email') extending WorkerHost, with a process(job) method that plays the same role as the plain processor function. Its dependencies, like a mailer, come in through the constructor like any other NestJS class.

If decorators like @Processor and constructor injection are new to you, NestJS fundamentals: controllers, services and modules covers how Nest wires these classes together.

// app.module.ts: one shared Redis connection for every queue
@Module({
  imports: [
    BullModule.forRoot({
      // No maxRetriesPerRequest: BullMQ sets null on worker connections itself,
      // and producers keep the default so add() fails fast if Redis is down.
      connection: { host: 'localhost', port: 6379 },
    }),
    EmailModule,
  ],
})
export class AppModule {}

// email.module.ts
@Module({
  imports: [
    BullModule.registerQueue({ name: 'email' }),
    SequelizeModule.forFeature([User]),
  ],
  providers: [EmailProcessor, UsersService, MailerService], // MailerService: your own email-sending service
})
export class EmailModule {}

// email.processor.ts
@Processor('email')
export class EmailProcessor extends WorkerHost {
  constructor(private mailer: MailerService) {
    super();
  }

  async process(job: Job): Promise<void> {
    switch (job.name) {
      case 'send-welcome':
        await this.mailer.sendWelcome(job.data);
        break;
      default:
        // Without this, an unknown job name would "complete" silently.
        throw new Error(`Unknown job name: ${job.name}`);
    }
  }
}

// users.service.ts
@Injectable()
export class UsersService {
  constructor(
    @InjectModel(User) private userModel: typeof User,
    @InjectQueue('email') private emailQueue: Queue,
  ) {}

  async createUser(input: CreateUserInput) {
    const user = await this.userModel.create({ ...input });
    await this.emailQueue.add('send-welcome', {
      userId: user.id,
      email: user.email,
    });
    return user;
  }
}

BullMQ vs Bull

Bull is BullMQ's predecessor from the same authors. Its README says it is in maintenance mode, getting bug fixes only, and points to BullMQ for new features. BullMQ is a rewrite in TypeScript, and it's where things like Job Schedulers, flows (parent and child jobs) and job deduplication were added. For a new Node.js project, use BullMQ. An existing Bull setup that works is still a safe, well-tested library, so there's no emergency, but plan any migration as a code change, since the APIs differ.

Mistakes that catch almost everyone once

• –Passing your own ioredis instance to a Worker without maxRetriesPerRequest: null, or setting it on the producer's Queue, where adding a job then hangs while Redis is down. Covered above, but worth repeating because it's that common.
• –Catching an error inside the processor and returning success anyway — BullMQ only knows a job failed if the processor actually throws. Swallowing the error and returning { success: false } tells BullMQ the job succeeded, and it will never retry.
• –Forgetting removeOnComplete and removeOnFail — by default BullMQ keeps every job in Redis forever, which quietly exhausts memory on a high-throughput queue.
• –Skipping graceful shutdown — if the process is killed mid-job, that job gets marked stalled and retried, which can mean the same email gets sent twice. worker.close() on SIGTERM waits for active jobs to finish before the process actually exits.
• –Retrying errors that can never succeed. A validation error with five attempts and exponential backoff just fails 15 seconds later (1 + 2 + 4 + 8). Throw UnrecoverableError instead.
• –Assuming a job runs exactly once. Retries and stalled-job recovery both re-run jobs, so side effects need to be idempotent.

Common questions

Does BullMQ retry failed jobs by default?

No. attempts defaults to 1, so a job that throws goes straight to failed. Set attempts, and usually a backoff, on each job or in the queue's defaultJobOptions.

How do I retry a failed job in BullMQ?

Automatically, with attempts and backoff. Manually, with job.retry() for one job or queue.retryJobs() for every failed job. Pass resetAttemptsMade: true to job.retry() if the job should get its full retries again.

How do I stop BullMQ from retrying a job?

Throw UnrecoverableError from the processor and the job fails immediately. If you use a custom backoff strategy, returning -1 from it does the same.

What's the difference between attempts and backoff?

attempts is how many times a job may run in total, including the first run. backoff is how long BullMQ waits between those runs.

Should I use Bull or BullMQ?

BullMQ for anything new. Bull is in maintenance mode and only gets bug fixes.

What's next

The shift BullMQ asks for is small: stop treating "send the email" as part of the request, and start treating it as a fact you record and act on later. The diagram at the top ends with "Sync to Elasticsearch", and that's the next post. For how queues fit with PostgreSQL, Redis and search, see how I'm learning backend development. I'm a frontend engineer who builds the UI on top of APIs like this. If you need that, see my work or get in touch.