After this lesson, you will be able to: Decide when to introduce a queue, compare pub/sub vs work queue, and pick from Redis / RabbitMQ / SQS / Kafka for a specific workload.
Queues are how systems handle work that doesn't need to happen RIGHT NOW. They also decouple producers from consumers, making both easier to scale and replace.
Some work shouldn't block the user. Sending a welcome email, generating a thumbnail, running a report, charging a card. The user submitted the form; they don't want to wait 8 seconds for the side effects. A queue lets the producer (web request handler) enqueue a job and return immediately; the consumer (background worker) processes it later. Bonus: if the consumer is down, the queue holds the work until it's back up. The system stays available even when a component is broken.
Work queue (point-to-point): each message is delivered to exactly ONE consumer. Used for background jobs (email, image processing). Multiple consumers compete for messages; scaling = adding more consumers. Pub/sub (broadcast): each message is delivered to ALL subscribers. Used for events (user created → notify analytics + CRM + audit log). Each subscriber is independent. The same broker (Redis, RabbitMQ, Kafka) can usually do both with different APIs.
Redis (with BullMQ / Sidekiq / Resque): tiny ops surface, perfect for small-to-medium work queues. Default for most startups. RabbitMQ: classic AMQP broker, mature, strong routing. Used in enterprises and Erlang/Elixir shops. AWS SQS: managed queue, near-zero ops, pay-per-message. Default if you're on AWS and don't want to operate anything. Kafka: high-throughput durable log, designed for analytics and event sourcing at scale. Overkill for typical CRUD work queues; necessary for event-streaming architectures. Vercel/Inngest/Trigger.dev: code-first workflow runners that abstract the queue away. Easiest start; some lock-in.
TypeScript example. BullMQ is the modern Node.js queue library.
// queue.ts (the producer side)import { Queue } from "bullmq";import { redis } from "./redis";export const emailQueue = new Queue("email", { connection: redis });// Anywhere in your app:await emailQueue.add("welcome",{ attempts: 5, backoff: { type: "exponential", delay: 1000 } },);// worker.ts (the consumer side, runs as a separate process)import { Worker } from "bullmq";import { redis } from "./redis";import { sendWelcomeEmail } from "./email";new Worker("email",async (job) => {if (job.name === "welcome") {await sendWelcomeEmail(job.data.userId, job.data.email);}},{ connection: redis, concurrency: 5 },);// Deploy the worker separately from the web app. Scale them independently.// BullMQ handles retries, dead-letter queues, scheduling, and a built-in dashboard.
Almost every queue gives at-least-once delivery: a message may be delivered MORE than once (e.g. the worker crashed after doing the work but before acking). Your consumer must be idempotent: processing the same message twice produces the same result. Patterns: track a processed_message_id table; use the message ID as a unique key on the side effect (Stripe idempotency keys); check 'has this already been done?' before doing it. If you skip idempotency, you'll occasionally send the same email twice. Sometimes that's fine; sometimes (payments, refunds) it's catastrophic.
Adding a queue for work that's already fast enough. Synchronous is simpler; only add a queue when latency or decoupling demands it. Forgetting dead-letter queues. Messages that fail repeatedly clog the queue if they don't get parked aside. Configure DLQs explicitly. Treating message delivery as exactly-once. Almost no real queue guarantees this; design for at-least-once. Sharing a queue between unrelated work types. Build separate queues per work type so a slow report job doesn't starve fast emails. Skipping observability. A queue dashboard (BullMQ board, RabbitMQ UI) is the first thing you'll need at 3am.
Pick the cleanest answer.
Sign in and purchase access to unlock this lesson.