Message Queue

In Turkish: Mesaj Kuyruğu
Pronunciation: MES-ij KYOO

Updated September 30, 20263 min read

In short

A message queue is a component that stores messages from one service until another is ready to process them, so parts of a system can work asynchronously.

What is a message queue?

A message queue lets one part of a system, the producer, send a message without waiting for the part that handles it, the consumer. The queue stores each message safely until a consumer picks it up, processes it, and acknowledges it, after which the message is removed. Messages are usually small pieces of data, such as JSON describing a task or an event.

This makes the system asynchronous and decoupled: the producer and consumer don't need to be running at the same time or at the same speed. If a sudden burst of orders arrives, messages simply wait in the queue while workers process them at a steady pace, and you can add more workers to go faster. Popular tools include RabbitMQ, Amazon SQS, Redis-based job queues, and Apache Kafka, which is technically a distributed log but is often used for similar jobs.

Think of the order rail in a restaurant kitchen: waiters clip tickets to the rail and return to their tables, and cooks take the tickets one by one as they become free. Message queues are commonly used for sending emails, processing images and videos, handling payments, and passing events between microservices.

Most queues guarantee at-least-once delivery, which means a message can occasionally arrive twice, for example if a consumer crashes before acknowledging it. Consumers should therefore be idempotent, so processing the same message twice has the same effect as processing it once, and messages that keep failing are usually moved to a dead-letter queue for inspection. A queue also differs from publish-subscribe (pub/sub): in a queue each message is handled by one consumer, while pub/sub delivers a copy of every message to each subscriber.

At a glance

Producers don't wait: messages sit in the queue, in order, until a consumer processes one and sends an ack, and only then is it removed.

Key takeaways

Producers send messages; consumers process them later, at their own pace.
The queue decouples services and absorbs traffic spikes.
Consumers acknowledge messages after processing, so work isn't lost if they crash.
At-least-once delivery means message handlers should be idempotent.
In a queue each message goes to one consumer; in pub/sub every subscriber gets a copy.

Example

Producing and consuming jobs with RabbitMQjavascript

// RabbitMQ via the amqplib package
const channel = await connection.createChannel();
await channel.assertQueue("emails", { durable: true });

// Producer: put a job on the queue and move on without waiting
const job = { to: "ada@example.com", template: "welcome" };
channel.sendToQueue("emails", Buffer.from(JSON.stringify(job)), { persistent: true });

// Consumer: take one job at a time and acknowledge it when done
await channel.prefetch(1);
await channel.consume("emails", async (msg) => {
  const task = JSON.parse(msg.content.toString());
  await sendEmail(task.to, task.template);
  channel.ack(msg); // only now is the message removed from the queue
});

Readers ask

What is the difference between a message queue and a message broker?

A message broker is the server software, such as RabbitMQ, that receives, stores, and routes messages. A message queue is one of the structures a broker manages, although people often use the two terms interchangeably.

Is Kafka a message queue?

Kafka is primarily a distributed event streaming platform that keeps messages in an ordered, replayable log instead of deleting them after delivery. It is often used for the same jobs as a message queue, but consumers track their own position and many independent consumer groups can read the same data.

Why use a message queue instead of calling a service directly?

A queue lets the caller respond immediately instead of waiting for slow work, keeps messages safe while the receiving service is down, and smooths out traffic spikes. The trade-off is extra infrastructure and the need to handle delays and duplicate messages.

Often compared

Message Queue vs Pub/SubIn a message queue each message goes to one consumer, so workers share the load, while pub/sub copies every message to each subscriber so many services react.