{"id":9656,"date":"2026-10-12T13:33:37","date_gmt":"2026-10-12T12:33:37","guid":{"rendered":"https:\/\/upcloud.com\/global\/?post_type=tutorial&p=9656"},"modified":"2026-10-12T13:33:37","modified_gmt":"2026-10-12T12:33:37","slug":"rabbitmq-monitoring-upcloud","status":"publish","type":"tutorial","link":"https:\/\/upcloud.com\/global\/resources\/tutorials\/rabbitmq-monitoring-upcloud\/","title":{"rendered":"Part 4: RabbitMQ Monitoring, Scaling, and Operations on UpCloud"},"content":{"rendered":"\n

The RabbitMQ cluster setup<\/strong><\/a> from Part 3 passes its initial tests. Nodes join, quorum queues replicate, and elections happen when a leader goes down. What comes next is operating that cluster for months. Most RabbitMQ problems won\u2019t show up during setup. They will appear under real workloads, such as when a consumer falls behind, when disk fills faster than expected, or when a node needs to be replaced.<\/p>\n\n\n\n

This part covers the operational aspects of running a three-node RabbitMQ cluster on UpCloud, with focusing on RabbitMQ monitoring, load balancing, and scaling. We\u2019ll route client traffic through the load balancer, write clients that handle failure correctly, read the metrics that matter, and work through the failure modes and maintenance tasks we\u2019ll encounter in practice.<\/p>\n\n\n\n

Traffic Routing with UpCloud Load Balancer<\/h2>\n\n\n\n

AMQP uses long-lived, stateful TCP connections. A typical application opens one AMQP connection, multiplexes several channels over it, and keeps that connection open for the lifetime of the process. This is meaningfully different from HTTP, where a load balancer can distribute individual requests across backends because each request uses a short-lived connection.<\/p>\n\n\n\n

For AMQP, the load balancer\u2019s primary job is failover routing. It provides clients with a single, stable endpoint and stops directing new connections to unhealthy nodes. Once a client connects to a broker node, it remains connected to that node for the lifetime of the connection. The load balancer\u2019s role ends at initial connection placement and failover.<\/p>\n\n\n\n

Setting up the UpCloud Load Balancer<\/h3>\n\n\n\n

For this RabbitMQ load balancing setup, create an UpCloud Load Balancer<\/a> in the same region as the RabbitMQ nodes. Configure a backend for each RabbitMQ node using its private SDN IP address and port 5672. The load balancer should use the private SDN network to reach the backends, not the public network.<\/p>\n\n\n\n

For the health check, use an HTTP check against the RabbitMQ management API\u2019s alarm endpoint on port 15672:<\/p>\n\n\n\n

GET \/api\/health\/checks\/alarms<\/p>\n\n\n\n

This endpoint returns HTTP 200 when the node has no active alarms and HTTP 503 when memory or disk alarms are active. A node with an active alarm is still technically running, but it\u2019s blocking publishers and shouldn\u2019t receive new client connections. Using this endpoint as the health check means the load balancer routes traffic away from nodes under pressure, not just nodes that are completely down.<\/p>\n\n\n\n

Configure the health check with a short interval (10-15 seconds) and a low threshold for marking nodes unhealthy (2-3 consecutive failures). RabbitMQ alarm states change quickly, and we want the load balancer to stop routing to an alarmed node before a queue of blocked publishers builds up.<\/p>\n\n\n\n

The health check needs credentials. Create a dedicated monitoring user with the monitoring management tag and no other permissions:<\/p>\n\n\n\n

rabbitmqctl add_user monitor ‘choose-a-monitoring-password’<\/p>\n\n\n\n

rabbitmqctl set_user_tags monitor monitoring<\/p>\n\n\n\n

The monitoring tag grants read-only access to the management API, which is enough for health checks.<\/p>\n\n\n\n

Separating AMQP and management traffic<\/h3>\n\n\n\n

Do not use the same load balancer rule for AMQP client traffic and management UI access. AMQP traffic (port 5672 or 5671 with TLS) belongs behind the load balancer. The management UI (port 15672) should be accessed directly via individual node IPs or a separate restricted route, not through the same load balancer that our application clients use.<\/p>\n\n\n\n

The reason is operational clarity. During an incident, we want direct access to each node\u2019s management API without going through a load balancer that may be routing around that node. If we only have load-balanced management access, we may not be able to inspect the node that is causing the problem.<\/p>\n\n\n\n

What the load balancer doesn\u2019t fix<\/h3>\n\n\n\n

The load balancer doesn\u2019t make queues available when they don\u2019t have quorum. If two of three nodes are down, the load balancer can route a client connection to the one surviving node, but quorum queues on that node won\u2019t accept writes because a majority of replicas isn\u2019t available. The client will connect successfully and then receive errors when it tries to publish. Quorum is a separate guarantee from connectivity.<\/p>\n\n\n\n

Client-Side Reliability<\/h2>\n\n\n\n

Our cluster can handle broker-level node failures. What happens to our application during those failures depends almost entirely on how we write the clients. connection retry logic, publisher confirms, and redelivery handling are all application-level responsibilities.<\/p>\n\n\n\n

Connection retries and backoff<\/h3>\n\n\n\n

AMQP connections close when a broker node restarts, a network partition occurs, or the load balancer routes the client to a different backend after a health check failure. The application must detect the closure and attempt to reconnect.<\/p>\n\n\n\n

Most AMQP client libraries support automatic connection recovery, but it\u2019s often disabled by default or not configured with sensible backoff. When a broker fails and then recovers, every client connected to it will try to reconnect at roughly the same time. Without backoff and jitter, the reconnect storm can overwhelm the recovering node.<\/p>\n\n\n\n

We should configure automatic connection recovery with exponential backoff. A reasonable starting interval is 2-5 seconds, increasing up to 30-60 seconds with randomized jitter:<\/p>\n\n\n\n

import pika<\/p>\n\n\n\n

import time<\/p>\n\n\n\n

import random<\/p>\n\n\n\n

def connect_with_backoff(host, credentials, max_attempts=10):<\/p>\n\n\n\n

    delay = 2<\/p>\n\n\n\n

    for attempt in range(max_attempts):<\/p>\n\n\n\n

        try:<\/p>\n\n\n\n

            parameters = pika.ConnectionParameters(<\/p>\n\n\n\n

                host=host,<\/p>\n\n\n\n

                credentials=credentials,<\/p>\n\n\n\n

                heartbeat=60,<\/p>\n\n\n\n

                blocked_connection_timeout=300<\/p>\n\n\n\n

            )<\/p>\n\n\n\n

            return pika.BlockingConnection(parameters)<\/p>\n\n\n\n

        except pika.exceptions.AMQPConnectionError:<\/p>\n\n\n\n

            if attempt == max_attempts – 1:<\/p>\n\n\n\n

                raise<\/p>\n\n\n\n

            jitter = random.uniform(0, delay * 0.2)<\/p>\n\n\n\n

            time.sleep(delay + jitter)<\/p>\n\n\n\n

            delay = min(delay * 2, 60)<\/p>\n\n\n\n

Publisher confirms<\/h3>\n\n\n\n

Publisher confirms are the only reliable mechanism for knowing that a message has been durably committed by the broker. Without them, publishing is fire-and-forget. A message in flight when the broker crashes is lost with no signal to the producer.<\/p>\n\n\n\n

We should enable confirm mode on a channel and handle acks and nacks:<\/p>\n\n\n\n

unconfirmed = {}<\/p>\n\n\n\n

next_seq = 1 # broker assigns sequence numbers starting at 1, incrementing per publish<\/p>\n\n\n\n

def on_delivery_confirmed(frame):<\/p>\n\n\n\n

    delivery_tag = frame.method.delivery_tag<\/p>\n\n\n\n

    multiple = frame.method.multiple<\/p>\n\n\n\n

    is_nack = frame.method.NAME == ‘Basic.Nack’<\/p>\n\n\n\n

    # ‘multiple’ means all tags up to and including delivery_tag are confirmed<\/p>\n\n\n\n

    tags = [t for t in list(unconfirmed) if t <= delivery_tag] if multiple else [delivery_tag]<\/p>\n\n\n\n

    for tag in tags:<\/p>\n\n\n\n

        msg = unconfirmed.pop(tag, None)<\/p>\n\n\n\n

        if is_nack and msg:<\/p>\n\n\n\n

            retry_or_dead_letter(msg)<\/p>\n\n\n\n

# Pass the Ack\/Nack callback directly to confirm_delivery (async Channel only)<\/p>\n\n\n\n

channel.confirm_delivery(ack_nack_callback=on_delivery_confirmed)<\/p>\n\n\n\n

# Track the sequence number before publishing; basic_publish returns None<\/p>\n\n\n\n

global next_seq<\/p>\n\n\n\n

seq = next_seq<\/p>\n\n\n\n

next_seq += 1<\/p>\n\n\n\n

unconfirmed[seq] = message_body<\/p>\n\n\n\n

channel.basic_publish(<\/p>\n\n\n\n

    exchange=”,<\/p>\n\n\n\n

    routing_key=’orders’,<\/p>\n\n\n\n

    body=message_body,<\/p>\n\n\n\n

    properties=pika.BasicProperties(delivery_mode=2),  # persistent<\/p>\n\n\n\n

    mandatory=True<\/p>\n\n\n\n

)<\/p>\n\n\n\n

A nack from the broker doesn\u2019t mean the message was lost. It means the broker couldn\u2019t commit it at that moment, often because of a transient overload or a failing node. The correct response to that is to retry with backoff, not to drop the message.<\/p>\n\n\n\n

Publisher confirms have a throughput cost because every write must be acknowledged before the producer can treat it as committed. For workloads where throughput matters, we can publish in batches and wait for confirms on the batch rather than confirming each message individually.<\/p>\n\n\n\n

Consumer acknowledgements<\/h3>\n\n\n\n

A consumer should acknowledge a message only after it has finished the work associated with that message. If the consumer crashes before acknowledging the message, RabbitMQ redelivers it to another consumer. If the consumer acknowledges early and then crashes, the work is lost.<\/p>\n\n\n\n

Let\u2019s consider setting a prefetch count with basic.qos before starting consumers. Without a prefetch limit, RabbitMQ will deliver all available messages to the first consumer that connects, which can overwhelm a slow consumer and cause unacknowledged counts to spike:<\/p>\n\n\n\n

channel.basic_qos(prefetch_count=20)<\/p>\n\n\n\n

def process_message(channel, method, properties, body):<\/p>\n\n\n\n

    try:<\/p>\n\n\n\n

        do_work(body)<\/p>\n\n\n\n

        channel.basic_ack(delivery_tag=method.delivery_tag)<\/p>\n\n\n\n

    except Exception:<\/p>\n\n\n\n

        # Reject and do not requeue; route to dead letter exchange<\/p>\n\n\n\n

        channel.basic_nack(delivery_tag=method.delivery_tag, requeue=False)<\/p>\n\n\n\n

channel.basic_consume(queue=’orders’, on_message_callback=process_message)<\/p>\n\n\n\n

Use requeue=False when a message fails processing and shouldn\u2019t be retried immediately. With requeue=True, a rejected message goes back to the front of the queue and will be immediately redelivered to the same or another consumer. For messages that consistently fail to process, this creates a tight retry loop that can overwhelm the queue and prevent other messages from being delivered. Route failures to a dead letter exchange instead.<\/p>\n\n\n\n

Handling unroutable messages<\/h3>\n\n\n\n

A message published to an exchange with a routing key that matches no bound queue is silently discarded by default. The mandatory flag changes this: when set, RabbitMQ returns unroutable messages to the producer via a basic.return callback instead of dropping them.<\/p>\n\n\n\n

If we can\u2019t always set mandatory on every publish call, configure an alternate exchange on the exchange itself. An alternate exchange receives any message that the primary exchange can\u2019t route:<\/p>\n\n\n\n

rabbitmqadmin -u admin -p yourpassword -V production \\<\/p>\n\n\n\n

declare exchange name=app.events type=topic durable=true \\<\/p>\n\n\n\n

arguments='{“alternate-exchange”:”app.unrouted”}’<\/p>\n\n\n\n

rabbitmqadmin -u admin -p yourpassword -V production \\<\/p>\n\n\n\n

declare exchange name=app.unrouted type=fanout durable=true<\/p>\n\n\n\n

rabbitmqadmin -u admin -p yourpassword -V production \\<\/p>\n\n\n\n

declare queue name=unrouted-messages durable=true \\<\/p>\n\n\n\n

arguments='{“x-queue-type”:”quorum”}’<\/p>\n\n\n\n

rabbitmqadmin -u admin -p yourpassword -V production \\<\/p>\n\n\n\n

declare binding source=app.unrouted destination=unrouted-messages routing_key=””<\/p>\n\n\n\n

Messages that reach the alternate exchange are usually a symptom of a routing configuration problem, a stale binding, or a producer publishing to the wrong key. Monitoring the depth of the unrouted messages queue gives us visibility into these issues rather than silently discarding the messages.<\/p>\n\n\n\n

Idempotency<\/h3>\n\n\n\n

RabbitMQ guarantees at-least-once delivery when the publisher confirms, and consumer acknowledgments are in use. That means a consumer may receive the same message more than once: when a connection drops before an ack is sent, when a consumer crashes mid-processing, or when a message is nacked with requeue. The redelivered flag in the message properties will be set to true for redeliveries. A false value guarantees the message has not been seen before. A true value means it may have been seen before, but it doesn\u2019t guarantee prior processing was completed, so we can\u2019t use it alone to skip work safely.<\/p>\n\n\n\n

This is why we should design consumers to handle duplicates safely. A common approach is to use the message\u2019s message_id property as an idempotency key and check whether that ID has already been processed before doing the work. Use a fast-lookup store for this check, such as Redis with a short TTL, rather than a full database write on every check.<\/p>\n\n\n\n

RabbitMQ Scaling Considerations<\/h2>\n\n\n\n

A three-node RabbitMQ cluster has a fixed capacity ceiling. When the workload grows beyond what the cluster can handle, we have a few options, and they aren\u2019t all the same.<\/p>\n\n\n\n

Signals that the cluster needs more capacity<\/h3>\n\n\n\n

Memory alarms activating under normal load (not just during traffic spikes) usually indicate that the memory threshold is too low or that the node genuinely doesn\u2019t have enough RAM for the workload. Check rabbitmqctl status to see the current memory usage and compare it to the threshold we set.<\/p>\n\n\n\n

Publisher flow control is the visible symptom of a constrained cluster. It can be triggered by memory pressure, disk pressure, or slow consumers, causing queue buildup. First, isolate which signal is triggering it before deciding on a response. Adding more nodes won\u2019t fix the flow control caused by a consumer that\u2019s too slow to drain a queue.<\/p>\n\n\n\n

Persistent queue backlogs, where the messages-ready count grows over hours without recovering, indicate that consumers can\u2019t keep up with the publish rate. This is an application-level problem (not enough consumer capacity) more often than it\u2019s an infrastructure problem.<\/p>\n\n\n\n

Why RabbitMQ doesn\u2019t scale horizontally without limits<\/h3>\n\n\n\n

Adding more nodes to a RabbitMQ cluster increases total resource capacity but doesn\u2019t linearly increase throughput for individual queues. Each quorum queue has a fixed replication group, and that group\u2019s leader handles all writes for that queue regardless of how many nodes the cluster has.<\/p>\n\n\n\n

Erlang\u2019s distribution protocol (which RabbitMQ uses internally) has overhead that grows with the number of nodes. Every node must exchange heartbeats, metadata, and cluster membership information with every other node. The metadata store uses Raft-based quorum replication, meaning every metadata write must be committed by a majority of nodes. With many nodes, metadata operations become a bottleneck.<\/p>\n\n\n\n

The RabbitMQ team\u2019s practical guidance is to run 3 or 5 nodes for most production workloads. This number is enough for quorum without significant inter-node overhead. Larger clusters are possible but should be validated against our specific throughput requirements.<\/p>\n\n\n\n

Vhosts as a workload isolation mechanism<\/h3>\n\n\n\n

Before adding nodes or splitting clusters, consider whether vhosts would help. A vhost is a logical namespace within a single RabbitMQ cluster. Each vhost has its own exchanges, queues, bindings, users, and policies. Applications using different vhosts don\u2019t share queue names or permissions, which is useful for multitenancy or separating workloads with different access controls.<\/p>\n\n\n\n

Vhosts share underlying cluster resources. However, they don\u2019t provide memory, CPU, or disk isolation. A workload in one vhost that fills the disk or triggers a memory alarm affects all vhosts on those nodes. This is why we should use vhosts when we need namespace separation and access control isolation, not when we need resource isolation.<\/p>\n\n\n\n

When separate clusters make more sense<\/h3>\n\n\n\n

Separate clusters make sense when workloads have meaningfully different requirements, and you want failure domains to be independent. A payment processing queue and a low-priority notification queue should probably be on different clusters, because a disk issue on the payment cluster shouldn\u2019t affect notifications, and vice versa. Running them together saves infrastructure cost but creates a coupling we may regret during an incident.<\/p>\n\n\n\n

Separate clusters also make sense when different teams own different workloads and need independent upgrade cadences, configuration ownership, or operational responsibility. A shared cluster that multiple teams depend on tends to slow down maintenance and create coordination overhead.<\/p>\n\n\n\n

RabbitMQ Monitoring: Key Metrics to Watch<\/h2>\n\n\n\n

RabbitMQ exposes a large amount of telemetry. Most of it is useful for debugging specific incidents, but a smaller set of metrics is worth watching continuously.<\/p>\n\n\n\n

Here are the metrics that matter for daily operations:<\/p>\n\n\n\n

    \n
  1. Messages ready<\/em> is the number of messages waiting in a queue to be delivered to a consumer. A queue where messages-ready consistently grows over time has consumers that can\u2019t keep up with the publish rate. A sudden spike followed by recovery is normal. A steady upward trend signals that consumers can\u2019t keep up.<\/li>\n\n\n\n
  2. Messages unacknowledged<\/em> is how many messages have been delivered to consumers but not yet acknowledged. This number is bounded by our prefetch configuration. If it stays consistently at the prefetch ceiling, we may not have enough consumer concurrency to process the current delivery rate.<\/li>\n\n\n\n
  3. Memory used per node compared to the memory threshold<\/em> shows how much headroom remains before flow control activates. We want an alert at roughly 80% of the threshold, which gives us time to investigate before publishers start blocking.<\/li>\n\n\n\n
  4. Disk free on the data volume compared to the alarm threshold<\/em> alerts us well before the alarm value. By the time the alarm activates, publishers are already blocked.<\/li>\n\n\n\n
  5. Channel count per connection<\/em> reveals misconfigured clients. A connection with hundreds of channels is almost always a client that is opening channels without closing them. Each channel is an Erlang process, and a leak here will drive the Erlang process count up over time.<\/li>\n\n\n\n
  6. Erlang process count<\/em> is a cluster-level health indicator. A stable cluster with consistent traffic has a roughly stable process count. A process count that grows over hours without corresponding traffic growth suggests a leak, usually from unclosed connections or channels.<\/li>\n\n\n\n
  7. Connection count<\/em> and connection events<\/em> can surface reconnect storms (sudden spike in new connections after a node restart), client misconfiguration (clients that keep connecting and immediately disconnecting), or unexpected traffic growth.<\/li>\n<\/ol>\n\n\n\n

    We can use a common open-source platform like Prometheus to set up monitoring.<\/p>\n\n\n\n

    Setting up the Prometheus plugin<\/h3>\n\n\n\n

    To send metrics from our RabbitMQ cluster to Prometheus, we need to first enable the rabbitmq_prometheus plugin on all three nodes:<\/p>\n\n\n\n

    rabbitmq-plugins enable rabbitmq_prometheus<\/p>\n\n\n\n

    The plugin exposes metrics at http:\/\/node-ip:15692\/metrics in Prometheus format. Now, configure the Prometheus scrape job to collect from all three nodes:<\/p>\n\n\n\n

    scrape_configs:<\/p>\n\n\n\n

      – job_name: rabbitmq<\/p>\n\n\n\n

        static_configs:<\/p>\n\n\n\n

          – targets:<\/p>\n\n\n\n

              – rmq1.internal:15692<\/p>\n\n\n\n

              – rmq2.internal:15692<\/p>\n\n\n\n

              – rmq3.internal:15692<\/p>\n\n\n\n

        metrics_path: \/metrics<\/p>\n\n\n\n

    The RabbitMQ team maintains official Grafana dashboard templates at grafana.com\/grafana\/dashboards<\/a>. We can search for \u201cRabbitMQ\u201d to find the current dashboards. The \u201cRabbitMQ-Overview\u201d dashboard (dashboard ID 10991)<\/a> is a good starting point. It covers queue depths, message rates, node resource usage, and connection counts in one view.<\/p>\n\n\n\n

    Make sure to add alert rules to our monitoring system for:<\/p>\n\n\n\n