The RabbitMQ cluster setup from Part 3 passes its initial tests. Nodes join, quorum queues replicate, and elections happen when a leader goes down. What comes next is operating that cluster for months. Most RabbitMQ problems won’t show up during setup. They will appear under real workloads, such as when a consumer falls behind, when disk fills faster than expected, or when a node needs to be replaced.
This part covers the operational aspects of running a three-node RabbitMQ cluster on UpCloud, with focusing on RabbitMQ monitoring, load balancing, and scaling. We’ll route client traffic through the load balancer, write clients that handle failure correctly, read the metrics that matter, and work through the failure modes and maintenance tasks we’ll encounter in practice.
Traffic Routing with UpCloud Load Balancer
AMQP uses long-lived, stateful TCP connections. A typical application opens one AMQP connection, multiplexes several channels over it, and keeps that connection open for the lifetime of the process. This is meaningfully different from HTTP, where a load balancer can distribute individual requests across backends because each request uses a short-lived connection.
For AMQP, the load balancer’s primary job is failover routing. It provides clients with a single, stable endpoint and stops directing new connections to unhealthy nodes. Once a client connects to a broker node, it remains connected to that node for the lifetime of the connection. The load balancer’s role ends at initial connection placement and failover.
Setting up the UpCloud Load Balancer
For this RabbitMQ load balancing setup, create an UpCloud Load Balancer in the same region as the RabbitMQ nodes. Configure a backend for each RabbitMQ node using its private SDN IP address and port 5672. The load balancer should use the private SDN network to reach the backends, not the public network.
For the health check, use an HTTP check against the RabbitMQ management API’s alarm endpoint on port 15672:
GET /api/health/checks/alarms
This endpoint returns HTTP 200 when the node has no active alarms and HTTP 503 when memory or disk alarms are active. A node with an active alarm is still technically running, but it’s blocking publishers and shouldn’t receive new client connections. Using this endpoint as the health check means the load balancer routes traffic away from nodes under pressure, not just nodes that are completely down.
Configure the health check with a short interval (10-15 seconds) and a low threshold for marking nodes unhealthy (2-3 consecutive failures). RabbitMQ alarm states change quickly, and we want the load balancer to stop routing to an alarmed node before a queue of blocked publishers builds up.
The health check needs credentials. Create a dedicated monitoring user with the monitoring management tag and no other permissions:
rabbitmqctl add_user monitor ‘choose-a-monitoring-password’
rabbitmqctl set_user_tags monitor monitoring
The monitoring tag grants read-only access to the management API, which is enough for health checks.
Separating AMQP and management traffic
Do not use the same load balancer rule for AMQP client traffic and management UI access. AMQP traffic (port 5672 or 5671 with TLS) belongs behind the load balancer. The management UI (port 15672) should be accessed directly via individual node IPs or a separate restricted route, not through the same load balancer that our application clients use.
The reason is operational clarity. During an incident, we want direct access to each node’s management API without going through a load balancer that may be routing around that node. If we only have load-balanced management access, we may not be able to inspect the node that is causing the problem.
What the load balancer doesn’t fix
The load balancer doesn’t make queues available when they don’t have quorum. If two of three nodes are down, the load balancer can route a client connection to the one surviving node, but quorum queues on that node won’t accept writes because a majority of replicas isn’t available. The client will connect successfully and then receive errors when it tries to publish. Quorum is a separate guarantee from connectivity.
Client-Side Reliability
Our cluster can handle broker-level node failures. What happens to our application during those failures depends almost entirely on how we write the clients. connection retry logic, publisher confirms, and redelivery handling are all application-level responsibilities.
Connection retries and backoff
AMQP connections close when a broker node restarts, a network partition occurs, or the load balancer routes the client to a different backend after a health check failure. The application must detect the closure and attempt to reconnect.
Most AMQP client libraries support automatic connection recovery, but it’s often disabled by default or not configured with sensible backoff. When a broker fails and then recovers, every client connected to it will try to reconnect at roughly the same time. Without backoff and jitter, the reconnect storm can overwhelm the recovering node.
We should configure automatic connection recovery with exponential backoff. A reasonable starting interval is 2-5 seconds, increasing up to 30-60 seconds with randomized jitter:
import pika
import time
import random
def connect_with_backoff(host, credentials, max_attempts=10):
delay = 2
for attempt in range(max_attempts):
try:
parameters = pika.ConnectionParameters(
host=host,
credentials=credentials,
heartbeat=60,
blocked_connection_timeout=300
)
return pika.BlockingConnection(parameters)
except pika.exceptions.AMQPConnectionError:
if attempt == max_attempts – 1:
raise
jitter = random.uniform(0, delay * 0.2)
time.sleep(delay + jitter)
delay = min(delay * 2, 60)
Publisher confirms
Publisher confirms are the only reliable mechanism for knowing that a message has been durably committed by the broker. Without them, publishing is fire-and-forget. A message in flight when the broker crashes is lost with no signal to the producer.
We should enable confirm mode on a channel and handle acks and nacks:
unconfirmed = {}
next_seq = 1 # broker assigns sequence numbers starting at 1, incrementing per publish
def on_delivery_confirmed(frame):
delivery_tag = frame.method.delivery_tag
multiple = frame.method.multiple
is_nack = frame.method.NAME == ‘Basic.Nack’
# ‘multiple’ means all tags up to and including delivery_tag are confirmed
tags = [t for t in list(unconfirmed) if t <= delivery_tag] if multiple else [delivery_tag]
for tag in tags:
msg = unconfirmed.pop(tag, None)
if is_nack and msg:
retry_or_dead_letter(msg)
# Pass the Ack/Nack callback directly to confirm_delivery (async Channel only)
channel.confirm_delivery(ack_nack_callback=on_delivery_confirmed)
# Track the sequence number before publishing; basic_publish returns None
global next_seq
seq = next_seq
next_seq += 1
unconfirmed[seq] = message_body
channel.basic_publish(
exchange=”,
routing_key=’orders’,
body=message_body,
properties=pika.BasicProperties(delivery_mode=2), # persistent
mandatory=True
)
A nack from the broker doesn’t mean the message was lost. It means the broker couldn’t commit it at that moment, often because of a transient overload or a failing node. The correct response to that is to retry with backoff, not to drop the message.
Publisher confirms have a throughput cost because every write must be acknowledged before the producer can treat it as committed. For workloads where throughput matters, we can publish in batches and wait for confirms on the batch rather than confirming each message individually.
Consumer acknowledgements
A consumer should acknowledge a message only after it has finished the work associated with that message. If the consumer crashes before acknowledging the message, RabbitMQ redelivers it to another consumer. If the consumer acknowledges early and then crashes, the work is lost.
Let’s consider setting a prefetch count with basic.qos before starting consumers. Without a prefetch limit, RabbitMQ will deliver all available messages to the first consumer that connects, which can overwhelm a slow consumer and cause unacknowledged counts to spike:
channel.basic_qos(prefetch_count=20)
def process_message(channel, method, properties, body):
try:
do_work(body)
channel.basic_ack(delivery_tag=method.delivery_tag)
except Exception:
# Reject and do not requeue; route to dead letter exchange
channel.basic_nack(delivery_tag=method.delivery_tag, requeue=False)
channel.basic_consume(queue=’orders’, on_message_callback=process_message)
Use requeue=False when a message fails processing and shouldn’t be retried immediately. With requeue=True, a rejected message goes back to the front of the queue and will be immediately redelivered to the same or another consumer. For messages that consistently fail to process, this creates a tight retry loop that can overwhelm the queue and prevent other messages from being delivered. Route failures to a dead letter exchange instead.
Handling unroutable messages
A message published to an exchange with a routing key that matches no bound queue is silently discarded by default. The mandatory flag changes this: when set, RabbitMQ returns unroutable messages to the producer via a basic.return callback instead of dropping them.
If we can’t always set mandatory on every publish call, configure an alternate exchange on the exchange itself. An alternate exchange receives any message that the primary exchange can’t route:
rabbitmqadmin -u admin -p yourpassword -V production \
declare exchange name=app.events type=topic durable=true \
arguments='{“alternate-exchange”:”app.unrouted”}’
rabbitmqadmin -u admin -p yourpassword -V production \
declare exchange name=app.unrouted type=fanout durable=true
rabbitmqadmin -u admin -p yourpassword -V production \
declare queue name=unrouted-messages durable=true \
arguments='{“x-queue-type”:”quorum”}’
rabbitmqadmin -u admin -p yourpassword -V production \
declare binding source=app.unrouted destination=unrouted-messages routing_key=””
Messages that reach the alternate exchange are usually a symptom of a routing configuration problem, a stale binding, or a producer publishing to the wrong key. Monitoring the depth of the unrouted messages queue gives us visibility into these issues rather than silently discarding the messages.
Idempotency
RabbitMQ guarantees at-least-once delivery when the publisher confirms, and consumer acknowledgments are in use. That means a consumer may receive the same message more than once: when a connection drops before an ack is sent, when a consumer crashes mid-processing, or when a message is nacked with requeue. The redelivered flag in the message properties will be set to true for redeliveries. A false value guarantees the message has not been seen before. A true value means it may have been seen before, but it doesn’t guarantee prior processing was completed, so we can’t use it alone to skip work safely.
This is why we should design consumers to handle duplicates safely. A common approach is to use the message’s message_id property as an idempotency key and check whether that ID has already been processed before doing the work. Use a fast-lookup store for this check, such as Redis with a short TTL, rather than a full database write on every check.
RabbitMQ Scaling Considerations
A three-node RabbitMQ cluster has a fixed capacity ceiling. When the workload grows beyond what the cluster can handle, we have a few options, and they aren’t all the same.
Signals that the cluster needs more capacity
Memory alarms activating under normal load (not just during traffic spikes) usually indicate that the memory threshold is too low or that the node genuinely doesn’t have enough RAM for the workload. Check rabbitmqctl status to see the current memory usage and compare it to the threshold we set.
Publisher flow control is the visible symptom of a constrained cluster. It can be triggered by memory pressure, disk pressure, or slow consumers, causing queue buildup. First, isolate which signal is triggering it before deciding on a response. Adding more nodes won’t fix the flow control caused by a consumer that’s too slow to drain a queue.
Persistent queue backlogs, where the messages-ready count grows over hours without recovering, indicate that consumers can’t keep up with the publish rate. This is an application-level problem (not enough consumer capacity) more often than it’s an infrastructure problem.
Why RabbitMQ doesn’t scale horizontally without limits
Adding more nodes to a RabbitMQ cluster increases total resource capacity but doesn’t linearly increase throughput for individual queues. Each quorum queue has a fixed replication group, and that group’s leader handles all writes for that queue regardless of how many nodes the cluster has.
Erlang’s distribution protocol (which RabbitMQ uses internally) has overhead that grows with the number of nodes. Every node must exchange heartbeats, metadata, and cluster membership information with every other node. The metadata store uses Raft-based quorum replication, meaning every metadata write must be committed by a majority of nodes. With many nodes, metadata operations become a bottleneck.
The RabbitMQ team’s practical guidance is to run 3 or 5 nodes for most production workloads. This number is enough for quorum without significant inter-node overhead. Larger clusters are possible but should be validated against our specific throughput requirements.
Vhosts as a workload isolation mechanism
Before adding nodes or splitting clusters, consider whether vhosts would help. A vhost is a logical namespace within a single RabbitMQ cluster. Each vhost has its own exchanges, queues, bindings, users, and policies. Applications using different vhosts don’t share queue names or permissions, which is useful for multitenancy or separating workloads with different access controls.
Vhosts share underlying cluster resources. However, they don’t provide memory, CPU, or disk isolation. A workload in one vhost that fills the disk or triggers a memory alarm affects all vhosts on those nodes. This is why we should use vhosts when we need namespace separation and access control isolation, not when we need resource isolation.
When separate clusters make more sense
Separate clusters make sense when workloads have meaningfully different requirements, and you want failure domains to be independent. A payment processing queue and a low-priority notification queue should probably be on different clusters, because a disk issue on the payment cluster shouldn’t affect notifications, and vice versa. Running them together saves infrastructure cost but creates a coupling we may regret during an incident.
Separate clusters also make sense when different teams own different workloads and need independent upgrade cadences, configuration ownership, or operational responsibility. A shared cluster that multiple teams depend on tends to slow down maintenance and create coordination overhead.
RabbitMQ Monitoring: Key Metrics to Watch
RabbitMQ exposes a large amount of telemetry. Most of it is useful for debugging specific incidents, but a smaller set of metrics is worth watching continuously.
Here are the metrics that matter for daily operations:
- Messages ready is the number of messages waiting in a queue to be delivered to a consumer. A queue where messages-ready consistently grows over time has consumers that can’t keep up with the publish rate. A sudden spike followed by recovery is normal. A steady upward trend signals that consumers can’t keep up.
- Messages unacknowledged is how many messages have been delivered to consumers but not yet acknowledged. This number is bounded by our prefetch configuration. If it stays consistently at the prefetch ceiling, we may not have enough consumer concurrency to process the current delivery rate.
- Memory used per node compared to the memory threshold shows how much headroom remains before flow control activates. We want an alert at roughly 80% of the threshold, which gives us time to investigate before publishers start blocking.
- Disk free on the data volume compared to the alarm threshold alerts us well before the alarm value. By the time the alarm activates, publishers are already blocked.
- Channel count per connection reveals misconfigured clients. A connection with hundreds of channels is almost always a client that is opening channels without closing them. Each channel is an Erlang process, and a leak here will drive the Erlang process count up over time.
- Erlang process count is a cluster-level health indicator. A stable cluster with consistent traffic has a roughly stable process count. A process count that grows over hours without corresponding traffic growth suggests a leak, usually from unclosed connections or channels.
- Connection count and connection events can surface reconnect storms (sudden spike in new connections after a node restart), client misconfiguration (clients that keep connecting and immediately disconnecting), or unexpected traffic growth.
We can use a common open-source platform like Prometheus to set up monitoring.
Setting up the Prometheus plugin
To send metrics from our RabbitMQ cluster to Prometheus, we need to first enable the rabbitmq_prometheus plugin on all three nodes:
rabbitmq-plugins enable rabbitmq_prometheus
The plugin exposes metrics at http://node-ip:15692/metrics in Prometheus format. Now, configure the Prometheus scrape job to collect from all three nodes:
scrape_configs:
– job_name: rabbitmq
static_configs:
– targets:
– rmq1.internal:15692
– rmq2.internal:15692
– rmq3.internal:15692
metrics_path: /metrics
The RabbitMQ team maintains official Grafana dashboard templates at grafana.com/grafana/dashboards. We can search for “RabbitMQ” to find the current dashboards. The “RabbitMQ-Overview” dashboard (dashboard ID 10991) is a good starting point. It covers queue depths, message rates, node resource usage, and connection counts in one view.
Make sure to add alert rules to our monitoring system for:
- memory usage above 80% of threshold
- disk free below 2x the alarm threshold
- any active alarms
- queue depth growing consistently over 15 minutes
- Erlang process count growing unexpectedly, etc.
Common Failure Modes
Here are a few common issues we might run into when using RabbitMQ in production.
Node crash
When a RabbitMQ node stops unexpectedly, the cluster detects the failure through missed heartbeats. This takes a few seconds. During that window, clients connected to the failed node see their connections close, and quorum queues with a leader on that node pause while an election happens.
From a healthy node, we can check what the cluster sees:
rabbitmqctl cluster_status
The failed node should appear in “Disk Nodes” but not “Running Nodes”. We can check whether any queue is missing quorum:
rabbitmqctl list_queues -p production name leader members online
A queue where online shows two members still has quorum and will elect a new leader if the old leader was lost. A queue with only one online member can’t elect a new leader and won’t accept writes until a second member returns.
Once the node comes back online and rejoins, we can verify the cluster is fully healthy:
rabbitmqctl cluster_status
rabbitmqctl list_queues -p production name leader members
With this, we can confirm that all three nodes are in “Running Nodes”.
It is also important to note that queue leaders do not redistribute automatically after a node recovers. The node that won the election during the outage retains leadership. If we want to rebalance leaders across nodes, we can use rabbitmq-queues to rebalance all after the cluster is fully healthy.
Disk full
RabbitMQ activates the disk alarm when free space on the data volume drops below the configured threshold. Once the alarm is active, all publishers on that node block. They don’t receive errors. Their publish calls simply don’t return until the alarm clears. From the application’s perspective, publishing appears to hang.
Consumers aren’t blocked by a disk alarm and can continue draining queues. If we can increase the rate of consumption to reduce queue depth, that frees disk space and may clear the alarm on its own.
For immediate relief, we can expand the MaxIOPS volume through UpCloud Hub. UpCloud lets us resize volumes without stopping the server. Once we’ve increased it, we can resize the filesystem to allow our server to use the new space:
resize2fs /dev/vdb
Then, we can check that the alarm cleared after the resize:
rabbitmqctl status | grep -A5 alarms
An empty alarms section means the alarm is no longer active and publishers will unblock.
If the disk filled because of unexpected message accumulation (a consumer that went offline and left messages building up), address the root cause before the next time. Add monitoring on queue depth and disk usage so we have warning time before the alarm activates.
Network partition
Under pause_minority (the setting from RabbitMQ cluster setup in Part 3), the minority node will pause when it loses connectivity to the other two nodes. The paused node stops accepting client connections and stops serving queue traffic.
To confirm a partition occurred, we can run:
rabbitmqctl cluster_status
We will need to look for content under “Network Partitions” in the output. The management UI also displays a warning prominently when the partition state is recorded.
After connectivity is restored, the paused node should restart itself automatically and rejoin the cluster. Once it does, we need to:
- Verify all three nodes appear in “Running Nodes”
- Check that queue memberships are intact
- Confirm no alarms are active
- Run a quick publish/consume test against each node to confirm the cluster is serving traffic normally.
A partition under pause_minority doesn’t lose committed messages, because only the majority partition continued accepting writes. The minority node’s view of the world is simply behind while it was paused.
Slow consumers
Slow consumers cause messages to accumulate in queues. As queue depth grows, RabbitMQ uses more memory. If the depth grows enough to trigger the memory alarm, publishers block even though the problem is on the consumer side.
We can check which queues are accumulating by running:
rabbitmqctl list_queues -p production name messages messages_ready messages_unacknowledged consumers
A queue, where messages_ready is large, and consumers is nonzero, but messages_unacknowledged is at the prefetch limit, indicates consumers that are receiving messages but processing them slowly. Adding more consumer instances is the immediate fix. If the processing bottleneck is downstream (a slow database, a slow external API), more consumers won’t help, and we need to address the downstream bottleneck.
A queue with zero consumers and a growing messages_ready count means consumers have gone offline. Check whether the consuming service is running and reconnecting.
Publisher flow control
Flow control activates when the broker needs to slow down incoming message rates to protect itself. It can be triggered by the memory alarm, the disk alarm, or a per-queue credit flow when a queue can’t keep up with the publish rate to it.
Flow control that activates during a known traffic burst and clears within a few minutes as the burst ends is probably transient and acceptable. Flow control that activates regularly under normal load means the cluster doesn’t have enough capacity for the workload.
We can check what triggered it by running:
rabbitmqctl status | grep -A10 alarms
rabbitmqctl list_queues -p production name messages messages_ready
If a memory or disk alarm is active, address the alarm. If no alarms are active but flow control is still visible, check whether a specific queue has an unusually large backlog that is causing per-queue flow control.
Backups and Recovery
When dealing with production systems, we must always prepare for failure. And failure can be both downtime and lost data. To prevent data loss, we need to have solid backup and recovery plans in place.
What UpCloud snapshots cover
An UpCloud snapshot captures the full disk state of a server volume at a point in time. This includes the OS, RabbitMQ binaries, configuration files, and the RabbitMQ data directory with Raft logs and message store.
A snapshot of a running RabbitMQ node isn’t guaranteed to be consistent because the Raft log may be in the middle of a write operation at the moment of the snapshot. For the most consistent snapshot, we should take it while the node is stopped. If we snapshot a running node, it may still be usable, but recovery may require RabbitMQ to replay the log from its last safe checkpoint.
Snapshots are useful for recovering a complete node (OS, config, and data together) after a server loss, and for fast rollback after a failed upgrade.
Definitions export as configuration backup
As in the RabbitMQ setup from Part 2, definitions export captures the broker topology: all vhosts, users, permissions, exchanges, queues, bindings, policies, and runtime parameters. It doesn’t capture messages in queues, active connections, or consumer state.
Export definitions from the management API:
curl -s -u admin:yourpassword \
http://localhost:15672/api/definitions \
> rabbitmq-definitions-$(date +%Y%m%d).json
Store this file in UpCloud Object Storage on a schedule:
s3cmd put rabbitmq-definitions-$(date +%Y%m%d).json \
s3://your-bucket/rabbitmq/definitions/
A daily export to Object Storage gives us a versioned record of our broker topology. This is primarily useful for disaster recovery (rebuilding a cluster from scratch), for detecting configuration drift, and for seeding a new cluster during a blue-green upgrade.
Restoring definitions to a running cluster
If topology has been accidentally modified or deleted, we can restore from a definitions export without stopping the cluster. The import is additive by default: it creates any missing objects but doesn’t delete objects that exist in the cluster but not in the file.
curl -s -u admin:yourpassword \
-H “Content-Type: application/json” \
-X POST http://localhost:15672/api/definitions \
-d @rabbitmq-definitions-20260520.json
Rebuilding a cluster from scratch
If we need to replace the entire cluster, the order of operations matters.
First, if any of the original nodes still exist, we will need to remove them from the cluster view before starting fresh:
rabbitmqctl forget_cluster_node [email protected]
rabbitmqctl forget_cluster_node [email protected]
If we are starting completely fresh:
# On the first new node
rabbitmqctl stop_app
rabbitmqctl reset
rabbitmqctl start_app
# Import definitions
curl -s -u admin:yourpassword \
-H “Content-Type: application/json” \
-X POST http://localhost:15672/api/definitions \
-d @rabbitmq-definitions-latest.json
# Then join the second and third nodes
# (same join_cluster sequence as Part 3)
When replacing a single failed node rather than the whole cluster, we will need to provision a new server, copy the Erlang cookie, start RabbitMQ, and join it to the cluster. We will have to remove the old failed node from the cluster’s knowledge with forget_cluster_node before or after joining the replacement:
# From a healthy node, remove the failed node
rabbitmqctl forget_cluster_node [email protected]
# On the new replacement node, after copying the Erlang cookie
rabbitmqctl stop_app
rabbitmqctl join_cluster [email protected]
rabbitmqctl start_app
Joining the cluster makes the node available but does not automatically restore quorum queue membership. Each queue’s replica set still references the old failed node until we explicitly grow it. We can use rabbitmq-queues grow to add the replacement node to the queues that lost a member:
# Add the replacement node as a replica to queues that need it
rabbitmq-queues grow [email protected] all –vhost-pattern “.*” –queue-pattern “.*”
# Verify replica counts have recovered
rabbitmqctl list_queues -p production name leader members online
Monitor the online column until all queues show three online members.
Rolling Restarts and Upgrades
Configuration changes, OS patches, and version upgrades all require restarting nodes. And the order in which we do it determines whether the cluster stays available throughout.
Safe restart sequence for a three-node cluster
When we need to restart all nodes for a configuration change or OS update, we must always restart nodes hosting the fewest queue leaders first, and the node hosting the most queue leaders last.
There is no single cluster leader. Each quorum queue independently elects its own leader, and a node may lead anywhere from zero to all of our queues. Restarting a node triggers a leader election only for the queues it leads, which pauses writes on those queues briefly. This is why leaving the most-loaded node for last minimizes total write disruption.
Find the current leader distribution before starting by running:
rabbitmqctl list_queues -p production name leader
Then, count how many queues each node leads. Start with the node leading the fewest. Restart one node, wait for it to fully rejoin, then move to the next:
# On the chosen node
systemctl restart rabbitmq-server
# From a healthy node, confirm it rejoined
rabbitmqctl cluster_status
Do not restart the next node until the previous one shows up in “Running Nodes”. Restarting two nodes simultaneously in a three-node cluster will drop the cluster below quorum, making queues unavailable for writes.
Minor version upgrades
Upgrading from one minor release to another within the same major version (for example, 4.1.x to 4.2.x) is generally supported as a rolling restart, provided the Erlang version stays within the supported range for both the old and new RabbitMQ releases.
To do that, update the package on one node, restart it, verify it rejoins successfully, and repeat for the remaining nodes. Follow the same sequence as a rolling restart: fewest queue leaders first, most queue leaders last.
# On each node, one at a time
apt-get update
apt-get install -y rabbitmq-server
systemctl restart rabbitmq-server
# Verify
rabbitmqctl status
rabbitmqctl cluster_status
Check the RabbitMQ release notes before upgrading to confirm there are no compatibility warnings for our specific release pair.
Major version upgrades
Rolling upgrades across most major RabbitMQ version pairs aren’t supported, but the 3.13.x to 4.x path is an exception: RabbitMQ officially supports rolling in-place upgrades from 3.13 to 4.x, provided all stable feature flags are enabled on the 3.13 cluster before starting, and you follow the documented node-by-node sequence. For other major version pairs (or when you want a lower-risk migration window), the supported path is a blue-green migration:
- Build a new cluster running the target version, using the same configuration as the existing cluster.
- Import definitions from the old cluster into the new one.
- Move consumers to the new cluster first. The RabbitMQ Federation plugin can bridge the two clusters so consumers on the new cluster pull messages from the old one during the transition.
- Once the old cluster’s queues are empty, switch producers to the new cluster.
- Decommission the old cluster once all traffic has moved.
This requires running two clusters simultaneously during the migration window, which has cost and operational overhead.
Erlang version upgrades
RabbitMQ runs on the Erlang/OTP runtime, and each RabbitMQ release requires a specific range of Erlang versions. The RabbitMQ Erlang Version Requirements page documents the supported Erlang range for each RabbitMQ release. Check it before upgrading Erlang on any node.
Because the Erlang VM must restart to use a new version, an Erlang upgrade requires the same restart sequence as a RabbitMQ upgrade: nodes hosting fewer queue leaders first, the most-loaded node last. If you are upgrading both Erlang and RabbitMQ at the same time, verify that the new Erlang version is compatible with both your current and target RabbitMQ versions before starting.
On Ubuntu, Erlang upgrades typically come through apt:
apt-get update
apt-get install -y erlang
systemctl restart rabbitmq-server
After restarting, confirm the new Erlang version and verify RabbitMQ is still healthy:
rabbitmqctl status | grep -E “Erlang|RabbitMQ”
rabbitmqctl cluster_status
Conclusion
A stable long-term RabbitMQ setup on UpCloud has a few consistent properties: a load balancer that routes clients to healthy nodes and stops sending traffic to alarmed ones, clients that use publisher confirms and handle reconnections correctly, monitoring that surfaces queue depth growth and resource alarms before they become incidents, and a clear runbook for the handful of operational tasks you will repeat: node replacements, rolling restarts, definition exports, and consumer backlog recovery.
Rethink the architecture if the cluster is regularly hitting flow control under normal load and adding capacity doesn’t resolve it, if the queue topology has become difficult to reason about because too many workloads share the same cluster, or if reliability requirements have grown to include multi-region availability.
A managed service makes sense when the team doesn’t have engineers who can own a messaging layer end to end, when on-call coverage for the broker isn’t feasible, or when the managed service cost is reasonable relative to the engineering time self-hosting requires. Running RabbitMQ on UpCloud gives us control, flexibility, and lower infrastructure costs at scale. It also means the cluster is ours to operate, debug, and maintain. If that tradeoff no longer makes sense, move to a managed service before a preventable incident makes the decision for us.
Discussion