{"id":9496,"date":"2026-11-12T16:45:52","date_gmt":"2026-11-12T16:45:52","guid":{"rendered":"https:\/\/upcloud.com\/global\/?post_type=tutorial&#038;p=9496"},"modified":"2026-11-12T16:45:52","modified_gmt":"2026-11-12T16:45:52","slug":"rabbitmq-cluster-setup-upcloud","status":"publish","type":"tutorial","link":"https:\/\/upcloud.com\/global\/resources\/tutorials\/rabbitmq-cluster-setup-upcloud\/","title":{"rendered":"Part 3: RabbitMQ Cluster Setup for High Availability on UpCloud"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">In Part 1, we covered <a href=\"https:\/\/upcloud.com\/global\/resources\/tutorials\/rabbitmq-high-availability-upcloud\/\">RabbitMQ high availability<\/a> on UpCloud and designed the three-node architecture for this series. In Part 2, we walked through the <a href=\"https:\/\/upcloud.com\/global\/resources\/tutorials\/rabbitmq-setup-upcloud\/\">RabbitMQ setup<\/a> for the first production-ready node, including dedicated MaxIOPS storage, scoped users and vhosts, explicit memory and disk limits, and a working definitions export. That node is a solid baseline, but it\u2019s still a single point of failure. In this part, we\u2019ll move on to the <strong>RabbitMQ cluster setup<\/strong> by adding two more nodes, bringing the cluster to three nodes.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Clustering changes what we need to think about. A single node can fail, and we can lose all the work, or, worse, it can force downtime for our application. A cluster has partition behavior, quorum requirements, leader election, replication policies, and recovery sequences. We\u2019ll work through all of them here. By the end, we\u2019ll have a functioning three-node RabbitMQ cluster on UpCloud with quorum queues distributed across all nodes, a tested partition handling strategy, and a verified failure response.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This part assumes the single node from Part 2 is running, correctly configured, and accessible over the private SDN network.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Planning a Three-Node RabbitMQ Cluster<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Before provisioning two more servers, it\u2019s worth being clear about what three nodes actually give us.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Quorum queues require a majority of their replicas to be available before they\u2019ll accept writes. In a three-node cluster where each node holds a replica, that means two nodes must be up and reachable. If one node fails, the remaining two form a majority and the cluster continues serving traffic. If two nodes fail simultaneously, there is no majority, and the queue stops accepting writes until enough nodes have returned.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">A four-node cluster doesn\u2019t improve this. With four nodes, we\u2019d need three available for quorum, which means we can still only lose one. We&#8217;d be running an extra server with no additional fault tolerance to show for it. Sure, we\u2019ll get additional throughput and workload capacity, but it doesn\u2019t help with availability. For most teams, three nodes is the right practical shape. It provides the minimum fault tolerance quorum queues needed without the overhead of additional nodes that don\u2019t change the math.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Five nodes give you stronger fault tolerance for business-critical infrastructure. With five nodes, quorum requires three, so any two can fail simultaneously, and the cluster continues serving writes. A three-node cluster can&#8217;t absorb a second failure when one node is already down for maintenance or a rolling upgrade, leaving your write path exposed until the first node returns. The cost is two additional servers and the coordination overhead that comes with a larger cluster. For most workloads, that cost is disproportionate. For systems where downtime carries serious business consequences, the five-node setup is the right architecture.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">When it comes to node sizing, the relationship between cluster size and quorum size has a direct implication on it. In normal operation, each node carries its share of the workload. When a node goes down, the remaining nodes absorb its client connections and handle quorum operations with one fewer member available. When sizing nodes, we should account for both normal baseline traffic and the additional load when a peer goes down and its connections are redirected. A node that runs at 70% CPU under normal load will be overwhelmed when its peers redirect traffic to it during a failure. That headroom is what lets the cluster absorb a node loss without degrading. Sizing for failure-mode load, with normal operating load as the baseline, is correct capacity planning and not a waste.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Preparing Nodes for the RabbitMQ Cluster<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">With the logic out of the way, let\u2019s get started with deploying and preparing the nodes to run the quorum.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Step 1: Provision two more servers<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">We\u2019ll need to create two additional <a href=\"https:\/\/upcloud.com\/global\/docs\/products\/cloud-servers\/\">UpCloud Cloud Servers <\/a>matching the configuration from Part 2: the same plan, storage layout, and OS image, and attached to the same SDN private network as the first node.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Give each server a unique, stable hostname before we install RabbitMQ. RabbitMQ identifies each node in the cluster by its Erlang node name, which takes the form rabbit@hostname. This name is written into the node\u2019s data directory during the first startup. If we rename a node after it has joined a cluster, we\u2019ll need to perform a reset-and-rejoin sequence. Set the hostnames once, before any RabbitMQ installation.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">On each new server:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">hostnamectl set-hostname rmq2.internal<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Use whatever naming convention fits the environment, but keep all three names distinct and stable. For an UpCloud <a href=\"https:\/\/upcloud.com\/global\/docs\/guides\/configure-sdn-private-networks\/\">SDN private network<\/a>, \/etc\/hosts is the simplest option when the node count is fixed.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">First, find the private SDN IP assigned to each server. Run this on each node:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">ip -4 addr show<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">The output will list several interfaces. Ignore lo (loopback) and the interface with a public IP. UpCloud also attaches a utility network interface with a \/22 subnet\u2014ignore that too. Our SDN private network is the remaining interface, typically with a \/24 subnet matching what we configured when creating the network:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">1: lo: ...<br>inet 127.0.0.1\/8 ... # loopback: skip<br>2: eth0: ...<br>inet 95.111.195.237\/22 ... # public interface: skip<br>3: eth1: ...<br>inet 10.10.9.200\/22 ... # UpCloud utility network (\/22): skip<br>5: eth3: ...<br>inet 10.0.11.4\/24 ... # SDN private network: this is the one<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">In this example, the SDN IP is 10.0.11.4. Make sure to note it down for each node before continuing.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><em>Alternatively, we could open the UpCloud control panel, navigate to each server, and check the IP address listed under our SDN network in the <strong>Network<\/strong> tab.<\/em><\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Once we have all three IPs, run this on all three nodes, replacing the addresses with our actual values:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">cat &gt;&gt; \/etc\/hosts &lt;&lt;'EOF'<br>&lt;rmq1-sdn-ip&gt; rmq1.internal rmq1<br>&lt;rmq2-sdn-ip&gt; rmq2.internal rmq2<br>&lt;rmq3-sdn-ip&gt; rmq3.internal rmq3<br>EOF<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Make sure to use private SDN IP addresses here. Erlang distribution, which RabbitMQ uses for inter-node communication, sends cluster traffic unencrypted by default. Exposing that traffic on a public interface puts cluster cookies, queue metadata, and message payloads in the open. Private SDN keeps inter-node traffic isolated to a network only your cluster nodes can reach.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Once \/etc\/hosts is updated on all three nodes, verify that each node can resolve the others by name, by running these commands:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">ping -c 2 rmq2.internal<br>ping -c 2 rmq3.internal<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">If resolution fails, clustering will fail with an opaque Erlang distribution error. We\u2019ll then need to fix \/etc\/hosts entries on all nodes before proceeding.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Step 2: Install RabbitMQ on the new nodes<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">On each of the two new servers, run the same installation steps from Part 2: install erlang and rabbitmq-server, enable the management plugin, and start the service.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">apt-get update -y<br>apt-get install -y erlang rabbitmq-server<br>rabbitmq-plugins enable rabbitmq_management<br>systemctl enable rabbitmq-server<br>systemctl start rabbitmq-server<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Next, apply the same rabbitmq.conf to each new node. Create the file if it doesn\u2019t exist:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">touch \/etc\/rabbitmq\/rabbitmq.conf<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Then write the following configuration:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\"># \/etc\/rabbitmq\/rabbitmq.conf\n\n# Flow control activates when broker memory exceeds 60% of available RAM.\n# Adjust based on node size. Do not set above 0.7 unless the node\n# is dedicated exclusively to RabbitMQ.\nvm_memory_high_watermark.relative = 0.6\n\n# Block publishers when free disk space drops below 1.5x total physical memory.\n# This gives us time to react before the node runs out of disk.\n# Use an absolute value instead if our data volume is much larger than memory:\n# disk_free_limit.absolute = 10GB\ndisk_free_limit.relative = 1.5\n\n# Bind the AMQP listener to all interfaces.\n# We will restrict this to the private interface in the next step.\nlisteners.tcp.default = 5672\n\n# Management UI port.\nmanagement.listener.port = 15672<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Restart RabbitMQ on each node to apply the configuration:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">systemctl restart rabbitmq-server<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">We don\u2019t need to manually create users or vhosts on these nodes. After the cluster forms, we\u2019ll import the definitions exported from Part 2, which recreates all users, vhosts, permissions, exchanges, queues, and policies.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Step 3: Copy the Erlang cookie<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">The Erlang cookie is a shared secret that Erlang nodes use to verify that they belong to the same cluster. Every node that should join the cluster must have an identical cookie. If the cookies don\u2019t match, join_cluster will fail with an authentication error.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">When RabbitMQ starts for the first time on a node, it generates a random cookie and writes it to \/var\/lib\/rabbitmq\/.erlang.cookie. We need to overwrite this generated value on nodes 2 and 3 with the value from node 1.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">On node 1, read the cookie and copy the value:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">cat \/var\/lib\/rabbitmq\/.erlang.cookie<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">On each of nodes 2 and 3, stop RabbitMQ, write the same cookie, fix the permissions, and restart them:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">systemctl stop rabbitmq-server<br>echo -n 'your-cookie-value-here' &gt; \/var\/lib\/rabbitmq\/.erlang.cookie<br>chown rabbitmq:rabbitmq \/var\/lib\/rabbitmq\/.erlang.cookie<br>chmod 400 \/var\/lib\/rabbitmq\/.erlang.cookie<br>systemctl start rabbitmq-server<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\"><em>We\u2019ll also need to ensure that the cookie file is readable only by its owner (using the chmod command above), as the Erlang runtime requires that. Wrong permissions could cause RabbitMQ to fail at startup or generate a new random cookie, which will break cluster authentication.<\/em><\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Bind inter-node communication to the private interface<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">By default, Erlang will use whatever interface the hostname resolves to for inter-node communication. On a server with both public and private interfaces, this may route cluster traffic over the public network.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">To fix that, in \/etc\/rabbitmq\/rabbitmq-env.conf, set two values on each node:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\"># \/etc\/rabbitmq\/rabbitmq-env.conf<br>NODE_IP_ADDRESS=&lt;this-node-sdn-ip&gt;<br>USE_LONGNAME=true<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Here\u2019s what these two do:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>USE_LONGNAME=true: tells Erlang to identify nodes by their fully-qualified hostname (rabbit@rmq2.internal) rather than the short form (rabbit@rmq2). Without it, join_cluster will fail with a {:node_name, :short} error when we reference a node by its .internal name.<\/li>\n\n\n\n<li>NODE_IP_ADDRESS=&lt;this-node-sdn-ip&gt;: pins Erlang\u2019s inter-node listener to the private SDN interface. Without it, Erlang binds to whatever interface the hostname resolves to, which may be the public IP on a multi-interface server.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">Restart the service on each node after applying this:<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">systemctl restart rabbitmq-server<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Step 4: Verify firewall rules on all nodes<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Before forming the cluster, make sure the inter-node ports are open between all three nodes on the private SDN network. RabbitMQ cluster communication requires three port ranges:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>4369<\/strong>: the Erlang Port Mapper Daemon (EPMD), which nodes use to discover each other\u2019s inter-node listening ports<\/li>\n\n\n\n<li><strong>25672<\/strong>: the actual inter-node communication port<\/li>\n\n\n\n<li><strong>35672\u201335682<\/strong>: used by CLI tools (rabbitmqctl, rabbitmq-diagnostics, etc.) to communicate with nodes<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">In our <a href=\"https:\/\/upcloud.com\/global\/docs\/guides\/managing-firewall\/\">UpCloud Firewall rules<\/a>, add rules for <strong>both inbound and outbound<\/strong> directions for these ports. Also, we have to remember UpCloud\u2019s firewall is stateless, which means it does not track connections, so an inbound allow rule alone is not enough. Return traffic will require a matching outbound rule.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">RabbitMQ Cluster Setup: Forming the Cluster<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">With all three nodes running the same cookie and able to resolve each other by hostname, we can form the cluster. To form it, designate one node as the initial member and join the other two to it.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Node 1 already exists and is running. On node 2, stop the RabbitMQ application (not the Erlang runtime), join the cluster, and restart:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">rabbitmqctl stop_app<br>rabbitmqctl join_cluster rabbit@rmq1.internal<br>rabbitmqctl start_app<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Repeat this on node 3:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">rabbitmqctl stop_app<br>rabbitmqctl join_cluster rabbit@rmq1.internal<br>rabbitmqctl start_app<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Both node 2 and node 3 can join via node 1 and don\u2019t need to join each other separately.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Verifying RabbitMQ Cluster Health<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Once both nodes have joined, check the cluster state from any of the three nodes:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">rabbitmqctl cluster_status<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Here\u2019s what it will look like:<\/p>\n\n\n\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" src=\"https:\/\/upcloud.com\/media\/rabbitmq-cluster-status-three-nodes-1024x930.png\" alt=\"RabbitMQ cluster status showing three running nodes and no network partitions\" class=\"wp-image-88056\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">In a healthy cluster, all three nodes appear in both \u201cDisk Nodes\u201d and \u201cRunning Nodes\u201d, and the \u201cNetwork Partitions\u201d section is empty:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">Cluster name: rabbit@rmq1.internal\nDisk Nodes\nrabbit@rmq1.internal\nrabbit@rmq2.internal\nrabbit@rmq3.internal\n\nRunning Nodes\nrabbit@rmq1.internal\nrabbit@rmq2.internal\nrabbit@rmq3.internal\n\nAlarms\n\n(none)\n\nNetwork Partitions\n\n(none)<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">If a node appears in \u201cDisk Nodes\u201d but not \u201cRunning Nodes\u201d, it\u2019s known to the cluster but not currently reachable.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">If the cluster status shows only one or two nodes, check the Erlang cookie on the missing node, verify name resolution, and confirm that the firewall isn\u2019t blocking ports 4369 and 25672.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Import definitions<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Now that the cluster exists, import the definitions we exported from Part 2 to recreate users, vhosts, permissions, and policies across the cluster. Run this from any one node. RabbitMQ will replicate the topology to all members:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">curl -s -u admin:mypassword -H \"Content-Type: application\/json\" -X POST http:\/\/localhost:15672\/api\/definitions -d @rabbitmq-definitions-20260520.json<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">After the import, verify from the management UI or CLI that users, vhosts, and policies appear correctly on all three nodes.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Common RabbitMQ Cluster Setup Issues<\/h3>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><thead><tr><th>Symptom<\/th><th>Likely cause<\/th><\/tr><\/thead><tbody><tr><td>join_cluster fails with nodedown<\/td><td>Hostname not resolvable, or EPMD blocked on port 4369<\/td><\/tr><tr><td>join_cluster fails with authentication_failure<\/td><td>Erlang cookie mismatch between nodes<\/td><\/tr><tr><td>Node joins but shows as stopped in cluster status<\/td><td>Inter-node port 25672 is blocked<\/td><\/tr><tr><td>Node appears in disk nodes but not running nodes after restart<\/td><td>Node has not yet restarted, or data directory permission issue<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<h2 class=\"wp-block-heading\">RabbitMQ Quorum Queues in Practice<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Quorum queues use a Raft-based replication model. When a quorum queue is declared, one replica becomes the leader, and the others become followers. The node where the queue is declared becomes the initial leader (though queue-leader-locator can change that, and placement is a hint rather than a guarantee). If that node fails, the surviving replicas vote among themselves, and whichever one wins a majority takes over. Only a replica that is already fully caught up can win that vote, so nothing confirmed to a publisher is lost during the handover. Because it takes a majority to elect a leader, a three-member queue keeps working even after the loss of one node, but not after the loss of two.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">All writes to that queue go through the leader. The leader appends each incoming message to its Raft log and sends the log entry to followers. Once a majority of replicas has acknowledged the entry, the write is considered committed, and the leader can send a publisher confirm if the producer requested one.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Followers don\u2019t serve writes directly. If a producer connects to a node that isn\u2019t the leader for a given queue, RabbitMQ routes the operation to the leader transparently. From the application\u2019s perspective, publishing works the same regardless of which node the client connects to.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">When the leader node goes down, the remaining followers hold an election. Raft elections require a majority to succeed, which means two of the three nodes in our cluster must be reachable and agree on the new leader. The election typically completes within a few seconds. During that window, the queue won\u2019t accept new writes. Producers using publisher confirms will receive an error or a nack and should retry. Consumers connected through the failed node will need to reconnect.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">After a new leader is elected, it catches up any followers who fell behind and resumes normal operations. A committed log entry survives a leader failure because it was already present on a majority of replicas before the failure.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Setting x-quorum-initial-group-size<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">When we declare a quorum queue, RabbitMQ places replicas on a subset of available nodes. The number of initial replicas is controlled by the x-quorum-initial-group-size argument on the queue declaration. If this argument isn\u2019t set, RabbitMQ defaults to placing replicas on all available nodes up to a maximum of 3.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">In a three-node cluster, the default behavior is appropriate. All three nodes hold a replica, and the queue can survive the loss of any one node. We don\u2019t need to set this argument manually for the standard case.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">We would set it explicitly if we have more than three nodes and want a specific queue to use a different group size, or if we\/re declaring queues during cluster provisioning and want consistent behavior regardless of how many nodes are currently running.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Declare a quorum queue with an explicit group size:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">rabbitmqadmin -u admin -p yourpassword -V production \\<br>declare queue name=orders durable=true \\<br>arguments='{\"x-queue-type\":\"quorum\",\"x-quorum-initial-group-size\":3}'<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">After declaring it, verify that the queue has members on all three nodes:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">rabbitmqctl list_queues -p production name type members<\/code><\/pre>\n\n\n\n<figure class=\"wp-block-image size-large\"><img decoding=\"async\" src=\"https:\/\/upcloud.com\/media\/rabbitmq-quorum-queue-members-three-nodes-1024x508.png\" alt=\"RabbitMQ quorum queue showing replicas across all three cluster nodes.\" class=\"wp-image-88095\" \/><\/figure>\n\n\n\n<p class=\"wp-block-paragraph\">The members column should list all three node names.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Streams as an Alternative<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Streams are a separate queue type in RabbitMQ 4.x, and the decision of when to use them instead of quorum queues comes down to the workload pattern.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Quorum queues are a work queue. Consumers compete for messages, and each message is delivered to exactly one consumer. Once a consumer acknowledges a message, it\u2019s removed from the queue. This model suits task dispatch, order processing, payment workflows, and notification pipelines (any pattern where each message is a unit of work that exactly one consumer should process).<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Streams behave more like a log. Messages are retained for a configurable period regardless of consumer acknowledgments. Multiple independent consumers can read from the same stream at different offsets. Consumers can replay from any position in the stream. This model fits audit trails, activity feeds, event fan-out where multiple services need to react to the same event independently, or high-throughput ingestion where consumers read sequentially and throughput matters more than per-message routing.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Stream replication is also Raft-based at the storage level but is optimized for sequential I\/O patterns. Each type fills a different role: streams handle replay and fan-out, quorum queues handle competing consumers. Most RabbitMQ deployments that need both will run both types on the same cluster.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">In this series&#8217; reference architecture, quorum queues are the default for all critical application messaging. Use streams selectively when the workload genuinely requires replay or fan-out semantics.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Policies and Replication<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">The x-quorum-initial-group-size argument controls replica placement when the queue is declared. For cluster-wide defaults and for existing queues, we must use policies.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">A RabbitMQ policy is a set of key-value settings applied to queues (or exchanges) matching a name pattern within a vhost. Policies are applied dynamically and take effect immediately without restarting the broker.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The relevant policy key for quorum queue replication is target-group-size. This sets the desired replica count for queues matching the policy and applies to both new and existing queues.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Apply a cluster-wide quorum policy:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">rabbitmqctl set_policy -p production QuorumDefaults \".*\" '{\"target-group-size\":3}' --apply-to queues --priority 0<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">This sets the target replication factor to 3 for all queues in the production vhost. RabbitMQ will expand or shrink replicas on existing queues to match the target the next time those queues are reconciled.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Verify that policies are applied:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">rabbitmqctl list_policies -p production<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Check that queues show the expected member count:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">rabbitmqctl list_queues -p production name type members policy<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">The members column should list three node names for each quorum queue, and the policy column should show the policy name if one matched.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Every write to a quorum queue is replicated to all members before it\u2019s committed. A queue with 3 members writes to 3 disks per message. Increasing the member count beyond that in a three-node cluster isn\u2019t possible anyway. Reducing to 1 or 2 would save write overhead, but removes the HA property the cluster was built to provide.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Network Partitions and Split-Brain<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">A network partition occurs when some nodes in a cluster can\u2019t communicate with others. From each node\u2019s perspective, the nodes it can\u2019t reach have failed. In a partition, both sides of the cluster may believe they are the healthy majority and continue accepting writes independently. When the partition heals, the two sides have diverged with no automatic safe way to reconcile them. That state is split-brain.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">RabbitMQ provides three modes for handling partitions, configured with cluster_partition_handling in rabbitmq.conf:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>ignore keeps each partition operating without restriction, which means split-brain can occur. This is the default and isn\u2019t appropriate for production.<\/li>\n\n\n\n<li>pause_minority pauses any node that detects it\u2019s in the minority partition (fewer than half of all cluster nodes are visible to it). In a three-node cluster where one node loses connectivity to the other two, the isolated node pauses itself and stops serving client traffic. The two remaining nodes form a majority and continue operating. When the isolated node regains connectivity, it resumes automatically with no divergence.<\/li>\n\n\n\n<li>autoheal lets partitions write freely and then, after the partition heals, picks a winning partition and restarts all nodes that were in the losing partition. The losing partition\u2019s writes since the partition started are discarded. This can cause data loss and require nodes to restart, making it an unreliable primary strategy.<\/li>\n<\/ul>\n\n\n\n<p class=\"wp-block-paragraph\">For a three-node cluster, use pause_minority:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">cluster_partition_handling = pause_minority<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Add this to rabbitmq.conf on all three nodes and restart to apply it.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">pause_minority works predictably in a three-node cluster because the math is unambiguous. Any single node that loses connectivity to both other nodes is in the minority and pauses. If two nodes lose connectivity to the third, those two are the majority and continue operating. There\u2019s no ambiguity and no possible split-brain.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Larger clusters lose that tidiness. A four-node cluster splits evenly into two groups of two, leaving each side with more than half the nodes, so both pause and the whole cluster goes offline; the same happens to a five-node cluster if no group of three survives intact. pause_minority is behaving as designed in both cases, since it trades availability for consistency and makes no promise that some partition will always survive.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Detecting partitions<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">When a partition has occurred, and a node is paused under pause_minority, rabbitmqctl cluster_status will show the affected node missing from \u201cRunning Nodes\u201d as seen from the healthy side. The Network Partitions section in the status output will show the detected partition when the connectivity issue becomes visible:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">rabbitmqctl cluster_status<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Look for anything under \u201cNetwork Partitions\u201d in the output. An empty partitions section means the cluster sees no current or unresolved partition state. When a paused node regains connectivity, it automatically restarts and rejoins the cluster. Verify the full cluster is healthy after recovery before resuming normal traffic.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Testing RabbitMQ Cluster Failover<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">Testing failure scenarios before real incidents happen is a necessary part of bringing a cluster into production. Three scenarios are worth running explicitly.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Stopping a follower node<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Find which node is the leader for our test queue:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">rabbitmqctl list_queues -p production name leader members<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Pick a node that is listed as a member but isn\u2019t the leader. Stop RabbitMQ on it:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\"># On the non-leader node<br>systemctl stop rabbitmq-server<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">From another node, check the cluster status:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">rabbitmqctl cluster_status<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">The stopped node should appear in \u201cDisk Nodes\u201d but not in \u201cRunning Nodes\u201d. Run a publish and consume test from the remaining two nodes to confirm the queue is still serving traffic:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">rabbitmqadmin -u admin -p yourpassword -V production \\<br>publish routing_key=orders payload=\"test-after-follower-stop\"<\/code><\/pre>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">rabbitmqadmin -u admin -p yourpassword -V production \\<br>get queue=orders<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">The queue should respond normally. Restart the stopped node and verify it rejoins:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">systemctl start rabbitmq-server<br>rabbitmqctl cluster_status<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">Stopping the leader node<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Using the same list_queues output from before, identify and stop the current leader. The cluster needs a few seconds to detect the failure, hold an election, and establish a new leader. During this window, write attempts to the affected queue will fail. Producers using publisher confirms should receive nacks or channel errors and retry.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Once the election completes, confirm the queue has a new leader on one of the remaining nodes:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">rabbitmqctl list_queues -p production name leader<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">The leader column should show a different node name. Test that the queue is again accepting and delivering messages, then restart the former leader and verify it rejoins as a follower.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Clean versus unclean shutdown<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">A clean shutdown via systemctl stop rabbitmq-server or rabbitmqctl stop leaves the node\u2019s data in a consistent state. When the node restarts, it rejoins the cluster and resyncs any log entries it missed.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">An unclean shutdown due to power loss, a kernel panic, or an OOM kill can leave the Raft log partially written. RabbitMQ handles this during startup by replaying the log from the last safe checkpoint. In most cases, the node recovers and rejoins without operator intervention.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">If a node fails to rejoin after an unclean shutdown, check the service logs:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code class=\"\">journalctl -u rabbitmq-server --since \"10 minutes ago\"<\/code><\/pre>\n\n\n\n<p class=\"wp-block-paragraph\">Look for Raft log errors or quorum membership messages. Allow the node to complete its recovery sequence before considering a reset. Forcing a reset with rabbitmqctl reset causes the node to discard all its queue data and rejoin as an empty member. Use it only when the node is genuinely unrecoverable, and we have confirmed that the other nodes hold all committed data.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">What HA Covers and Doesn\u2019t<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">It\u2019s worth being precise about what we\u2019ve built and where its limits are.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">What the cluster handles automatically<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">When a single node goes down cleanly or crashes, the cluster continues serving traffic for all quorum queues that still have a majority of replicas available. RabbitMQ elects a new leader for any queue whose leader was on the failed node. The load balancer stops sending new client connections to the unavailable node. Consumers connected through the failed node need to reconnect and will have their unacknowledged messages redelivered.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">When the node comes back, it rejoins the cluster, resyncs its queue replicas, and begins accepting connections again. No operator action is required for a normal single-node failure and recovery.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">What the cluster handles depending on configuration<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Network partitions are handled according to our cluster_partition_handling setting. With pause_minority, the cluster correctly suspends the minority partition and avoids split-brain. With the default ignore setting, a partition can result in both sides accepting writes that can\u2019t be reconciled when the partition heals.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">What the cluster doesn\u2019t protect against<\/h3>\n\n\n\n<p class=\"wp-block-paragraph\">Messages that a producer published but didn\u2019t receive a publisher confirm for are at risk during any failure. The confirm signal indicates that the message has been committed on a majority of replicas. Without confirms, the producer has no way to know whether the broker accepted the message before the failure occurred. Enable publisher confirms on all producers and retry on nack or connection failure.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Disk exhaustion isn\u2019t handled automatically. When a node\u2019s data volume fills past the disk alarm threshold, RabbitMQ blocks publishers to protect the Raft log. The cluster continues running, but no new messages can be written until disk space is freed or the volume is expanded. Monitor disk usage and alert before reaching the alarm threshold.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Application-level problems sit outside what clustering can fix. A producer that floods a queue, a consumer that never acknowledges, or a client that opens thousands of channels and doesn\u2019t close them will affect the cluster regardless of how many nodes it has. Clustering provides infrastructure-level fault tolerance. Correct client behavior, monitoring, and operational ownership of the messaging layer remain our responsibility.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Conclusion<\/h2>\n\n\n\n<p class=\"wp-block-paragraph\">We now have a running three-node RabbitMQ cluster on UpCloud. The cluster uses quorum queues replicated across all three nodes, private SDN networking for inter-node traffic, and pause_minority partition handling to prevent split-brain. We\u2019ve tested single-node failure and leader election, and we understand the conditions under which the cluster will and won\u2019t protect our messages.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The configuration decisions that matter most in this setup are the partition handling mode, the quorum group size, and the replication policy. Those three settings determine what the cluster actually guarantees in the event of a failure. The default partition handling (ignore) and the disk-free limit require explicit values in production. Everything else we set in Part 2 carries forward correctly.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Part 4 covers <a href=\"https:\/\/upcloud.com\/global\/resources\/tutorials\/rabbitmq-monitoring-upcloud\/\">RabbitMQ monitoring, scaling, and operations on UpCloud<\/a>: routing client traffic through the UpCloud Load Balancer, surfacing useful signals before something breaks, and handling the operational sequences we\u2019ll encounter in production, including node replacement, rolling upgrades, and consumer backlog management.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\"><\/p>\n","protected":false},"author":82,"featured_media":0,"comment_status":"open","ping_status":"closed","template":"","community-category":[223,259],"class_list":["post-9496","tutorial","type-tutorial","status-publish","hentry"],"acf":[],"_links":{"self":[{"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/tutorial\/9496","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/tutorial"}],"about":[{"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/types\/tutorial"}],"author":[{"embeddable":true,"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/users\/82"}],"replies":[{"embeddable":true,"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/comments?post=9496"}],"version-history":[{"count":7,"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/tutorial\/9496\/revisions"}],"predecessor-version":[{"id":10312,"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/tutorial\/9496\/revisions\/10312"}],"wp:attachment":[{"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/media?parent=9496"}],"wp:term":[{"taxonomy":"community-category","embeddable":true,"href":"https:\/\/upcloud.com\/global\/wp-json\/wp\/v2\/community-category?post=9496"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}