Announcing a New Version of RabbitMQ for Pivotal Cloud Foundry
What developers love about RabbitMQ is that it is “messaging that just works.” In other words, they can focus on using communication in their applications -- not on minutiae such as establishing connections or having to worry about guaranteeing availability of senders and receivers.
But let’s face it: in the cloud-native era, “just works” is a progressively higher bar to reach. Historically, RabbitMQ was easier to get running and operate than legacy messaging middleware. Today, developers expect the service to “just be there” when they want it.
Today, we’re proud to announce an important new capability of RabbitMQ for Pivotal Cloud Foundry (PCF.) Developers can now provision their own dedicated clusters of RabbitMQ with a CF create-service call. This reflects our focus on continuing to improve developer self-service automation in Pivotal Cloud Foundry.
Typical Use Cases for RabbitMQ for PCF
RabbitMQ for Pivotal Cloud Foundry is an integrated distribution of RabbitMQ that allows Pivotal Cloud Foundry platform operators to provide a “messaging-as-a-service” offering to their application developers. RabbitMQ for PCF implements the On-Demand Service Broker interface, which provides a high degree of self-service automation. This allows developers to provision their own instances of RabbitMQ clusters. Typically developers will instantiate these dedicated RabbitMQ clusters to provide communications for new cloud-native as well as migrated and replatformed applications.
Platform operators make on-demand instances available through plans they pre-configure. Operators can choose which organizations and spaces the plan will be available to. They can set quotas for each plan, and for the number of RabbitMQ instances in general so they can easily manage the resources that get consumed. Finally, a plan defines the number of nodes in a cluster, the instance types for each node, and AZ placement for nodes. These plans are then made available in the service marketplace for developers. When a developer creates an on-demand service instance of RabbitMQ, the service broker will instantiate the appropriate number of VMs and install the correct RabbitMQ images. The nodes are then configured into a RabbitMQ cluster, and a smoke test is run to ensure everything is working correctly. At this point, the cluster is available to bind to relevant applications.
RabbitMQ for PCF also supports a pre-provisioned multi-tenant deployment of a RabbitMQ cluster. Operators will want to pre-provision the cluster to support applications that utilize Spring Cloud Services, Push Notification Service for PCF, and Spring Cloud Data Flow. Pre-provisioned clusters are also commonly used to support messaging for other enterprise resources, such as bridging to legacy messaging infrastructure. Developers receive a RabbitMQ vhost instance when they issue a cf create-service command that names a multi-tenant plan.
For deployments that make heavy use of a multi-tenant service, RabbitMQ for PCF allows operators to pre-provision multiple multi-tenant plans. This allows for isolation of intensive workloads to their own pre-provisioned cluster.
Implementing RabbitMQ Patterns with On-Demand Clusters
Here are examples of patterns that developers can utilize with on-demand clusters of RabbitMQ to meet their application messaging needs.
Simple and work queues:
In this pattern, developers define a single queue that one or more producer apps post messages to, and one or more consumers fetch messages from. Producer and consumer applications will both need to bind to the RabbitMQ service instance.
Typically deploying a single node of RabbitMQ should be sufficient for most workloads. This pattern can be configured to provide message persistence in case the node were to fail and needs to start up again. You can also configure the RabbitMQ Federation or Shovel plugins to pass messages to other RabbitMQ message brokers, which will both increase availability of messages as well as potentially scale throughput. Other ways to scale the system up and down include adding or removing additional producers and consumers, such as through using the cf scale command. For example, this allows you to instantiate more workers to consume and process messages off a queue. You can also increase the size of the instance type for the single node.
Publish & subscribe systems:
In this pattern, developers will define multiple queues, and make use of the exchange feature for RabbitMQ to route messages from producers to the appropriate queue. Consumers will pull messages off the appropriate queue for processing.
Developers may find it advantageous to deploy multiple nodes of RabbitMQ in a cluster. In addition to providing more processing power for queues, this gives developers more control over how to architect processing power for consumers and producers. For example, developers can choose to dedicate one node for receiving messages published by producers that will not have any queues explicitly declared on it. Developers would declare the queues on the other nodes, and the publishing node will route messages appropriately as defined in its exchange. Consumers will pull messages off the nodes dedicated to hosting the queues. This way developers can explicitly spread the processing load depending on message type.
To increase the availability of messages in the cluster, users can configure queues to mirror between nodes. Note that mirroring between nodes will impact the performance of RabbitMQ.
Other ways to scale this system include increasing the number of producers or consumers such as through the cf scale command. Operators can also configure the plan with more nodes in the cluster to reduce the number of queues serviced per node. Performance-sensitive systems may wish to disable queue mirroring between nodes. It’s important to note the potential trade-off between performance and availability.
The exchange features of RabbitMQ allow for sophisticated routing of messages. It’s possible to separate message types that might have higher performance or resilience requirements and routing them to queues configured accordingly. Utilize the Federation or Shovel plugins for even more sophisticated ways to route messages to other RabbitMQ brokers.
Developers looking to explore the full capabilities of RabbitMQ architecture should check out the new book by Gavin Roy, “RabbitMQ in Depth”, available from Manning Publications.
Find out more
- Check out the documentation for RabbitMQ for PCF 1.10 to learn more.
- View the webinar “Messaging for Cloud-Native and Migrated Apps: What’s New with RabbitMQ for PCF”.
- Get RabbitMQ for PCF 1.10
About the Author
