One of the goals for the Spring XD 1.2 release was to obtain the baseline performance metrics on a typical cluster of machines and then optimize stream performance where necessary. Spring XD is a unified, distributed, and extensible system for data ingestion, real time analytics, batch processing, and data export. Our testing drove several optimizations to increase streaming performance. The benchmarks found that a single, single threaded Spring XD stream can handle over 2 million (100 byte) events a second, using Apache Kafka as a transport. If the business case allows the stream to run on one container, XD can handle ~13 million (100 byte) events a second using direct (in-memory) binding.
To gain a perspective on these results, the graph below shows XD’s performance in megabytes per second, compared to Kafka’s own performance benchmarks with various message sizes.
This graph shows that a customer will get all of the benefits of Spring XD with near-native performance for the underlying transport. Spring XD 1.2 uses the new Spring Integration Kafka adapter, which offers a richer feature set than the standard Kafka client library. The other takeaway from our tests is that these results were obtained using Spring XD out of the box, requiring minimal configuration to set up the Kafka batch size, mode, and the connectivity to the Kafka transport.
For anyone interested in reproducing these numbers, please refer to the engineering blog, which describes in detail the tests performed and infrastructure used.
- More about Spring XD
- Read the technical post about Spring XD performance
- Read the Spring XD 1.2 release announcement
About the Author
Sabby Anandan, Product Manager in the Spring Team at Pivotal. He focuses on building products that address the challenges faced with iterative development and operationalization of data-intensive applications at scale. Before joining Pivotal, Sabby worked in engineering and management consulting positions. He holds a Bachelor's degree in Electrical and Electronics from the University of Madras and a Master's degree in Information Technology and Management from the Carnegie Mellon University.Follow on Twitter More Content by Sabby Anandan