Benchmarking Stream Performance With Spring XD 1.2 and Apache Kafka

June 17, 2015 Sabby Anandan

sfeatured-XD-KafkaOne of the goals for the Spring XD 1.2 release was to obtain the baseline performance metrics on a typical cluster of machines and then optimize stream performance where necessary. Spring XD is a unified, distributed, and extensible system for data ingestion, real time analytics, batch processing, and data export. Our testing drove several optimizations to increase streaming performance. The benchmarks found that a single, single threaded Spring XD stream can handle over 2 million (100 byte) events a second, using Apache Kafka as a transport. If the business case allows the stream to run on one container, XD can handle ~13 million (100 byte) events a second using direct (in-memory) binding.
To gain a perspective on these results, the graph below shows XD’s performance in megabytes per second, compared to Kafka’s own performance benchmarks with various message sizes.

image03

image01

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

This graph shows that a customer will get all of the benefits of Spring XD with near-native performance for the underlying transport. Spring XD 1.2 uses the new Spring Integration Kafka adapter, which offers a richer feature set than the standard Kafka client library. The other takeaway from our tests is that these results were obtained using Spring XD out of the box, requiring minimal configuration to set up the Kafka batch size, mode, and the connectivity to the Kafka transport.

For anyone interested in reproducing these numbers, please refer to the engineering blog, which describes in detail the tests performed and infrastructure used.

Learn more:

About the Author

Sabby Anandan

Sabby Anandan, Product Manager in the Spring Team at Pivotal. He focuses on building products that address the challenges faced with iterative development and operationalization of data-intensive applications at scale. Before joining Pivotal, Sabby worked in engineering and management consulting positions. He holds a Bachelor's degree in Electrical and Electronics from the University of Madras and a Master's degree in Information Technology and Management from the Carnegie Mellon University.

Follow on Twitter Follow on Linkedin More Content by Sabby Anandan
Previous
Apache Geode Update
Apache Geode Update

It’s been a few months since Project Geode, the open source core of Pivotal Gemfire, was announced. Since t...

Next
New Spring XD Releases And Beta Release Of Flo For Spring XD
New Spring XD Releases And Beta Release Of Flo For Spring XD

The Spring XD engineering team has some big announcements regarding Spring XD 1.2 and 1.1.3 along with Flo ...

How do you measure digital transformation?

Take the Benchmark