Case Study of Batch Processing With Spring Cloud Data Flow Server in Cloud Foundry - Bruce Thelen, CoreLogic

December 14, 2017

Problem: CoreLogic RiskMeter is a platform for insurance underwriters to assess natural hazard risk to insured properties. RiskMeter is going through a replatforming as part of CoreLogic’s cloud native transformation with help from Pivotal Labs. The replatformed Riskmeter is implemented in Spring Boot and deployed on Cloud Foundry as a cloud native 12 factor app. While executing this replatforming, the team discovered the need for recreating the old application’s batch processing features. In this new cloud native version of the application, there was a strong desire to be able to blue green deploy the application without disrupting running batches. Solution: In order to accomplish this, we chose to use Spring Cloud Data Flow Server with Spring Cloud Task to execute a Spring Batch Job. The batch job updates its status via a StepExecutionListener sending messages to a RabbitMQ message queue which is read via a websocket connection from the Angular frontend thus providing a real time progress bar. Specific Items To Emphasize As Lessons Learned: Decomposing the system to extract the short lived microservice that composes the batch run Injecting configuration into Spring Cloud Data Flow Server Injecting configuration into Spring Tasks Techniques for blue/green deployment of Spring Tasks Realtime batch status progress bar via websockets to Angular front end Conclusion: Previous CoreLogic projects which implemented homegrown batch processing relied heavily on complex infrastructure, deployment processes (i.e. allocating dedicated VM instances, configuring app servers), and long development cycles. In one case, it took approximately a year to implement a similar system. Using Cloud Foundry, Spring Cloud Data Flow Server, Spring Task, and Spring Batch, we were able to decompose our app and deploy the batch solution in about two months without having to write custom and complex batch management tooling. Bruce Thelen, Principal Software Architect, CoreLogic Filmed at SpringOne Platform 2017

Previous
State or Events? Which Shall I Keep? - Jakub Pilimon, Kenny Bastani
State or Events? Which Shall I Keep? - Jakub Pilimon, Kenny Bastani

Tired of fancy abstractions without actual down to earth implementation? In this talk we will go through bo...

Next Video
Cloud Event Driven Architectures with Spring Cloud Stream 2.0 - Oleg Zhurakousky, Gary Russell
Cloud Event Driven Architectures with Spring Cloud Stream 2.0 - Oleg Zhurakousky, Gary Russell

Event driven have never left the main stage, but with the advent of cloud-native, distributed architectures...