Chicago RDBMS Data Migration: Modernizing Traditional ETL into microservices
Speakers: ANUPAMA PRADHAN SR TECHNOLOGY ARCHITECT, HCSC JEFF CHERNG ADVISORY DATA ENGINEER, PIVOTAL Extract, transform, load (ETL) has always been complex and expensive for moving massive data sets from one data source to another. This is especially true if the source system is a traditional RDBMS with complicated relationships between tables. Most of the time, traditional ETL processes are implemented with batch, monolithic, and tightly coupled approaches. As the result, traditional ETL processes are often considered fragile, hard to maintain, not easy to tune, and often introduce high data latency between source and destination systems. In this session, Anupama Pradhan and Jeff Cherng will cover how to modernize traditional RDBMS ETL processes to cloud-native event driven microservices pipelines by using Cloud Foundry, Spring Cloud Stream, and RabbitMQ/Kafka. The pipelines will be able to handle high volume data sets and complex database queries, yet with low data latency between the source RDBMS and destination data store. In addition, the design will be highly tunable and scalable. The session will also cover analysis of performance metrics based on implementations of real world use cases.