For Super Bowl XLIX, we felt it would be cool to see if people were tweeting #Patriots more or less than #Seahawks. So, we built a real-time analysis that would give us a “live” view of Twitter reactions to each play as the game unfolded.
The real story, however, is how we set this whole analysis system up in just one day.
Building this Super Bowl application was timely for our team since we’ve recently been working on a solution to capture real-time analytics—this project seemed like a great way to leverage and test the work we’ve been doing.
The image above depicts the swings in the number of tweets for either team during the game. The system analyzed over 150,000 tweets during the last hour of the game, and, with the architecture below, the entire system was built from scratch in less than a day.
Real-time Analysis of Super Bowl Tweets Using Microservices on Cloud Foundry
At a high-level, we identified 3 core microservices. The first microservice ingests the raw tweets from Twitter’s garden hose (or stream API) and then passes them to the second microservice. The second microservice performs the real-time analytics on the tweets coming through the system by searching for either #Patriots or #Seahawks and then updating a count for whichever hashtags it found. The last microservice displays a dashboard, graphically depicting the count data.
Advantages of a Microservice Architecture
By breaking the solution into microservices, we could quickly build the unique pieces that we needed while leveraging existing services. For example, we were able to quickly put together a solution for ingesting the tweets from Twitter by using Spring XD. Spring XD lets you quickly create a solution for data ingestion by enabling the user to define a source for the data (Twitter in this case) and then configure a sink to send the data to (we used RabbitMQ).
The stream analytics microservice was also quickly pulled together—we simply needed to configure it to perform the analytics on the data as it passed through the system. By using RabbitMQ and REST to communicate between these services, we gained the flexibility of hosting these on either the same Pivotal Cloud Foundry (PCF) environment or to use separate environments if we preferred. For example, we decided to host the dashboard on Pivotal’s public Cloud Foundry offering, called Pivotal Web Services, while hosting the other 2 microservices on our internal, private PCF environment.
Another key advantage of decoupling these core functions into microservices is that we could quickly do development iterations in an agile fashion. For example, we were able to quickly make changes to the dashboard microservice without affecting the operation of the other microservices, and the dashboard took the majority of our one day development timeline.
Advantages of Using Cloud Foundry
Since we were using Cloud Foundry, we would test the changes locally and then quickly push these changes from our desktop computer using the “cf push” command. Cloud Foundry also made it easy to use a blue-green deployment so that changes to a microservice can be pushed while providing a safety net where developers can roll back to the prior version if any problems occur.
Hosting our microservices on PCF made it simple to scale up the individual components, as necessary. Of course, we expected traffic during the Super Bowl game to be significantly higher than we saw during development and testing. With our platform, we were able to scale the number of instances of the stream analytics microservice by simply issuing the Cloud Foundry “cf scale” command. As tweets arrived through RabbitMQ, they were spread to the multiple instances of the stream analytics service, and any available instance would grab the latest tweet and individually process it, spreading the load across all the instances.
By leveraging both a microservice architecture and Cloud Foundry we were able to quickly put together a solution to compare the number of the tweets mentioning either #patriots or #seahawks. Cloud Foundry also made it easy to update services as we were developing and made it trivial to scale components up as needed.
- Webinar: Made for Each Other: Microservices + PaaS
- Webinar: Building “Bootiful” Microservices with Spring Boot
- Podcast: All Things Pivotal Podcast Episode #8: A Chat About Microservices With Matt Stine
- Podcast: Using Service Brokers in Pivotal Cloud Foundry
- Slides: Developing Microservices for PaaS with Spring and Cloud Foundry
- Pivotal Cloud Foundry Product/Download/Docs and Blog Posts
- Spring XD Project, Getting Started, Documentation, Videos, and More
About the AuthorMore Content by Allan Baril