How We Found Out The #Patriots Beat The #Seahawks on Twitter Too

February 6, 2015 Allan Baril

For Super Bowl XLIX, we felt it would be cool to see if people were tweeting #Patriots more or less than #Seahawks. So, we built a real-time analysis that would give us a “live” view of Twitter reactions to each play as the game unfolded.

image00

The real story, however, is how we set this whole analysis system up in just one day.

Building this Super Bowl application was timely for our team since we’ve recently been working on a solution to capture real-time analytics—this project seemed like a great way to leverage and test the work we’ve been doing.

The image above depicts the swings in the number of tweets for either team during the game. The system analyzed over 150,000 tweets during the last hour of the game, and, with the architecture below, the entire system was built from scratch in less than a day.

Real-time Analysis of Super Bowl Tweets Using Microservices on Cloud Foundry

At a high-level, we identified 3 core microservices. The first microservice ingests the raw tweets from Twitter’s garden hose (or stream API) and then passes them to the second microservice. The second microservice performs the real-time analytics on the tweets coming through the system by searching for either #Patriots or #Seahawks and then updating a count for whichever hashtags it found. The last microservice displays a dashboard, graphically depicting the count data.

image01

Advantages of a Microservice Architecture

By breaking the solution into microservices, we could quickly build the unique pieces that we needed while leveraging existing services. For example, we were able to quickly put together a solution for ingesting the tweets from Twitter by using Spring XD. Spring XD lets you quickly create a solution for data ingestion by enabling the user to define a source for the data (Twitter in this case) and then configure a sink to send the data to (we used RabbitMQ).

The stream analytics microservice was also quickly pulled together—we simply needed to configure it to perform the analytics on the data as it passed through the system. By using RabbitMQ and REST to communicate between these services, we gained the flexibility of hosting these on either the same Pivotal Cloud Foundry (PCF) environment or to use separate environments if we preferred. For example, we decided to host the dashboard on Pivotal’s public Cloud Foundry offering, called Pivotal Web Services, while hosting the other 2 microservices on our internal, private PCF environment.

Another key advantage of decoupling these core functions into microservices is that we could quickly do development iterations in an agile fashion. For example, we were able to quickly make changes to the dashboard microservice without affecting the operation of the other microservices, and the dashboard took the majority of our one day development timeline.

Advantages of Using Cloud Foundry

Since we were using Cloud Foundry, we would test the changes locally and then quickly push these changes from our desktop computer using the “cf push” command. Cloud Foundry also made it easy to use a blue-green deployment so that changes to a microservice can be pushed while providing a safety net where developers can roll back to the prior version if any problems occur.

Hosting our microservices on PCF made it simple to scale up the individual components, as necessary. Of course, we expected traffic during the Super Bowl game to be significantly higher than we saw during development and testing. With our platform, we were able to scale the number of instances of the stream analytics microservice by simply issuing the Cloud Foundry “cf scale” command. As tweets arrived through RabbitMQ, they were spread to the multiple instances of the stream analytics service, and any available instance would grab the latest tweet and individually process it, spreading the load across all the instances.

By leveraging both a microservice architecture and Cloud Foundry we were able to quickly put together a solution to compare the number of the tweets mentioning either #patriots or #seahawks. Cloud Foundry also made it easy to update services as we were developing and made it trivial to scale components up as needed.

Learn More:

About the Author

Biography

More Content by Allan Baril
Previous
MySQL Service Now Highly Available on Pivotal Cloud Foundry
MySQL Service Now Highly Available on Pivotal Cloud Foundry

The Pivotal Cloud Foundry team is excited to announce that the Pivotal Services catalog now includes a high...

Next
Pivotal Philanthropy Helps Viral Video Game With A Strong Message On To The App Store
Pivotal Philanthropy Helps Viral Video Game With A Strong Message On To The App Store

Last year, two young women went to the Girls Who Code Summer Immersion Program and came out with a viral vi...