How to Deliver an Event-driven Architecture

July 12, 2017 Richard Seroter

Do you have that "friend" on Facebook who uses all caps to share even the most inane observation (e.g. "JUST ATE TOAST FOR DINNER”, “WHAT’S THE PLURAL OF WALRUS")? Thankfully, we can mute those crazies. But isn't it interesting how social media and mobile platforms condition us to process streams of events? We "follow" people to tap into their musings, adjust our overall feeds on the fly, and engage when we're notified. It's fluid and interactive. Our enterprise systems? Not so much. Most have rigid, pre-defined communication paths and coupling between components used in business processes. This makes it harder to detect trends in your data, make quick business changes, and create more relevant relationships with your customers. It's getting better, though. Many of your peers have accelerated their move to an event-driven architecture and use Cloud Foundry as the platform to get them there. Let's look at some specific things we do to help you build a more adaptable business.

What is an event-driven architecture?

"Events" typically refer to things that happen. For instance, a sensor reports a temperature change, a user clicks their mouse, a customer deposits a check into a bank account. In an event-driven architecture, systems emit and process events in a loosely-coupled fashion. These environments promote asynchronous communication and adaptability.

An event-driven architecture transforms how you generate, process, store, and access data. While possibly more complex to understand, an event-driven architecture helps you embrace unpredictability and reduces dependencies between components. As a result, you can adjust to business changes faster. Martin Fowler of ThoughtWorks does an expert job explaining these core principles in a recent blog post and follow-up video.

So in this model, events are core to your architecture. Being "event-driven" means much more than installing Apache Kafka or using serverless Functions. It involves a catalog of capabilities that together help you emit and process events. Here's one visualization of those capabilities.

Event_Driven_Architecture_EDA_Diagram.png

You'll notice that I call out six areas:

  • Source/sink. In an event-driven architecture, virtually everything generates events: devices, apps, user activity, and more. Many other things consume the result of events, often through server-push protocols like Web Sockets.

  • Server-side applications. This is your code. Things you build to take in events. It may take the form of web apps, microservices, functions, event processing apps, or rich data pipelines.

  • Messaging. How do events zip around the architecture? You may use direct calls between components (with the help of service discovery), or, use a bus. That bus may be in-memory and prioritize low-latency over durability. Or it may store messages and route to per-consumer queues. Another choice is an event stream broker that uses a distributed commit log. Finally, it's useful to have a component that talks to SaaS apps or off-prem application services.

  • Storage and analytics. An event-driven architecture changes how you think about storage. If you do event sourcing, then event records get stored in a database (or databases!) and a materialized view reveals the current state. That current state could get loaded into an application cache, and regenerated on demand. You may use an in-memory data grid to load data from mainframes or legacy systems to get high-performance reads. Caching opens up lots of possibilities. Companies embrace an event-driven model in order to be more responsive. So, you may use a data warehouse and train a machine learning model to process events and generate insight used by apps or data pipelines.

  • Application runtime. What's going to actually run all your event-driven apps? You likely need a platform that keeps long-running apps online, executes short-lived tasks or functions, handles load balancing, maintains a fresh list of routes, and quickly scales apps up and down.

  • Event logging and monitoring. A downside of an event-driven architecture is the lack of predictability. How was this event handled by downstream systems? Why are these systems returning inconsistent results? Your architecture must have a robust method for tracing event paths, monitoring app performance, and making troubleshooting tolerable.

You won't become event-driven overnight. Obviously. Like most technology changes, it's a journey. But it's likely that you need a technology overhaul in order to effectively introduce event processing to your architecture. Enter Cloud Foundry. Cloud Foundry is uniquely positioned to run a whole host of your event-driven apps.

What Cloud Foundry offers you.

For years, Cloud Foundry gave you a platform to run modern applications. It's also become an ideal place to run the wide variety of components that make up your event-driven architecture. Here are seven ways Cloud Foundry makes your event-driven apps better.

  1. Build and deploy stream processing apps using Spring Cloud Stream and Spring Cloud Data Flow. Your event driven apps may be extremely loosely coupled. Each may have no idea what's upstream or downstream. Spring Cloud Stream helps you fully abstract your code from the underlying messaging engine. This library runs great in Pivotal Cloud Foundry with a RabbitMQ or Kafka backend. Want to quickly assemble Spring Boot apps into an event-driven pipeline? Spring Cloud Data Flow is a game-changer, and is quickly becoming an integrated part of Pivotal Cloud Foundry. Stay tuned!

  1. Use brokers to get on-demand data services. The best platforms make data services a first-class citizen. Pivotal Cloud Foundry has you covered here. Get on-demand instances of MySQL, RabbitMQ, Pivotal Cloud Cache, and Redis directly from Pivotal. Discover even more partner offerings from the likes of anynines, Crunchy, Aerospike, Redis Labs, Hazelcast, and more.  Why does this matter? Each event-driven microservice can have its own relational database and cache, all fully managed by the platform. This gets you away from shared monolithic repositories and lets you control the assets in your service domain.

  1. Write apps in any language you want. Sure, Cloud Foundry is a great place for event-driven Spring Boot apps. But you can also deploy web or worker apps written in the .NET Framework, .NET Core, Node.js, Ruby, PHP, Python, or Go. If you want, ship your apps in Docker containers. Heck, write scripts and kick them off as Cloud Foundry Tasks. Use the right language and libraries for each stage of your event processing.

  2. Depend on elastic, dynamic runtime. Cloud Foundry is all about easy software delivery, and operations at scale. Take a web app or background app and instantly scale it up and down. The platform takes care of updating all network routes and firewall rules. Scale manually, or set up an autoscaler policy. Have confidence that your event-driven app can handle any volume of traffic you want to throw at it.

  1. Get context by using built-in monitoring, logging, and distributed tracing. While it may be trickier to observe the health of an event-driven environment, it shouldn't be impossible. PCF Metrics gives you rich infrastructure metrics correlated with your application logs. See how each application instance is behaving, and scale accordingly! For good measure, Pivotal Cloud Foundry also offers Trace Explorer so that you can see the call graph for your event-driven services.

Here again, our ecosystem shines. Extend Pivotal Cloud Foundry with industry leading offerings from Dynatrace, New Relic, Blue Medora, SignalFx, Honeycomb, and more.

  1. Kick off tasks or functions. Sometimes, your code needs to react to something, and then shut down. It's not continuously processing a stream of data. That's fine. Cloud Foundry offers Cloud Foundry Tasks for one-off or scheduled jobs. Trigger via API, CLI, or via Applications Manager, these tasks might perform a data migration, warm a cache, or send notifications. And now you also have Spring Cloud Function for more serverless-style apps that respond to HTTP or messaging triggers.

  2. Get close to the data or services you want, by running Cloud Foundry anywhere. The Big Three cloud providers have some fantastic native event-driven services. Our service broker makes it easy to connect to Azure Event Hubs, Google Cloud PubSub, Amazon SQS, and more. You can use any these services regardless of where your Pivotal Cloud Foundry instance is running. We make it easy to run Cloud Foundry anywhere, and tap into the native services that help you ingest, process, and analyze events.

One platform, ready for your existing apps or more modern, event-driven ones. Pivotal Cloud Foundry gives you what's needed to build a responsive, maintainable architecture. Even if you don't use Cloud Foundry, make sure that you consider all the capabilities you need to satisfy your event-driven ambitions.

Take Cloud Foundry for a spin and see all this in action. Try a free trial with Pivotal Web Services, or download the all-in-one virtual machine called PCF Dev. See something missing from our architecture above? Let us know in the comments!

 

About the Author

Richard Seroter

Richard Seroter is a Senior Director of Product for Pivotal, an 11-time Microsoft MVP for cloud, an instructor for developer-centric training company Pluralsight, the lead InfoQ.com editor for cloud computing, and author of multiple books on application integration strategies. As a Senior Director of Product at Pivotal, Richard heads up product marketing and helps customers see how to transform the way they build software. Richard maintains a regularly updated blog (seroter.wordpress.com) on topics of architecture and solution design and can be found on Twitter as @rseroter.

Follow on Twitter More Content by Richard Seroter
Previous
Introducing Pivotal MySQL*Web, Pivotal’s New Open Source Web-Based Administration UI for MySQL for Pivotal Cloud Foundry
Introducing Pivotal MySQL*Web, Pivotal’s New Open Source Web-Based Administration UI for MySQL for Pivotal Cloud Foundry

Pivotal MySQL*Web is a Pivotal open source project intended to handle the administration of MySQL for Pivot...

Next
Microsoft Azure Partner of the Year Award: And the Winner is.......
Microsoft Azure Partner of the Year Award: And the Winner is.......

Amid 16,000 partners at Microsoft’s Inspire conference last week, Pivotal was recognized as “Azure Consumpt...

How do you measure digital transformation?

Take the Benchmark