Available today, Spring Flo is now out of beta and is generally available.
In traditional Spring fashion, Flo helps developers build software—making integration easier, improving the speed and quality of development, and addressing organizational needs. This release includes new capabilities, including the composition of batch workflows—allowing developers to run batch jobs sequentially, in parallel, or both. Downloads and documentation are located here, and the remainder of this post provides a background on Flo, explains the challenges it addresses, reviews the Flo solution and features then talks about the journey ahead.
A Quick Background: Spring XD, Flo for Spring XD, and Spring Cloud Data Flow
For those that are unfamiliar, Spring XD’s purpose is to simplify big data app development for real-time streams or batch pipelines in use cases like Internet of Things, complex event processing, closed loop analytics, and batch workflow orchestration. Spring XD is composed of several moving parts—it is both a runtime environment and a development framework, providing data ingestion, real-time analytics, batch processing, and data export in a unified, extensible, and distributed model.
Not too long ago, one of our Spring engineering leaders, Andy Clement conceived a concept for visualizing, creating, and monitoring streaming and batch data pipelines on Spring XD. Months later, Flo for Spring XD was born and moved into beta. As explained in this short Flo for Spring XD overview video and the documentation, Flo for Spring XD is an HTML5 canvas app that runs on top of Spring XD and provides a GUI for creating, analyzing, and monitoring pipelines. To provide forward-looking context, there is a Cloud Native redesign of Spring XD underway, and it is called Spring Cloud Data Flow—this project is the next candidate to adopt Flo.
The Challenges That Flo Addresses
As Flo has moved from Beta to GA, it has focused more on addressing several development challenges, while also addressing some organizational needs.
First, developers wanted to better address complexity. Most developers use a wide variety of integration technologies, applications, frameworks, and tool chains. Each of these serves a subset of their needs, makes it more cumbersome to do work, and falls short of meeting all data integration requirements. Wherever possible, they wanted a way to consolidate the “user experience” for development and use fewer tools that were more integrated.
Second, users wanted to be able to develop faster without sacrificing quality or extensibility. This meant creating and visualizing jobs through a drag-n-drop GUI in addition to a DSL or shell and a RESTful API. Given their desire to have the GUI operate at a higher level than code, it would allow them to build pipelines in seconds or minutes instead of hours or days. While they were interested in speed, they still wanted to be able to go back in and customize or extend the default components.
Lastly, users needed to address organizational needs, and this included several perspectives. From a development perspective, they wanted to be able to re-use patterns that repeat within their applications and across their enterprises. With regards to learning curves, they wanted new developers to be productive day one—through a GUI—and graduate to more sophisticated work via the DSL or shell. In looking at other needs, they wanted to allow different groups to access and use data pipelines without friction. For example, they wanted business users to quickly pipe a new data stream into their data lake for analysis, supporting a quicker response to the market and competition—normally, analysts don’t have the skillset of a developer yet they need to get insights at the fastest rate possible. As well, they wanted to make it easy for IT operations groups to access and monitor activity across various data pipelines.
Flo For Spring XD: An Integrated GUI With Graphical Canvas, DSL, and Shell
To solve these challenges, Flo for Spring XD provides an intuitive GUI that is simple to adopt and use. Flo also includes all the basic elements of an integration-flow designer such as connectors, control nodes, palettes, state transitions, and graph topologies—importantly, there is a textual shell, DSL support, and a graphical canvas designed for creating and reviewing comprehensive workflows.
Because of its unique GUI, Flo for Spring XD supports a hybrid model of development. This satisfies both classes of developers—those who need a simpler, faster, easier to use, drag-and-drop canvas and those that need extreme control. As depicted in the screenshot below, a “textual shell” (the text area above the canvas) replicates most of the features supported in Spring XD’s shell. For many developers, it is much easier to build comprehensive workflows using the “textual shell” than the drag and drop interface. The shell is fully DSL compliant and bundled with context-specific autocomplete and content assists, operating much like an IDE.
The graphical canvas and textual shell also stay in sync. Many applications show either a text input or a graphical input. When they do show both, these apps rarely keep the two in sync. Flo for Spring XD takes a very different approach and actually keeps the textual representation synchronized with the graphical representation. So, modifying the graphical canvas in the GUI will update the text immediately and vice versa.
Importantly, the GUI enables someone who is initially comfortable with the graphical canvas to see the DSL created before their eyes as they edit it. Then, they can gradually become more familiar with the DSL syntaxes. As well, they can edit the DSL and see changes on the graphical canvas. This teaches them to become a power user and streamlines development work on a series of similar pipelines. Developers can copy the DSL and paste it into a new pipeline rather than start from a blank canvas. The copied text would create a new graphic structure to then tweak.
This approach addresses all of the challenges portrayed by Spring XD users since beta and covered in the prior section—making development faster and simpler while addressing organizational needs like pattern re-use.
Flo for Spring XD: Features
The new release of Flo for Spring XD allows Spring XD users to:
- Create, manage, and monitor stream and batch pipelines using DSL, a graphical canvas, or both
- Build and orchestrate comprehensive workflows using streams and batch jobs together
- Develop integration flows with named channels, taps, and composite modules
- Write pipelines via DSL with content-assist and auto-complete
- Perform health monitoring
- Capture granular, component-level metrics within the pipeline itself or for monitoring
- See a visual representation of modules across a distributed deployment
- Use auto-adjustment and grid-layout capabilities in the GUI for easier and more efficient organization of pipelines
Flo’s Journey Ahead
At its core, Flo has been generally envisioned to adapt to any Spring project that uses the notion of visual graphs, flowcharts, workflows, or pipelines and wants to interact with these elements through a simple, easy-to-use GUI.
Spring Cloud Data Flow, the Cloud Native redesign of Spring XD, is the next immediate candidate to adapt Flo and will be fully featured in an upcoming release of Spring Cloud Data Flow.
Further adding to the Flo family, Flo for Spring Integration and Flo for Spring Batch are also variants in the current release pipeline, and future releases are targeting Flo add-ons for distributed tracing, granular metrics, alerting, and enhanced monitoring.
Learning More
- Read more about Spring XD or visit the site
- Check out a recent article on Flo for Spring XD or see the Documentation
- Find out more about Spring Cloud Data Flow or listen to a podcast about it.
About the Author
More Content by Sabby Anandan
