A Quick Overview of Flow


I suppose any blog that claims to target the subject of flow should first define what it means by "flow". There is no argument that the word "flow" is heavily overloaded in the world of technology and related practices and disciplines. It is even more overloaded when you lump in fields such as architecture, healthcare, and—not surprisingly—plumbing. So, when talking about event-driven architectures, and especially event-driven integration, what do I mean by "flow"?

First, I provided a very high level overview in a thread on Twitter a few months ago:

Click on the tweet to see the whole thread.

In a blog we can explore the definition in a bit more detail.

Defining Flow

My book, Flow Architectures: The Future of Streaming and Event-Driven Integration, defines flow as follows:

Flow is networked software integration that is event-driven, loosely coupled, and highly adaptable and extensible. It is principally defined by standard interfaces and protocols that enable integration with a minimum of conflict and toil. To my knowledge, there is no universally agreed upon standard to do this today. However, we shall see that flow will drive fundamental changes to the way we integrate businesses and other institutions. I also argue that it is inevitable.

This definition leaves a lot to be desired in terms of the mechanics of flow, so let’s define some key properties that differentiate flow from other integration options. In information theory—which studies the quantification, storage, and communication of information—the sender of a set of information is called the producer, and the receiver is known as the consumer. (The terms source and receiver are also used, but I prefer producer and consumer). With this in mind, flow is the movement of information between disparate software applications and services characterized by the following:

  • Consumers (or their agents) request streams from producers through self-service interfaces
  • Producers (or their agents) choose which requests to accept or reject
  • Once a connection is established, consumers do not need to actively request information—it is automatically pushed to them as it is available
  • Producers (or their agents) maintain control of the transmission of relevant information—i.e., what information to transmit when, and to whom
  • Information is transmitted and received over standard network protocols—including to-be-determined protocols specifically aligned with flow mechanics 

Flow allows an information consumer to locate and request data from independently operated information producers without requiring those producers—or agents that manage flow on the producer’s behalf, such as serverless cloud services or network services that manage flow—to know of the consumer’s existence ahead of time. Flow also allows producers (or agents working on their behalf) to maintain control over the format, protocols, and policies by which the information will be shared. (pp. 3-4)

The "agents" I mention are important, as it is quite likely that the mechanisms that enable and control flow will be decoupled from the actual event producers and consumers. For example, you might leverage a cloud service that lets you trigger Lambda-style functions as consumers of events you receive through that service. Much of the connectivity and security of your flow systems might be delegated to such agents.

Flow and Integration

Flow is defined as a style of integration between applications rather than as an architecture of any individual application. Why is that?

The answer lies in the focus of flow on systems: collections of agents that interact with one another to produce a desired outcome. Flow is ostensibly about the movement of real-time state changes between disparate software agents. It is especially applicable to that movement between agents owned by disparate organizations. However, flow also requires standard interfaces and protocols to enable that integration—which do not yet exist. 

When retail goods suppliers push real-time inventory to a major retailer like Walmart, that is event-driven integration. When stock markets push current buy and ask data to high-frequency traders, that is event-driven integration. When a manufacturing robot sensor detects a critical misalignment between parts, that is event-driven integration. Today, all three use completely different interfaces and protocols to complete those tasks.

Interfaces and Protocols

The flow future is one in which events are streamed using standard protocols, including one (or a very few) standard metadata schemas that allow consistent understanding of what will be found in the event payload. Metadata would include things like the payload schema and/or media type, the source of the event, a timestamp for when the event occurred, and anything else that helps set context for what the payload represents and how it is represented.

Payloads will also likely be standardized over time, though this will more likely happen on an industry-by-industry basis, with different schemas being adopted in different contexts as agreed upon by the parties involved. Custom payloads, of course, will also be possible, but less useful for inter-organization integration.

Finally, the protocols are not very useful without interfaces that enable two agents to establish communication with each other and determine which stream of events should be sent over that connection. Thus, the second major component that turns event-driven integration into flow is a standard interface definition.

All indications are that this interface (or these interfaces) will be publish-and-subscribe style interfaces that enable producers to simply place each new event into the appropriate topic or topics, and consumers to indicate their interest in receiving events from those topics. The exact form of the interfaces is impossible to know today, but I believe we'll see some new ways of identifying the topics consumers wish to subscribe to, and new infrastructure to make those connections as simple, yet efficient, as possible.

What Flow Isn't

Flow is not a "new" event-driven architecture for applications. It does not replace Kafka-style log-driven queues, or workflow automation approaches like Amazon Step Functions, for example. It's not a product you can buy, nor an open source project you can download. It's not even a standard you can adopt today. It is simply a concept that is almost certain to be brought to fruition.

Why I believe that so strongly is a topic for another post, but I will say that there are several signs in the existing event-driven software development market that point to flow. Flow is not an application architecture, but it does reflect the evolution of event-driven systems architecture across the Internet as a whole. You do not adopt a specific architecture today to gain value from flow tomorrow, rather you adopt a technique for moving real-time event data across your application portfolio.

If this seems a bit complicated, I apologize. It really is not. The book goes into much more detail about flow and its nuances, and has many, many examples to help you understand the key concepts. I encourage you to pick up a copy to read more.

Future posts will go into more depth about how I came to the conclusions I came to in the book. You'll learn more about the Wardley mapping process I used to evaluate this user need (real-time event-driven integration) and about how I devised a use of Promise Theory to make my map clearer and more defensible. You'll also see how my mapping led me to the list of component categories I used in the book to evaluate today's market and how it might evolve to achieve the flow vision.

If you have questions or comments, comment here or reach out to me on Twitter (@jamesurquhart) or LinkedIn (https://www.linkedin.com/in/jurquhart/). My tag line these days is "I write to learn", so I can very sincerely say I welcome all constructive feedback, positive or negative. 

Comments

Post a Comment

Popular posts from this blog

How AWS EventBridge API Destinations Are A Flow Innovation

Image
The new API destinations feature of AWS EventBridge introduces a simple mechanism for defining how events processed by EventBridge should be packaged and sent to third party web APIs using standard HTTP verbs, such as GET, POST, and others. (Most commercial APIs are REST APIs, which use these verbs.) By defining some basic connection, authentication, and data formatting information using straight forward wizards and templating mechanisms, EventBridge can automatically create the right HTTP calls every time it processes a relevant event. I highly recommend reading this blog post from AWS's James Beswick to get a better understanding of how this all works. Why is this important news? Because AWS has just made nearly the entire API economy a valid set of consumers for your event streams without requiring you to write any code. Do you want Twiilio to consume a customer's exceptions stream so that it can text them the information in a timely manner? Just create an API destinatio
Read more

Flow and the Future of Software Development

Image
Think about the ways in which the software development process resembles the flow of water running downhill. Ideas are the tiny drops of potential value that gather into features and architectures which then feed backlogs. From there, developers create code and configuration—thus adding value—which  then merges with other flows. Work done in the combined flow gains more and more value until it is presented to those that convert all that value into desired outcomes: profit, or perhaps mission success. OK, it's a bit of a corny analogy, but it represents my mental model of the challenges that developers face while creating any significant enterprise or commercial software system. First, the challenge is to convert potential energy (a worthy idea or demonstrated need) into kinetic energy (the increasing value of the solution to the end user) via a metaphorical force of gravity (the desire for the aforementioned profit or mission success). The other thing I picture is that water runnin
Read more

Two Ways To Evaluate Which Event-Driven Architecture is Right for Your Flow Systems

Image
The event-driven platforms market is a confusing place. In fact, there is an entire appendix in Flow Architectures: The Future of Streaming and Event-Driven Integration dedicated to evaluating much of that market and it's almost a third of the book. Thus, it is no surprise that developers and architects not experienced with eventing systems struggle to identify the best platform for their applications. I've identified one method for making a selection in the book, which I will update in this blog post to reflect what I have learned since the book came out. I used a model based on the types of event streams, which works extremely well if you are going to consume a stream over which you have little control. (It also works for selecting technologies within a scope you control, but I am more interested in how you "future-proof" your decisions for a flow future.)  Another interesting way to slice the market comes from researcher Jamie Brandon, who looked at the problem fr
Read more