The database language SQL has proven to be an invaluable asset for most developers building software applications. Yet, the world as we know it has changed dramatically since SQL was created in 1974. Together with my colleague Dr. Matthias J. Sax (ASF committer and PMC for Apache Kafka, Apache Flink, and Apache Storm) I wrote an article on how the future of SQL is being shaped by changes in how data—and notably data in motion—is used to build applications today.
Head over to The Future of SQL: Databases Meet Stream Processing on the Confluent blog to start reading.
Figure: Today’s world never sleeps. As software engineers, we are increasingly building architectures in which automated systems drive other automated systems directly, 24x7. And this requires a change of engineering strategy: The modern database, as shown on the right, must support a “data-active, query-passive” approach in which queries execute and passively wait for new data to arrive from upstream queries and data sources.
We start with a brief history of SQL and how it’s been used with data systems through to the modern day and then cover the rise of Apache Kafka and data in motion. We then take a deep dive into several key areas where SQL—originally designed for data at rest—needs to meet data in motion so that we engineers can build the modern architectures that companies need today and in the future.
Figure: An example of where data in motion meets data at rest: a stream of customer location changes is continuously being joined to a table with customer profile data. See the linked article for full explanation.
If you have finished reading the article, you might wonder what is happening next? There’s a lot of activity in the technical community in this area. Matthias and I have been participating on behalf of Confluent in INCITS, the U.S. national body for a variety of technical standards, including the SQL standard. Specifically, we have been working in the technical committee DM32 (Data Management and Interchange) that develops standards for database languages. Here, we are collaborating in a Streaming SQL working group together with vendors of traditional databases such as Microsoft, Oracle, and IBM, cloud database players like Snowflake., and vendors of stream processing systems like Alibaba and Google. This list of participants, many of which are industry leaders with both technical pedigree as well as strong market reputation, presents a clear validation of the increasing importance of event streaming and streaming databases in modern software architectures.
So if you want to learn even more, my article is actually part one of a series of Readings in Streaming Database Systems. Feel free to check out the full series:
- Overview: Readings in Streaming Database Systems
- Part 1: The Future of SQL: Databases Meet Stream Processing (my article)
- Part 2: 4 Key Design Principles and Guarantees of Streaming Databases
- Part 3: How Do You Change a Never-Ending Query?