Apache Kafka

Customers are becoming digital, and digital needs data NOW. Fast forward, data analytics quickly evolved from legacy Data Warehouses to Big Data, but always looking at "what happened", a retrospective view. Businesses need fresh data to be faster and make smarter decisions, and that means real-time data. I am happy to introduce "Stream Processing Landscape", a general guideline to understand how Apache Flink, a leading open source engine for real-time data, is setting the pace for data streaming processing and fits into the enterprise ecosystem. 🟢 Structured: well-know and well-governed data source, structured data is pointed to be around 20% of all the corporate data in the world. It is mainly stored in databases, with defined schemas. Apache Flink commonly leverages a CDC strategy to consume data in real-time from databases. 🟢 Unstructured: the remaining 80% of the enterprise data is a combination of unstructured data formats, in many ways. Cost-effective and scalable Big Data solutions, such as cloud-native storage and Apache Kafka, helps companies to safe guard and store years of logs, machine data and images. 🟢 Enterprise Apps: streaming data should integrate with the application ecosystem, triggering actions for best next offers, ad-hoc advertisement and up-selling opportunities. Specialized solutions for marketing, point of sales and enterprise management can be augmented with real-time data and AI. 🟢 Data Ecosystem: Apache Flink leverages most the robust and mature frameworks and engines currently available in the data ecosystem, including open data formats like Apache Parquet, and new Big Data management approaches such as Apache Iceberg and Fluss for Lakehouses. These open standards ensure interoperability and freedom of choice to adopt any tool, any vendor. #StreamProcessing #RealtimeData #ApacheFlink #ApacheIceberg #Lakehouse

Apache Kafka is getting native support for queues—and it’s a game changer for the #datastreaming community. With the #opensource Kafka 4.0 release, #ApacheKafka went beyond logs and stream processing by introducing queue-style semantics through KIP-932. This new capability unlocks parallelism that isn’t limited by partition count—perfect for real-world #jobqueue and #taskdistribution scenarios. The key innovation is the introduction of share groups, allowing multiple consumers to process messages from the same partition. This enhances Kafka’s scalability for event-driven applications and simplifies patterns previously handled by external #messagequeues like RabbitMQ or ActiveMQ. Kafka’s new queue support adds: - Fine-grained control with message-level acknowledgements (ack, release, reject) - High-throughput flexibility beyond the limits of traditional consumer groups - A powerful hybrid model combining queue semantics with Kafka’s durable event log For use cases like inventory systems, sales event processing, or real-time analytics, this makes Kafka a unified solution—blending the benefits of #streamprocessing with the needs of flexible, scalable queuing. This isn’t production-ready yet—no Dead Letter Queue (DLQ) support, no delayed retries, and no strict ordering per key. But for teams already invested in Kafka, it avoids the operational overhead of maintaining separate queue infrastructure for simpler tasks. Kafka continues evolving into the central nervous system for modern #cloudnative data architectures. Deep dive slide deck and video recording: https://lnkd.in/eHihr_Yr What’s the first use case you’d move to Kafka queues?

Apache Flink + Apache Hudi: Changelog Mode for Streaming ETL. When building real-time data architectures, streaming engines like Apache Flink consumes change events from sources like Apache Kafka or ApachePulsar. But how do we persist those changes efficiently, support downstream incremental queries, and build layered analytics in a lakehouse? Enter Apache Hudi’s changelog mode! At the heart of this setup is Flink’s Dynamic Table abstraction - a powerful construct that allows you to treat "unbounded changelog streams" as continuously updating tables. - Flink reads from a source stream and materializes it as a Dynamic Table. - Continuous queries transform this table, outputting another Dynamic Table. - Hudi acts as the sink, consuming the changelog stream and storing it as a Merge-on-Read (MOR) table. The innovation here is that Hudi doesn’t just store the data. It also serves as a materialization layer for the changelog stream, supporting native INSERT, UPDATE, and DELETE operations. How It Works? Flink’s Dynamic Table introduces a metadata column called "RowKind", which can take values like: +I  - INSERT -U / +U - UPDATE_BEFORE / UPDATE_AFTER -D - DELETE Hudi maps this directly via the "_hoodie_operation" column in changelog mode. This allows: ✅ Row-level changes to be encoded into log files ✅ Streaming readers to consume the data before compaction ✅ Preservation of order to support accurate incremental views This capability has been available since Hudi 0.9 and is actively used in production across streaming ETL pipelines. Traditional batch ETL pipelines suffer from high latency and heavy resource costs. With this Flink + Hudi pattern, you unlock: 🌟 Low-latency data ingestion 🌟 Fine-grained updates via changelogs 🌟 SQL-based lakehouse transformations (bronze → silver → gold) Flink handles the stateful compute, and Hudi handles durable, versioned storage. I am presenting all about this architecture in the Data Streaming Summit 2025 (organized by StreamNative) next week. Join me (link in comments). #dataengineering #softwareengineering

Game-changer feature. Diskless topics 🚀🚀🚀