Hot And Cold Data With Apache Kafka, Tiered Storage, And Iceberg Data (R)evolution podcast

Hot and cold data with Apache Kafka, Tiered Storage, and Iceberg

10M ago 48:58

Content provided by Aiven. All podcast content including episodes, graphics, and podcast descriptions are uploaded and provided directly by Aiven or their podcast platform partner. If you believe someone is using your copyrighted work without your permission, you can follow the process outlined here https://ppacc.player.fm/legal.

Utilizing the true potential of data streaming is key to business success.

In this Data (R)evolution episode, we're joined by Josep Prat and Filip Yonov to dive into the transformative features of Apache Kafka and its evolving role in data architecture. They discuss the critical importance of collaboration and feedback in enhancing Kafka's capabilities, the future of "lake house" technology, exciting updates from the Open Source Program Office (OSPO), and the importance of Kafka's readiness to support evolving data formats—making it a backbone for modern data ecosystems.

Key Takeaways:

Community collaboration and contribution are essential for the continuous improvement and testing of Apache Kafka's capabilities
The evolution of Apache Kafka into a more versatile platform, combined with object storage and open table formats, can significantly enhance real-time data streaming, analytics, and the future of "lake house" technology
Tiered storage in Kafka facilitates more efficient and cost-effective data management by decoupling storage from computing

Resources:

Watch the full interview on our YouTube: https://www.youtube.com/@Aiven_io
Check out our website for more information: https://aiven.io/
Check out Aiven AI Database Optimizer
Want to be on our mailing list? Sign up here: https://aiven.io/resources
Follow us on LinkedIn: https://www.linkedin.com/company/aiven/
Sign up for our newsletter for more insights on this topic: https://aiven.io/newsletter
Connect with Filip Yonov on LinkedIn: https://www.linkedin.com/in/filipyonov/
Connect with Josep Prat on LinkedIn: https://www.linkedin.com/in/jlprat/

Timestamps:

[05:49] Kafka servers have theoretical storage limits

[09:29] Test storage proposal process for Apache Kafka

[17:38] LinkedIn conducted an experiment merging Xcode versions

[22:11] Data lake evolving into lake house architectures

[25:00] Broker pushes data to remote storage, plugin handles retrieval and format translation

[26:40] Kafka excels at high-speed, high-volume data

[32:18] Kafka data consumption evolving with new options

[40:19] Managing metadata for conversion on community level

[47:45] Kafka's potential as a widely used API

11 episodes