Hot and cold data with Apache Kafka, Tiered Storage, and Iceberg

Data (R)evolution

#Tech #Business #Aiven #Data #Cloud Technologies #Technology #Digital Revolution

13:44

Legendary actor and mental health advocate Glenn Close is on a quest to change how we think about mental health, starting with her decision to speak out about her own family's struggles — a brave choice considering the stigma that pervades the topic. This week, we're revisiting this sweeping conversation with TEDWomen curator Pat Mitchell, where Close shares the inspiration behind the advocacy group she founded to combat the crisis, underscoring the transformative power of community and the critical need for comprehensive mental health care systems. Want to help shape TED’s shows going forward? Fill out our survey ! Hosted on Acast. See acast.com/privacy for more information.…

about a year ago 48:58

MP3•Episode home

Utilizing the true potential of data streaming is key to business success.

In this Data (R)evolution episode, we're joined by Josep Prat and Filip Yonov to dive into the transformative features of Apache Kafka and its evolving role in data architecture. They discuss the critical importance of collaboration and feedback in enhancing Kafka's capabilities, the future of "lake house" technology, exciting updates from the Open Source Program Office (OSPO), and the importance of Kafka's readiness to support evolving data formats—making it a backbone for modern data ecosystems.

Key Takeaways:

Community collaboration and contribution are essential for the continuous improvement and testing of Apache Kafka's capabilities
The evolution of Apache Kafka into a more versatile platform, combined with object storage and open table formats, can significantly enhance real-time data streaming, analytics, and the future of "lake house" technology
Tiered storage in Kafka facilitates more efficient and cost-effective data management by decoupling storage from computing

Resources:

Watch the full interview on our YouTube: https://www.youtube.com/@Aiven_io
Check out our website for more information: https://aiven.io/
Check out Aiven AI Database Optimizer
Want to be on our mailing list? Sign up here: https://aiven.io/resources
Follow us on LinkedIn: https://www.linkedin.com/company/aiven/
Sign up for our newsletter for more insights on this topic: https://aiven.io/newsletter
Connect with Filip Yonov on LinkedIn: https://www.linkedin.com/in/filipyonov/
Connect with Josep Prat on LinkedIn: https://www.linkedin.com/in/jlprat/

Timestamps:

[05:49] Kafka servers have theoretical storage limits

[09:29] Test storage proposal process for Apache Kafka

[17:38] LinkedIn conducted an experiment merging Xcode versions

[22:11] Data lake evolving into lake house architectures

[25:00] Broker pushes data to remote storage, plugin handles retrieval and format translation

[26:40] Kafka excels at high-speed, high-volume data

[32:18] Kafka data consumption evolving with new options

[40:19] Managing metadata for conversion on community level

[47:45] Kafka's potential as a widely used API

11 episodes

Hot and cold data with Apache Kafka, Tiered Storage, and Iceberg

Data (R)evolution

published about a year ago

MP3•Episode home

Utilizing the true potential of data streaming is key to business success.

Key Takeaways:

Community collaboration and contribution are essential for the continuous improvement and testing of Apache Kafka's capabilities
The evolution of Apache Kafka into a more versatile platform, combined with object storage and open table formats, can significantly enhance real-time data streaming, analytics, and the future of "lake house" technology
Tiered storage in Kafka facilitates more efficient and cost-effective data management by decoupling storage from computing

Resources:

Watch the full interview on our YouTube: https://www.youtube.com/@Aiven_io
Check out our website for more information: https://aiven.io/
Check out Aiven AI Database Optimizer
Want to be on our mailing list? Sign up here: https://aiven.io/resources
Follow us on LinkedIn: https://www.linkedin.com/company/aiven/
Sign up for our newsletter for more insights on this topic: https://aiven.io/newsletter
Connect with Filip Yonov on LinkedIn: https://www.linkedin.com/in/filipyonov/
Connect with Josep Prat on LinkedIn: https://www.linkedin.com/in/jlprat/

Timestamps:

[05:49] Kafka servers have theoretical storage limits

[09:29] Test storage proposal process for Apache Kafka

[17:38] LinkedIn conducted an experiment merging Xcode versions

[22:11] Data lake evolving into lake house architectures

[25:00] Broker pushes data to remote storage, plugin handles retrieval and format translation

[26:40] Kafka excels at high-speed, high-volume data

[32:18] Kafka data consumption evolving with new options

[40:19] Managing metadata for conversion on community level

[47:45] Kafka's potential as a widely used API

11 episodes

#Tech #Business #Aiven #Data #Cloud Technologies #Technology #Digital Revolution

All episodes

Data (R)evolution

1
Making Open Source More Secure and Sustainable 36:26

41 weeks ago36:26

36:26

Open-source software is a cornerstone of modern tech infrastructure, yet its sustainability is often precarious. In this episode of Data (R)evolution, Josep Prat and Floor Drees join host Francesco Tisiot to dive into the critical need for sustainable open-source. They emphasize the importance of supporting open-source projects to ensure their stability in the linguistic challenges around the word "free" in open source, and they share real-world threats, such as a recent attack on the XZ package. Tune in to learn about the evolving landscape of open source, its challenges, and how we can all contribute to its future security and viability. Key Takeaways: Ensure the sustainability of open source projects by encouraging multiple contributors and reducing dependency on a single maintainer Companies should use open source for financial benefits and actively contribute code, time, and resources to these projects Evaluating the health of open-source projects involves a combination of numerical data, public information, and community understanding Resources: Watch the full interview on our YouTube: https://www.youtube.com/@Aiven_io Check out our website for more information: https://aiven.io/ Check out Aiven AI Database Optimizer Want to be on our mailing list? Sign up here: https://aiven.io/resources Follow us on LinkedIn: https://www.linkedin.com/company/aiven/ Sign up for our newsletter for more insights on this topic: https://aiven.io/newsletter Connect with Josep Prat on LinkedIn: https://www.linkedin.com/in/jlprat/?originalSubdomain=de Connect with Floor Drees on LinkedIn: https://www.linkedin.com/in/floordrees/ Timestamps: [05:59] Open source promotes transparency; closed source doesn't. [13:30] JavaScript ecosystem heavily reliant on single maintainers. [22:42] Encourage open source support from corporations. [26:11] It’s critical to choose sustainable projects and contribute to open source. [31:07] EU framework law responds to open source security. [34:36] How to ensure open source sustainability for community benefit.…

Data (R)evolution

1
Aiven's Vision for Enterprise Solutions and Sustainability 34:04

42 weeks ago34:04

34:04

At the forefront of everything we do at Aiven, is our customers. And we do this by focusing on enterprise readiness, trust, and compliance, bringing AI closer to daily operations, and allowing customers to leverage their existing cloud agreements. In this Data (R)evolution episode, we're joined by Oskari Saarenmaa, Aiven’s CEO and Co-Founder, to talk about Aiven's future plans, including the innovative “Bring Your Own Cloud” (BYOC) approach and its potential advantages. We explored key features such as vector search, AI enhancements, and new serve capabilities. Additionally, we looked at Ivan's sustainability efforts and their commitment to reducing the carbon footprint of their services. This episode is filled with technical insights and forward-thinking ideas. Key Takeaways: Aiven prioritizes enterprise-grade security and compliance, integrating advanced technical controls and achieving certifications to build customer trust The "Bring Your Own Cloud" (BYOC) feature allows customers to deploy Aiven technologies on their existing cloud accounts, maximizing cost savings and native cloud capabilities EverSQL's acquisition enhances Aiven's platform by optimizing SQL database performance through AI, enabling developers to focus on value-adding tasks and improving overall efficiency Resources: Watch the full interview on our YouTube: https://www.youtube.com/@Aiven_io Check out our website for more information: https://aiven.io/ Check out Aiven AI Database Optimizer Want to be on our mailing list? Sign up here: https://aiven.io/resources Follow us on LinkedIn: https://www.linkedin.com/company/aiven/ Sign up for our newsletter for more insights on this topic: https://aiven.io/newsletter Connect with Oskari Saarenmaa on LinkedIn: https://www.linkedin.com/in/oskarisaarenmaa/?originalSubdomain=fi Timestamps: 03:47 Founders started consulting, saw need for scalability 08:29 Cloud growth challenges open source business models 10:21 EverSQL optimizes databases, helps users with performance 15:06 Engineer initiated carbon footprint tracking for product 17:26 Enterprise evolution from self-service to platform 25:31 Aiven offers diverse AI tools and technologies 29:09 Developing PGAas with visual identity, technical features 32:19 Basic technical tooling, trust, experience enable partnership 33:29 Acknowledgment of role and sustainability in business…

Data (R)evolution

1
Fueling Startup Innovation through Aiven’s Cluster Program 18:23

43 weeks ago18:23

18:23

Navigating the startup landscape in the cloud era is crucial. But luckily, the Aiven platform empowers startups to succeed in this competitive and dynamic environment. In this Data (R)evolution episode, Aiven’s Ian Massingham and Angie Byron discuss the transformative impact of cloud computing on startups and the benefits of leveraging open-source technology. They dive into the common challenges early-stage startups face, the importance of agility and flexibility in scaling, and the role of Aiven's Cluster program in supporting startup growth. Our discussion highlights the synergy between open-source best practices, multi-cloud capabilities, and the strategic allocation of resources—essential factors for thriving in the modern startup ecosystem. Key Takeaways: Leveraging open-source technologies enables startups to quickly prototype, scale, and avoid reinventing the wheel using proven solutions backed by vast developer communities. Flexibility to pivot and adapt rapidly is crucial for startups to find product-market fit and address unexpected challenges and opportunities as they grow. Programs like Aiven's Cluster offer valuable resources and credits that allow early-stage startups to build robust data infrastructure on multiple cloud platforms without substantial initial capital investment. Resources: Watch the full interview on our YouTube: https://www.youtube.com/@Aiven_io Check out our website for more information: https://aiven.io/ Check out Aiven AI Database Optimizer Want to be on our mailing list? Sign up here: https://aiven.io/resources Follow us on LinkedIn: https://www.linkedin.com/company/aiven/ Sign up for our newsletter for more insights on this topic: https://aiven.io/newsletter Connect with Ian Massingham on LinkedIn: https://www.linkedin.com/in/imassingham/?originalSubdomain=uk Connect with Angie Byron on LinkedIn: https://www.linkedin.com/in/webchick/?originalSubdomain=ca Timestamps: 00:00 Startup market transformed by affordable cloud technology 03:43 Open source and startups create a great mix 08:13 Aiven provides data infrastructure for cloud services 10:33 Aiven streamlines open source tech for efficiency 15:06 Startups benefit from agile cloud and open-source 16:59 The Cluster program supports startups with resources and mentoring…

Data (R)evolution

1
Navigating Kafka: Challenges, Solutions, And The Future Of Real-Time Data Streaming 33:58

44 weeks ago33:58

33:58

The complexities of real-time data streaming and auto-scaling in the cloud are no joke. But three are solutions to make it more simple and efficient. In this Data (R)evolution episode, Matan Mizrahi and Filip Yonov join us to discuss how Kafka effectively handles and optimizes data streams, the pursuit of standardized APIs, and the vital role of AI in optimizing complex systems. They also share how AI and open-source tools are transforming the landscape. Tune in to hear about the evolution of data streaming and the transformative impact of AI and cloud-native solutions. Key Takeaways: Emphasize the need for customer-defined scaling logic based on metrics and auto-scaling events for efficient Kafka management. The importance of integrating AI, like the Ivan AI database optimizer, to enhance workload optimization and complex system management in Kafka. The significance of strategic partner relationships in providing advanced software solutions and necessary knowledge and experience to mitigate risk in critical data infrastructure projects. Resources: Watch the full interview on our YouTube: https://www.youtube.com/@Aiven_io Check out our website for more information: https://aiven.io/ Check out Aiven AI Database Optimizer Want to be on our mailing list? Sign up here: https://aiven.io/resources Follow us on LinkedIn: https://www.linkedin.com/company/aiven/ Sign up for our newsletter for more insights on this topic: https://aiven.io/newsletter Connect with Matan Mizrahi on LinkedIn: https://www.linkedin.com/in/matan-mizrahi-2106/?originalSubdomain=il Connect with Filip Yonov on LinkedIn: https://www.linkedin.com/in/filipyonov/?originalSubdomain=nl Check out Claroty: https://claroty.com/ Timestamps: 05:13 Challenges in running and optimizing streaming systems 09:10 Using stream data to produce and consume 12:22 Product manager solves customer problems with new capabilities 16:10 Agreement on system complexity and standard API 18:22 Kafka not yet fully cloud native 23:30 Simplifying cloud setup through productization and self-service 26:17 Hope for AI integration with Kafka, improved management 28:24 AI optimizations, reduce customer pain, lower barriers 32:05 Matan emphasized importance of experience and knowledge…

Data (R)evolution

1
The Rise of Multi-Cloud, Open source, and BYOC 42:07

45 weeks ago42:07

42:07

Modernizing platforms for the AI-driven future is essential. The Aiven platform empowers businesses to thrive in this ever-evolving digital landscape. In this Data (R)evolution episode, Aiven’s Bob Krygowski and Stu Garrow explain why trust is the cornerstone of successful data platforms and the importance of managing multiple initiatives. They tackle the challenges of scaling a company, the role of organizational sponsorship, and the innovative "Bring Your Own Cloud" (BYOC) concept. Our discussion covers multi-cloud flexibility, open-source technology, and cloud portability—key elements for modernizing platforms in an AI-driven future. Key Takeaways: Trust and organization are crucial when managing multiple initiatives on a data platform, highlighting the need for a center of excellence or enablement approach. The "Bring Your Own Cloud" model offers automation and security while enabling organizations to leverage existing accounts and negotiate discounts. Open source technology on platforms like Ivan aids in achieving multi-cloud flexibility, addressing skill set gaps, and supporting cloud independence and AI initiatives. Resources: Watch the full interview on our YouTube: https://www.youtube.com/@Aiven_io Check out our website for more information: https://aiven.io/ Check out Aiven AI Database Optimizer Want to be on our mailing list? Sign up here: https://aiven.io/resources Follow us on LinkedIn: https://www.linkedin.com/company/aiven/ Sign up for our newsletter for more insights on this topic: https://aiven.io/newsletter Connect with Bob Krygowski on LinkedIn: https://www.linkedin.com/in/bobkrygowski/ Connect with Stu Garrow on LinkedIn: https://www.linkedin.com/in/stugarrow/?originalSubdomain=sg Timestamps: [03:56] Data platform enables trust and accessibility [08:35] Aiven’s platform enables, supports, and modernizes products [13:38] Security, compliance, and cost savings in multi-cloud [19:52] Utilize multiple cloud providers for optimal solutions [23:08] Customer self-service workflow simplifies cloud implementation [26:20] Abstraction simplifies the complexity of data initiatives [32:06] Open source solves multi-cloud problems and democratizes skills [33:47] Open source enables developer skillset progression [37:48] New initiatives will require data for success…

Data (R)evolution

1
Hot and cold data with Apache Kafka, Tiered Storage, and Iceberg 48:58

46 weeks ago48:58

48:58

Utilizing the true potential of data streaming is key to business success. In this Data (R)evolution episode, we're joined by Josep Prat and Filip Yonov to dive into the transformative features of Apache Kafka and its evolving role in data architecture. They discuss the critical importance of collaboration and feedback in enhancing Kafka's capabilities, the future of "lake house" technology, exciting updates from the Open Source Program Office (OSPO), and the importance of Kafka's readiness to support evolving data formats—making it a backbone for modern data ecosystems. Key Takeaways: Community collaboration and contribution are essential for the continuous improvement and testing of Apache Kafka's capabilities The evolution of Apache Kafka into a more versatile platform, combined with object storage and open table formats, can significantly enhance real-time data streaming, analytics, and the future of "lake house" technology Tiered storage in Kafka facilitates more efficient and cost-effective data management by decoupling storage from computing Resources: Watch the full interview on our YouTube: https://www.youtube.com/@Aiven_io Check out our website for more information: https://aiven.io/ Check out Aiven AI Database Optimizer Want to be on our mailing list? Sign up here: https://aiven.io/resources Follow us on LinkedIn: https://www.linkedin.com/company/aiven/ Sign up for our newsletter for more insights on this topic: https://aiven.io/newsletter Connect with Filip Yonov on LinkedIn: https://www.linkedin.com/in/filipyonov/ Connect with Josep Prat on LinkedIn: https://www.linkedin.com/in/jlprat/ Timestamps: [05:49] Kafka servers have theoretical storage limits [09:29] Test storage proposal process for Apache Kafka [17:38] LinkedIn conducted an experiment merging Xcode versions [22:11] Data lake evolving into lake house architectures [25:00] Broker pushes data to remote storage, plugin handles retrieval and format translation [26:40] Kafka excels at high-speed, high-volume data [32:18] Kafka data consumption evolving with new options [40:19] Managing metadata for conversion on community level [47:45] Kafka's potential as a widely used API…

Data (R)evolution

1
Empower Professionals to Perform Better with AI 37:13

47 weeks ago37:13

37:13

Adopting cloud technology to gain velocity and agility in the marketplace allows companies to focus on what matters most. However, this transition isn't without its risks. In this episode of Data (R)evolution, hosts Janki Patel Westenberg and Francesco Tisiot are joined by Dave Li, previous Senior Software Engineer at Helthjem, and Oded Valin, Co-Founder & CEO at EverSQL, to discuss the application of AI in database management, the importance of expertise in cloud adoption, and the need for streamlined tools in an increasingly data-driven world. Key Takeaways: Balance expert partnerships with internal knowledge to tackle scalability and performance efficiently. Trust AI recommendations to improve efficiency but maintain user control to avoid unexpected costs. Balance expert partnerships with internal knowledge to tackle scalability and performance efficiently. Resources: Watch the full interview on our YouTube: https://www.youtube.com/@Aiven_io Check out our website for more information: https://aiven.io/ Want to be on our mailing list? Sign up here: https://aiven.io/resources Follow us on LinkedIn: https://www.linkedin.com/company/aiven/ Sign up for our newsletter for more insights on this topic: https://aiven.io/newsletter Connect with Oded on LinkedIn: https://www.linkedin.com/in/valin/?originalSubdomain=il Connect with Dave on LinkedIn: https://www.linkedin.com/in/daveli/?originalSubdomain=no Visit Helthjem: https://www.helthjem.no/ Timestamps: [03:36] Why cloud adoption is beneficial for smaller companies [07:45] The hidden costs and expertise required for Cloud [17:57] Why human involvement will always be necessary in the manual parcel delivery process [28:32] Automate SQL optimization for inexperienced database users [34:34] Janki and Francesco’s top takeaways…

Data (R)evolution

1
Effective Strategies for Managing Security Incidents and Compliance 46:47

48 weeks ago46:47

46:47

In a world that's increasingly dependent on digital ecosystems, how do you ensure that your platform stands up to the demands of security and compliance? In this episode of Data (R)evolution, we are joined by Jamie Arlen, Aiven's CISO, and Chris Hill, CISO at Avaya. With rising demands for transparency and data security, Jamie and Chris discuss maintaining compliance with standards like Fedramp, PCI, GDPR, and HIPAA, while emphasizing the need for trust within teams and partnerships, proactive preparation for security incidents, and the continuous effort required for maintaining compliance and security. Key Takeaways: Building and maintaining compliance requires continuous effort and thorough documentation, acting as the foundation of trust between businesses and their customers. Proactive preparation for worst-case scenarios, supported by a strong and skilled team, is crucial for maintaining security and reducing the impact of potential incidents. Understanding the distinctions between reliability and resiliency allows organizations to better manage and respond to both steady operations and unexpected challenges. Resources: Watch the full interview on our YouTube: https://www.youtube.com/@Aiven_io Check out our website for more information: https://aiven.io/ Want to be on our mailing list? Sign up here: https://aiven.io/resources Follow us on LinkedIn: https://www.linkedin.com/company/aiven/ Sign up for our newsletter for more insights on this topic: https://aiven.io/newsletter Connect with Jamie on LinkedIn: https://www.linkedin.com/in/jamesarlen/?originalSubdomain=ca Visit Avaya: https://www.avaya.com/en/ Timestamps: [08:21] Auditors need receipts for provable compliance [12:04] Create a detective team that prevents and responds to threats [13:08] Teamwork allows you to progress rapidly in a short period of time [18:15] Prepare for bad days and maintain compliance readiness [20:30] The difference between resilience and reliability [23:21] Use observability platforms to monitor threats [30:44] Practical information sharing for common understanding [33:21] Adapting operations for US government compliance expectations [42:48] Janki and Francesco’s top takeaways…

Data (R)evolution

1
What's Happening in the Caching World: Redis, Valkey and DragonFly 16:34

49 weeks ago16:34

16:34

In the world of software development, caching is a crucial yet complex component. What happens when one of the most popular caching systems, Redis, changes its license, sending shockwaves through the community? And how does a company pivot to not one but two solutions to ensure high-performance data management? Today's discussion dives deep into one of the most critical and often-overlooked components of the software industry—caching. Our hosts sit down with Josep Prat, the Director of the Open Source Program Office at Aiven, and John Kennedy, the Head of Databases, to explore the complexities of caching, the recent Redis license change, and their pioneering work with Dragonfly and the Valkey project. Key Takeaways: Caching must be planned early in infrastructure design to avoid costly performance bottlenecks Redis' license change has pushed companies like Aiven to explore alternatives, such as the Valkey fork Aiven's collaboration with Dragonfly offers high-scale, enterprise-grade caching solutions compatible with Redis, enabling seamless user migration Resources: Watch the full interview on our YouTube: https://www.youtube.com/@Aiven_io Check out our website for more information: https://aiven.io/ Check out Aiven AI Database Optimizer Want to be on our mailing list? Sign up here: https://aiven.io/resources Follow us on LinkedIn: https://www.linkedin.com/company/aiven/ Sign up for our newsletter for more insights on this topic: https://aiven.io/newsletter Connect with Josep on LinkedIn: https://www.linkedin.com/in/jlprat/?originalSubdomain=de Connect with John on LinkedIn: https://www.linkedin.com/in/john-kennedy-5216b112/?originalSubdomain=uk Timestamps: [01:00] Caching is a complex problem in IT [04:38] Redis license changed, causing repercussions for users [07:40] Subset of users not allowed to use Redis, seeking Valkey alternative under Linux Foundation [12:00] Dragonfly accelerates evolving projects and offers a smooth transition [14:31] Josep discusses Valkey Fork, Linux Foundation, Redis, and Dragonfly…

Data (R)evolution

1
Building Smart Data Platforms with AI to Drive Business Innovation 22:00

50 weeks ago22:00

22:00

Imagine a world where databases not only store vast amounts of data but autonomously optimize themselves, predict future trends, and seamlessly integrate with AI. In this Data (R)evolution episode, we share a fascinating conversation with Soumya Bijjal, the VP of Product at Aiven, and John Joseph Kennedy, the Head of Databases at Aiven, about the future of AI, databases, and data platforms. They dissect the current AI landscape, the challenges companies face, and the transformative potential of integrating AI with data systems. Key Takeaways: Focus on advanced data processing and streaming technologies like Kafka to maintain data accuracy and timeliness. Tools like Aiven AI Database Optimizer provide automated recommendations for optimizing queries and indexes, which helps accelerate performance and reduce cloud costs without requiring extensive DBA expertise. Companies should experiment with new technologies to stay ahead while relying on tried-and-tested solutions for critical production workloads. Resources: Watch the full interview on our YouTube: https://www.youtube.com/@Aiven_io Check out our website for more information: https://aiven.io/ Check out Aiven AI Database Optimizer Want to be on our mailing list? Sign up here: https://aiven.io/resources Follow us on LinkedIn: https://www.linkedin.com/company/aiven/ Sign up for our newsletter for more insights on this topic: https://aiven.io/newsletter Connect with Soumya on LinkedIn: https://www.linkedin.com/in/soumyabijjal/?originalSubdomain=fi Connect with John on LinkedIn: https://www.linkedin.com/in/john-kennedy-5216b112/?originalSubdomain=uk Timestamps: [01:28] Soumya’s feelings towards AI [06:34] Stream processing tech offers trial and error [08:37] Data quality, trust, and AI optimization impact [12:21] Balancing new technologies while ensuring production safety [14:49] The EverSQL acquisition [18:17] Janki and Francesco’s top takeaways…

Data (R)evolution

1
Welcome to Data (R)evolution: Shaping the Future of Technology 1:54

1 year ago1:54