Content provided by The Data Flowcast. All podcast content including episodes, graphics, and podcast descriptions are uploaded and provided directly by The Data Flowcast or their podcast platform partner. If you believe someone is using your copyrighted work without your permission, you can follow the process outlined here https://ppacc.player.fm/legal.
Player FM - Podcast App
Go offline with the Player FM app!
icon Daily Deals

Customizing Airflow for Complex Data Environments at Stripe with Nick Bilozerov and Sharadh Krishnamurthy

27:40
 
Share
 

Manage episode 469915069 series 2948506
Content provided by The Data Flowcast. All podcast content including episodes, graphics, and podcast descriptions are uploaded and provided directly by The Data Flowcast or their podcast platform partner. If you believe someone is using your copyrighted work without your permission, you can follow the process outlined here https://ppacc.player.fm/legal.

Keeping data pipelines reliable at scale requires more than just the right tools — it demands constant innovation. In this episode, Nick Bilozerov, Senior Data Engineer at Stripe, and Sharadh Krishnamurthy, Engineering Manager at Stripe, discuss how Stripe customizes Airflow for its needs, the evolution of its data orchestration framework and the transition to Airflow 2. They also share insights on scaling data workflows while maintaining performance, reliability and developer experience.

Key Takeaways:

(02:04) Stripe’s mission is to grow the GDP of the internet by supporting businesses with payments and data.

(05:08) 80% of Stripe engineers use data orchestration, making scalability critical.

(06:06) Airflow powers business reports, regulatory needs and ML workflows.

(08:02) Custom task frameworks improve dependencies and validation.

(08:50) "User scope mode" enables local testing without production impact.

(10:39) Migrating to Airflow 2 improves isolation, safety and scalability.

(16:40) Monolithic DAGs caused database issues, prompting a service-based shift.

(19:24) Frequent Airflow upgrades ensure stability and access to new features.

(21:38) DAG versioning and backfill improvements enhance developer experience.

(23:38) Greater UI customization would offer more flexibility.

Resources Mentioned:

Nick Bilozerov -

https://www.linkedin.com/in/nick-bilozerov/

Sharadh Krishnamurthy -

https://www.linkedin.com/in/sharadhk/

Apache Airflow -

https://airflow.apache.org/

Stripe | LinkedIn -

https://www.linkedin.com/company/stripe/

Stripe | Website -

https://stripe.com/

Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow #MachineLearning

  continue reading

56 episodes

iconShare
 
Manage episode 469915069 series 2948506
Content provided by The Data Flowcast. All podcast content including episodes, graphics, and podcast descriptions are uploaded and provided directly by The Data Flowcast or their podcast platform partner. If you believe someone is using your copyrighted work without your permission, you can follow the process outlined here https://ppacc.player.fm/legal.

Keeping data pipelines reliable at scale requires more than just the right tools — it demands constant innovation. In this episode, Nick Bilozerov, Senior Data Engineer at Stripe, and Sharadh Krishnamurthy, Engineering Manager at Stripe, discuss how Stripe customizes Airflow for its needs, the evolution of its data orchestration framework and the transition to Airflow 2. They also share insights on scaling data workflows while maintaining performance, reliability and developer experience.

Key Takeaways:

(02:04) Stripe’s mission is to grow the GDP of the internet by supporting businesses with payments and data.

(05:08) 80% of Stripe engineers use data orchestration, making scalability critical.

(06:06) Airflow powers business reports, regulatory needs and ML workflows.

(08:02) Custom task frameworks improve dependencies and validation.

(08:50) "User scope mode" enables local testing without production impact.

(10:39) Migrating to Airflow 2 improves isolation, safety and scalability.

(16:40) Monolithic DAGs caused database issues, prompting a service-based shift.

(19:24) Frequent Airflow upgrades ensure stability and access to new features.

(21:38) DAG versioning and backfill improvements enhance developer experience.

(23:38) Greater UI customization would offer more flexibility.

Resources Mentioned:

Nick Bilozerov -

https://www.linkedin.com/in/nick-bilozerov/

Sharadh Krishnamurthy -

https://www.linkedin.com/in/sharadhk/

Apache Airflow -

https://airflow.apache.org/

Stripe | LinkedIn -

https://www.linkedin.com/company/stripe/

Stripe | Website -

https://stripe.com/

Thanks for listening to “The Data Flowcast: Mastering Airflow for Data Engineering & AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.

#AI #Automation #Airflow #MachineLearning

  continue reading

56 episodes

All episodes

×
 
Loading …

Welcome to Player FM!

Player FM is scanning the web for high-quality podcasts for you to enjoy right now. It's the best podcast app and works on Android, iPhone, and the web. Signup to sync subscriptions across devices.

 

icon Daily Deals
icon Daily Deals
icon Daily Deals

Quick Reference Guide

Listen to this show while you explore
Play