Go offline with the Player FM app!
Embracing Data Mesh and SQL Sensors for Scalable Workflows at lastminute.com with Alberto Crespi
Manage episode 489814020 series 2948506
The flexibility of Airflow plays a pivotal role in enabling decentralized data architectures and empowering cross-functional teams.
In this episode, we speak with Alberto Crespi, Data Architect at lastminute.com, who shares how his team scales Airflow across 12 teams while supporting both vertical and horizontal structures under a data mesh approach.
Key Takeaways:
(02:17) Defining responsibilities within data architecture teams.
(04:15) Consolidating multiple orchestrators into a single solution.
(07:00) Scaling Airflow environments with shared infrastructure and DevOps practices.
(10:59) Managing dependencies and readiness using SQL sensors.
(14:23) Enhancing visibility and response through Slack-integrated monitoring.
(19:28) Extending Airflow’s flexibility to run legacy systems.
(22:28) Integrating transformation tools into orchestrated pipelines.
(25:54) Enabling non-engineers to contribute to pipeline development.
(27:33) Fostering adoption through collaboration and communication.
Resources Mentioned:
https://www.linkedin.com/in/crespialberto/
lastminute.com | Website
https://lastminute.com
https://airflow.apache.org/
https://www.getdbt.com/
https://github.com/astronomer/astronomer-cosmos
https://slack.com/
https://kubernetes.io/
https://www.atlassian.com/software/confluence
https://slack.com/
https://www.astronomer.io/events/roadshow/london/
https://www.astronomer.io/events/roadshow/new-york/
https://www.astronomer.io/events/roadshow/sydney/
https://www.astronomer.io/events/roadshow/san-francisco/
https://www.astronomer.io/events/roadshow/chicago/
Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.
#AI #Automation #Airflow #MachineLearning
62 episodes
Embracing Data Mesh and SQL Sensors for Scalable Workflows at lastminute.com with Alberto Crespi
The Data Flowcast: Mastering Apache Airflow ® for Data Engineering and AI
Manage episode 489814020 series 2948506
The flexibility of Airflow plays a pivotal role in enabling decentralized data architectures and empowering cross-functional teams.
In this episode, we speak with Alberto Crespi, Data Architect at lastminute.com, who shares how his team scales Airflow across 12 teams while supporting both vertical and horizontal structures under a data mesh approach.
Key Takeaways:
(02:17) Defining responsibilities within data architecture teams.
(04:15) Consolidating multiple orchestrators into a single solution.
(07:00) Scaling Airflow environments with shared infrastructure and DevOps practices.
(10:59) Managing dependencies and readiness using SQL sensors.
(14:23) Enhancing visibility and response through Slack-integrated monitoring.
(19:28) Extending Airflow’s flexibility to run legacy systems.
(22:28) Integrating transformation tools into orchestrated pipelines.
(25:54) Enabling non-engineers to contribute to pipeline development.
(27:33) Fostering adoption through collaboration and communication.
Resources Mentioned:
https://www.linkedin.com/in/crespialberto/
lastminute.com | Website
https://lastminute.com
https://airflow.apache.org/
https://www.getdbt.com/
https://github.com/astronomer/astronomer-cosmos
https://slack.com/
https://kubernetes.io/
https://www.atlassian.com/software/confluence
https://slack.com/
https://www.astronomer.io/events/roadshow/london/
https://www.astronomer.io/events/roadshow/new-york/
https://www.astronomer.io/events/roadshow/sydney/
https://www.astronomer.io/events/roadshow/san-francisco/
https://www.astronomer.io/events/roadshow/chicago/
Thanks for listening to “The Data Flowcast: Mastering Apache Airflow® for Data Engineering and AI.” If you enjoyed this episode, please leave a 5-star review to help get the word out about the show. And be sure to subscribe so you never miss any of the insightful conversations.
#AI #Automation #Airflow #MachineLearning
62 episodes
All episodes
×
1 Transforming Customer Education in Data Engineering at Astronomer with Marc Lamberti 22:19

1 Embracing Data Mesh and SQL Sensors for Scalable Workflows at lastminute.com with Alberto Crespi 30:09

1 The AI-Ready Pipeline: Reimagining Airflow at Veyer® Logistics with Anu Pabla 23:21

1 Streamlining AI and ML Operations at IBM with BJ Adesoji and Ryan Yackel 24:44

1 Inside the Custom Framework for Managing Airflow Code at Wix with Gil Reich 31:02

1 Modernizing Legacy Data Systems With Airflow at Procter & Gamble with Adonis Castillo Cordero 22:13

1 Building an End-to-End Data Observability System at Netflix with Joseph Machado 38:54

1 Why Developer Experience Shapes Data Pipeline Standards at Next Insurance with Snir Israeli 30:28

1 Data Quality and Observability at Tekmetric with Ipsa Trivedi 22:49

1 Introducing Apache Airflow® 3 with Vikram Koka and Jed Cunningham 27:28

1 Airflow in Action: Powering Instacart's Complex Ecosystem 25:14

1 From ETL to Airflow: Transforming Data Engineering at Deloitte Digital with Raviteja Tholupunoori 27:42

1 A Deep Dive Into the 2025 State of Airflow Survey Results with Tamara Fingerlin of Astronomer 23:26

1 Airflow’s Role in the Rise of DataOps with Andy Byron 26:15

1 The Software Risk That Affects Everyone and How To Address It with Michael Winser and Jarek Potiuk 28:27
Welcome to Player FM!
Player FM is scanning the web for high-quality podcasts for you to enjoy right now. It's the best podcast app and works on Android, iPhone, and the web. Signup to sync subscriptions across devices.