20 subscribers
Go offline with the Player FM app!
Podcasts Worth a Listen
SPONSORED


1 81: From Nothing to a Billion: The Leadership Playbook They Don’t Teach You w/ Harry L Allen 36:51
Lessons in Data Engineering: Scaling, AI, and Open Source with Sandy Ryza
Manage episode 465428005 series 2816966
In this episode of Product by Design, Kyle chats with Sandy Ryza, lead engineer on the Dagster project, author, and thought leader in data engineering. Sandy shares his journey through the world of data—from building big data tools at Cloudera to working as a data scientist, product manager, and engineer—and how those experiences led him to help create Dagster, an open-source data orchestration platform.
We discuss:
- The evolution of data engineering and the growing complexity of modern data pipelines.
- The role of AI and unstructured data in shaping the future of data platforms.
- How organizations should think about data platforms to avoid costly rework.
- Best practices for managing data complexity using software engineering principles.
- The future of open-source tools in data infrastructure and the push toward interoperability.
Sandy Ryza
Sandy is a lead engineer, author, and thought leader in the domain of data engineering. Sandy co-wrote “Advanced Analytics with PySpark” and “Advanced Analytics with Spark”. He led ML and data science teams at Cloudera, Remix, Clover Health, and KeepTruckin.
Sandy is currently the lead engineer on the Dagster project, an open-source data orchestration platform used in MLOps, data science, IOT and analytics. Sandy is a regular speaker at data engineering and ML conferences.
Links from the Show:
Twitter: @s_RYZ
Dagster: dagster.io
Book: Advanced Analytics with Spark – O'Reilly
Podcast Recommendation: Empire (British Empire & Ottoman Empire history)
Books Sandy is Reading: The Shortest History of India, The Sun Also Rises, Werner Herzog’s Autobiography
More by Kyle:
Follow Prodity on Twitter and TikTok
Follow Kyle on Twitter and TikTok
Sign up for the Prodity Newsletter for more updates.
Prodity on Medium
Like our podcast, consider Buying Us a Coffee or supporting us on Patreon
114 episodes
Manage episode 465428005 series 2816966
In this episode of Product by Design, Kyle chats with Sandy Ryza, lead engineer on the Dagster project, author, and thought leader in data engineering. Sandy shares his journey through the world of data—from building big data tools at Cloudera to working as a data scientist, product manager, and engineer—and how those experiences led him to help create Dagster, an open-source data orchestration platform.
We discuss:
- The evolution of data engineering and the growing complexity of modern data pipelines.
- The role of AI and unstructured data in shaping the future of data platforms.
- How organizations should think about data platforms to avoid costly rework.
- Best practices for managing data complexity using software engineering principles.
- The future of open-source tools in data infrastructure and the push toward interoperability.
Sandy Ryza
Sandy is a lead engineer, author, and thought leader in the domain of data engineering. Sandy co-wrote “Advanced Analytics with PySpark” and “Advanced Analytics with Spark”. He led ML and data science teams at Cloudera, Remix, Clover Health, and KeepTruckin.
Sandy is currently the lead engineer on the Dagster project, an open-source data orchestration platform used in MLOps, data science, IOT and analytics. Sandy is a regular speaker at data engineering and ML conferences.
Links from the Show:
Twitter: @s_RYZ
Dagster: dagster.io
Book: Advanced Analytics with Spark – O'Reilly
Podcast Recommendation: Empire (British Empire & Ottoman Empire history)
Books Sandy is Reading: The Shortest History of India, The Sun Also Rises, Werner Herzog’s Autobiography
More by Kyle:
Follow Prodity on Twitter and TikTok
Follow Kyle on Twitter and TikTok
Sign up for the Prodity Newsletter for more updates.
Prodity on Medium
Like our podcast, consider Buying Us a Coffee or supporting us on Patreon
114 episodes
All episodes
×
1 Engineering, Entrepreneurship, and Evolving Teams - with Jason Monberg 1:10:22

1 Structured Thinking for Unstructured Data: A Conversation with Founder Kirk Marple 44:29

1 Revolutionizing Investment Research with AI 1:04:18

1 Lessons in Data Engineering: Scaling, AI, and Open Source with Sandy Ryza 46:28

1 Redefining the Digital Landscape: JB Benjamin on Privacy, Blockchain, and the Future of Technology 1:03:42

1 Scaling Authorization with Jake Moshenko: Lessons for Product Leaders 39:40

1 Character Over Skills: A New Framework for Growth and Leadership with David CM Carter 45:35

1 The Power of Visual Thinking in Product Management with Christoph Steinlehner 48:32

1 Personalized Nutrition: How AI, Data, and 3D Printing Are Transforming Health with Ari Tulla of Elo 54:38

1 The Intersection of Authenticity and Innovation: UX Insights from Oak Theory Founders Hannah Ryu and Veronica Shelton 46:29

1 Cybersecurity, Identity, and Building Trust in the Age of AI: Insights from CEO Aaron Painter 50:42

1 The Subterranean Drone Revolution: Transforming Grain Storage with Dr. Lorenzo Conti 48:12

1 Stop Making Products that Suck - A Conversation with Debbie Levitt 1:31:20

1 Mastering Product Development and Team Dynamics - With CEO Ben Johnson 52:37

1 Episode 100 Celebration! Five Lessons from the Last Five(ish) Years 30:51

1 Bridging the Human-Technology Gap with AI Avatars - A Conversation with Founder Chuck Rinker 1:01:15

1 Navigating Tech Innovations and Strategic Partnerships with Varag Gharibjanian 1:00:36

1 The Rise of Video and the Future of Virtual Communication 48:54

1 Navigating the AI Revolution in Video Creation: A Deep Dive with Vikram Chalana 58:02

1 Extending Human Potential: Creating Personal AIs with Founder Suman Kanuganti 57:08

1 Revolutionizing Product Management with Ryan Glasgow, Founder and CEO of Sprig 1:00:01

1 Engineering the Future of Product Management: A Deep Dive with Hubert Palan 53:00

1 Fostering Product Excellence and Customer-Centricity with Vidya Dinamani 59:20

1 Amplifying Your Intelligence Through AI - With Kevin Surace 1:14:33

1 Creating a Brand - Content, Empathy, and Communication 46:16


1 Product Consulting, Driving Outcomes, and Managing a Product Career 54:51

1 Mobile Gaming and the Future of AR, VR, and the Metaverse 1:07:10

1 Design by Constraints - Creating Holistic Experiences and Authentic Brands 48:38

1 Synthetic Agents and Democratizing AI - A Conversation with Ahmed Reza, CEO and Founder of Yobi 1:07:37
Welcome to Player FM!
Player FM is scanning the web for high-quality podcasts for you to enjoy right now. It's the best podcast app and works on Android, iPhone, and the web. Signup to sync subscriptions across devices.