Artwork

Content provided by Demetrios. All podcast content including episodes, graphics, and podcast descriptions are uploaded and provided directly by Demetrios or their podcast platform partner. If you believe someone is using your copyrighted work without your permission, you can follow the process outlined here https://ppacc.player.fm/legal.
Player FM - Podcast App
Go offline with the Player FM app!

Designing for Forward Compatibility in Gen AI // Rohit Agarwal // #189

1:00:17
 
Share
 

Manage episode 383047456 series 3241972
Content provided by Demetrios. All podcast content including episodes, graphics, and podcast descriptions are uploaded and provided directly by Demetrios or their podcast platform partner. If you believe someone is using your copyrighted work without your permission, you can follow the process outlined here https://ppacc.player.fm/legal.

MLOps podcast #189 with Rohit Agarwal, CEO of Portkey.ai, Designing for Forward Compatibility in Gen AI. // Abstract For two whole years of working with a large LLM deployment, I always felt uncomfortable. How is my system performing? Are my users liking the outputs? Who needs help? Probabilistic systems can make this really hard to understand. In this talk, we'll discuss practical & implementable items to secure your LLM system and gain confidence while deploying to production. // Bio Rohit is the Co-founder and CEO of portkey.ai which is an FMOps stack for monitoring, model management, compliance, and more. Previously, he headed Product & AI at Pepper Content which has served ~900M generations on LLMs in production. Having seen large LLM deployments in production, he's always happy to help companies build their infra stacks on FM APIs or Open-source models. // MLOps Jobs board https://mlops.pallet.xyz/jobs // MLOps Swag/Merch https://mlops-community.myshopify.com/ // Related Links Website: https://portkey.ai ⁠ --------------- ✌️Connect With Us ✌️ ------------- Join our slack community: https://go.mlops.community/slack Follow us on Twitter: @mlopscommunity Sign up for the next meetup: https://go.mlops.community/register Catch all episodes, blogs, newsletters, and more: https://mlops.community/ Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/ Connect with Rohit on LinkedIn: https://www.linkedin.com/in/1rohitagarwal/ Timestamps: [00:00] Rohit's preferred coffee [00:15] Takeaways [03:22] Please like, share, and subscribe to our MLOps channels! [05:16] Rohit's current work [06:37] The Portkey landscape [09:13] Compute unit is no longer a Cloud resource, it's a Foundational Model [11:09] Hang-ups at high-scale models and how to combat them [15:22] Complexity of the Apps evolving [19:54] Rohit's working relationships with the agents [22:52] Fine-tuning reliability [24:38] Small language models can outperform larger ones [26:38] Market map at Portkey [34:37] AI Gateway [37:59] Worker Bee and Queen Bee [39:27] Security and Compliance [43:11] Idea of Data Mesh [45:57] Forward compatibility [49:59] Decoupling AI Gateway from the code [56:05] Hardest design decisions to make since creating Portkey [58:52] Wrap up

  continue reading

441 episodes

Artwork
iconShare
 
Manage episode 383047456 series 3241972
Content provided by Demetrios. All podcast content including episodes, graphics, and podcast descriptions are uploaded and provided directly by Demetrios or their podcast platform partner. If you believe someone is using your copyrighted work without your permission, you can follow the process outlined here https://ppacc.player.fm/legal.

MLOps podcast #189 with Rohit Agarwal, CEO of Portkey.ai, Designing for Forward Compatibility in Gen AI. // Abstract For two whole years of working with a large LLM deployment, I always felt uncomfortable. How is my system performing? Are my users liking the outputs? Who needs help? Probabilistic systems can make this really hard to understand. In this talk, we'll discuss practical & implementable items to secure your LLM system and gain confidence while deploying to production. // Bio Rohit is the Co-founder and CEO of portkey.ai which is an FMOps stack for monitoring, model management, compliance, and more. Previously, he headed Product & AI at Pepper Content which has served ~900M generations on LLMs in production. Having seen large LLM deployments in production, he's always happy to help companies build their infra stacks on FM APIs or Open-source models. // MLOps Jobs board https://mlops.pallet.xyz/jobs // MLOps Swag/Merch https://mlops-community.myshopify.com/ // Related Links Website: https://portkey.ai ⁠ --------------- ✌️Connect With Us ✌️ ------------- Join our slack community: https://go.mlops.community/slack Follow us on Twitter: @mlopscommunity Sign up for the next meetup: https://go.mlops.community/register Catch all episodes, blogs, newsletters, and more: https://mlops.community/ Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/ Connect with Rohit on LinkedIn: https://www.linkedin.com/in/1rohitagarwal/ Timestamps: [00:00] Rohit's preferred coffee [00:15] Takeaways [03:22] Please like, share, and subscribe to our MLOps channels! [05:16] Rohit's current work [06:37] The Portkey landscape [09:13] Compute unit is no longer a Cloud resource, it's a Foundational Model [11:09] Hang-ups at high-scale models and how to combat them [15:22] Complexity of the Apps evolving [19:54] Rohit's working relationships with the agents [22:52] Fine-tuning reliability [24:38] Small language models can outperform larger ones [26:38] Market map at Portkey [34:37] AI Gateway [37:59] Worker Bee and Queen Bee [39:27] Security and Compliance [43:11] Idea of Data Mesh [45:57] Forward compatibility [49:59] Decoupling AI Gateway from the code [56:05] Hardest design decisions to make since creating Portkey [58:52] Wrap up

  continue reading

441 episodes

All episodes

×
 
Loading …

Welcome to Player FM!

Player FM is scanning the web for high-quality podcasts for you to enjoy right now. It's the best podcast app and works on Android, iPhone, and the web. Signup to sync subscriptions across devices.

 

Quick Reference Guide

Copyright 2025 | Privacy Policy | Terms of Service | | Copyright
Listen to this show while you explore
Play