AI Evaluations Crash Course In 50 Minutes (2025) | Hamel Husain Behind The Craft podcast

Content provided by Peter Yang. All podcast content including episodes, graphics, and podcast descriptions are uploaded and provided directly by Peter Yang or their podcast platform partner. If you believe someone is using your copyrighted work without your permission, you can follow the process outlined here https://ppacc.player.fm/legal.

Behind the Craft »
AI Evaluations Crash Course in 50 Minutes (2025) | Hamel Husain

22h ago 52:29

MP3•Episode home

Today, I want to share a new episode with Hamel Husain.

Hamel has trained 2,000+ PMs and engineers from companies like OpenAI, Anthropic, and Google on how to run AI evals. In my new episode, he shares a free master class on how to build evals for a real AI agent in just 50 minutes using a simple spreadsheet. I learned a lot from Hamel and I think you will too.

Hamel and I talked about:

(00:00) What the most valuable part of evals is

(01:25) Live walkthrough: Analyzing 100 real production traces

(09:50) Creating the eval criteria using a simple spreadsheet

(24:44) Why binary pass/fail ratings beat 1-5 scores every time

(28:52) The agreement metric trap that fools most PMs

(30:08) True positive and negative rates explained

(36:00) How to set up continuous evals in production

Get the takeaways: https://creatoreconomy.so/p/ai-evaluations-crash-course-in-50-minutes-hamel-husain

Where to find Hamel:

X: https://x.com/HamelHusain

Website: https://hamel.dev/

📌 Subscribe to this channel – more interviews coming soon!

76 episodes

#Tech #Peter Yang