📌 Copy the AI system I use to run a one-person $1M+ business: https://www.behindthecraft.com
🤝 Subscribe to my practical AI newsletter read by 150K+ professionals: https://creatoreconomy.so
Shreya and Hamel teach the industry standard course for AI evaluations taken by 4,500+ students from OpenAI, Google, and more. In our interview, I asked them to give live feedback on the AI evals that I built for my skills and demo how anyone can run evals with ChatGPT/Claude and their free evals skill. Don’t miss this up to date primer on evals from the real experts. 🙂
P.S., I asked Shreya and Hamel to give my audience 25% off the next cohort of their AI evals course. It has 4.7 stars with 900 reviews which is almost unheard of on Maven. Sign up via the link here: https://maven.com/parlance-labs/evals?utm_campaign=peteryang&utm_medium=affiliate&utm_source=maven&promoCode=PETERYANG
We talked about:
(00:00) Why AI evals are completely different now with the latest models
(02:30) Demo: Live audit of the evals that I built for my AI skills
(05:14) Top down vs. bottom up evals and why AI sucks at the latter
(08:16) Spinning up AI agents to grade each eval criteria
(14:32) Demo: Using Shreyas AI skill to run evals in Claude
(23:46) How to review output labels to identify failures
(34:37) How to turn failures into reusable evals
(43:19) Where human judgement is still needed
Resources mentioned:
Free eval skills from Shreya: https://github.com/ai-evals-course/evals-skills
Do automated evals work?: https://parlance-labs.com/blog/posts/auto-evals/
Thanks to our sponsors:
Wispr Flow — 4x faster than typing with your voice — https://ref.wisprflow.ai/peteryang
📌 Get the takeaways: https://creatoreconomy.so/p/how-to-build-better-ai-evals-with-claude-code-shreya-hamel
Where to find Shreya and Hamel:
Shreya: https://www.sh-reya.com/
Hamel: https://hamel.dev/
Subscribe to this channel - more interviews coming soon!
