))
20250702 - How to pass an AI coding benchmark: train on the questions
Manage episode 492185026 series 3662020
Content provided by David Gerard. All podcast content including episodes, graphics, and podcast descriptions are uploaded and provided directly by David Gerard or their podcast platform partner. If you believe someone is using your copyrighted work without your permission, you can follow the process outlined here https://ppacc.player.fm/legal.
AI models pass SWE-Bench from memory Text version: https://pivot-to-ai.com/2025/07/02/how-to-pass-an-ai-coding-benchmark-train-on-the-questions/
Patreon: https://www.patreon.com/davidgerard Ko-Fi: https://ko-fi.com/A1529D5 Buy me nice things: https://www.amazon.co.uk/hz/wishlist/ls/3Q8VZW46J6DM6 Get an extremely cool Pivot to AI shirt or mug: https://pivot-to-ai.redbubble.com
Source:
The SWE-Bench Illusion: When State-of-the-Art LLMs Remember Instead of Reason https://arxiv.org/abs/2506.12286
Previously on Pivot to AI:
OpenAI o3 beats FrontierMath — because OpenAI funded the test and had access to the questions https://pivot-to-ai.com/2025/01/20/openai-o3-beats-frontiermath-because-openai-funded-the-test-and-had-access-to-questions/ AI benchmarks are self-promoting trash — but regulators keep using them https://pivot-to-ai.com/2025/02/25/ai-benchmarks-are-self-promoting-trash-but-regulators-keep-using-them/ Apple: ‘Reasoning’ AIs fail hard if they actually have to think https://pivot-to-ai.com/2025/06/08/apple-reasoning-ais-fail-hard-if-they-actually-have-to-think/ video: https://www.youtube.com/watch?v=gSx9pI5so30&list=UU9rJrMVgcXTfa8xuMnbhAEA
Full Pivot to AI playlist: https://www.youtube.com/playlist?list=UU9rJrMVgcXTfa8xuMnbhAEA
110 episodes
Share
