Capital de TokensThe judge got it right where a small test failed
Watch on YouTube Another signal emerged in TerminalBench. The judge’s preferred candidate solved eighteen out of fifty tasks during the search; another solved nineteen. When the full evaluation was repeated, the judge’s favorite averaged 36.7 percent and the other, 28.1. A one-task difference in a small sample had pointed
Another signal emerged in TerminalBench. The judge’s preferred candidate solved eighteen out of fifty tasks during the search; another solved nineteen. When the full evaluation was repeated, the judge’s favorite averaged 36.7 percent and the other, 28.1. A one-task difference in a small sample had pointed to the wrong candidate. The researchers observed problems in the second candidate’s code: a verifier disabled by default and a terminal tool at risk of failing. The judge could detect those clues without running the entire test.
Full episode: https://youtu.be/aJ5Qe2Sup5I
🤖 AI-generated content: the script, voices, and images in this episode were produced with artificial intelligence tools.
#Shorts