In discusses when not to trust LLM judges for agent evaluation.
This is a dev post classified by Jev as AI dev tools (a free resource), kept by the Dev Radar because it carries real work, not commentary.
Great paper from Amazon. In discusses when not to trust LLM judges for agent evaluation. (bookmark it) A common way to compare task agents is to have an LLM user simulator talk to each one and an LLM judge score the transcript. This paper from Amazon shows that gate fails in two specific ways. 1. Satisfaction does not track success. 57.5% of conversations the raters marked satisfied had failed the customer's task. 2. Close calls go wrong. The ranking holds across agents of very different ability, but among near-equal agents the gate picks the lower-reward one on 31% of pairs, compared wit
Posted by DAIR.AI (132.4k followers) 5 days ago · 157 likes · 12k views · view the original post on X. Kept by the Dev Radar as AI dev tools. Tools mentioned: academy.dair.ai.
More dev work like this
- Hallmark has been installed 50k+ times! — @nutlope
- Your AI coding agents need a workflow, not more babysitting. — @DanKornas
- Register by October 7 and save >> https://bit.ly/4vkntyD — @linuxfoundation
- Nimble is able to process images! — @madiator
- Your agent already solved this. Finding the session is the problem. — @DanKornas
- Trains your own large language model from scratch using plain PyTorch. — @tom_doerr
- gpt 5.6 terra = gpt 6 astra — @notjazii
- your rtx 3060 was running bonsai 2 at 26 tok/s this morning and now does 40 tok/s, i… — @sudoingX
Every post is read and classified by Jev (TypeSafe): what it is, which market it belongs to, and whether the link is a real tool. 12.2k posts from 4.7k X accounts over the last 21 days, 1.4k tools, 12 markets. Collected every 5 minutes, fully re-ranked every hour — last update 2026-09-19 18:39 UTC. Full method.