It's standard practice to store a fixed reference answer for every eval case. When the…
This is a dev post classified by Jev as Testing & observability (a free resource), kept by the Dev Radar because it carries real work, not commentary.
Nice work from Adobe. It's standard practice to store a fixed reference answer for every eval case. When the underlying data changes daily, that stored answer goes stale. Adobe researchers write each reference answer as a Python function instead. The function runs against the live system at evaluation time, so the expected answer follows the data, and an upstream API change makes the test fail visibly. An LLM judge then splits the agent's response and the computed answer into atomic facts and scores precision and recall, whatever the output format. Against expert labels, this raises agree
Posted by DAIR.AI (132.4k followers) 2 days ago · 84 likes · 11.5k views · view the original post on X. Kept by the Dev Radar as Testing & observability. Tools mentioned: academy.dair.ai.
More dev work like this
- What does observability look like when a judgment costs nothing? — @hugorcd
- When agent traces turn into a debugging pile, this repo gives you a place to inspect them. — @DanKornas
- I’m glad @strawgate's standards for coffee are lower than his standards for code review… — @kayvz
- Gitar finds bugs, fixes them under your team's rules, and verifies every fix by running… — @SonarSource
- OPENAI 🔥: Chrome extensions are now supported in the ChatGPT desktop browser! — @testingcatalog
- AI at scale needs visibility to match. — @splunk
- Quick tip from Visual Studio on YouTube... See the Visual Studio Debugger Agent use a… — @VisualStudio
- Agent regressions are hard to catch when traces are read-only. — @DanKornas
Every post is read and classified by Jev (TypeSafe): what it is, which market it belongs to, and whether the link is a real tool. 12.2k posts from 4.7k X accounts over the last 21 days, 1.4k tools, 12 markets. Collected every 5 minutes, fully re-ranked every hour — last update 2026-09-19 18:39 UTC. Full method.