Bad benchmarks and evals: Senior SWE-Bench, napkin math, and winter tires
发生了什么
We're going to look at three different kinds of benchmarks, one set of calculations for baseline numbers for performance "napkin math" estimates, one set of AI model evals, and one on car tires. To build my intuition for things, I like thinking about them before seeing the explanation, so these are presented with the benchmark information first and the explanation later in case you want to think about your answer be…
摘要按规则整理自下方来源原文