Reproduce it, or it doesn't count: why training-side decontamination can't be verified, and what an evaluation-side rule looks like
Since OpenAI retired SWE-bench Verified in February (every frontier model tested could reproduce reference fixes for some tasks; underspecified tests rewarded knowing the intended fix), I've been...