code execution evals
1 statements across 1 episodes · 1 bullish · 0 bearish · 1 people on the record · first statement Apr 2, 2025 by Guy Gur-Ari · across every show →
Everything said about code execution evals, oldest first
Apr 2, 2025 positive
Ground-truth evaluations without code execution provide significant mileage for AI
“So code execution, like checking the correctness of solutions and running tests automatically can help, although not, although you can get a lot of mileage out of evals that don't have code execution in them that just compare against ground truth.”