benchmarks

“As flawed as these benchmarks are in text, they're way worse in audio.”

“And I think the work around over, you know, overfitting on the test, I think is like that. 100% is happening.”

“Let's just reproduce, you know, 500 to a thousand examples for every single one of these data sets ourselves and just make sure that this data is definitely not in the, you know, the training set. So we did that and then we're able to like now be confident abo…”

“The argument that I tried to lay out in this post is that more people should make benchmarks that are tailored to them.”

“The labs that are not that frontier will keep measuring themselves on last year's benchmarks. And then the labs that are actually frontier will tell you about benchmarks you've never heard of.”

“I think one of the problems is, like, the benchmarks, they always report, like, the best event. So they report, like, you know, what's the best metric they got out of 64 attempts? But in healthcare, we're more interested in, like, what's the worst event, right…”