vals

“And I think long-term that will be actually the biggest bottleneck, our ability to take companies and their evals and make them legible because that's how we'll figure out what signal we hill climb on and where we actually adopt.”

“We also have this internal system called Steve. Steve the Economic Vals employee. And so that, that's been a mechanism by which we're able to actually take more of the human work over time and put it into Steve.”

“Now we built up a team, but we've also really invested heavily in infrastructure. And so we're able to run evaluations in a massively distributed way running effectively the maximum possible rate limits with every model we get access to.”

“At VALS, one very early decision we made was the decision to never sell training data to labs. It's often a place that we're pushed. When we start working with a new lab to actually source and sell for them a bunch of training data.”

“In that month we spent roughly 1.5 million dollars worth of tokens. This is free, by the way. I, no, I don't want to but it was actually 10 X more we were spending in tokens than employee salary for that month.”