The Ledger, every show

“I think it's the future for the short term as the knowledge of how to use AI gets disseminated throughout industry. We have basically a concentration of a bunch of people in San Francisco who really know how to deploy agents, eval agents, Be AI first in engine…”

“It doesn't annoy me, no, because the, you know, we end every week with millions more in the bank, right? So it's funny how you can have, I've been at other Companies where I've seen, you know, interesting financial engineering and accounting, and people can ha…”

“I wouldn't say that open source model improvements cannibalize our core business because data is most valuable on the frontier of model performance, so each of our customers has their own unique goals and is purchasing eval and training data sets To fill gaps …”

“I buy it. I think it's also self-serving towards Mercore, in that we think that every specialized model will need enterprise specific eval and training data to show the model how it's performing in its setting, and I think it depends on I think the diversity a…”

“So the trend that we see is we're, as a product team, constantly fighting to reduce surface area and simplify things, and we also see a higher ratio of PMs to ENG because engineering is less bottlenecked.”

“I can do things very quickly now, you know, it's like there's, ah, skill issues have almost gone away. So now it's all about, ah, judgment, and am I doing what is going to drive the most business value?”

“We've biased towards more senior hires. Who are better at understanding what drives the business forward, finding, kind of, like, really crocking how we operate, how we make more revenue, how we deliver better services to our customers, how we keep our custome…”

“I want very careful never to delegate judgment or decision making to models because it's, they make you think that it's doing the right thing, but You have to be paranoid with them still, right? You still have to, like, double check everything, and that's what…”

“This is just, like, VC-subsidized work that labs love. The problem is scaling it beyond a few data points or what one founder or full-time employees can do, and this is the position that we're in, is we're having to compete against Basically founder-led annota…”

“There's never going to be that 90% for security because the goal posts are always going to move.”

“Yeah, I think so, but I think it might play out similar to driverless cars, where it's really hard to scale physical things as opposed to software, so it might be more of a, more of like a Waymo robo-taxi-cruise type moment than a ChatGPT moment, but I think t…”

“I can answer this from kind of like a, how it affects the product team. We would love to move like our biggest challenge is moving down market so that every single enterprise can efficiently run human data projects for eval and training. And that'll diversify …”

“I hope that we can move towards a future of better accounting of the outcomes being driven by token spend. Because even here, I think in a company like Salesforce we have so many, a company of that size, certainly you're getting, you should have different spen…”

“We can't spend money fast enough to service all of the demand that we have.”

“Get a real internship as soon as possible because whatever you learn in school is probably going to be outdated quickly.”

“I'm not convinced that 90% of enterprise workflows can be handled by open models or Frontier models right now.”

“I don't think there's an ROI problem right now. I think we're in a period of exploration and experimentation where there's more tolerance, more patience To get that ROI calculation right now.”

“To be honest, I let the team do whatever is best for them, and this is a trend I've just observed amongst almost everybody, is that cloud design has done a great job. People really like using it, it's easy to use, and we've just had a natural movement towards …”

“Evals and training data are the primary bottleneck to model performance right now.”

“Probably the same answer as before in that, like, the three years out opportunity of robotics.”

“To an extent, we're already in this world. It's not that successful though for the small players always. So how I would describe it is we're facing what looks like a cottage industry of founders doing annotation themselves, right?”

“Well, in our Apex benchmarks, we're getting closer to around 50% of Long Horizon workflows. Top models are scoring around around that much.”

“I think macro, the percentage will increase over time to more than three percent.”

“We made a tool that's maximally flexible, has all sorts of, we had, like, hundreds of different projects running on it. That's just chaos to manage, and what we needed to do sooner was to put guardrails on the type of services that we support, and work closer …”

“I think that it's a knowledge dissemination problem, so I think the, that, that's one way to look at it. The other way is, why not hire someone to just do this agent deployment at your own company? And I just don't think the skill is out there yet. I don't thi…”

“I'm proud that of the people I work closest with on my teams, I've only had attrition to founding. And we've had quite a bit of it.”

“We've moved away from take-home assignments. We do one take-home assignment, which is, like, can you just, like, use an agent to go, you're on your own for a bit of time, go use an agent, you know, give, produce this artifact for me, and we'll look at it.”

“As the headcounts increased, you know, like more than 10 X in the last year, we just want to be careful not to grow one faster than the other.”

“It's the direction we have been heading, which has reduced concentration, and it's the direction that we'll continue to head as every enterprise begins to have human data work for their proprietary use cases.”

“I think that real world, like physical data is going to grow significantly over the next three years.”

“The data types also change very frequently. So we're, we've moved from supervised fine tuning to preference ranking to Rubric based annotation to now RL environments across a whole bunch of different modalities.”

“The data type that's growing the fastest for us is environments.”

“We see very Rapidly increasing demand for cyber defensive capabilities via data, and very interesting data types”