Insight
Foody: Success measurement bottlenecks economy-wide AI automation
“And so in many ways, the barrier to applying agents to the entire economy To automate every workflow is how do we measure success? How do we eval it and write the PRDs for everything that we want agents to do, which Mercore is obviously a huge part of doing.”
Assertion Not checkable as stated
Mercor scaled from $1M to $400M revenue run rate in 16 months
“We grew from one to four hundred million in revenue run rate in 16 months, and it's been an extraordinary journey and super exciting.”
Prediction Not checkable as stated
Foody: Human AI eval work will last as long as human advantages
“And I think that that road to improving models will last for as long as there is anything in the economy that humans can do, which models can't, and be a huge portion of what the future of work looks like.”
Prediction Not checkable as stated
Foody: The entire economy will likely become an RL environment machine
“It speaks to conversations I've had with a lot of researchers and executives at top labs, which is that it's highly likely that the entire economy will become an RL environment machine.”
Assertion Not checkable as stated
Foody: Some Fortune 500s fear evaluating AI automation in their businesses
“There are certain enterprises we talk to that are almost like fearful, not wanting to engage, not wanting to, you know, eval their businesses because that'll provide the evidence that their value chain is being automated. And there's others that, I mean, liter…”
Prediction Not checkable as stated
Foody: AI superintelligence is more than three years away
“Like, I don't think it's, I know there's been some executives at big labs that say we'll have super intelligence in three years, but I think the truth is that it's a longer road.”
Prediction Not checkable as stated
Foody: AI will automate majority of knowledge work tasks in 10 years
“Like, I think we'll be able to automate a majority of knowledge work tasks in, in the next 10 years for sure.”
Assertion Supported
Foody: AI labs are broadly shifting from RLHF to RLAIF
“What everyone is generally moving towards is reinforcement learning from AI feedback instead of human feedback, where you have instead the human defined some sort of success criteria, some way to measure that... And it's far more scalable and data efficient, a…”
Prediction Not checkable as stated
Foody: Software automation will create a unified global labor market within 10 years
“When we're able to Automate that matching problem at the cost of software. It makes way for this global unified labor market that every candidate applies to and every company hires from facilitating a perfect flow of information in the economy. And I think tha…”
Opinion
Foody: Traditional expert networks struggle to meet AI post-training demands
“One core difference is that alpha sites would generally be a one-off call versus a lot of our work is really hiring people for projects, right? Of how do they work on something for a longer period of time? And so that That's, I think, one of the reasons that s…”
Insight
Foody: Top 10% of human evaluators drive majority of model improvement
“And there's also this really interesting dynamic where in a set of a hundred people that we hire, oftentimes the top 10% of people will drive majority of the model improvement.”
Assertion Not checkable as stated
Mercor had no sales or marketing personnel for its first 18 months
“For the last year and a half of the business, we've had no one in sales and marketing.”
Assertion Not checkable as stated
Foody: Mercor is lifetime profitable and has never burned net capital
“We bootstrapped the company to a million dollar revenue run rate and have always remained super capital efficient. Like we've never burned money. We were lifetime profitable.”
Insight
Foody: Efficient post-training datasets, not 10x pre-training, drive AI progress
“And it's not going to be, you know, 10 X more pre-training data that gets those capabilities. It's much more going to be all of the post-training data sets that are far more data efficient and thoughtful that help us get there.”
Insight
Foody: AI evals are the product requirement documents for models
“If the model is the product, then the eval is the product requirement document.”
Prediction Not checkable as stated
Foody: AI labs and apps will use evals as sales collateral
“I think labs will increasingly use labs as well as application layer companies will increasingly use evals to demonstrate the capabilities of their models and their products.”
Insight
Foody: Human AI data market is bound by human-model capability gap
“Effectively, the market is bound by the amount of things where humans can do something that models can't.”
Insight
Foody: AI evals and RL environments share the exact same data type
“There's not actually a nuance in the data type. It's more just a different semantic way of what describing what it's being used for. But ultimately it's just some stasis point for like, how do you measure what good looks like?”
Insight
Foody: Software development has virtually unlimited elastic demand
“Like in accounting, I think realistically we only need so much accounting in the world, right? Like maybe there's areas where we can do more and that'll be good. But it doesn't feel like the world needs a hundred times more accounting. On the other hand, in so…”
Assertion Not checkable as stated
Foody: Tens of thousands work on AI post-training at any given time
“Tens of thousands at any given time, hundreds of thousands more generally. I mean, it's huge. And the most exciting thing is that it's growing really quickly.”
Disclosure
Mercor hires Harvard Lampoon staff and Emmy winners to train AI
“Like we hired all the people from the Harvard Lampoon a couple of months ago, their comedy club to help with making models funnier. And so do all sorts of stuff like that, hiring Emmy award-winning screenwriters and everything across the board on creative capa…”
Disclosure
Mercor's median AI trainer pay is $95/hr, reaching up to $500/hr
“I mean, so our median pay rate in the marketplace is 95 dollars an hour, but it can flex up well up into like 500 dollars an hour based on the depth of someone's expertise.”
Assertion Not checkable as stated
Foody: Traditional AI crowdsourcing platforms typically pay around $30 per hour
“If you look at the economics of the crowdsourcing companies, oftentimes they would pay like 30 dollars an hour to town as sort of the average.”
Assertion Not checkable as stated
Mercor hit its $50M run rate forecast two weeks after pitching
“Where I remember when we were talking to Benchmark before they led our series A, we were at 1.5 million in run rate. And I said we'd be at fifty million in run rate by the end of the year. And they said we were absolutely insane, right? As anyone, anyone would…”