The Exchanges

Q our world to focus on big data as if it's just numbers and not other forms of data because you're really describing, I mean, what you describe your work as doing is applying machine learning to text and natural language. So how does that How did, how did you kind of, how does that work? And then we can talk a little bit more about, um, how you got there.

A Yeah. So how does it work? I mean, language is just an encoding of concepts, right? And anything that can be encoded can be measured. And so I, I was, uh, sharing the story the other day. We were actually originally started out looking at Kickstarter projects, right? So we started out with this question, could we just look at the text, um, Of a Kickstarter project and some of its, you know, metadata around the text and predict, you know, before it was ever published whether it was going to raise money. And we didn't look at the quality of the idea. We didn't look at whether a celebrity endorsed it. Turns out, ah, we got over 90% predictive on minute zero of a project as to whether it was going to hit its fundraising goal based solely on things like how long is the text and what kind of fonts are you using and how many headings do you have.

AI assessment note: “language is just an encoding of concepts, right? And anything that can be encoded can be measured.”

Q our world to focus on big data as if it's just numbers and not other forms of data because you're really describing, I mean, what you describe your work as doing is applying machine learning to text and natural language. So how does that How did, how did you kind of, how does that work? And then we can talk a little bit more about, um, how you got there.

A Yeah. So how does it work? I mean, language is just an encoding of concepts, right? And anything that can be encoded can be measured. And so I, I was, uh, sharing the story the other day. We were actually originally started out looking at Kickstarter projects, right? So we started out with this question, could we just look at the text, um, Of a Kickstarter project and some of its, you know, metadata around the text and predict, you know, before it was ever published whether it was going to raise money. And we didn't look at the quality of the idea. We didn't look at whether a celebrity endorsed it. Turns out, ah, we got over 90% predictive on minute zero of a project as to whether it was going to hit its fundraising goal based solely on things like how long is the text and what kind of fonts are you using and how many headings do you have.

AI assessment note: “language is just an encoding of concepts, right? And anything that can be encoded can be measured.”

Q The, the techniques you're describing, is it the same underlying technique applied to all different domains, but do you have to also train each Corpus on a different, um, domain? Like there's special, like there's inside language in each industry, or are they, are there also universals across all of them?

A Um, that's a really good question. Uh, you don't know until you train is the short answer to the question. So, uh, we have a set of NLP libraries that look for common attributes of text, and we always start out any new vertical by turning them on the, on the documents and seeing what happens. So things like sentence length, Almost always interesting. Things like the density of verbs and adjectives, almost always interesting. Document length, almost always interesting. But the specific phrases that matter and what it means to write a job listing is very different than what it means to predict whether a patient is going to become ill, right? And so this, the specifics matter, um, the goals matter. So if it's a document that's intended for broad consumption, It really probably shouldn't be longer than 607 hundred words. If it's, ah, a stock prospectus where you're giving a company some information about how their stocks are likely to perform, it's gonna be pages and pages. And so, you know, the, the specific benchmarks that you're looking for often vary vertical by vertical, but the principles of the kinds of things you look for, um, are pretty similar.

AI assessment note: “benchmarks that you're looking for often vary vertical by vertical, but the principles... are pretty similar”

Q The ability to really zero in via just the text, did that surprise you?

A Um, I mean, we started off with a hypothesis that it would be that way. Uh, and that, you know, sometimes how you say things is more influential than what you're actually saying, right? And it's counterintuitive to any of us who've built products before, because you like to think you're leading with a strong vision. We weren't surprised. Um, we were curious as we started to apply the technology to some other verticals, whether it would extend, um, you know, our first Big area has really been in the area of job listings, where we've looked to see the first real product application, where we've looked at listings now from over 10,000 different companies. We've measured who's applied to which listings, and we do see the content matters. We do see some tailoring by geography. Turns out what works in New York is different than what works in San Francisco. We see a lot of tailoring by industry, so what works to hire in tech is very different than what it looks like to hire a claims adjuster, um, or someone in retail. Right, so you see some differentiation, but in all cases, you know, depending on how you're slicing and dicing the categories, that text leads, you know, we, we've looked at real estate a little bit, um, prior to launching our, our, uh, jobs application, and we've seen the same principles apply.

AI assessment note: “We weren't surprised. Um, we were curious as we started to apply the technology”

Q What are some other scenarios where you could use sort of this natural language text analysis to make more, predict interesting things?

A Uh, yeah, so people are really starting to think broadly about this. We saw, uh, a New York City-based company, uh, helping people optimize the sale of their New York City apartments recently using, uh, the right phrases. Um, we've seen people do things in healthcare that I think are really interesting. It's not a known vertical to me, but, uh, looking at the kind of notes that doctors take about a patient and predicting the patient's likelihood of having a major insurance incident over the next You know, 12 to 15 months. It's really interesting things in actuarial science. Like, I think anytime people are producing text, which, by the way, in businesses, whatever your business is, text is actually the thing you produce the most of.

AI assessment note: “helping people optimize the sale of their New York City apartments recently using, uh, the right phrases”

Q years. I mean, and in the early days, they didn't have this, this kind of corpus to train the algorithms on, on, obviously. So they had to use different kinds of techniques. Like, where does your work fit, and how do you see how it fits in the evolution of natural language? Like, how is that, how has it been, and where we are, where are we now, kind of?

A Yeah, I mean, I think, uh, In core natural language processing, empirical strategies have always been really important. So when I was a grad student years ago writing a dissertation, collecting data was just a lot more work, right? So I had to go and record people in the field, and I had to transcribe things. It feels like ancient now, actually, but I actually finished my PhD 12 years ago. It wasn't that ancient. Um, the fact that the internet has codified everything over the last 15 or 20 years, at least in, in English and most Western languages, uh, means that you have this ready set of corpora available for you. The tricky part is collecting the text and the outcomes that go with it. The outcomes are the part that's hard. Finding the content is easy.

AI assessment note: “when I was a grad student... collecting data was just a lot more work”

Q years. I mean, and in the early days, they didn't have this, this kind of corpus to train the algorithms on, on, obviously. So they had to use different kinds of techniques. Like, where does your work fit, and how do you see how it fits in the evolution of natural language? Like, how is that, how has it been, and where we are, where are we now, kind of?

A Yeah, I mean, I think, uh, In core natural language processing, empirical strategies have always been really important. So when I was a grad student years ago writing a dissertation, collecting data was just a lot more work, right? So I had to go and record people in the field, and I had to transcribe things. It feels like ancient now, actually, but I actually finished my PhD 12 years ago. It wasn't that ancient. Um, the fact that the internet has codified everything over the last 15 or 20 years, at least in, in English and most Western languages, uh, means that you have this ready set of corpora available for you. The tricky part is collecting the text and the outcomes that go with it. The outcomes are the part that's hard. Finding the content is easy.

AI assessment note: “when I was a grad student years ago... collecting data was just a lot more work”

Q That's great. Did you guys, I have to ask, did you, um, kind of put any Kickstarter projects up there yourselves just to give it a whirl?

A We were asked this a lot during our fundraising. We did look at pitch decks, by the way. One of the things I will come back to your question. Um, one of the things that's been fascinating about having the beta out there in the world is the ways people are using it. So of course they're using it for job listings, but people are using it for everything. Like just a couple days ago, I had a material science professor write to me saying, I put all my course syllabi through. And I was like, really? Like, how did that work for you? I can't imagine that that was a good result. And he's like, oh, I threw out all of the job parts. I just looked at gender bias. Wow. Because that was a component that I needed for what I was doing.

AI assessment note: “I will come back to your question. Um, one of the things that's been fascinating”