The Exchanges

Q and they'll figure out how to, Right, for iOS and stuff like that. How do you think about that in the, in the ML world? Because there is this sort of dogma right now, I feel, that people believe that only somebody who's trained in LLM before can work on an LLM-centric company or things like that. So I'm just sort of curious, you know, how you think about that.

A Yeah, actually, my thinking about this was already pretty clear because of having seen how OpenAI went about this. If you look at the people who Did GPT-III. Eventually they went on to start Anthropic. All of them are basically from physics background. The CEO Dario's a physics PhD and Jared Kaplan is a physics professor. He wrote the scaling loss paper. Uh, so they basically open AI really succeeded in bringing these extremely talented physics people who wanted into machine learning and, and they came into the sort of language models and scaling. And that paid, paid off well for them. And similarly, their whole engineering team, if you look at an engineering team, it's just Dropbox and Stripe people. All like software engineers, solid people who can build infrastructure front end, and now they're doing AI work. So it's always been clear to me that, uh, in order to work in AI, both research and product, you do not need to already have been in AI. And that's being shown clearly with our, uh, with, with Johnny Ho, our co-founder, he was not in AI. He was a competitor programmer, a trader. He had worked at Quora for a year, but, uh, he's as good as anybody can get in picking up new things. Uh, so the other thing also is that LLMs are sort of in this weird territory where the people who use the LLMs for building stuff, Understand it better than the people who actually did gradient …

AI assessment note: “in order to work in AI, both research and product, you do not need”

Q I'll ask a dangerous question, but what do you, what do you think the future of search looks like? Right? Five years plus out. Do we get, do we still have monolithic horizontal providers if the players change? Um, do we get more embedded apps like contextual search as a feature in different places? Are there agents that do things for you? Like, what do you think it looks like?

A I think there's this phrase that's becoming popular. Uh, I think Karpathy was the first one who tweeted it and Satya Nadella is also using it. It's called an answer engine instead of a search engine that directly tries to answer your question instead of providing you a bunch of links or just snippets from the first link. Uh, so we believe in that, like perplexity is the first conversational answer engine, uh, like truly the first. I think nobody built it before us. I believe answer engines will become its own segment, market segment. Like, just like you have a default search engine, there'll be a default answer engine over time if these things really work. And The burden of like getting ranking and search right will, uh, reduce in the sense answer engines can do more heavy lifting than search engines. As these things get really good and like way fewer hallucinations, um, and even, even if they do hallucinate, people can still go and click on these links. They will eventually prefer this experience over the regular like 10 links or 20 links UI that Google has. So that, that's something I'm pretty confident about. Uh, and, and I think the sort of asking follow up questions will become more of the norm. The number of queries and perplexity that go to at least one follow up has been increasing ever since we released the chat UI. So that, that will keep going up. People will get use…

AI assessment note: “It's called an answer engine instead of a search engine that directly tries to answer”

Q and before the Sydney news and before Bing integrated all these things. So you were quite early to realizing that You know, this is going to be a really core piece of search. Perplexity's mission, I believe, is to create the world's most trusted information service. Um, how do you think about important product points around factual accuracy, um, bias in presenting aggregate information for users and things like that?

A Yeah, I guess you're also from a PhD background, right? So when you write your first paper, the thing your advisors teach you is, uh, you only have to write things that you can actually cite. Anything else that you write in the paper is your opinion, not, not a scientific fact. And so that sort of stuck with us pretty closely. Um, and that's sort of why we did the first version where it's citation powered search. So, uh, for factual accuracy, our first step towards that was making sure you can only say stuff that you can cite. Uh, this is a pretty subtle point here. It's not just that We want to retrofit citations into a chatbot. Uh, that's not what perplexity is. In fact, it's more like a citation first service that it'll never say anything that it cannot say. So if you, if, if people have tried to play with it as if it's like chat GPT where like, tell me who are you or like things like that. And even for those questions, it will still go to a search engine, pull up stuff and come back with an answer. It's not going to say I'm, I'm perplexity. I'm like a bot design. How are you doing? Or something like that. So, um, This is because of our obsession about factual accuracy. Like even if it doesn't have a personality or, or a character in it, uh, we don't, we don't care. We want, we only care about the other thing, which is obsession about truth and, uh, accuracy. Yet the second …

AI assessment note: “for factual accuracy, our first step towards that was making sure you can only say stuff that you can cite”

Q mind. I remember, even as you're prototyping these things, you were talking about indexing aspects of Twitter or other sort of data feeds and then providing search on top of them. Um, how did you end up building a team that can iterate that rapidly as well as a culture of fast iteration, like other specific things that, that you all do as a team to help, help reinforce that?

A Yeah, I'll, I'll take the first part of this question and then I'll also let Dennis answer this because he's a big part of why this is happening. Uh, we both are basically from an academic background. So in general, the culture and academia is to, you have hundreds of ideas and you just need to try them out pretty quickly, uh, run a lot of experiments really quickly and get the results and iterate. So we come from that background, both of us. So, so that's not really new to us. It's just that When it comes to trying out in products, it, it, it's not just a result you get from running an experiment. You actually have to go to users and make them use it and, uh, talk to companies or customers or potential, you know, people in a company who would be using a product and get feedback. So there's that aspect of operational work that needs to be done to get results for experiments. And there's this aspect of quickly doing the engineering to get it to a state where you can show it to people. So both of these things had to come together and that's why the company exists. And Dennis is incredibly good at engineering and recruiting. Um, and so we found our other co-founder, Johnny Ho, who's also like a big reason why perplexity operates this fast and full credit to Dennis for helping recruit him. He was the world number one at competitive programming. Uh, it's like being Magnus Carlsen of…

AI assessment note: “we both are basically from an academic background... Dennis is incredibly good at engineering and recruiting”

Q In this, um, era of fewer clicks, If you, if the summary is in chat that's just giving you the answer, how does that impact the relationship of search or answer engines with publishers? Does that remove the incentive to publish information on the internet? Does it become more adversarial? How do you guys think about that?

A Uh, probably not. I feel like, so it's sort of like whether you cite a paper or not sort of thing, right? Like you would cite a paper, the paper was really good. It's actually going to bring back the whole concept of page rank even more. By the way, the concept of page rank was inspired by academic citations, uh, from what I've read, uh, like a very important paper tends to get cited. So when you're in this sort of citation based search interface now, The better content a publisher has, the more likely it'll get cited by an LLM. Unless humans figure out answer engine optimization or LLM optimization, which, which I hope fully they don't invest effort into. But in general, um, it's unlikely, uh, that it's as easy as SEO with just keywords because LLMs are going to be much smarter in understanding relevance to a query. So I think it's just going to incentivize people to publish higher quality content. In order to get cited by an LLM powered answer, like Substack or things like that try to do, like you, you, you, you want to own your content and you publish it and make sure it's high quality and you have your own like set of subscribers. Um, I, so, so people put a lot of effort into that more than writing tweets. So something like that is likely to happen with this interface too. Um, but it's unclear exactly how to make all this monetize at scale, like You know, the click based ad…

AI assessment note: “I think it's just going to incentivize people to publish higher quality content.”

Q That makes sense. I guess in addition to that, uh, or maybe related, you've done an impressive amount of research in reinforcement learning. What's unique about the way that Perplexity uses reinforcement learning and how does it tie into these plans?

A We like RLHF, like Reinforcement Learning from Human Feedback, where we use the contractors to, we collect feedback from the users on whether they like the summaries, the completions or not, and like we use contractors to use, like, like do some ratings themselves. And these days, even LLMs can be prompted to do the work the contractors do. Anthropics wrote in a paper on that. So all these things are getting really, uh, very efficient to do. So that's sort of how we have been thinking about Reinforcement learning right now, but, uh, we haven't gone beyond that to think of like agents and browsers and things like that. Uh, we, we'll probably focus more on the first part for the next at least six months to a year.

AI assessment note: “We like RLHF, like Reinforcement Learning from Human Feedback, where we use the contractors”