The Ledger, every show

“Somewhere where I disagree with him is that you need to own the IDE. I think like he made kind of some good points about, you know, not having platform risk in the long term, but some of the, you know, features that were mentioned, like suggesting diffs, for e…”

“So I think that even if a delta exists, in twenty-twenty-four, the delta between proprietary and open source won't be large enough that a startup like us, with a lot of data that we've collected, can take the data that we have, fine-tune an open source model, …”

“GPT-IV itself has been trained on human eval, and we know this because GPT-IV is able to predict the exact doc string in many of the problems. I've seen it predict, like, the specific example values in the doc string, which is extremely improbable for it to ju…”

“And to the best of my knowledge, I think that's the first example that I'm aware of a LLM search engine model that's effectively connected to, like, a large enough index that I would consider, like, an internet scale. So, so I think we were the first to releas…”

“In the future, you know, in the future, I think programming is just going to be really just the problem solving. Like you come up with an idea, you come up with like the basic design for the algorithm in your head, and you just tell the AI, hey, just like, jus…”

“What really shocks us is that a lot of the people who do that they're coming from ChatGPT. So they tried it in ChatGPT with ChatGPT-IV. It didn't work. Maybe it required like some multi-step reasoning. Maybe it required to like, Some internet context or someth…”

“I think that GPT-IV, my hypothesis is that the jump from four to 4.5, or four to five, will be smaller than the jump from Three to four.”

“Like, I think it's generally been shown that if you have the space to just put The raw files inside of a big context window. That is still better than chunking and retrieval. It just is.”

“We've seen emerging capabilities in the find model, whereby training it on high quality code, it can actually, like, reason better. It went from not being able to solve like, World problems where like riddles where like with like temporal and like low, like pl…”

“It worked so well that it actually worked better than Google Lens which released its V-One around the same time.”

“Really who everyone's competing with is ChatGPT which only has, like, that one web interface, and, like, ChatGPT is really the bar.”

“All of our models are at the top of the big code leaderboard by far. It's not close, particularly in languages other than Python. We have a 10 point gap between us and the next best model on Java, JavaScript, I think C-sharp multilingual.”

“I was also looking into image captioning, where like you give a model an image, and then it tells you what's in the image, but it turns out that what people want is the exact opposite. People want to give a description of an image, and then have the AI generat…”

“I think T-Zero is the first model that did large-scale instruction tuning from diverse data sources in the fall of twenty-twenty-one. This is before InstructGPT. This is before Flan T-Five, which came out in twenty-twenty-two. This is, I think, the very, very …”

“Paul Graham actually picked it for us.”

“At NVIDIA, They know that they're going to win regardless. So they don't care where you get the GPUs from. They're like, they're truly neutral, unlike various sales reps that you might encounter at various like clouds and, you know, hardware companies, et cete…”

“But with int eight, there's not necessarily a Speed increase. It's just the storage optimization.”

“So we have these great quantization libraries that, you know, for the most part are able to get the size down with not that much quality loss, but there is some, like the quantized models currently are actually worse than the non-quantized ones.”

“And Replit is great, and people use that feature. But yeah, I think there's more we can do in terms of, like, having something a bit closer to code interpreter where it's able to run the code and then, like, recursively iterate on it.”

“They actually implemented a custom feature for us in Faster Transformer which is one of their libraries... They implemented streaming generation for T-Five-based models, which we were running at the time up until we switched to GPT in In February, March of thi…”

“So as, I think there's always an opportunity for us to become more general if we wanted. But We've been along this path of, like, what is the best, most advanced reasoning engine that's connected to your code base, that's connected to the internet, that we can…”