BERT

“Probably BERT. And before that, it was like ResNet for computation, like images, computer vision classification. So ResNet already need to run on NVIDIA K-eighty, which is kind of one of the first SEU on AWS and other places. And, but way over, but even at thi…”

“At least at the time Transformers came out, BERT came out and all that, nobody really thought this was a this was a path to emergent reasoning capabilities, a path to AGI itself.”

“So at Roblox for many, many years, we were pushing very early AI BERT models, primitive type models to drive safety.”

“BERT and MUM, people underestimate how much, because we measure search quality so religiously, some of the biggest jumps in search quality in that period where search went ahead of everyone else was because of BERT and MUM. We built transformers and used it im…”

“So, right, so you went from needing to train thousands about thousands of documents to 10 documents, maybe, to extract things like dollar amounts from invoice totals simple things like that.”

“We use Base-Ten to run some I would say some LAMA, some BERT model for classification.”

“Generative AI is an example we were doing, you know, we had Bert, we had mum as ranking improvements way back in the early days, really as the technology was new, that enables us to get our feet wet with it.”

“For a vast majority of enterprises, they should probably be using something off the shelf, fine tuning BERT models. If it's a vision, they should be fine tuning resonant or using something like clip, like the less work they can do the better.”

“We took it and we applied it to model embeddings where instead of zebras and horses, we have like BERT embeddings and GPT embeddings, or like two completely different models with different architectures. So I think these are GTR, which is a T five based retrie…”

“We started with this BERT model that Google had put in open domain, which was trained on all of the internet's, you know, data and knowledge. And we would then take those models and then for every customer of ours, we'd actually build custom embeddings, you kn…”

“I think we built the first large language model on BERT that was in, in our space ever in 2017, 20 18.”

“It is the probably the largest scale rollout of transformers yet after Google rolled out BERT for search”

“Google was running transformers even in their search workload since 2018, 2019. The advent of BERT, which was one of the most most well-known, most popular transformers before we got to the GPT madness is, has been their, in their production search workloads f…”

“In 2019, the earthquake had already happened. Deep learning models and so on have already been grappled with. Large language models and transformer models like BERT and others started being used by the more mainstream engineering cohorts.”

“I think one thing that I remember specifically was the Transformers paper paper came out, but I think it started making a lot of noise when BERT was out. Like when people started seeing what BERT could essentially do. That was kind of Like a switch that went o…”

“I think one of the things that became very apparent early on when playing with BERT was unlike all the prior generations of models, it had a large amount of world knowledge that we didn't deliberately encode into it. It wasn't in, you know, the fine tuning dat…”

“Having said that, if you look at the number of connections that the state-of-the-art machine learning systems for some of these problems have, they're more than many animals. So we're actually at the point where the, you know, they have hundreds of millions or…”