The Ledger
Every statement that passed quotation and attribution checks. Mix any filter with any other: certainty 1/5, debate potential 5/5, or both at once.
why aren't all 27 resolved? a statement only gets an assessment when the public record can support or contradict it. opinions and what-ifs never can, and 0 checkable ones are still open, waiting for their date. predictions held up or didn't; assertions are supported or contradicted. on every card: ▮▮▮▮▮ certainty · ▮▮▮▮▮ debate potential. speakers are clickable
Google frames Bard as an experiment because LLMs aren't fully baked products
“It's part of why we talked about Bard as an experiment at this point. We don't think that this technology is yet ready to be a fully baked product.”
Information retrieval is not the majority use case for Google Bard
“Some people, and it turns out not the majority use case, but some people are going to use it to try to find information.”
Persistent memory for LLMs will take much longer to solve than expected
“And so I am beyond Fascinated and intrigued by the research that's taking place around memory and being able to reference things from previous conversations, but I think it's going to take way longer than we expect.”
LLM hallucination is a core generative feature, not a bug
“And so finding the way of harnessing that as a feature rather than a bug, I think is going to be challenging, especially when one of the perceived shortcomings of your product is actually going to be, is like the core of what makes it function. I don't know th…”
Google Bard refuses pediatric medical dosage questions to avoid hallucination risk
“How much Tylenol should I give to my kid when they have a fever? Very well-intentioned question, but you want to make sure that the answer that you provide is as tuned toward not hallucinating as possible. And so for early versions, and even to now, we think t…”
Jack Krawczyk credits ChatGPT for proving consumer excitement for generative AI
“I have to give credit to ChatGPT going out into the world and showing people there's more than just the risk of this technology. There's excitement that exists.”
Building a new product inside big tech is nothing like a startup
“I don't like making comparisons to starting a new product inside of a large company is like building a startup. Like, it is nothing like building a startup.”
LLM logic problems are best solved by having models implicitly execute code
“The way that you start solving math and some logic-oriented problems is under the hood implicitly ask the model to write and execute code.”
Google Bard aims to eliminate unintended hallucinations, not all hallucinations
“And our approach to it is, we're not, quote unquote, solving hallucination. We're solving Unintended hallucination, which would be something like, what's the distance between point A and point B? Or what's one plus one? Like, that should always be two.”
LLMs are creative idea expanders, not Q&A engines
“And as we were experimenting with the technology, it became clear this isn't a simple question and answer oriented technology. It really is a way to help ideas come to life, to really help exploratory ideas manifest. It takes your imagination and expands it in…”
Embedding LLMs in existing products constrains open-ended user behavior
“The challenge when you put a language model into an existing product is there's going to be a natural constraint of that. If all of a sudden I'm in Google Docs, and I want to ask for vacation ideas that I want to explore, or I'm trying to figure out what are s…”
Google initially delayed real-time streaming in Bard to enable content filtering
“Like today, before I came here, we just talked about Bard being able to respond in real time. Other language models have been able to do that, but we have elected to present results in piecemeal, because it allows us to do a certain amount of filtering.”
Asking 'why not today' forces prioritization without creating interpersonal conflict
“And early on, we started saying things like, why tomorrow, why not today? Because what that ends up forcing isn't this, like, uncomfortable clash of, like, how dare you make it make something I'm asking for not a priority. It's, hey, share your priorities. Lik…”
Polishing AI products internally lets the fast-moving market pass you by
“If you spend all your time trying to build the world's greatest product, the world's gonna move past you.”
Google Bard no longer hallucinates on time and weather queries
“Which, happy to report, Bard does not hallucinate now on time and weather.”
Evaluation suites have replaced traditional PRDs in AI product development
“You're, in language models, your eval is your prior product requirements document. It's a probabilistic based system, and so the way you construct what you will evaluate it against effectively dictates what the product is that's going to be built.”
A lack of clear value, not awareness, limits global LLM adoption
“Three quarters of the world still does not use this technology. It's not an awareness problem. It's a finding the right value to get them to use the technology.”
Google initially delayed streaming Bard responses to complete real-time copyright checks
“One of the things that we started to see in some of these outputs as you start to kind of stream them out is, well, even though it was a function of probability and not regurgitation of copyrighted material, you would get a response, and then at the end of the…”
Many users prefer delayed, complete AI responses because it signals thoughtfulness
“A not insignificant amount of people will tell you, I actually just like that it gives me the whole answer when it's ready. Like, it makes me think that it, that it's thinking. That it's being thoughtful about the response that it provides.”
Google Bard declines sensitive breaking news queries to avoid inaccurate summaries
“We're making the decision to basically say, it feels like a higher risk to respond with something That is not a fair summary than just saying, we'd rather take the approach to say, like, I can't help with that yet.”
Candidates who suffered through slow shipping processes make the best AI hires
“And like, the people that have experienced Pain, in various regards, are the ones that have been by far the most successful.”
Reputation is the most critical capital for executing inside large enterprises
“Inside of a large company perspective, like, your reputation is the most critical capital that exists to getting things done, and you gotta be willing to risk it.”
LLM inference costs will decrease significantly in relatively short order
“I think you're gonna see the cost of inference tend to go down. Inference is the cost to actually serve and run the model. I think you're gonna see those things go down in relatively short order.”
Google initially experimented with integrating Meena and LaMDA into Assistant
“When I first got to Google in 2020, We were experimenting with a technology that was then called MENA, one of the first language models that was out available in the market. MENA became Lambda, and what we were experimenting with was, how do we get this very c…”
Google delayed Bard's coding capabilities because the initial version lacked quality
“We knew that code, for example, was something that people were using language models for, but we didn't think it was good enough, and so at some point we made the decision to say, look, code's not ready, so code launched a couple weeks after we launched”
LLMs are poor at direct arithmetic but excel at natural language translation
“These things are really bad at doing math because they write compelling text, but they're really good at translating language, natural language to code.”
Bard evaluates generated responses sentence-by-sentence against the web
“We recently launched something that we call the V-two of our Google It button, which when you receive a response on on BARD, and it happens to be something that you want a factual answer around, we go sentence by sentence and find, is there content from around…”