Nov 30, 2023 · 1h 23m · in-depth
The Bard blueprint | Creating value, shipping fast, and advancing AI | Jack Krawczyk (Google)
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
In this episode of In Depth, Google Product Director Jack Krawczyk discusses the creation, launch, and rapid evolution of Google Bard, offering tactical product management frameworks for building probabilistic AI systems within a major tech company.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. Brett holds 17.4% of the talking time here. How this is scored →
Brett as informed peer 3.6 Guest teaching 5.3 Guest disagreement 0.3 Brett pushing back 0.2
speaking balance: gold is Brett, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 36:39 Emphatic rejection of easy consensus
Jack immediately rejects Brett's prompt asking if gaining organizational alignment on launching an imperfect product was easy, delivering a blunt 'No' before explaining the friction.
Hardest push from Brett ▶ 16:22 Challenging conventional big company delivery timelinesBrett presses Jack on the contrast between Google's traditional delivery cadence and Bard's rapid shipping pace, questioning how a small team bypassed corporate inertia.
Biggest teaching moment ▶ 44:29 Teaching implicit code execution mechanicsJack educates Brett on how language models bridge their inherent mathematical flaws by writing and executing code behind the scenes rather than calculating directly.
Brett holds their own ▶ 16:22 Layer cake analogy of deep tech platformsBrett demonstrates strong domain synthesis by laying out his technology layer cake thesis, comparing Bard's foundation to multi-decade stacks like Vision Pro and search indexing.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | Brett as informed peer | Guest teaching | Guest disagreement | Brett pushing back | Why |
|---|---|---|---|---|---|---|
| The Origin Story and Genesis of Google Bard | 4 | 5 | 0 | 0 | Brett asks about the origin story and why Bard was chosen as a standalone form factor rather than embedded directly into existing products. Jack provides an insightful breakdown contrasting constrained surfaces like Docs/Gmail with unconstrained direct LLM interfaces. | |
| Defining Early Product Boundaries and Mitigating Safety Risks | 3 | 6 | 1 | 0 | Brett asks how early product boundaries and safety were determined. Jack explains the distinction between adversarial safety risks and inadvertent safety risks like dosage questions. | |
| Red Teaming and Managing Adversarial Prompts | 3 | 6 | 0 | 0 | Brett inquires about the mechanism of policy boundaries, and Jack explains modern red-teaming practices and handling jailbreaks. | |
| Industry Dynamics and the Window for AI Alignment | 4 | 5 | 1 | 1 | Brett presses on why Bard shipped so quickly after years of internal research. Jack reframes the speed by clarifying that transformers date back to 2017 and ChatGPT opened the market readiness window. | |
| Fostering Curiosity and Redefining Computing Paradigms | 4 | 5 | 0 | 0 | Brett asks about taking risks within a massive company like Google. Jack explains how mission-driven curiosity reshapes risk assessment when computing shifts from doing things for you to with you. | |
| Driving Fast Execution Through Passion and User Impact | 5 | 5 | 1 | 1 | Brett formulates a detailed analogy of technology layer cakes and asks how the team managed startup-like velocity. Jack highlights user research stories like autistic users crafting email responses. | |
| Product Differentiation and Possibility-First Design | 4 | 5 | 0 | 0 | Brett asks how Bard differentiated itself given ChatGPT's prior release. Jack elaborates on the North Star principle of acting as a possibility generator rather than an answer generator. | |
| Establishing Team Principles and a Culture of Urgency | 3 | 5 | 0 | 0 | Brett asks about the development of operating principles. Jack describes executive leadership modeling behavior and the refusal to accept unnecessary delays. | |
| Eliminating False Precision and Fostering Team Ownership | 3 | 5 | 0 | 0 | Brett asks whether operating norms were implicit or explicit. Jack explains how rigid swimlanes create false precision in rapidly evolving fields. | |
| Learning Through the Trough of Despair | 4 | 5 | 0 | 0 | Brett circles back to how large companies navigate the friction of shipping imperfect software. Jack explains the necessity of pushing through the trough of despair to capture real-world feedback. | |
| Aligning on Speed Over Perfection at Launch | 4 | 5 | 1 | 1 | Brett asks if it was difficult to align internal teams around launching an imperfect product. Jack responds with an emphatic 'No' before explaining how Google's research background facilitated understanding RLHF. | |
| Pitching a Standalone AI Experiment Inside Google | 4 | 5 | 1 | 0 | Brett asks about the initial internal pitch for Bard. Jack dispels startup comparisons inside big tech while emphasizing the importance of precise hypothesis articulation. | |
| Evolving Product Management for Open-Ended Technologies | 3 | 6 | 0 | 0 | Brett asks what felt new versus familiar in product management. Jack explains that traditional PRDs fail with open-ended technologies where use cases cannot be cleanly predetermined. | |
| Solving LLM Mathematical Constraints with Implicit Code Execution | 4 | 6 | 0 | 0 | Brett asks how the team mapped technological limits. Jack outlines implicit code execution as the tactical solution to LLMs failing at arithmetic. | |
| Roadmapping and Merging Collaboration with Task Execution | 4 | 5 | 0 | 0 | Brett asks about roadmapping and prioritization frameworks. Jack discusses combining generative collaboration with task execution in Assistant with Bard. | |
| Contextualizing Hallucinations and Grounding Responses | 4 | 5 | 1 | 1 | Brett explores engagement versus user acquisition. Jack uses the 'is the sky blue?' prompt to challenge binary views of hallucinations and explain Google's double-check grounding approach. | |
| Balancing Copyright Compliance with Output Delivery UX | 3 | 5 | 0 | 0 | Brett asks about UI decisions like batched responses versus token streaming. Jack shares how copyright validation checks led to surprising user preferences for batch delivery. | |
| Managing Breaking News and Sensitive Query Challenges | 3 | 5 | 0 | 0 | Brett asks about difficult unsolved technical problems. Jack discusses handling breaking news, sensitive topics, and maintaining bias to action during hiring. | |
| Overcoming Negative Public Narratives Through Real User Stories | 3 | 5 | 0 | 0 | Brett asks how the team endured harsh initial press coverage. Jack describes relying on internal support and direct feedback from real users like plumbers and ESL parents. | |
| Transforming Internal Communication from Email to Chat | 3 | 5 | 0 | 0 | Brett inquires about disseminating user feedback. Jack discusses shifting team communication from email silos to real-time chat channels. | |
| Near-Term Advances vs. Long-Term Memory and Indexing Hurdles | 4 | 6 | 0 | 0 | Brett asks what LLM problems will be solved soon versus take long. Jack details why personal indexing and long-term memory are significantly harder than token window expansions. | |
| Developing AI Fluency and Embracing Probabilistic Primitives | 4 | 6 | 0 | 0 | Brett asks how product managers should upskill for AI. Jack recommends reading foundational papers and mastering probabilistic primitives rather than treating hallucinations purely as flaws. |
Statements from this episode (27)
Google initially experimented with integrating Meena and LaMDA into Assistant
“When I first got to Google in 2020, We were experimenting with a technology that was then called MENA, one of the first language models that was out available in the market. MENA became Lambda, and what we were experimenting with was, how do we get this very c…”
LLMs are creative idea expanders, not Q&A engines
“And as we were experimenting with the technology, it became clear this isn't a simple question and answer oriented technology. It really is a way to help ideas come to life, to really help exploratory ideas manifest. It takes your imagination and expands it in…”
Embedding LLMs in existing products constrains open-ended user behavior
“The challenge when you put a language model into an existing product is there's going to be a natural constraint of that. If all of a sudden I'm in Google Docs, and I want to ask for vacation ideas that I want to explore, or I'm trying to figure out what are s…”
Google Bard refuses pediatric medical dosage questions to avoid hallucination risk
“How much Tylenol should I give to my kid when they have a fever? Very well-intentioned question, but you want to make sure that the answer that you provide is as tuned toward not hallucinating as possible. And so for early versions, and even to now, we think t…”
Jack Krawczyk credits ChatGPT for proving consumer excitement for generative AI
“I have to give credit to ChatGPT going out into the world and showing people there's more than just the risk of this technology. There's excitement that exists.”
Google frames Bard as an experiment because LLMs aren't fully baked products
“It's part of why we talked about Bard as an experiment at this point. We don't think that this technology is yet ready to be a fully baked product.”
Information retrieval is not the majority use case for Google Bard
“Some people, and it turns out not the majority use case, but some people are going to use it to try to find information.”
Google initially delayed real-time streaming in Bard to enable content filtering
“Like today, before I came here, we just talked about Bard being able to respond in real time. Other language models have been able to do that, but we have elected to present results in piecemeal, because it allows us to do a certain amount of filtering.”
Asking 'why not today' forces prioritization without creating interpersonal conflict
“And early on, we started saying things like, why tomorrow, why not today? Because what that ends up forcing isn't this, like, uncomfortable clash of, like, how dare you make it make something I'm asking for not a priority. It's, hey, share your priorities. Lik…”
Polishing AI products internally lets the fast-moving market pass you by
“If you spend all your time trying to build the world's greatest product, the world's gonna move past you.”
Google delayed Bard's coding capabilities because the initial version lacked quality
“We knew that code, for example, was something that people were using language models for, but we didn't think it was good enough, and so at some point we made the decision to say, look, code's not ready, so code launched a couple weeks after we launched”
Google Bard no longer hallucinates on time and weather queries
“Which, happy to report, Bard does not hallucinate now on time and weather.”
Building a new product inside big tech is nothing like a startup
“I don't like making comparisons to starting a new product inside of a large company is like building a startup. Like, it is nothing like building a startup.”
LLMs are poor at direct arithmetic but excel at natural language translation
“These things are really bad at doing math because they write compelling text, but they're really good at translating language, natural language to code.”
LLM logic problems are best solved by having models implicitly execute code
“The way that you start solving math and some logic-oriented problems is under the hood implicitly ask the model to write and execute code.”
Evaluation suites have replaced traditional PRDs in AI product development
“You're, in language models, your eval is your prior product requirements document. It's a probabilistic based system, and so the way you construct what you will evaluate it against effectively dictates what the product is that's going to be built.”
A lack of clear value, not awareness, limits global LLM adoption
“Three quarters of the world still does not use this technology. It's not an awareness problem. It's a finding the right value to get them to use the technology.”
Bard evaluates generated responses sentence-by-sentence against the web
“We recently launched something that we call the V-two of our Google It button, which when you receive a response on on BARD, and it happens to be something that you want a factual answer around, we go sentence by sentence and find, is there content from around…”
Google Bard aims to eliminate unintended hallucinations, not all hallucinations
“And our approach to it is, we're not, quote unquote, solving hallucination. We're solving Unintended hallucination, which would be something like, what's the distance between point A and point B? Or what's one plus one? Like, that should always be two.”
Google initially delayed streaming Bard responses to complete real-time copyright checks
“One of the things that we started to see in some of these outputs as you start to kind of stream them out is, well, even though it was a function of probability and not regurgitation of copyrighted material, you would get a response, and then at the end of the…”
Many users prefer delayed, complete AI responses because it signals thoughtfulness
“A not insignificant amount of people will tell you, I actually just like that it gives me the whole answer when it's ready. Like, it makes me think that it, that it's thinking. That it's being thoughtful about the response that it provides.”
Google Bard declines sensitive breaking news queries to avoid inaccurate summaries
“We're making the decision to basically say, it feels like a higher risk to respond with something That is not a fair summary than just saying, we'd rather take the approach to say, like, I can't help with that yet.”
Candidates who suffered through slow shipping processes make the best AI hires
“And like, the people that have experienced Pain, in various regards, are the ones that have been by far the most successful.”
Reputation is the most critical capital for executing inside large enterprises
“Inside of a large company perspective, like, your reputation is the most critical capital that exists to getting things done, and you gotta be willing to risk it.”
LLM inference costs will decrease significantly in relatively short order
“I think you're gonna see the cost of inference tend to go down. Inference is the cost to actually serve and run the model. I think you're gonna see those things go down in relatively short order.”
Persistent memory for LLMs will take much longer to solve than expected
“And so I am beyond Fascinated and intrigued by the research that's taking place around memory and being able to reference things from previous conversations, but I think it's going to take way longer than we expect.”
LLM hallucination is a core generative feature, not a bug
“And so finding the way of harnessing that as a feature rather than a bug, I think is going to be challenging, especially when one of the perceived shortcomings of your product is actually going to be, is like the core of what makes it function. I don't know th…”