Jack Krawczyk
Senior Product Director, Meta · 1 appearance on the record.
computed by AI from the episodes · how this works → · full disclaimer →
executiveoperator@JackK ↗LinkedIn ↗
Jack Krawczyk is best known for serving as the product lead for Google’s generative AI chatbot Bard (later renamed Gemini). Over his career, he has held senior product leadership roles at Meta, Google, Pandora, StumbleUpon, WeWork, and UnitedMasters.
27statements → 6claims → 3claims resolved → 67%fully supported → 4/5average certainty → 2.07/5average debate potential → ≈4.5/5argument clarity, estimated →
2 supported 1 partly supported 0 contradicted 3 not checkable as stated how the 6 claims stand · each chip opens the sources
2 predictions · 4 assertions · 3 opinions · 13 insights · 5 disclosures · every statement was checked. The predictions and assertions are the 6 claims: statements the public record can support or contradict. 3 are resolved, and 3 name no date, number or outcome precise enough to check. Everything else (opinions, insights, what ifs, disclosures) can never be settled by the record, so it carries no assessment.
The record, in short
What the tape says about how Jack argues and how the claims held up. Everything they said, and everything said about them, is in the tabs below.
Their most notable supported claim
Expressed certainty vs assessment result
weighted support: a fully supported claim counts one, a partly supported claim counts half. Each filled bar is clickable and opens exactly those claims; "none yet" means nothing said at that certainty level has resolved yet
How they sound: not measured why? →
We measure speaking style by listening to the audio itself, and a fair number needs at least 2,000 words from one person on tape we have measured. There is too little of Jack Krawczyk on measured tape to publish a rate. This says nothing about how they speak.
Everything Jack Krawczyk said on In Depth that made the record, most notable first. Filter by type, assessment or year in the ledger →
Google frames Bard as an experiment because LLMs aren't fully baked products
“It's part of why we talked about Bard as an experiment at this point. We don't think that this technology is yet ready to be a fully baked product.”
Information retrieval is not the majority use case for Google Bard
“Some people, and it turns out not the majority use case, but some people are going to use it to try to find information.”
Persistent memory for LLMs will take much longer to solve than expected
“And so I am beyond Fascinated and intrigued by the research that's taking place around memory and being able to reference things from previous conversations, but I think it's going to take way longer than we expect.”
LLM hallucination is a core generative feature, not a bug
“And so finding the way of harnessing that as a feature rather than a bug, I think is going to be challenging, especially when one of the perceived shortcomings of your product is actually going to be, is like the core of what makes it function. I don't know th…”
Google Bard refuses pediatric medical dosage questions to avoid hallucination risk
“How much Tylenol should I give to my kid when they have a fever? Very well-intentioned question, but you want to make sure that the answer that you provide is as tuned toward not hallucinating as possible. And so for early versions, and even to now, we think t…”
Jack Krawczyk credits ChatGPT for proving consumer excitement for generative AI
“I have to give credit to ChatGPT going out into the world and showing people there's more than just the risk of this technology. There's excitement that exists.”
Building a new product inside big tech is nothing like a startup
“I don't like making comparisons to starting a new product inside of a large company is like building a startup. Like, it is nothing like building a startup.”
LLM logic problems are best solved by having models implicitly execute code
“The way that you start solving math and some logic-oriented problems is under the hood implicitly ask the model to write and execute code.”
Google Bard aims to eliminate unintended hallucinations, not all hallucinations
“And our approach to it is, we're not, quote unquote, solving hallucination. We're solving Unintended hallucination, which would be something like, what's the distance between point A and point B? Or what's one plus one? Like, that should always be two.”
LLMs are creative idea expanders, not Q&A engines
“And as we were experimenting with the technology, it became clear this isn't a simple question and answer oriented technology. It really is a way to help ideas come to life, to really help exploratory ideas manifest. It takes your imagination and expands it in…”
Embedding LLMs in existing products constrains open-ended user behavior
“The challenge when you put a language model into an existing product is there's going to be a natural constraint of that. If all of a sudden I'm in Google Docs, and I want to ask for vacation ideas that I want to explore, or I'm trying to figure out what are s…”
Google initially delayed real-time streaming in Bard to enable content filtering
“Like today, before I came here, we just talked about Bard being able to respond in real time. Other language models have been able to do that, but we have elected to present results in piecemeal, because it allows us to do a certain amount of filtering.”
Asking 'why not today' forces prioritization without creating interpersonal conflict
“And early on, we started saying things like, why tomorrow, why not today? Because what that ends up forcing isn't this, like, uncomfortable clash of, like, how dare you make it make something I'm asking for not a priority. It's, hey, share your priorities. Lik…”
Polishing AI products internally lets the fast-moving market pass you by
“If you spend all your time trying to build the world's greatest product, the world's gonna move past you.”
Google Bard no longer hallucinates on time and weather queries
“Which, happy to report, Bard does not hallucinate now on time and weather.”
Evaluation suites have replaced traditional PRDs in AI product development
“You're, in language models, your eval is your prior product requirements document. It's a probabilistic based system, and so the way you construct what you will evaluate it against effectively dictates what the product is that's going to be built.”
A lack of clear value, not awareness, limits global LLM adoption
“Three quarters of the world still does not use this technology. It's not an awareness problem. It's a finding the right value to get them to use the technology.”
Google initially delayed streaming Bard responses to complete real-time copyright checks
“One of the things that we started to see in some of these outputs as you start to kind of stream them out is, well, even though it was a function of probability and not regurgitation of copyrighted material, you would get a response, and then at the end of the…”
Many users prefer delayed, complete AI responses because it signals thoughtfulness
“A not insignificant amount of people will tell you, I actually just like that it gives me the whole answer when it's ready. Like, it makes me think that it, that it's thinking. That it's being thoughtful about the response that it provides.”
Google Bard declines sensitive breaking news queries to avoid inaccurate summaries
“We're making the decision to basically say, it feels like a higher risk to respond with something That is not a fair summary than just saying, we'd rather take the approach to say, like, I can't help with that yet.”
Candidates who suffered through slow shipping processes make the best AI hires
“And like, the people that have experienced Pain, in various regards, are the ones that have been by far the most successful.”
Reputation is the most critical capital for executing inside large enterprises
“Inside of a large company perspective, like, your reputation is the most critical capital that exists to getting things done, and you gotta be willing to risk it.”
LLM inference costs will decrease significantly in relatively short order
“I think you're gonna see the cost of inference tend to go down. Inference is the cost to actually serve and run the model. I think you're gonna see those things go down in relatively short order.”
Google initially experimented with integrating Meena and LaMDA into Assistant
“When I first got to Google in 2020, We were experimenting with a technology that was then called MENA, one of the first language models that was out available in the market. MENA became Lambda, and what we were experimenting with was, how do we get this very c…”
Appearances (1)
| Episode | Date | Speaking time |
|---|---|---|
| The Bard blueprint | Creating value, shipping fast, and advancing AI | Jack Krawczyk (Goog | Nov 30, 2023 | 1h 2m |