Colvin: GenAI telemetry carries vastly more sensitive PII than traditional metrics
Samuel Colvin · Agent Engineering with Pydantic + Graphs — with Samuel Colvin, CEO of Pydantic Logfire · Feb 6, 2025 · at 37:21
Samuel Colvin, founder of Pydantic Logfire, explains why LLM observability creates unprecedented data privacy and PII challenges compared to traditional telemetry tools like Datadog.
0:00 / 0:24exact quote · 24.1s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“With Gen AI, that, that distinction doesn't exist because it's all just like messed up in the text. If you have that same patient asking an LLM how to, what drug they should take or how to stop smoking, you can't extract the PII and not send it to the Observability platform. So the sensitivity of the data that's going to end up in observability platforms is going to be at, like, basically different order of magnitude to what's in a, what you would normally send to Datadog.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Samuel Colvin
Colvin: Agent framework engineering quality lags far behind Python ecosystem
“Looking in general at the ecosystem of agent frameworks, the engineering quality is far below that of the rest of the Python ecosystem.”
Colvin: LLMs are 100x faster under four specific engineering conditions
“My take is that there are four, four things where if you can cover all four of these things, LLMs are not like three X faster or five X faster. They're like a hundred X faster.”
Pydantic built a VIP issue scorer after closing an OpenAI founder's ticket
“Basically this started off because one of the OpenAI co-founders created an issue on Pydantic. And we just closed it and said it was wrong. And so we have this that, like, injects itself and tries to summarize someone and it gives them a, like, brutal score of…”
Colvin: Agent Frameworks Exist Because AI Models Are Not Clever Enough
“Agents, agent frameworks, graphs, all of this stuff is basically making up for the fact that right now the models are not that clever.”
Colvin: Logfire migrated from ClickHouse to Timescale before choosing DataFusion
“We don't use ClickHouse. We started building our database with ClickHouse, moved off ClickHouse onto Timescale, which is a Postgres extension to do analytical databases. And then moved off Timescale onto Data Fusion, and we're basically now building, it's Data…”
Colvin: Gen AI observability will merge into general-purpose observability platforms
“Web observability stopped being a thing, not because the web stopped being a thing, but because all observability had to do web. If you were talking to people in 2010 or 2012, they would have talked about cloud observability. Now that's not a term because all …”