Jul 9, 2026 · 1h 9m · neon-show
The Hidden Layer Every Al Product Needs Today | Atin Sanyal Founder, Galileo
0:00 / 0:00
▶ Watch on YouTube →
gold bands on the timeline = statements, start to end. Hover to read, click to jump. CC turns on captions
Galileo co-founder Atin Sanyal explores the critical role of data quality, evaluation infrastructure, and specialized small language models in building reliable, production-grade enterprise AI and autonomous agents.
How this conversation actually went
Every chapter scored 0–10 on four independent dynamics. Hover any point for the reasoning behind the score. How this is scored →
Siddhartha as informed peer 4.5 Guest teaching 4.6 Guest disagreement 0.8 Siddhartha pushing back 0.9
speaking balance: gold is Siddhartha, purple is the guest (3 minute bins)
Sharpest disagreement ▶ 35:30 Atin rejects single-query LLM optimization as sufficient for agents
Atin directly counters the premise that cheaper frontier LLMs eliminate the need for SLMs, arguing that multi-step agent interactions compound latency and infrastructure choke points exponentially.
Hardest push from Siddhartha ▶ 34:51 Siddharth challenges the long-term viability of SLMsSiddharth forcefully presses Atin on why anyone would need small language models when LLM prices are plummeting 80% annually and frontier model speeds are accelerating.
Biggest teaching moment ▶ 12:25 Atin deconstructs tokenization and sequential generationAtin provides a comprehensive technical explanation of how neural networks act as memory machines generating probabilistic sequences over dictionary vocabularies.
Siddhartha holds their own ▶ 29:55 Siddharth connects AI observability directly to SRE history and SplunkSiddharth showcases his enterprise software knowledge by accurately framing LLM observability within traditional SRE telemetry and citing industry precedent like Splunk.
the scores for every segment, with the reasoning behind each
| Chapter | Topic | Siddhartha as informed peer | Guest teaching | Guest disagreement | Siddhartha pushing back | Why |
|---|---|---|---|---|---|---|
| Architecting Uber's Michelangelo Platform and Mission-Critical AI | 3 | 4 | 0 | 0 | Siddharth asks straightforward questions about Atin's background at Uber and Michelangelo platform architecture. Atin provides a detailed technical breakdown of streaming ML and early mission-critical AI systems. | |
| Founding Galileo, Chris Ray's Advice, and Data Bottlenecks | 3 | 5 | 1 | 0 | Atin narrates founding Galileo after advice from Stanford professor Chris Ré, explaining feature stores and why uncurated data is catastrophic for production AI. Siddharth largely listens as Atin breaks down data infrastructure. | |
| Demystifying Language Models, Tokens, and Quantifying Uncertainty | 2 | 6 | 0 | 0 | Siddharth prompts Atin to explain language models and tokens for a layman audience. Atin gives a foundational masterclass on seq-to-seq models, attention mechanisms, and token probability distributions. | |
| Early Product Traction and Fine-Tuning Small Models | 4 | 5 | 1 | 0 | Siddharth asks about the technical origins of AI observability and Galileo's first enterprise customers. Atin explains statistical uncertainty quantification and early fine-tuning workflows on smaller BERT-scale models. | |
| Data Curation Bottlenecks and the ChatGPT Inflection Point | 5 | 4 | 1 | 2 | Siddharth presses on what data scientists were actually building in 2021-2022 and clarifies whether Galileo produced data or filtered it. Atin clarifies that Galileo acted as a data debugging filter to remove labeling errors. | |
| Enterprise Adoption Curves and Developing the Luna Model | 4 | 6 | 1 | 0 | Siddharth inquires about the enterprise adoption curve and why Galileo created its Luna model. Atin explains why LLM-as-a-judge approaches choke at production scale and describes distilling models into specialized evaluators. | |
| Overcoming Latency in Observability and Defining Evals | 6 | 5 | 0 | 1 | Siddharth demonstrates solid domain familiarity, comparing AI observability challenges to traditional SRE practices and citing Splunk. Atin elaborates on sub-100ms latency benchmarks and in-house inference optimization using LoRA. | |
| The Strategic Case for Small Language Models (SLMs) | 7 | 5 | 3 | 5 | Siddharth offers sharp, informed pushback, challenging Atin's advocacy for SLMs by noting that frontier LLM costs drop 80% yearly and latency is shrinking. Atin pushes back with economic and compounded-latency arguments regarding complex agent multi-step chains. | |
| The Observability Flywheel and Evals-Driven Development | 5 | 5 | 0 | 1 | Siddharth inquires about the revenue mix between offline evals and online observability, joking about the classic 'works on my machine' dilemma. Atin details the continuous evals-driven development flywheel. | |
| Production Agent Use Cases and Open-Source Agent Control | 4 | 4 | 0 | 0 | Siddharth asks for tangible production agent use cases across enterprise verticals. Atin explains real-world implementations in sales intelligence and SRE root-cause analysis, highlighting open-source Agent Control. | |
| Engineering to Enterprise Sales: Building Galileo's GTM | 5 | 3 | 1 | 1 | Siddharth asks how an engineer unlearns big tech habits to build an enterprise GTM motion selling to Fortune 50 clients. Atin details early customer discovery interviews and identifying the core ICP. | |
| Transition to CPO and Designing Future AI Product Interfaces | 5 | 4 | 0 | 0 | Siddharth asks what defines a great product when software and code are commoditized into tokens. Atin shares his product philosophy on natural language, voice interfaces, and headless software. | |
| The Cisco Acquisition and Galileo's Future Outlook | 5 | 4 | 2 | 2 | Siddharth congratulates Atin on the Cisco acquisition and briefly interjects that protocols like MCP already address agent interactions. Atin reframes MCP as only a small tool-calling piece within the larger agent observability fabric. |
Statements from this episode (16)
Sanyal: AI outputs are mission-critical because AI is now the product interface
“I think it's the first time in human history that AI has become the face of the product. A user is directly talking to an AI and working with an AI. So every output of an AI system is mission critical. And with agents doing actions and making critical decision…”
Uber built the world's first machine learning feature store
“We had worked on the feature store at Uber, and we had built the world's first feature store, and we literally coined the term.”
Sanyal: Most enterprise data scientists are essentially data curators
“The main thing they actually work on, though, is curating data, because at the end of the day, you're not really a scientist sitting building new model architectures. Most data scientists are essentially data curators.”
Sanyal: ChatGPT commoditized modeling skills, turning AI adoption into software engineering
“Ever since ChatGPT, I've seen the adoption of AI in enterprises increase its velocity manifold. And the reason is because the niche skill has been commoditized. Now you have a black box and the whole point is how do you build a software and infrastructure laye…”
Sanyal: Compliance, security, and resilience are the main enterprise AI bottlenecks
“So banks and telecom companies and healthcare companies, while their engineering and innovation has stepped up in its velocity, the main bottleneck now is compliance, security, and just general resilience.”
Sanyal: LLM-as-a-judge approaches choke and fail to scale in production
“LLMs as judges, which was the traditional way that people were using to evaluate, and that has a whole history of it. They don't scale. They don't scale in production. They don't allow you to do full scale observability, which means intercepting every single i…”
Sanyal: Enterprises Will Not Deploy Action-Taking Agents Without Observability
“No one's going to put an agentic system, especially if they do actions and tasks, they're not going to do it without any kind of observability.”
Sanyal: SLMs will always maintain an inference cost advantage over LLMs
“It is so
cheap to run a single inference on an SLM versus an LLM. So that gap will always be there.”
Sanyal: LLM evaluation does not require general-purpose reasoning models
“We feel like if you really focus on the fundamental problem that, hey, evaluations for language models is a very task-specific, constrained problem, and it does not require you to use you know, general purpose reasoning for that, and then there's fine-tuning y…”
Sanyal: AI observability faces a trilemma of cost, latency, and quality
“Observability is a bit of a different ballgame because it has a different set of challenges. It's an infrastructural problem and latency and cost and quality. Those are the three things, and they are a bit of opposing forces with each other. It's like this tri…”
Sanyal: AI developers should build evaluation suites before building applications
“In fact, you build the evals before you build the app. That kind of becomes, that's why people say that, hey, evals is the new weapon for product managers because they are the ones who are defining the app's behavior.”
Sanyal: VP of data science is a diminishing role in enterprises
“In our case, our ICPs were the VPs of engineering or rather VPs of data science back in the day. It was a role that apparently is a diminishing breed.”
Sanyal: Software and code are commoditized into tokens by fast AI generation
“In an era where you can build an, you know, a website and a web application with a Postgres database behind in 30 seconds, Software is just commoditized. Software is just tokens. Code is tokens.”
Sanyal: In the end, humans will just talk to software to work
“In the end, we'll enter an era where we'll talk to software, and software will do work for us, and everything in the middle is a means to an end.”
Sanyal: Software could become completely headless and ambient within 5-10 years
“Ideally maybe 10 years down the line or perhaps five to 10 years down the line if we can build a system which is completely headless, where you just start kind of going back to Iron Man, right? Where you just talk to the walls and they answer you and they get …”
Sanyal: MCP addresses only a drop in the ocean of agent interactions
“Well, MCP is one stab at solving a very specific problem of agent interactions, which is to selecting the right tools and sort of automating that process. There's a lot more to be solved. It's a drop in the ocean of the overall set of interactions.”