Rajpal: Simulation testing revealed an early partner's real failure was over-refusal
Shreya Rajpal · ⚡️Snowglobe: Simulations for your AI · Sep 25, 2025 · at 4:11
Shreya Rajpal shares an example of how simulation testing showed an enterprise customer that over-refusal, not toxicity, was their chatbot's actual problem.
0:00 / 0:32exact quote · 32.4s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“Organizations that we're, we were, we had as our design partner, we you know, they were like, oh, we're very worried about toxicity, and we want toxicity guardrails, and we did all of this testing for them in production, and toxicity was actually not a real concern for them. What ended up being an actual concern that only immersion simulation was, you know, over refusal, that their, like, a chatbot was so conservative that it was just refused, like, pretty, you know, requests that should have been pretty benign. And so, you know, that allows them to like, okay, you don't need toxicity guardrails. You more kind of need to align your system to, you know, not over refuse.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Shreya Rajpal
Rajpal: Anthropic Claude models had regressions from serving architecture changes
“Anthropix kind of cloud models kind of had a regression, right? Because they changed to a new serving architecture.”
Rajpal: Foundation models rarely generate toxic outputs without explicit jailbreaks
“Most of the stuff that the frameworks will recommend is actually stuff that the model providers are already working on. So toxicity, unless you're doing, unless somebody is very explicitly trying to jailbreak what you've built, you know, you won't run into the…”
Rajpal: AI simulations should prioritize product KPIs over generic safety metrics
“So I would actually say that like a lot of the things to simulate are more aligned with like product KPIs or product metrics that actually make Whatever AI system you're building very sticky, rather than, you know, focusing more on, like, traditional safety se…”
Rajpal: Fine-tuning open-source models on synthetic data closes proprietary capability gaps
“Not out of the box, but with a lot of that fine tuning and that the training, et cetera, you are able to kind of close the gap and even have better performance on metrics.”
Rajpal: AI agents resemble autonomous vehicle architectures with cascading ML units
“What patterns really worked well in self-driving cars, which is weirdly a very similar system to, you know, agents of today where you have like these cascading kind of like units that are all machine learning based and, you know, they all kind of like feed int…”