Brady: Multi-model fallbacks require maintaining distinct prompts per LLM provider
James Brady · How To Hire AI Engineers (ft. James Brady and Adam Wiggins of Elicit) · Jun 21, 2024 · at 21:36
James Brady of Elicit discusses the hidden engineering maintenance cost of using model shadowing or provider fallbacks between OpenAI and Anthropic.
0:00 / 0:11exact quote · 11.2s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“You definitely need to have a different prompt if you want to stay within a few percentage points degradation, like I said before. And that certainly comes at a cost.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from James Brady
Brady: Prompt Engineering Won't Be a Durable Differentiating Skill
“I don't think that prompt engineering is going to be a kind of durable differential skill that people will hold. I do think that the way that you set up the ML problem to kind of ask the right questions, if you see what I mean, rather than the specific phrasin…”
Brady: Elicit uses checked exceptions in Python to handle edge cases
“We use checked exceptions inside our Python code base, which means that we can use the type system to make sure we are handling, properly handling, all of the various things that could be going wrong, all the different exceptions that could be getting raised. …”
Brady: Centralized AI gateways restrict Elicit's model and prompt experimentation
“For illicit where really the secret source of the real secret source is which models we're using, how we're using them, how we're combining them, how we're thinking about the user problem, how we're thinking about all these pieces coming together. You really n…”
Brady: Elicit does not run an ML-focused interview for AI engineers
“We don't have an ML-focused interview for the AI engineer role at all, actually.”
Brady: Fault-tolerant engineering discipline and ML curiosity are in natural tension
“I think the fault first mindset and the ML curiosity attitude could be somewhat in tension, right? Because for example, the stereotypical, stereotypical version of someone that is great at building fault tolerant systems has probably been doing it for a decade…”