Brady: Elicit uses checked exceptions in Python to handle edge cases
James Brady · How To Hire AI Engineers (ft. James Brady and Adam Wiggins of Elicit) · Jun 21, 2024 · at 14:10
James Brady, head of engineering at Elicit, describes their engineering practices for handling non-deterministic LLM outputs in Python.
0:00 / 0:30exact quote · 30.9s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“We use checked exceptions inside our Python code base, which means that we can use the type system to make sure we are handling, properly handling, all of the various things that could be going wrong, all the different exceptions that could be getting raised. So checked exceptions are not, not really particularly popular. Actually, there's not many people that are big fans of them for our particular use case to really make sure that we've not just forgotten to handle, you know, this particular type of error. We have found it useful to force us to think about all the different edge cases that come up.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from James Brady
Brady: Prompt Engineering Won't Be a Durable Differentiating Skill
“I don't think that prompt engineering is going to be a kind of durable differential skill that people will hold. I do think that the way that you set up the ML problem to kind of ask the right questions, if you see what I mean, rather than the specific phrasin…”
Brady: Centralized AI gateways restrict Elicit's model and prompt experimentation
“For illicit where really the secret source of the real secret source is which models we're using, how we're using them, how we're combining them, how we're thinking about the user problem, how we're thinking about all these pieces coming together. You really n…”
Brady: Elicit does not run an ML-focused interview for AI engineers
“We don't have an ML-focused interview for the AI engineer role at all, actually.”
Brady: Fault-tolerant engineering discipline and ML curiosity are in natural tension
“I think the fault first mindset and the ML curiosity attitude could be somewhat in tension, right? Because for example, the stereotypical, stereotypical version of someone that is great at building fault tolerant systems has probably been doing it for a decade…”
Brady: Early-twenties engineers today show strikingly higher capability and maturity
“The maturity and capabilities and just kind of general put togetherness of people at that age now is strikingly different to where I was then.”
Brady: Effective AI engineers need SWE fundamentals, ML curiosity, and a fault-first mindset
“The three things that we say are most important for a highly effective AI engineer first of all, conventional software engineering skills, which is kind of a given, but definitely worth mentioning. The second thing is a curiosity and enthusiasm for machine lea…”