Biggio: Forcing function calls over voice yields reliable UI control
Ilan Biggio · Building AGI in Real Time (OpenAI Dev Day 2024) · Oct 4, 2024 · at 16:56
OpenAI engineer Ilan Biggio explains an architectural pattern for using OpenAI's Realtime API to control user interfaces.
0:00 / 0:41
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“Speech to text is really interesting because you can prevent, you can prevent responses like audio responses and force function calls. And so you can do stuff like UI control that is like super, super reliable... If you like cut out the audio outputs and make it so it always has to output a function, like you can end up with pretty, pretty reliable, like commands like a command architecture.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Ilan Biggio
Biggio: Real-time video/vision is likely the next Realtime API feature
“To use ChatGPT's voice mode as an example, like we've demoed the video, right? Like real-time image, right? So I'm not actually sure what timelines are, but I would expect, if I had to guess, that like that is probably the next thing that we're going to be mak…”