Clark: Data availability and engineering form the base of AI's needs hierarchy
Scott Clark · a16z Podcast | AI, from 'Toy' Problems to Practical Application · Jan 2, 2019 · at 31:23
Scott Clark, co-founder and CEO of SigOpt, outlines a framework for enterprise AI adoption, placing basic data readiness at the foundation and optimization at the top.
0:00 / 0:25exact quote · 25.7s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“The data problem is the first, like, layer in Maslow's hierarchy of AI. Like, you need to actually have the data. Then you need to be able to understand the business context of what you're aiming for and Do a lot of the data engineering to make sure that you can actually leverage that data in some way that it's not just in filing cabinets somewhere. Then there's the tooling and the infrastructure for actually like training some of these more sophisticated algorithms. And then actually optimization just sits at the very top.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Scott Clark
Clark: AI testing should use many weak estimators to detect behavioral differences
“Instead of trying to come up with a small number of strong estimators for performance, where we want to be able to conclusively say A is better than B, Instead, what we want is a large number of potentially weak estimators to be able to determine whether or no…”
Clark: Untuned deep learning models perform worse than tuned simple algorithms
“An untuned, sophisticated system will underperform a tuned simple system.”
Clark: System trust, not performance, limits enterprise AI value
“The thing that's holding back people getting value from these AI systems is not performance. It's not about squeezing out that last half a percent from some eval function or some performance metric. It's about being able to confidently trust these systems.”
Clark: High-level LLM evaluations mask undesired AI system behaviors
“We're seeing people do the exact same thing again today with LLMs, where they're focusing on these high-level metrics, these end outputs, these performance evals, and that ends up masking all of these potentially undesired behaviors within the system itself.”
Clark: Evaluating end-to-end AI performance hides upstream system failures
“And what I think a lot of firms are running into right now is if you're only looking at that last step, if you're only looking at the system's performance as a whole, it can be very difficult to understand when, where, and why behaviors are shifting within thi…”