Tamir: Direct OpenAI or Anthropic APIs outperform cloud intermediary platforms
Mike Tamir · How Shopify Implements AI Across Sales and Product with the Head of AI at Shopify · Apr 26, 2024 · at 14:53
Mike Tamir (Head of AI at Shopify) discusses the trade-offs between accessing foundation models directly versus using cloud aggregators like AWS Bedrock or Azure.
0:00 / 0:18exact quote · 18.6s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“Using a first party solution with say OpenAI or with Anthropic is going to have different performance, different latency than if you do it via an intermediary, like a cloud. So you maybe get some simplicity by not having to onboard different vendors and different, different support, but there is going to be an impact in terms of performance.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Mike Tamir
Tamir: On-premise infrastructure is cheaper for AI training if already established
“Maybe you buy your own infrastructure, which is also a cheaper answer. If you have, if you already have an infrastructure, an on-prem infrastructure system, that's really going to depend on what your AI training expectations are anticipated.”
Tamir: Serving 70B open-source LLMs forces teams to buy reserved GPUs
“What if you want to do something like a seventy billion Lama or minstrel or mixed or any of these, those are going to be very hard to get on an ad hoc basis. And you're going to end up having to pay the price for a reserve instance, if you want to be able to s…”
Tamir: E-commerce platforms should use dynamic discounts, never dynamic pricing
“You never want to do dynamic pricing. You want to do dynamic discounts as a product framing bit of advice”
Tamir: Search must balance direct and complementary relevance for optimal revenue
“And maybe what you would want to do is have one metric that shows for direct relevance and another that shows complementary relevance, right? And then it is somewhat a product decision of how much you want to balance these, but it's also something that you can…”
Tamir: Successful ML teams require serving engineers and embedded product managers
“You need to have engineering for serving. You need to have your ML scientists and engineers, and you need to have product. You need to have product every year in the trenches with your ML builders.”
Tamir: Per-token commercial LLM costs are unscalable for every enterprise query
“And if you pay a commodity LLM, which right now does usually beat out open source LLMs, you're going to end up paying per token and you can't scalably, if you have the right size of a business, do this for every query or every kind of use case.”