Tay: Complex architecture modifications fail due to an implementation lottery
Yi Tay · The 10,000x Yolo Researcher Metagame — with Yi Tay of Reka · Jul 5, 2024 · at 1:11:30
Yi Tay, co-founder of Reka, discusses why certain AI architecture ideas succeed while others are ignored by the research and engineering community.
0:00 / 0:31exact quote · 31.2s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“A lot of architecture changes, right, the moment they are, like, tedious to implement, like, nobody, like, SuiGuru is a simple thing, right? [4306] Yi Tay: Just split it and then get it. [4307] Yi Tay: It's a very simple thing to implement. [4309] Yi Tay: Maybe that's why it's caught on, because it has, like, a additional boost that's for the simplicity of it, right? [4314] Yi Tay: So there's also, like, a bit of, like, [4315] Yi Tay: Implementation Lottery, if you will, right? [4317] Yi Tay: A little bit of like, if you propose like some very complicated thing for like, easy in PyTorch. [4323] Yi Tay: Yeah, nobody will use that, right?”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Yi Tay
Yi Tay: Gap Between Closed AI Labs and Open-Source Is Increasing
“I think the gap is definitely increasing.”
Yi Tay: IR and RecSys research lags significantly behind NeurIPS and ICML
“Also the IR community and the retrieval community is also like always behind the mainstream. And then now it's just probably gotten even more worse because of ILM and stuff. So, okay, I'm getting into Hottick territory, but it's just, like, certain conferences…”
Tay: Frontier AI researchers cannot maintain standard nine-to-five work-life balance
“You cannot be, like, checking out on, like, Friday, Saturday, Sunday, and, like, work at, like, nine to five if you want to, like, Make progress, or like, some people are just so good at detaching, like, ok, like, you know, like, eight pm, I'm not going to, my…”
Tay: Long context architecture is the future of AI over RAG
“And, yeah, I mean, I think long context is definitely the future, rather than rec. But I mean, they could be used in conjunction, like,”
DeepMind Abandoned AlphaProof to Run Gemini End-to-End for IMO Math
“We wanted to try to, like, use, actually use Gemini as an end-to-end model. Basically, no, no second system with alpha proof. No second system. In, text out.”
Tay: Most specialized tools will be subsumed directly into model parameters
“Then the most I can see in the future is there'll be a model then that, that is, there's something that really cannot be subsumed by a model. Then you just use a tool or something, right? But my prediction is that I think most things can be subsumed by the mod…”