Cheah: AI Engineers Do Not Need ML Math to Build Products
Eugene Cheah · RWKV: Reinventing RNNs for the Transformer Era · Aug 31, 2023 · at 1:33:57
Eugene Cheah (UIlicious CTO / RWKV contributor) explains the practical skills required for application-layer AI engineers versus research scientists.
0:00 / 0:51exact quote · 51.3s
▶ Watch the full episode on YouTube → 720p mp4 · rendered on demand · StarZero watermark
“Frankly, for an AI engineer, you don't need it. You, your main thing that you needed to do was to, frankly, just play around with ChatGPT, or all the alternatives, be aware of the alternatives, because be very mercenary, swap out to Cloudia if it's better for you, or swap out to an open source if it's better for you, and just play around the prompts. Learn, learn prompting techniques like one shot, two shot, few shots and then from there on, you can start building your agents, stacking, stacking your prompts, and in sequences, and stuff like that, and you are able to build applications that do anything in terms of the AI space, and all this without knowing all this nerdy stuff, or the hard engineering, because that's all you really need to actually build a product for the user.”
quote is from the automated transcript, cleaned for reading: filler sounds and stutters are removed, nothing is rephrased. names can be misheard (the analysis reads context, assessments check outside sources). how →
More from Eugene Cheah
Cheah: Hybrid SSM-transformer models outperform pure baselines of both
“None of us understand why a hybrid with a state-based model, the RWA state space, and transformer performs better than the baseline of both. It's like when you train one, you expect, and then you replace, you expect the same results. That's our pitch. That's o…”
Cheah: Non-positional attention architectures remain stable beyond trained context
“One key advantage of this alternate attention mechanic that is not based on token position is that the model don't suddenly become crazy when you go past the eight K training context or a million context. It is actually still stable. It's still, it's able to r…”
Cheah: Standard Transformers Will Never Scale to Ten Million Tokens
“I think what was quick, I think it was rather quick after I concluded that transformer as it is will not scale to ten million tokens.”
Foreign Language Data Degrades English Benchmark Scores on Small LLMs
“Adding in a foreign data set is actually a loss, because once you're below a certain param count, so we're talking about the seven important, right? The more you add that's more in line with your evals, the more it will degrade, and they just exclude it.”
RWKV Matches GPT-NeoX Performance at Equal Parameter and Data Scales
“RWKV is a modern recursive neural network with transformer-like level of LMM performance, which can be trained in a transformer mode. And this part has already been benchmarked against GPT-NeoX in the paper, And it has similar training performance compared to …”