tokenization
2 statements across 2 episodes · 1 bullish · 0 bearish · 2 people on the record · first statement Sep 7, 2023 by Andrew Feldman · across every show →
Everything said about tokenization, oldest first
Sep 7, 2023
Feldman: Poor tokenization schemes heavily bias foundation models toward English
“One of the things we learned was that it's really important taking into account
The language that has fewer tokens, especially, you know, Arabic, Hebrew, Hindi, their characters sometimes require two bits rather than one bit, and so the tokenization scheme has…”
May 16, 2024 positive
Suno builds on transformers and focuses innovation on audio tokenization
“We don't make it a secret that these are just transformers. This is somewhat our backgrounds doing text before, but also transformers scale nicely. A lot of work ends up being done for you by the open source text community, which is always really nice. We can …”