StarZero All shows Try the podcast clipper →

Dan Fu

Assistant Professor, UC San Diego · 1 appearance on the record.

computed by AI from the episodes · how this works → · full disclaimer →

academicscientistengineerexecutive@realDanFu ↗danfu.org ↗

Dan Fu is an Assistant Professor of Computer Science and Engineering at UC San Diego and VP of Kernels at Together AI. He completed his Ph.D. at Stanford University and focuses on hardware-aware machine learning systems, efficient sequence architectures, and GPU performance engineering.

8statements → 4claims → 1claims resolved → 4/5average certainty → 2.5/5average debate potential →

1 supported 0 partly supported 0 contradicted 1 not yet assessed 2 not checkable as stated how the 4 claims stand · each chip opens the sources

1 prediction · 3 assertions · 1 opinion · 3 insights · every statement was checked. The prediction and assertions are the 4 claims: statements the public record can support or contradict. 1 is resolved, 1 is not yet assessed, and 2 name no date, number or outcome precise enough to check. Everything else (opinions, insights, what ifs, disclosures) can never be settled by the record, so it carries no assessment.

The record, in short

What the tape says about how Dan argues and how the claims held up. Everything they said, and everything said about them, is in the tabs below.

Their most notable supported claim

Assertion Supported
Fu: Stanford and Arc Institute's DNA SSM Made the Cover of Science
“One of those gated, SSM gated states-based models ended up on the cover of science because a great group of folks went and trained some DNA models. So that's Michael Polley, Eric Yuen from Stanford and the Arc Institute.”
Dan Fu Dec 24, 2024 ▶ 17:31 2024 in Post-Transformer Architectures: State Space Models, RWKV [Latent Space LIVE! @ NeurIPS 2024]

Everything Dan Fu said on Latent Space that made the record, most notable first. Filter by type, assessment or year in the ledger →

Assertion Not checkable as stated
Fu: Embedding model quality barely matters for final RAG performance
“We had this experience over and over again where you could have any, an embedding model of any quality, so you could have a really, really bad embedding model, or you could have a really, really good one by, and by any measure of good, and for the final RAG ap…”
Dan Fu Dec 24, 2024 ▶ 33:00 2024 in Post-Transformer Architectures: State Space Models, RWKV [Latent Space LIVE! @ NeurIPS 2024]
Insight
Fu: Modern GPU compute primitives should be matrices, not floats
“We basically built a whole library just around this basic idea that all your basic compute primitives should not be a float, but it should be a matrix and everything should just be matrix compute.”
Dan Fu Dec 24, 2024 ▶ 30:29 2024 in Post-Transformer Architectures: State Space Models, RWKV [Latent Space LIVE! @ NeurIPS 2024]
Prediction Open · timeframe Dec 2029
Fu: Real-time long-context video generation cannot use quadratic attention
“You're certainly not going to do a giant quadratic attention computation to try to run that.”
Dan Fu Dec 24, 2024 ▶ 31:33 2024 in Post-Transformer Architectures: State Space Models, RWKV [Latent Space LIVE! @ NeurIPS 2024]
Insight
Fu: Efficient AI Architectures Are Dead on Arrival Without Hardware Co-Design
“Even if your model is theoretically more efficient, if somebody goes and runs it and it's two times slower one of the things that, that we've learned is that if you're in that situation, it's just going to be dead on arrival. So you want to be designing your a…”
Dan Fu Dec 24, 2024 ▶ 15:51 2024 in Post-Transformer Architectures: State Space Models, RWKV [Latent Space LIVE! @ NeurIPS 2024]
Opinion
Dan Fu: Nobody is actually submitting 2M token prompts into LLMs
“Nobody is actually putting in a two million context prompt into these models.”
Dan Fu Dec 24, 2024 ▶ 37:33 2024 in Post-Transformer Architectures: State Space Models, RWKV [Latent Space LIVE! @ NeurIPS 2024]
Assertion Not checkable as stated
Fu: AI21's Jamba Is the State of the Art Non-Transformer Model
“AI-II trained this hybrid MOE called Jamba that, that, that seems, that is currently the state of the art for these non-transformer architectures.”
Dan Fu Dec 24, 2024 ▶ 16:50 2024 in Post-Transformer Architectures: State Space Models, RWKV [Latent Space LIVE! @ NeurIPS 2024]
Insight
Fu: Changing one PyTorch line requires a week of CUDA development
“If we decided to change one thing in PyTorch, like one line of PyTorch code is like a week of CUDA code at least.”
Dan Fu Dec 24, 2024 ▶ 29:38 2024 in Post-Transformer Architectures: State Space Models, RWKV [Latent Space LIVE! @ NeurIPS 2024]
Assertion Supported
Fu: Stanford and Arc Institute's DNA SSM Made the Cover of Science
“One of those gated, SSM gated states-based models ended up on the cover of science because a great group of folks went and trained some DNA models. So that's Michael Polley, Eric Yuen from Stanford and the Arc Institute.”
Dan Fu Dec 24, 2024 ▶ 17:31 2024 in Post-Transformer Architectures: State Space Models, RWKV [Latent Space LIVE! @ NeurIPS 2024]

Appearances (1)

EpisodeDateSpeaking time
2024 in Post-Transformer Architectures: State Space Models, RWKV [Latent Space LIVE! @ Neu Dec 24, 2024 24m
Made with StarZero

Turn any episode into a week of clips.

This entire site, over 200 episodes transcribed, diarized, checked and made playable, runs on the StarZero media pipeline. Drop in your own episode and the podcast clipper finds the moments worth sharing, cuts them, captions them, and reframes them for every feed.

Try the podcast clipper Learn more