The Ledger
Every statement that passed quotation and attribution checks. Mix any filter with any other: certainty 1/5, debate potential 5/5, or both at once.
why aren't all 18 resolved? a statement only gets an assessment when the public record can support or contradict it. opinions and what-ifs never can, and 0 checkable ones are still open, waiting for their date. predictions held up or didn't; assertions are supported or contradicted. on every card: ▮▮▮▮▮ certainty · ▮▮▮▮▮ debate potential. speakers are clickable
Zhang: Meta Failed at Training MoE Models for Llama Series
“The reason why Lama open-sourced the MOE model, because I think they tried to train our MOE model, but they failed. So that, that's why they didn't open source MOE mode for Lama series.”
Zhang: XGrammar outperforms Outlines and is integrated into TensorRT-LLM
“And I think Xgrammar's performance is better than the outline's, and also in the TensorFlow RTLM, the latest release, TensorFlow RTLM also integrates Xgrammar as the backend for the constructed coding.”
Yining Zhang: DeepSeek V3 scores 94.6 on GSM8K, outperforming Llama 405B
“Yeah, I think even they use the FP-A to quantization, the benchmark result is very good, such as something like GSM-HK. The score is nearly 94.6. It's so high, you know. I think it's higher than every other open source AIM, even the LAMA 400 zero five billion.”
Zhang: Baidu and ByteDance Internal Models Use DeepSeek-Like MoE Architectures
“As far as I know, some companies such as Baidu or Baidu Dance, they are internal, the dominant AOM, they use the MOE architecture, and their ways, I think, is similar to the DeepSeq MOE model.”
Zhang: SGLang outperforms vLLM and has better usability than TensorRT-LLM
“I think for the common use case, maybe not, not the DeepSeq VIII, for the common use case, I think SGLAN's performance is better than FLM, and its usability is better than TensorFlow TLM.”
Zhang: SGLang achieved 3x throughput over vLLM in mid-2024 benchmarks
“At that time, I think its performance is maybe three times, is throughout, put it, three times than FLM.”
Zhang: DeepSeek-V3 is currently the leading open-source LLM
“Yeah, because DeepSeq VIII, I think, is currently considered the leading open source LLMs based on the benchmark results and the chat area results.”
Zhang: Llama 405B sees very few enterprise users compared to 70B
“I think at the base time, something like LAMA-Seventy-B is more common. I think LAMA-Seventy-B has released the 400 zero five billion weights, but I think there are just a few users use that.”
Zhang: vLLM does not support DeepSeek MLA while SGLang does
“Something like DeepSeq V-II, they proposed attention parent named MLA, multi-latent attention, and I think SGLAN is the only framework to support that. Maybe LightLM and TRTM also support, but VLM doesn't support.”
Zhang: DeepSeek team officially recommends SGLang as its inference engine
“That's why SGLAN is the recommended LLM engine by the DeepSeq team.”
SGLang was the first framework to support prefix caching
“At 2024, January, they support something like Redix cache. It's a prefix caching technology. I think SGLAN is the first framework that supports prefix cache.”
Zhang: TensorRT-LLM is blazing fast but difficult to extend
“And TensorFlow RTM, I think it's blazing fast. Its performance is so good, but it's not easy to do some secondary development. If you want to add some new feature, it's a little hard.”
Zhang: SGLang achieves higher cache hit rates using block size of one
“Redix cache, I think it's the technology of the prefix caching, and it is a special case for something like block size is one, you know, for VLM or for other frameworks, they use something block size, 32, and SGLAN use the block size one. I think if you use th…”
Zhang: DeepSeek-V3 cannot run on a single 8xH100 GPU node
“You need, I think 671 gigabytes for the weights, and you also need an extra memory for the KV cache, so it's not possible to run that on H-one hundred.”
Zhang: MoE Inference Optimization Will Be Essential in 2025
“So I think at this new year, the MOE inference optimization will be very essential, and yeah, Yeah, so important.”
Zhang: SGLang creators Lianmin Zheng and Ying Sheng work at xAI
“Lian Ming and Yin are the XAI's member of the technical staff.”