Baseten

“I think at the base time, something like LAMA-Seventy-B is more common. I think LAMA-Seventy-B has released the 400 zero five billion weights, but I think there are just a few users use that.”

“We use Base-Ten to run some I would say some LAMA, some BERT model for classification.”

“So we find it better to, like, okay, we're going to replace this, you know, we're going to replace this layer with a layer from another model that's using, like, GQA, for instance. And then just for the right training, you can get it to have the same acceptanc…”

“I think that we've increasingly seen a lot of demand for the sort of pay per token APIs just because everyone wants to try open models. And then once they find a use case that's really sticky then they move over to Dedicated.”

“Some of the GPU kernels that were on GLM-Five-two within our inference engine is written by GLM-Five-two. And the trace and the kernels were guided by GLM-Five-two as the driver.”

“It is possible that the model in which I quantized more information is going to perform better because the quantization errors have canceled out. And so what Joshua showed in his mathematical proof where he had like a verifier in is that you can predict which …”

“We have, like, in our endpoint, like, if a model was to output the same exact token, like, four plus times, we just call the generation, we say, like, oh, sorry, this, like, try again, or, like, we will re-process the request. Because we know then, like, if it…”

“The challenge with tool calling more and more seems to be that the companies want certain tool calling, which is a very sensitive thing to train. And because you're dealing with all of the JSON outputs, if it doesn't like close the end of the request in a very…”

“And more often than not, it's like way cheaper if you're pushing like millions of tokens per hour, if you just pay per hour instead of pay per token.”