Prompt Caching

“Usually I will put my additional information at the beginning of the prompt. And that is helpful for two reasons. One, it can get cached. So subsequent calls to the LM with that same context at the top of the prompt are cheaper because the model provider store…”

“Anyone who's doing, who's working with a ton of context has to put them at the end because they're swapping them out. That's just how it is. I don't see any way around it.”