Sander Schulhoff: recommendation

Transcript context

…Got it. How much context is too much context? You call it additional information, so let's just call it that. Should you just go hog wild and just dump everything in there? What's your advice? I would say so. Yeah, that is pretty much my advice, especially in the conversational setting. I mean, frankly when you're not paying per token and maybe latency is not quite as important, but in that product- focused setting when you're giving additional information, it is a lot more important to figure out exactly what information you need. Otherwise, things can get expensive pretty quickly with all those API calls, and also slow. So latency and costs become big factors in deciding how much additional information is too much additional information. And so, usually I will put my additional information at the beginning of the prompt, and that is helpful for two reasons. One, it can get cached. So subsequent calls to the LM with that same context at the top of the prompt are cheaper because the model provider stores that initial context for you as well as the embeddings for it. So it saves a ton of computation from being done. And so that's one really big reason to do it at the beginning. And then the second is that sometimes if you put all your additional information at the end of the prompt and it's super, super long, the model can forget what its original task was and might pick up some question in the additional information to use instead. With the additional information, if you put at the top, do you put in XML brackets?…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.