Lenny Rachitsky: belief

Transcript context

…So recently I was working on a project for a medical coding startup where we were trying to get the GenAIs, GPT‑4 in this case, to perform medical coding on a certain doctor's transcript. And so I tried out all these different prompts and ways of showing the AI what it should be doing, but at the beginning of my process, I was getting little to no accuracy. It wasn't outputting the codes in a properly formatted way. It wasn't really thinking through well how to code the document. And so what I ended up doing was taking a long list of documents that I went and coded myself, or I guess got coded, and I took those and I attached reasonings as to why each one was coded in the way it was. And then I took all of that data and dropped it into my prompt, and then went ahead and gave the model a new transcript it had never seen before. And that boosted the accuracy on that task up by, I think, 70%. So massive, massive performance improvements by having better prompts and doing prompt engineering well. Awesome. I'm in that bucket too. I just find there's so much value in getting better at this stuff, and the stuff we're going to talk about is not that hard to start to put some of these things in practice. Another quick context question is just you have these two modes for thinking about prompt engineering. I think to a lot of people, they think of prompt engineering as just getting better at when you use Claude or ChatGPT, but there's actually more. So talk about these two modes that you think about. So this was actually a bit of a recent development for me, in terms of thinking through this and explaining it to folks. But the two modes are, first of all, there's the conversational mode in which most people do prompt engineering. And that is just, you're using Claude, you're using ChatGPT, you say, "Hey, can you write me this email?" It does a poor job, and you're like, "Oh, no, make it more formal," or, "Add a joke in there," and it adapts its output accordingly. And so I refer to that as conversational prompt engineering because you're getting it to improve its output over the course of a conversation. Notably, that is not where the classical concept of prompt engineering came from. It actually came a bit earlier from a more, I guess, AI engineer perspective where you're like, "I have this product I'm building. I have this one prompt or a couple different prompts that are super critical to this product. I'm running thousands, millions of inputs through this prompt each day. I need this one prompt to be perfect." And so a good example of that, I guess going back to the medical coding, is I was iterating on this one single prompt. It wasn't over the course of any conversation. I just take this one prompt and improve it, and there's a lot of automated techniques out there to improve prompts, and keep improving it over and over again until it's something I've satisfied with, and then never change it. And I guess only change it if there's really a need for it, but those are the two modes. One is the conversational. Most people are doing this every day. It's just normal chatbot interactions. And then there is the normal mode. I don't really have a good term for it. [inaudible 00:11:16]-…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.