Cat Wu: preference

Transcript context

…We've covered evals a bunch. There's this trend of just like, that is the future of product management is writing evals because essentially it's what does success look like? Okay, cool. Let me actually concretely define it and then we'll know. How much of your time are you spending writing evals, would you say? I think the importance of evals varies a bit based on the feature that you're working on and or like what the problem you're trying to solve is. So there are a lot of folks on our team who do spend a lot of time working on evals. We have a small pod of folks who collaborate very closely with research to more precisely understand our cloud code behaviors and what the largest areas of improvement are and trying to measure those pretty concretely. I personally jump into evals when there's a feature that I think needs a bit more product definition. And often the output of this is, okay, here are like five evals that I made. This is how you run them. These are the ones that succeed and these are the ones that don't. And this is like the prompt that I've used to increase the success rate. It varies a lot though, based on the exact feature. Not every feature needs it, but I think features such as memory benefit a lot from it. This point you made about people being very good at evaluating a model is so interesting. It's almost like a human eval of just like, okay, they understand where it's spiking or it's maybe lacking. Is there anyone specific that you want to shout out that's very good at this?…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.