Julie Zhuo: evaluation

Transcript context

…I'm one of those people that says thank you to the Waymo when I leave and just like thanks ChatGPT when I'm in voice mode, just like thank you, that was really helpful. So along these lines, I know there's a lot of ways to go here, but just in terms of skills that are important to a manager, Which do you think are most valuable to develop in working with agents in AI systems? I think about things like clarity communication, just like what comes to mind when you think about like, here's the things you want to double down on as you're learning to be manager that will also help you be really good at AI tooling and working with agents. The first is defining the goal and defining the outcome and being really, really crystal clear on what does success look like. I mean, there's obviously lots and lots of, like if you ask a company to do this, we'll know that this is challenging for humans, right? I think a lot of times when you talk about why is alignment so difficult at a big company, it often comes down to this question, which is, Different people may have different pictures of what success looks like. And even if I describe in human words, oh, you know, Lenny, I want to build this product and it's going to be amazing. Or this podcast episode, which you asked me, well, lots of people to hear it and take away things. That's very general. Like, how do we get even more specific so that we know without question whether we've hit it or not? This is actually a really, really difficult problem. It's a difficult question for us because again, we tend to think very high level. So figuring out how to boil it down so that an agent can really understand what success and failure looks like is a lot of the game. And I think this also relates to things like, well, that's why we have to write evals. And that's why they're so important because they're helping us understand what is the objective criteria. And these days I work in data and my company is all about trying to automate data analysis. And the forever question goes, the whole point of data and the whole point of metrics and KPIs is we're trying to put a little bit more of an objective measure or get as crystal clear as possible about what success looks like. And I think it's really an art more than it is like a science, but that's like the first thing. I think if you're really unclear about what success looks like, the prompt, you're probably not going to get the most amazing work. I think that's true for managing teams and it's very much true for managing AIs. Okay, so let me actually flip this on you and talk about another trend that we're seeing, which is this kind of flattening of orgs, managers being let go, everyone's becoming an IC again. I just had the CEO of Airtable on the podcast and his whole shtick was that CEOs have to become ICs again. He's coding more than he's ever coded again. And his feeling is you have to know what's possible by being in the weeds in order to figure out what your product should be. I also just saw a stat that Google let go of so many of their middle managers of smaller teams. It's just like this flattening trend. So do we even need managers, I guess, is one question in the future. And then just thoughts on how this will play out.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.