Inbal Shani: evaluation

Transcript context

…What are the success metrics for Copilot? It's such an interesting problem of measuring efficiency gains and productivity gains for engineers. What metrics do you focus on to tell you this is doing what we want it to be doing? We are in a world that there are no right metrics. There is no one metric to rule them all. It's a combination of the things that you're looking to measure out of adopting AI. You can think about measuring code quality using AI. Can you improve the code quality? Can you improve the security of your code by introducing, for example, our GitHub Advanced Security using AI? So there's a lot of different metrics that if you combine them together are providing you with that developer productivity, and then there's developer productivity, developer collaboration, time gain, that if you're bringing them together are yielding the developer happiness. The biggest area of focus for us right now is the definition of productivity because we have so many users that are coming to GitHub and they're looking for that productivity gain. But one of the things we're very conscious of as we continue evolving Copilot and AI across the software development lifecycle across all the tools, productivity is not the right metrics against each one of these components. When we're implementing AI to GitHub Advanced Security, writing more secure code is the right element. It's like how many secrets were we able to prevent from leaking? How many issues in the code we're able to detect and find and fix before you ship that? So I think that the world of the metrics for AI is really a big one. We are working a lot with our customers as they're asking the same questions and they're running their own experiments within their companies to figure out what is it that they want to measure. The most easiest one is time, but time is, it's funny what I'm going to say, but time is not quantifiable as a success metrics because you can write really bad code really fast. So it's really about how are we taking time and translating it to efficiency, to productivity, to something that are harder to measure, and then how does that lead to develop core happiness, which is another harder metrics to measure. So the combination of all these input metrics leading to eventually developer happiness is what we're focusing on. Yeah, it's so interesting because engineers could end up spending more time coding because they're enjoying it more. They're getting more done. Also, more lines of code isn't a good metric because you don't want to know if those are good lines of code. That's a classic way to not measure engineers well. So to me, it feels like developer happiness is the ultimate metric. We had Nicole on the podcast who came up with this framework, DORA, that's an interesting way of just measuring developer experience, developer happiness.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.