Gina Gotthilf: evaluation

Transcript context

…Awesome. Awesome. So I think that's a really important takeaways. Just there's a lot of time to do stuff and don't stress if things aren't moving as fast as you want. I'm curious, what's a mess-up or a big mistake maybe that you made or your teammate that was like, "Oh wow, that was a big waste of time." Yeah, look, a lot of things didn't work out. More than 50% of our A/B tests didn't work out. We made bets that didn't make sense. I will say though, that in the spirit of A and B sides, I, and I think in general, we are really good at forgetting the B stuff. I talk so much about all the stuff that worked that it's hard to remember all of those moments that didn't actually work. And the thing that I tend to talk about, which is this mistake that we made as the growth team is almost like one of those, when you get asked in an interview, what's your biggest weakness, and you're like, "I'm a perfectionist." It's like one of those things that actually makes you sound good because it's the story about how my team really wanted to implement badges. We spent a lot of time playing all the games that were popular at the time, trying to understand how those gamification growth hacks that we could find in those apps would potentially overlay onto Duolingo and how we would do that. And badges was just pervasive in all of the top games. And so it seemed like a no-brainer, but since we ranked all of our experiments in terms of ROI and return being like how many users we think we're going to get from this DAUs, and time investments, it never made sense to focus on this because we thought that the time sink would be too high. So I actually ended up not letting the team run this experiment for six months so that we focused on lower hanging fruit. So that's a mistake on my end. Then we decided to run this experiment in the most lean way possible. We're like, "You know what? There's MVEs. There's minimum viable experiments. We don't have to run a whole badges thing. We can just do something more simple and actually see if that leads to growth in an interesting way, and then we'll know." And we ran this very simple experiment that was like, you signed up and then you get a badge. And it was like this girl with a balloon. I don't know, she was happy or whatever. And of course in retrospect, it led to no results, because no one is proud of signing up. It's not an exciting moment, and you don't even have badges to collect. You can't show it to other people. None of the things that make badges compelling were there, but we were like, "Okay, well, we tested it didn't work." And then we moved on. So we moved on for another, I don't know, eight months and we didn't look back. And then when we did look back, first of all at that point we discovered that we hadn't been dogfooding, which also was embarrassing. Looking back, we hadn't been dogfooding in the growth team. We just come up with hypotheses. We were super careful about prioritizing them and making sure that we were doing the best possible write ups and all these things. But the dogfooding piece, I didn't come from a product background. I was a marketer and I didn't really understand the term dogfooding, but when we thought, we had a conversation, we were like, "You know what? ogfooding piece, I didn't come from a product background. I was a marketer and I didn't really understand the term dogfooding, but when we thought, we had a conversation, we were like, "You know what? If we had just tested that, we would've all known that this was a super lame badge." And I was like, "Why are we not testing our experiments?" And so that became part of our practice. It's still relevant. I just had a conversation yesterday with engineers at Latitud. I haven't explained what we built yet, where we're building it. Maybe we'll get there, but I was talking yesterday, [inaudible 00:55:35] at Latitud, and they're awesome. In terms of product team, we have the number eight employee at NewBank. You might've heard of NewBank, but it's this massive banking fintech in Latin America. And we have people from other fintechs. And then we have this guy who was a lead PM at Twilio, and I was explaining to them why we should be dogfooding. And they were all like, "Oh yeah, we should dogfood." It's just easy to forget stuff like that. So that was a mistake that we could have probably gotten to the growth that we got to with badges much earlier on. And not only did we get to growth with badges, but it became this amazing treasure trove of opportunity because once you have badges and people want them, you can now ask people to do anything. Go find friends, go buy things, whatever it is. And so we impacted almost all metrics across the company positively, including some we hadn't expected, but it's easy to talk about a mistake that ended up being a win. So that's why I compared it to the interview thing in the beginning. But we tried making Duolingo a social app really early on and failed. It was called Dual Duels. Dual Duels. You could duel.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.