Dylan Field: evaluation

Transcript context

…I imagine, that was a long day and our interview started at the end. What happened with that launch? I know you guys had to pull some stuff back. I imagine it taught you a lot. What happened? What did you learn? We had this feature that internally we called First Draft, and for some reason we changed the name to Make Design, which first of all, by the way, wrong name. We never intended it to be like, here's your design, you're done. It was really a starting point and we knew that, and this was early on in our AI journey, and the approach was basically nothing with fancy training or user data. It was all about, okay, you've got an LLM assembling legal pieces, and doing that according to a prompt. It's very basic in the way we built it, and it gets to choose some pretty cool outputs so you could edit the outputs and change colors, typography, smooth parts of the theme. I think that the industry then, even though it wasn't that long ago, was in a very different place in terms of the conversation around AI than we are today, but also, people put us through as paces in the ways that we hadn't fully done. One of the things they found was that if you typed in make me a weather app, it would make you something that looked pretty much similar to the Apple weather app. Given that that was under our control and that was really about, we should have had better QA and really looked at all the subcomponents more closely, I felt like maybe I would've felt differently if it was, we had trained this model and now we got to tweak some of the ways that we're post-training or whatever. But with the approach we were using, I was like, this was preventable. This is a QA failure, and so I pulled it. It was actually during our second Config, because we did the main one and then we went to Singapore and did a second. If I was tired during the last podcast we did together, I was even more tired then because the Singapore time zone shift is brutal from SF. Yeah, I'm sure we could have had better communication about the way we did it, but I thought it was the right thing to do. I would've done the same if you teleported me back. Then, we were interested after we did a lot of QA. I think that maybe takeaways from that, first of all, you got to put it through as paces, especially when you've got a wide surface area that can be explored through something like this. You really have to understand what are the inputs, make sure you did the QA work, and pushing the product and the team to hold up that high bar. I actually do this QA work. That's a big problem for a lot of AI companies these days. They're just so non-deterministic, there's all this autonomy you got to give them. How do you do this? Do you work with someone else that does a bunch of work for you or is it a team that just is really good at AI QA?…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.