Laura Burkhauser: evaluation

Transcript context

…Yeah, interesting. Hard thing to automate. We have done some of that stuff. It's been a minute since I last did a version of that. I was also struck, and our original use case was a little bit different. So with Waymark, we have this basically TV commercial maker for small business. And we've, for a long time, had a tool that pulls in all the images that the small business has published. to their website or their Facebook page or whatever. And then we make this library. And that was a great convenience factor even five years ago when there was not much we could do with it beyond just be like, here's what we collected for you so you don't have to go collect it for yourself and upload it. But obviously with AI, there's a lot more we can do. But the aesthetic quality of an image was always a really hard thing to evaluate. Like in early days, you could caption it, but did it look good? Like models who had a hard time with that. I'm not even sure that you should be trying to automate that. I don't know. Like I went to this dinner with the CEO of Midjourney and he's like, the reason why we have the best, like still have aesthetically like the best image generation is because I have my thumb on the scale. And like Google just lets like some kind of democratic panel or automation decide what the best, what the best image is. And the best image is always some generic pretty blonde lady or whatever when you ask for something. I thought that was pretty funny. But all of that is to say that like I, This may be an unpopular opinion, so fun for you because it's a little bit controversial, but I don't think that you can't, you should underestimate the importance of vibes in like aesthetic evals. And just when we first built Studio Sound, which by the way, like it's still our internal model and we still think it's better than we recently evaled it against all the other new Studio Sound providers and we still chose ours, even though in other cases we have thrown ours out and taken another kind of model That's obviously better, but we kept Studio Sound. Studio Sound was originally built by a cellist who just had a really good ear for things, and he did what we might now be calling eval, but he like... Our original eval thing was this guy would listen to different models and be like, this is better, this is better. Now, when he left Descript, then we had to actually write an eval that was like, what are the 37 different things that make one form of background noise removal better than another form of background noise removal? But I don't know. I think that it's reasonable to say in something that is like primarily judgment-based, We're just going to have a human do this and we're going to have a human do this forever. Someone that we know that we've vetted as having good taste is going to make these decisions. Yeah. Okay. That's quite interesting. Could you do just a quick overview of the frontier model landscape as you see it? Like you started to a minute ago when you said nano bananas, best first face swap and then cling this other thing. Is there a like expanded version of that you would say is kind of, here's how users should generally orient themselves to their options.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.