Nathan Labenz: belief

Transcript context

…Yeah, that's a great question and I think one that lots of folks think about. Maybe my first question for you would be, How were the results you were getting from your existing process? So you mentioned, first of all, that these days you mostly just do prompting, but when you were doing fine-tuning with SFT, do you feel like you were seeing the models improve substantially on that? Do you feel like it was-- and then this is a very high bar, which I would imagine wouldn't clear, but did you feel like it was behaving as well as your creative team and matching the quality of those examples it was given post-training? I would definitely not say it was matching the best work that our creative team could do, but definitely a notable improvement on the base model. I would say our typical complaint was probably most often that, and this would definitely vary through different generations, but more recently it was like able to do the job perfectly well, so to speak. And I think that's true today too with prompting, but few moments where you're like, damn, that was awesome. Incredible turn of phrase or, you know, nailed it, really nailed it, you know, in the way that sometimes you just get something from the creative team that's like, oh, wow, like that was a really good creative idea that impressed me, surprised me, delighted me. I wouldn't say we see too much of that coming from models even today. Okay. Yeah. Okay. So that was going to be my next question. So yeah, even with the latest Frontier models, you know, that sort of spark or wow moment, it sounds like is not something you see commonly.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.