Alessio Fanelli: evaluation

Transcript context

…Yeah, I'm sure they do. As soon as we, as soon as we do it, I'm like, Once we've wrapped up what we're doing in San Francisco, I'm sure we'll give it a go. I spoke to some guys today, actually, about fine tuning 405B, who might be able to allow us to do it very, like, very easily. I don't want to have to basically do all the setup myself. So, yeah, that might happen sooner rather than later. Anything from the releases today that you're super excited about? So prompt caching, I'm guessing when you're like dealing with a lot of codebases, that might be helpful. Is there anything with vision fine tuning related to like more like UI related development? Yeah, definitely. Yeah, I mean like we were talking, it's funny, like my co founder Sam, who you've met, and I were talking about the idea of doing vision fine tuning. Like, way back, like, well over a year ago, before Genie existed as it does now when we, when we collected our original dataset to do what we do now whenever there were image links and links to, like like, graphical resources and stuff, we also pulled that in as well. We never had the opportunity to use it, but it's something we have in storage. And, again, like, when we have the time, it's something that I'm super excited, particularly on the UI side. To be able to, like, leverage, particularly if you think about one of the things, I mean, not to sidetrack, but one of the things we've noticed is, I know Swebench is, like, the most commonly talked about thing, and honestly, it's a very, it's an amazing project, but, One of the things we've learned the most from actually shipping this product to users is, It's a pretty bad proxy at telling us how competent the model is, so, for example, When people are doing, like, React development using Genie, For us, it's impossible to know whether what it's written has actually done, you know, done what it wanted to. So at least even using, like, the fine tuning provision to be able to help eval, like, what we output is already something that's very useful. But also, in terms of being able to pair, here's a UI I want, here's the code that actually, like, represents that UI, is also going to be super useful as well, I think. In terms of generally, what have I been most impressed by? The distillation thing is awesome. I think we'll probably end up using it in places. But what it shows me more broadly about OpenAI's approach is they're going to be building a lot of the things that we've had to hack together internally, in terms from a tooling point of view, just to make our lives so much easier. And I've spoken to, you know, John, the head of fine tuning, extensively about this. But there's a bunch of tools that we've had to build internally for things like dealing with model lineage, dealing with dataset lineage, because it gets so messy so quickly, that we would love OpenAI to build. Like, absolutely would love them to build it. ernally for things like dealing with model lineage, dealing with dataset lineage, because it gets so messy so quickly, that we would love OpenAI to build. Like, absolutely would love them to build it. It's not, it's not what gives us our edge, but it certainly means that then we don't have to build it and maintain it afterwards. So, it's a really good first step, I think, in, like, the overall maturity of the fine tuning product and API in terms of where they're going to see those early products. And I think that they'll be continuing in that direction going on. Did you not, so there's a very active ecosystem of LLLmaps tools. Mm hmm. Did you not evaluate those before building your own?…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.