Shawn Wang: recommendation

Transcript context

…Yeah, we'll link on the episodes. Yeah, this is like a review of all the episodes that we've done, which I have in my head. This is one of the slides that I did on my Dev Day talk. So we're moving on from benchmarks to capabilities. And I think I have a useful categorization that I've been kind of selling. I'd be curious on your feedback or edits. I think there's basically like, I kind of like the thought spot. MMLU is a model of what's mature, what's emerging, what's frontier, what's niche. So mature is stuff that you can just rely on in production, it's solved, everyone has it. So what's solved is general knowledge, MMLU. And what's solved is kind of long context, everyone [01:10:00] has 128K. Today O1 announced 200K, which is Very expensive. I don't know what the price is. What's solved? Kind of solved is RAG. There's like 18 different kinds of RAG, but it's mostly solved. Bash transcription, I would say Whisper, is something that you should be using on a as much as possible. And then code generation, kind of solved. There's different tiers of code generation, and I really need to split out single line autocomplete versus multi file generation. I think that is definitely emerging. So on the emerging side, tool use, I would still kind of solve. Consider emerging, maybe, maybe more mature already. But they only launched for short output this year. Yeah, yeah, yeah. I think emerging is fine.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.