Andrew Lee: evaluation

Transcript context

…Other, so you mentioned, I think five providers, Anthropic, OpenAI, Gemini, Deepseek, and Kimmy. Not on that list were Grok, whatever the new meta models are called, and GLM or Minimax. Like, are there any other, how are you kind of, where are you drawing the line? How are you thinking about who's in and who's out? It is so hard to stay up to date on this stuff. We have the ability internally to test models pretty quickly. It's harder to actually ship things in production because, for example, the way thinking blocks work is different across different providers. And if you have bugs, you might just tune the prompts and things. So we haven't shipped that many, but we've tested GLM internally. We've tested the Google models, Kimi, DeepSeek, probably some others I'm not thinking of. I think right now, and most of this is initially vibes, right? You go in there and you play around with it and you're like, Is this close enough to the frontier that we want to put some effort in here? And usually the answer is no. And I think the ones that we've, like Kimi, DeepSeek, and Google, and plus obviously OpenAI are the ones where like, Okay, actually this is pretty close to the frontier, so it's worth doing. But there'll probably be others in that list. I have not been paying a huge amount of attention to Grok. Maybe I should be paying more attention to them. I don't hear a lot of other developers using their models, but they sure seem to be investing a lot. So I don't know, maybe that'll change. Yeah, we can't, in my view, we can't count Elon out of any race until he bows out himself. So, but I, would also agree I don't use it much. I just had occasion to use it a fair amount while riding in the Tesla over the last week. And it's not bad, you know, and the voice mode is pretty good. Definitely still feels a little that's also part of, you know, it's it's not just the model, it's also the integration. But I would say my experience using Grok in the console of the Tesla is definitely rougher, you know, than my experience using Anthropic and OpenAI and Google models.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.