Evidence receipt / evaluation
Published · transcript-backedReza Shabani: evaluation
3 May 2023 Latent Space Training a SOTA Code LLM in 1 week and Quantifying the Vibes — with Reza Shabani of Replit
“I mean, I think part of it is that there's a lot of comments and there's also a lot of natural language in, in a lot of code right.”
Source trail
Everything needed to verify it.
- Speaker
- Reza Shabani
- Attribution
- Verified speaker
- Claim type
- evaluation
- Recorded
- 3 May 2023
- Publisher
- Latent Space
Transcript context
…So this is a replica model. Same question. What is the square of bananas? Banana. And it answers unknown. And this being one of the, the thing that Amjad was talking about, which you guys are. Finding as a discovery, which is, it's better on pure natural language questions, even though you trained it on code. Exactly. Yeah. Hmm. Is that because there's a lot of comments in, No. I mean, I think part of it is that there's a lot of comments and there's also a lot of natural language in, in a lot of code right. In terms of documentation, you know, you have a lot of like markdowns and restructured text and there's also just a lot of web-based code on, on replica, and HTML tends to have a lot of natural language in it. But I don't think the comments from code would help it reason in this way. And, you know, where you can answer questions like based on instructions, for example. Okay. But yeah, it's, I know that that's like one of the things. That really shocked us is the kind of the, the fact that like, it's really good at, at natural language reasoning, even though it was trained on, on code. Was this the reason that you started running your model on hella swag and…
Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.