Public evidence record
Published podcast speaker
Claim ledger
8 transcript-backed records
01 / prediction
“Yeah, so I think that's a very, very good question. So I don't think we are running out of data on Earth.”
02 / evaluation
“Yeah, 30 trillion tokens. So I think what's exciting about Red Pajama V2 is not only the number of tokens, but we start to kind of learn from Red Pajama V1.”
03 / observation
“So, but the one problem of Lama is the data recipe is being described in a pretty detailed way in the paper, but the data is actually not there. So, and our original thinking is how about we take the recipe and we try to do our best effort reproduction and try to put it out, such that we can learn from our mistakes in the reproduction together, right?”
04 / belief
“If you go back to the original Google paper of federated learning, I think that's very different from what people are talking about today when they say federated.”
05 / belief
“If you go to AWS, you do know which region you are in, right? So I think one thing that we are trying to do is you have this disaggregated cloud, not only about location or geographically where they are, but about this reliability and also this diversity of this infrastructure.”
06 / evaluation
“It's just very expensive. So I think what those like state-based models trying to do is try to push the size of the state, right?”
07 / prediction
“So, and how can you make the whole thing really, really fast? So I think for the next couple years, yeah, we will see a whole bunch of new embeddings maybe of different size and much, much faster than today.”
08 / evaluation
“I don't know, but I wouldn't take that for granted that they should be the same, right? So that's one of the hypothesis that, so we have no opinion on that because I think that's the result of the study, not the assumption.”