Speakers in the public record
Claim mix
belief 11evaluation 8recommendation 5
Evidence policy
Every row below preserves an exact excerpt. Identified speakers are linked; unresolved voices are labeled and excluded from people counts.
Claim ledger
The useful parts, with receipts.
24 published records
“You know, use Mistral to rephrase some existing part of your data sets, generate more tokens, anything like that, or any other form of synthetic data that you choose to mention? I think you also mentioned the large world model paper, right?”
- Publisher
- Latent Space
“One thing I see in the recent papers that have been coming out is this sort of concept of multi-stage training data. And if you're doing full fine tuning, maybe the move or the answer is don't train 500 billion tokens on just code, because then yeah, it's going to massively overfit to just code.”
- Publisher
- Latent Space
“Yeah, I think, you know, the one thing that makes this sort of generative AI era very different from the sort of data science-y type era is that it is very non-deterministic and it's hard to control.”
- Publisher
- Latent Space
“There's no minimum, I would say, or at least, I can't make such a strong statement as to say that that does not exist.”
- Publisher
- Latent Space
“I think that's how one of your tweets, you trained on about 200 million tokens for the AP model to the context extension.”
- Publisher
- Latent Space
“I think provably solvable is always something that I know is extremely difficult, but from a heuristical standpoint, as well as like having like some sort of statistical efficiency on like how you can converge to the downstream tasks and improve the performance that way in a targeted manner, I do think there are papers that try to do that.”
- Publisher
- Latent Space
“Yeah, I think Alibi, we haven't compared with that one specifically, mostly because I've noticed some of the newer architectures don't actually employ it a lot.”
- Publisher
- Latent Space
“I think people that are watching Fallout on Amazon Prime right now can maybe feel nostalgia just looking at it.”
- Publisher
- Latent Space
“If I think about the progression of the evaluations, it start to force the model to actually understand like the totality of the context.”
- Publisher
- Latent Space
“You know, people always trying to define the difference between ML and AI. And I think in AI, we definitely care a lot more about out of domain generalization and that's all under the umbrella of learning, but it is a very specific kind of learning.”
- Publisher
- Latent Space
“I think people understand that with stable diffusion, you have these LoRa patches for different types of styles.”
- Publisher
- Latent Space
“Any other things that you want to bring up, maybe how people are using gradient, anything like that, I think that will help have a clearer picture for people.”
- Publisher
- Latent Space
“I think being on Twitter and looking at all these new headlines is really helpful, but then it only gets you a very surface level understanding.”
- Publisher
- Latent Space
“I think there's definitely a few of us ex-finance people moving into tech and then finding ourselves gravitating towards data and AI.”
- Publisher
- Latent Space
“Like how diverse are your embeddings space to the original corpus of the model, and then train on top of that to retain its abilities. And then finally, for the chat data set, making sure that it's attending to all the information that would be expected to really stretch its capabilities, because you could create like a long context data set where every single time the last 200 tokens could answer the entire question, and that's never going to make the model attend to anything.”
- Publisher
- Latent Space
“Because nothing against submitting research papers to like ICLR or ICML, knowing the state of the art, those are like six months late, right?”
- Publisher
- Latent Space
“Underrated specific instance would be the DeepSeek paper where I'd never seen it before, but the multi-head latent attention. That was really unexpected to me because I thought I'd seen every way that people wanted to cut mixture of experts into interesting ways.”
- Publisher
- Latent Space
“You can get around the long context sometimes where you can do retrieval augmented generation or you do hierarchical recursive summarization, whereas evolution in like a session, because that state variable could undergo pretty rapid changes.”
- Publisher
- Latent Space
“We definitely have a call to action to get more people to work together with us for long context evaluations. That is sort of the it topic throughout even meta or Google or any of the other folk are focusing on because I think we lack an understanding of that within the community.”
- Publisher
- Latent Space
“We used GPT-4 to rephrase certain aspects of the chat data, reformatting it or kind of generating new types of tokens and language and types of data that the model could see.”
- Publisher
- Latent Space
“Obviously, you know, you don't build everything that people ask you to build, but we know what's useful, right? Because I think that you're totally right there.”
- Publisher
- Latent Space
“Yeah, I think there's a huge resurgence in what I would call model alchemy to a certain extent, because you're taking all of these LoRa's and you're mixing them together.”
- Publisher
- Latent Space
“Yeah, in terms of, you know, all the literature out there, I would say, honestly, it's probably still TBD as to like the trade offs between the approach we did, which is more of a curriculum learning approach after the fact versus inherently training a model with a long context throughout, because I just don't think people have looked at the scaling properties of it in deep detail.”
- Publisher
- Latent Space
“We do have historical precedent, where the original code bomb was trained further from Mama 2, and it just lost all its language capability, basically, right? So I don't want to call that project like deem it as a failure, but it wasn't a really successful generalization exercise, because, you know, these models are about flexibility and being like generic to a certain extent.”
- Publisher
- Latent Space