hardware / uses
GPT-4
“We used GPT-4 to rephrase certain aspects of the chat data, reformatting it or kind of generating new types of tokens and language and types of data that the model could see.”
Public evidence record
Mark Huang
Published podcast speaker
- Claims
- 14
- Episodes
- 1
- Shows
- 1
- Named items
- 1
Books, apps, and tools
The evidenced stack.
Claim ledger
What Mark said.
14 transcript-backed records
01 / belief
There's no minimum, I would say, or at least, I can't make such a strong statement as to say that that does not exist.
“There's no minimum, I would say, or at least, I can't make such a strong statement as to say that that does not exist.”
- Speaker
- Mark Huang
- Publisher
- Latent Space
02 / belief
I think provably solvable is always something that I know is extremely difficult, but from a heuristical standpoint, as well as like having like some sort of statistical efficiency on like how you can converge to the downstream tasks and improve the performance that way in a targeted manner, I do think there are papers that try to do that.
“I think provably solvable is always something that I know is extremely difficult, but from a heuristical standpoint, as well as like having like some sort of statistical efficiency on like how you can converge to the downstream tasks and improve the performance that way in a targeted manner, I do think there are papers that try to do that.”
- Speaker
- Mark Huang
- Publisher
- Latent Space
03 / belief
Yeah, I think Alibi, we haven't compared with that one specifically, mostly because I've noticed some of the newer architectures don't actually employ it a lot.
“Yeah, I think Alibi, we haven't compared with that one specifically, mostly because I've noticed some of the newer architectures don't actually employ it a lot.”
- Speaker
- Mark Huang
- Publisher
- Latent Space
04 / belief
If I think about the progression of the evaluations, it start to force the model to actually understand like the totality of the context.
“If I think about the progression of the evaluations, it start to force the model to actually understand like the totality of the context.”
- Speaker
- Mark Huang
- Publisher
- Latent Space
05 / recommendation
Like how diverse are your embeddings space to the original corpus of the model, and then train on top of that to retain its abilities. And then finally, for the chat data set, making sure that it's attending to all the information that would be expected to really stretch its capabilities, because you could create like a long context data set where every single time the last 200 tokens could answer the entire question, and that's never going to make the model attend to anything.
“Like how diverse are your embeddings space to the original corpus of the model, and then train on top of that to retain its abilities. And then finally, for the chat data set, making sure that it's attending to all the information that would be expected to really stretch its capabilities, because you could create like a long context data set where every single time the last 200 tokens could answer the entire question, and that's never going to make the model attend to anything.”
- Speaker
- Mark Huang
- Publisher
- Latent Space
06 / evaluation
Because nothing against submitting research papers to like ICLR or ICML, knowing the state of the art, those are like six months late, right?
“Because nothing against submitting research papers to like ICLR or ICML, knowing the state of the art, those are like six months late, right?”
- Speaker
- Mark Huang
- Publisher
- Latent Space
07 / evaluation
Underrated specific instance would be the DeepSeek paper where I'd never seen it before, but the multi-head latent attention. That was really unexpected to me because I thought I'd seen every way that people wanted to cut mixture of experts into interesting ways.
“Underrated specific instance would be the DeepSeek paper where I'd never seen it before, but the multi-head latent attention. That was really unexpected to me because I thought I'd seen every way that people wanted to cut mixture of experts into interesting ways.”
- Speaker
- Mark Huang
- Publisher
- Latent Space
08 / recommendation
You can get around the long context sometimes where you can do retrieval augmented generation or you do hierarchical recursive summarization, whereas evolution in like a session, because that state variable could undergo pretty rapid changes.
“You can get around the long context sometimes where you can do retrieval augmented generation or you do hierarchical recursive summarization, whereas evolution in like a session, because that state variable could undergo pretty rapid changes.”
- Speaker
- Mark Huang
- Publisher
- Latent Space
09 / recommendation
We definitely have a call to action to get more people to work together with us for long context evaluations. That is sort of the it topic throughout even meta or Google or any of the other folk are focusing on because I think we lack an understanding of that within the community.
“We definitely have a call to action to get more people to work together with us for long context evaluations. That is sort of the it topic throughout even meta or Google or any of the other folk are focusing on because I think we lack an understanding of that within the community.”
- Speaker
- Mark Huang
- Publisher
- Latent Space
10 / recommendation
We used GPT-4 to rephrase certain aspects of the chat data, reformatting it or kind of generating new types of tokens and language and types of data that the model could see.
“We used GPT-4 to rephrase certain aspects of the chat data, reformatting it or kind of generating new types of tokens and language and types of data that the model could see.”
- Speaker
- Mark Huang
- Publisher
- Latent Space
11 / evaluation
Obviously, you know, you don't build everything that people ask you to build, but we know what's useful, right? Because I think that you're totally right there.
“Obviously, you know, you don't build everything that people ask you to build, but we know what's useful, right? Because I think that you're totally right there.”
- Speaker
- Mark Huang
- Publisher
- Latent Space
12 / evaluation
Yeah, I think there's a huge resurgence in what I would call model alchemy to a certain extent, because you're taking all of these LoRa's and you're mixing them together.
“Yeah, I think there's a huge resurgence in what I would call model alchemy to a certain extent, because you're taking all of these LoRa's and you're mixing them together.”
- Speaker
- Mark Huang
- Publisher
- Latent Space
13 / evaluation
Yeah, in terms of, you know, all the literature out there, I would say, honestly, it's probably still TBD as to like the trade offs between the approach we did, which is more of a curriculum learning approach after the fact versus inherently training a model with a long context throughout, because I just don't think people have looked at the scaling properties of it in deep detail.
“Yeah, in terms of, you know, all the literature out there, I would say, honestly, it's probably still TBD as to like the trade offs between the approach we did, which is more of a curriculum learning approach after the fact versus inherently training a model with a long context throughout, because I just don't think people have looked at the scaling properties of it in deep detail.”
- Speaker
- Mark Huang
- Publisher
- Latent Space
14 / evaluation
We do have historical precedent, where the original code bomb was trained further from Mama 2, and it just lost all its language capability, basically, right? So I don't want to call that project like deem it as a failure, but it wasn't a really successful generalization exercise, because, you know, these models are about flexibility and being like generic to a certain extent.
“We do have historical precedent, where the original code bomb was trained further from Mama 2, and it just lost all its language capability, basically, right? So I don't want to call that project like deem it as a failure, but it wasn't a really successful generalization exercise, because, you know, these models are about flexibility and being like generic to a certain extent.”
- Speaker
- Mark Huang
- Publisher
- Latent Space