app / likes
ChatGPT app
“This is why I like the ChatGPT app, because it gives the AI a home in your computer where you can focus on it, rather than just being another tab in my mess of internet options.”
Public evidence record
Nathan Lambert
Published podcast speaker
- Claims
- 117
- Episodes
- 4
- Shows
- 2
- Named items
- 3
Books, apps, and tools
The evidenced stack.
app / likes
ChatGPT
“This is why I like the ChatGPT app, because it gives the AI a home in your computer where you can focus on it, rather than just being another tab in my mess of internet options.”
book / recommends
Season of the Witch
“It’s a great book, Season of the Witch; I recommend it. A bunch of my SF friends who do get out recommended it to me.”
Claim ledger
What Nathan said.
19 transcript-backed records
01 / evaluation
There are now tons of tech companies in China that are releasing very strong frontier open weight models, to the point where I would say that DeepSeek is kind of losing its crown as the preeminent open model maker in China, and the likes of Z.
“There are now tons of tech companies in China that are releasing very strong frontier open weight models, to the point where I would say that DeepSeek is kind of losing its crown as the preeminent open model maker in China, and the likes of Z.”
- Speaker
- Nathan Lambert
- Publisher
- Lex Fridman Podcast
02 / evaluation
I think the tool use point is the one that’s stopping them from being most general purpose because, with something like Claude Code or ChatGPT with search, the autoregressive chain is interrupted with an external tool, and I don’t know how to do that with the diffusion setup.
“I think the tool use point is the one that’s stopping them from being most general purpose because, with something like Claude Code or ChatGPT with search, the autoregressive chain is interrupted with an external tool, and I don’t know how to do that with the diffusion setup.”
- Speaker
- Nathan Lambert
- Publisher
- Lex Fridman Podcast
03 / evaluation
I think in education, a lot of it needs to be, at this point, what I like— —because language models are so good at the math.
“I think in education, a lot of it needs to be, at this point, what I like— —because language models are so good at the math.”
- Speaker
- Nathan Lambert
- Publisher
- Lex Fridman Podcast
04 / evaluation
While I’m talking, I’ll say that the Chinese open language models tend to be much bigger and that gives them this higher peak performance as MoEs, whereas a lot of these things that we like a lot, whether it was Gemma or Nemotron, have tended to be smaller models from the US, which is starting to change.
“While I’m talking, I’ll say that the Chinese open language models tend to be much bigger and that gives them this higher peak performance as MoEs, whereas a lot of these things that we like a lot, whether it was Gemma or Nemotron, have tended to be smaller models from the US, which is starting to change.”
- Speaker
- Nathan Lambert
- Publisher
- Lex Fridman Podcast
05 / evaluation
The dense model holds most of the weights if you count them in a transformer model, so you can get really big gains from those mixture of experts on parameter efficiency at training and inference because you get this efficiency by not activating all of these parameters.
“The dense model holds most of the weights if you count them in a transformer model, so you can get really big gains from those mixture of experts on parameter efficiency at training and inference because you get this efficiency by not activating all of these parameters.”
- Speaker
- Nathan Lambert
- Publisher
- Lex Fridman Podcast
06 / evaluation
Anthropic has research on this where they show that if you put certain phrases in at pre-training, you can then elicit different behavior when you’re actually using the model because they’ve poisoned the pre-training data, as of now, I don’t think anybody in a production system is trying to do anything like this.
“Anthropic has research on this where they show that if you put certain phrases in at pre-training, you can then elicit different behavior when you’re actually using the model because they’ve poisoned the pre-training data, as of now, I don’t think anybody in a production system is trying to do anything like this.”
- Speaker
- Nathan Lambert
- Publisher
- Lex Fridman Podcast
07 / evaluation
If you’re going to upload model weights, it doesn’t really matter because anyone that’s serving it in an application and cares a lot about serving is going to, when serving it, if they’re using it for a specific task, they’re going to tailor it to that and it doesn’t matter that it’s saying it’s ChatGPT.
“If you’re going to upload model weights, it doesn’t really matter because anyone that’s serving it in an application and cares a lot about serving is going to, when serving it, if they’re using it for a specific task, they’re going to tailor it to that and it doesn’t matter that it’s saying it’s ChatGPT.”
- Speaker
- Nathan Lambert
- Publisher
- Lex Fridman Podcast
08 / evaluation
There’s not full agreement in the community, but for us that means releasing the training data, releasing the training code, and then also having open weights like this. And we’ll get into the details of the models and again and again as we try to get deeper into how the models were trained, we will say things like the data processing, data filtering data quality is the number one determinant of the model quality.
“There’s not full agreement in the community, but for us that means releasing the training data, releasing the training code, and then also having open weights like this. And we’ll get into the details of the models and again and again as we try to get deeper into how the models were trained, we will say things like the data processing, data filtering data quality is the number one determinant of the model quality.”
- Speaker
- Nathan Lambert
- Publisher
- Lex Fridman Podcast
09 / evaluation
Because the model weights for DeepSeek-R1 are openly available and the license is very friendly, the MIT license commercially available, all of these midsize companies and big companies are trying to be first to serve R1 to their users.
“Because the model weights for DeepSeek-R1 are openly available and the license is very friendly, the MIT license commercially available, all of these midsize companies and big companies are trying to be first to serve R1 to their users.”
- Speaker
- Nathan Lambert
- Publisher
- Lex Fridman Podcast
10 / evaluation
We can make these big jumps, but it just takes a long time to push the frontier of open source. And fundamentally, I would say that that’s because open source AI does not have the same feedback loops as open source software.
“We can make these big jumps, but it just takes a long time to push the frontier of open source. And fundamentally, I would say that that’s because open source AI does not have the same feedback loops as open source software.”
- Speaker
- Nathan Lambert
- Publisher
- Lex Fridman Podcast
11 / evaluation
As reinforcement learning is so much less compute, like it is a richer signal in terms of its impact. Because if they could do what RLHF is doing at pre-training, they would, but they don't know how to have that effect in like a stable manner.
“As reinforcement learning is so much less compute, like it is a richer signal in terms of its impact. Because if they could do what RLHF is doing at pre-training, they would, but they don't know how to have that effect in like a stable manner.”
- Speaker
- Nathan Lambert
- Publisher
- Latent Space
12 / evaluation
It's really tricky to actually do that. I think that people just keep using GPT-4 because it's really cheap.
“It's really tricky to actually do that. I think that people just keep using GPT-4 because it's really cheap.”
- Speaker
- Nathan Lambert
- Publisher
- Latent Space
13 / evaluation
I think InstructGPT does something where they like try to get the RL model to match the instruction tuning dataset because they were really happy with that dataset to constrain the distribution.
“I think InstructGPT does something where they like try to get the RL model to match the instruction tuning dataset because they were really happy with that dataset to constrain the distribution.”
- Speaker
- Nathan Lambert
- Publisher
- Latent Space
14 / evaluation
I think the reason this really is done on a deep level is that you're not actually trying to model any contestable preference in this.
“I think the reason this really is done on a deep level is that you're not actually trying to model any contestable preference in this.”
- Speaker
- Nathan Lambert
- Publisher
- Latent Space
15 / evaluation
Is this text bad? That's not that surprising, I think, because you could use like a hundred times smaller language model and do much better at filtering than RLHF.
“Is this text bad? That's not that surprising, I think, because you could use like a hundred times smaller language model and do much better at filtering than RLHF.”
- Speaker
- Nathan Lambert
- Publisher
- Latent Space
16 / evaluation
However, reinforcement learning proved highly effective, particularly given its cost and time effectiveness. So you don't really know exactly what the costs and time that Meta is looking at, because they have a huge team and a pretty good amount of money here to release these Llama models.
“However, reinforcement learning proved highly effective, particularly given its cost and time effectiveness. So you don't really know exactly what the costs and time that Meta is looking at, because they have a huge team and a pretty good amount of money here to release these Llama models.”
- Speaker
- Nathan Lambert
- Publisher
- Latent Space
17 / evaluation
Like, I don't know exactly if they're doing this now, but you can kind of see why doing RLHF at scale and prioritizing a lot of different endpoints would be hard because these are all things I'd be interested in if I was scaling up a big team to do RLHF and like what is going into the preference data.
“Like, I don't know exactly if they're doing this now, but you can kind of see why doing RLHF at scale and prioritizing a lot of different endpoints would be hard because these are all things I'd be interested in if I was scaling up a big team to do RLHF and like what is going into the preference data.”
- Speaker
- Nathan Lambert
- Publisher
- Latent Space
18 / evaluation
There's essentially the thing in my mind that I can't get past is the difference between the control you get in training a reward model and then training a policy because essentially everything you want your reward model to do might not be everything that you train the policy to do in the RLHF step where you have like the two different prompt distributions.
“There's essentially the thing in my mind that I can't get past is the difference between the control you get in training a reward model and then training a policy because essentially everything you want your reward model to do might not be everything that you train the policy to do in the RLHF step where you have like the two different prompt distributions.”
- Speaker
- Nathan Lambert
- Publisher
- Latent Space
19 / evaluation
I think technical terms are like deduplication, so you don't wanna pass the model, the same text, even if it came from different websites and there's tons more that goes into this.
“I think technical terms are like deduplication, so you don't wanna pass the model, the same text, even if it came from different websites and there's tons more that goes into this.”
- Speaker
- Nathan Lambert
- Publisher
- Latent Space