High Signal Podcasts Evidence ledger
Method
Browse
How the evidence works

Public evidence record

Sholto Douglas

Published podcast speaker

Claims
59
Episodes
3
Shows
1
Named items
1

Books, apps, and tools

The evidenced stack.

Browse the grouped index →

other / uses

Gwern's book reviews

“The book reviews is a good suggestion. I actually use, like, Gwern's book reviews as a way to recommend books to people.”

Dwarkesh Podcast · 25 Mar 2025

Evidence receipt · Source ↗

Claim ledger

What Sholto said.

42 transcript-backed records

01 / belief

In particular I think the intellectual ceiling goes quite—contra what I was saying before, which is we've demonstrated this incredible complexity of math, and programming problems… I do think that the type of task and setting that AlphaZero worked in this two-player perfect information game basically is incredibly friendly to RL algorithms.

“In particular I think the intellectual ceiling goes quite—contra what I was saying before, which is we've demonstrated this incredible complexity of math, and programming problems… I do think that the type of task and setting that AlphaZero worked in this two-player perfect information game basically is incredibly friendly to RL algorithms.”
Speaker
Sholto Douglas
Publisher
Dwarkesh Podcast

02 / belief

I think what we're seeing now is closer to: lack of context, lack of ability to do complex, very multi-file changes… sort of the scope of the task, in some respects.

“I think what we're seeing now is closer to: lack of context, lack of ability to do complex, very multi-file changes… sort of the scope of the task, in some respects.”
Speaker
Sholto Douglas
Publisher
Dwarkesh Podcast

05 / belief

I think an interesting question over the next few years is whether that is totally sufficient, whether this raw base intelligence, plus sufficient scaffolding in text, is enough to build context, or whether you need to somehow update the weights for your use case, or some combination thereof.

“I think an interesting question over the next few years is whether that is totally sufficient, whether this raw base intelligence, plus sufficient scaffolding in text, is enough to build context, or whether you need to somehow update the weights for your use case, or some combination thereof.”
Speaker
Sholto Douglas
Publisher
Dwarkesh Podcast

06 / belief

If you made an extremely efficient transform implementation on TPU, or Trainium, or Incuda, then I think there's a pretty high likelihood that you'll get a job offer.

“If you made an extremely efficient transform implementation on TPU, or Trainium, or Incuda, then I think there's a pretty high likelihood that you'll get a job offer.”
Speaker
Sholto Douglas
Publisher
Dwarkesh Podcast

07 / belief

I think of this product exponential in some respects where you need to be designing for a few months ahead of the model, to make sure that the product you build is the right one.

“I think of this product exponential in some respects where you need to be designing for a few months ahead of the model, to make sure that the product you build is the right one.”
Speaker
Sholto Douglas
Publisher
Dwarkesh Podcast

12 / belief

As you use more compute, and as you train on more, and more difficult tasks, your rate of improvement of biology for example is going to be somewhat bound by the time it takes a cell to grow in a way that your rate of improvement on math isn't, for example. So, yes, but I think for many things we'll be able to parallelize widely enough, and get enough iteration loops.

“As you use more compute, and as you train on more, and more difficult tasks, your rate of improvement of biology for example is going to be somewhat bound by the time it takes a cell to grow in a way that your rate of improvement on math isn't, for example. So, yes, but I think for many things we'll be able to parallelize widely enough, and get enough iteration loops.”
Speaker
Sholto Douglas
Publisher
Dwarkesh Podcast

17 / belief

I think you do that quite proactively in terms of you deliberately introduce people that you think will be interesting to each other and this kind of stuff, so… yeah.

“I think you do that quite proactively in terms of you deliberately introduce people that you think will be interesting to each other and this kind of stuff, so… yeah.”
Speaker
Sholto Douglas
Publisher
Dwarkesh Podcast

18 / belief

I think making sure that, like, there is a great debate of ideas on not, not just AI, but on other fields, and everything is incredibly high leverage in value.

“I think making sure that, like, there is a great debate of ideas on not, not just AI, but on other fields, and everything is incredibly high leverage in value.”
Speaker
Sholto Douglas
Publisher
Dwarkesh Podcast

21 / belief

I think a wonderful research project to do, if someone is out there listening to this, would be to try and take some of the techniques that Trenton's team has worked on and try and disentangle the neurons in the Mistral paper, Mixtral model, which is open source.

“I think a wonderful research project to do, if someone is out there listening to this, would be to try and take some of the techniques that Trenton's team has worked on and try and disentangle the neurons in the Mistral paper, Mixtral model, which is open source.”
Speaker
Sholto Douglas
Publisher
Dwarkesh Podcast

23 / belief

I think directions like publishing the constitution that you expect your model to abide by–trying to make sure that you RLHF it towards that, and ablate that, and have the ability for everyone to offer feedback and contribution to that–is really important.

“I think directions like publishing the constitution that you expect your model to abide by–trying to make sure that you RLHF it towards that, and ablate that, and have the ability for everyone to offer feedback and contribution to that–is really important.”
Speaker
Sholto Douglas
Publisher
Dwarkesh Podcast

24 / belief

You could imagine biology papers going here, math papers going here, and all of a sudden your breakdown is ruined. But that vision transformer one, where the class separation is really clear and obvious, gives I think some evidence towards the specialization hypothesis.

“You could imagine biology papers going here, math papers going here, and all of a sudden your breakdown is ruined. But that vision transformer one, where the class separation is really clear and obvious, gives I think some evidence towards the specialization hypothesis.”
Speaker
Sholto Douglas
Publisher
Dwarkesh Podcast

25 / belief

I think Sasha Rush has a great tweet where he basically plots the curve of the cost of attention respective to the cost of really large models and attention actually trails off.

“I think Sasha Rush has a great tweet where he basically plots the curve of the cost of attention respective to the cost of really large models and attention actually trails off.”
Speaker
Sholto Douglas
Publisher
Dwarkesh Podcast

26 / belief

Maybe not the user, but we should be able to understand and interpret what these values are doing and the information that is transmitting. I think that's a really important goal for the future.

“Maybe not the user, but we should be able to understand and interpret what these values are doing and the information that is transmitting. I think that's a really important goal for the future.”
Speaker
Sholto Douglas
Publisher
Dwarkesh Podcast

28 / belief

I think the most important part to illustrate is this cycle of coming up with an idea, proving it out at different points in scale, and interpreting and understanding what goes wrong.

“I think the most important part to illustrate is this cycle of coming up with an idea, proving it out at different points in scale, and interpreting and understanding what goes wrong.”
Speaker
Sholto Douglas
Publisher
Dwarkesh Podcast

31 / belief

At the moment, I think most of the labs are somewhat compute bound in that there are always more experiments you could run and more pieces of information that you could gain in the same way that scientific research on biology is somewhat experimentally throughput-bound.

“At the moment, I think most of the labs are somewhat compute bound in that there are always more experiments you could run and more pieces of information that you could gain in the same way that scientific research on biology is somewhat experimentally throughput-bound.”
Speaker
Sholto Douglas
Publisher
Dwarkesh Podcast

32 / belief

I think that's really worth exploring. For example, one of the evals that we did in the paper had it learn a language in context better than a human expert could, over the course of a couple of months.

“I think that's really worth exploring. For example, one of the evals that we did in the paper had it learn a language in context better than a human expert could, over the course of a couple of months.”
Speaker
Sholto Douglas
Publisher
Dwarkesh Podcast

37 / belief

I think one thing that I didn't fully convey before was that I think a lot of like good research comes from working backwards from the actual problems that you want to solve.

“I think one thing that I didn't fully convey before was that I think a lot of like good research comes from working backwards from the actual problems that you want to solve.”
Speaker
Sholto Douglas
Publisher
Dwarkesh Podcast

38 / belief

” There's also someone who works on Anthropic's performance team now, Simon Boehm, who has written in my mind the reference for optimizing a CUDA map model on a GPU.

“” There's also someone who works on Anthropic's performance team now, Simon Boehm, who has written in my mind the reference for optimizing a CUDA map model on a GPU.”
Speaker
Sholto Douglas
Publisher
Dwarkesh Podcast

40 / belief

There’s one mode of thinking in which fine-tuning is specialized, you've got this latent bundle of capabilities and you're specializing it for this particular use case that you want. I think I'm not sure how true or not that is.

“There’s one mode of thinking in which fine-tuning is specialized, you've got this latent bundle of capabilities and you're specializing it for this particular use case that you want. I think I'm not sure how true or not that is.”
Speaker
Sholto Douglas
Publisher
Dwarkesh Podcast

41 / belief

I think the traditional story for why distillation is more efficient is during training, normally you're trying to predict this one hot vector that says, “this is the token that you should have predicted.

“I think the traditional story for why distillation is more efficient is during training, normally you're trying to predict this one hot vector that says, “this is the token that you should have predicted.”
Speaker
Sholto Douglas
Publisher
Dwarkesh Podcast

42 / belief

I think there's an important aspect of shots on goal there, so to speak. Where just choosing to go to conferences itself is putting yourself in a position where luck is more likely to happen.

“I think there's an important aspect of shots on goal there, so to speak. Where just choosing to go to conferences itself is putting yourself in a position where luck is more likely to happen.”
Speaker
Sholto Douglas
Publisher
Dwarkesh Podcast
Search evidence