High Signal Podcasts Evidence ledger
Method
Browse
How the evidence works

Public evidence record

Dries Smit

Published podcast speaker

Claims
10
Episodes
1
Shows
1
Named items
0

Claim ledger

What Dries said.

10 transcript-backed records

01 / evaluation

Then if you train it using reasoning, you can actually do way more because now it has more time to actually reason and figure out what you're actually asking and form new extractions for solving a specific problem case as opposed to just regurgitating what it's seen on the internet?

“Then if you train it using reasoning, you can actually do way more because now it has more time to actually reason and figure out what you're actually asking and form new extractions for solving a specific problem case as opposed to just regurgitating what it's seen on the internet?”
Speaker
Dries Smit
Publisher
Machine Learning Street Talk

02 / evaluation

So the main constraint was we only had 3 games, 3 public games and 3 private games to evaluate on. So we had to make sure or that to make sure that whatever the design is is is you don't assume too much about the the environments and the games because it on the private leaderboard or like the public leaderboard beforehand, you could actually see that I wasn't even close to the top because it was super easy to overfit.

“So the main constraint was we only had 3 games, 3 public games and 3 private games to evaluate on. So we had to make sure or that to make sure that whatever the design is is is you don't assume too much about the the environments and the games because it on the private leaderboard or like the public leaderboard beforehand, you could actually see that I wasn't even close to the top because it was super easy to overfit.”
Speaker
Dries Smit
Publisher
Machine Learning Street Talk

03 / evaluation

We tried other methods such as directly predicting like more of a transductive method where you just have the input frames as context into a long sequence model and you just predict the actions. But that doesn't also it doesn't generalize well and it doesn't really make intuitive sense a lot because if you play the game, for example, the 1st game here, the maze level, you would intuitively play 1 or 2 actions and you know think about your path and you'll go to the end position.

“We tried other methods such as directly predicting like more of a transductive method where you just have the input frames as context into a long sequence model and you just predict the actions. But that doesn't also it doesn't generalize well and it doesn't really make intuitive sense a lot because if you play the game, for example, the 1st game here, the maze level, you would intuitively play 1 or 2 actions and you know think about your path and you'll go to the end position.”
Speaker
Dries Smit
Publisher
Machine Learning Street Talk

04 / prediction

they have the intention, they have something that the coding agents don't have because you know, the bull case, wouldn't it be amazing if you could stick the requirements in and the agents themselves would kind of understand, oh I see what they meant and they could evolve the requirements.

“they have the intention, they have something that the coding agents don't have because you know, the bull case, wouldn't it be amazing if you could stick the requirements in and the agents themselves would kind of understand, oh I see what they meant and they could evolve the requirements.”
Speaker
Dries Smit
Publisher
Machine Learning Street Talk

05 / belief

For example, you have this reasoning tokens which has like abstract representation of objects, which we then manifest or write down in like an English language textual tokens. So I would say that's more abstract, you identify objects, you try and find out what the mechanics is, dynamics of the game, what the goal is.

“For example, you have this reasoning tokens which has like abstract representation of objects, which we then manifest or write down in like an English language textual tokens. So I would say that's more abstract, you identify objects, you try and find out what the mechanics is, dynamics of the game, what the goal is.”
Speaker
Dries Smit
Publisher
Machine Learning Street Talk

06 / belief

I think the 1st 4 places was basically brute force algorithms that just search over a large space of actions but do some basic form of filtering and you can get a very good score.

“I think the 1st 4 places was basically brute force algorithms that just search over a large space of actions but do some basic form of filtering and you can get a very good score.”
Speaker
Dries Smit
Publisher
Machine Learning Street Talk

07 / belief

I think we have a clear benchmark, which we know humans, which is general in some sub domain, which we can say whatever that domain is can score that score and that is the medium score.

“I think we have a clear benchmark, which we know humans, which is general in some sub domain, which we can say whatever that domain is can score that score and that is the medium score.”
Speaker
Dries Smit
Publisher
Machine Learning Street Talk

08 / commitment

Exactly what I did was I just used a action model that learns, yeah, which frames are valid for a given transition and then it was more about the engineering of being able to learn within 100,000 actions because that was kind of the max action limit we could get out in the time limit we had.

“Exactly what I did was I just used a action model that learns, yeah, which frames are valid for a given transition and then it was more about the engineering of being able to learn within 100,000 actions because that was kind of the max action limit we could get out in the time limit we had.”
Speaker
Dries Smit
Publisher
Machine Learning Street Talk

09 / evaluation

Like we have to make sure it's working, but it is a case that we're gradually we are understanding less and less of our own code base. And we're struggling even with reviewing some of the changes is you might use a codex to help review some of it because it's such a broad change or we need to split it up.

“Like we have to make sure it's working, but it is a case that we're gradually we are understanding less and less of our own code base. And we're struggling even with reviewing some of the changes is you might use a codex to help review some of it because it's such a broad change or we need to split it up.”
Speaker
Dries Smit
Publisher
Machine Learning Street Talk

10 / recommendation

You play a game, you solve it as in like a large compute or action budget and then you have to play it again and you have to do like a speed run through it to improve it. So I think that is important, but I also think action efficiency is a practical step to reduce just brute force solutions.

“You play a game, you solve it as in like a large compute or action budget and then you have to play it again and you have to do like a speed run through it to improve it. So I think that is important, but I also think action efficiency is a practical step to reduce just brute force solutions.”
Speaker
Dries Smit
Publisher
Machine Learning Street Talk
Search evidence