High Signal Podcasts Evidence ledger
Method
Browse
How the evidence works

Public evidence record

Erik Schluntz

Published podcast speaker

Claims
28
Episodes
1
Shows
1
Named items
0

Claim ledger

What Erik said.

28 transcript-backed records

01 / belief

I think some way, like, right now, it just takes, I think we sort of force people to do the best practices of writing out sort of these full JSON schemas, but it would be really nice if you could just pass in a Python function as a tool.

“I think some way, like, right now, it just takes, I think we sort of force people to do the best practices of writing out sort of these full JSON schemas, but it would be really nice if you could just pass in a Python function as a tool.”
Speaker
Erik Schluntz
Publisher
Latent Space

03 / belief

I would say on the positives, like, there is really sort of incredible progress that's happened in the last five years that I think will be a big unlock for robotics.

“I would say on the positives, like, there is really sort of incredible progress that's happened in the last five years that I think will be a big unlock for robotics.”
Speaker
Erik Schluntz
Publisher
Latent Space

05 / belief

I think the computer use demo that we released is an extension of that. It has the same bash and edit tools, but it also has the computer tool that lets it get screenshots and move the mouse and keyboard.

“I think the computer use demo that we released is an extension of that. It has the same bash and edit tools, but it also has the computer tool that lets it get screenshots and move the mouse and keyboard.”
Speaker
Erik Schluntz
Publisher
Latent Space

06 / belief

I think that for robotics, the limiting factor is going to be reliability, that these models are really good at doing these demos of doing laundry or doing dishes.

“I think that for robotics, the limiting factor is going to be reliability, that these models are really good at doing these demos of doing laundry or doing dishes.”
Speaker
Erik Schluntz
Publisher
Latent Space

07 / belief

I think there's still need for sort of many different varied evals. Like sometimes you do really care about just sort of greenfield code generation.

“I think there's still need for sort of many different varied evals. Like sometimes you do really care about just sort of greenfield code generation.”
Speaker
Erik Schluntz
Publisher
Latent Space

08 / belief

Now, I would say that, you know, this is definitely an area of future research, especially if we talk about these problems that are going to take a human more than four hours.

“Now, I would say that, you know, this is definitely an area of future research, especially if we talk about these problems that are going to take a human more than four hours.”
Speaker
Erik Schluntz
Publisher
Latent Space

09 / belief

I think there's been a lot more progress in the software in the last few years. And I think a lot of the humanoid robot companies now are really trying to build amazing hardware.

“I think there's been a lot more progress in the software in the last few years. And I think a lot of the humanoid robot companies now are really trying to build amazing hardware.”
Speaker
Erik Schluntz
Publisher
Latent Space

11 / belief

I think examples are really good in things like descriptions, like so many, like just using the Linux command line, like how many times I do like dash dash help or look at the man page or something.

“I think examples are really good in things like descriptions, like so many, like just using the Linux command line, like how many times I do like dash dash help or look at the man page or something.”
Speaker
Erik Schluntz
Publisher
Latent Space

12 / belief

There's certain things that it's not very good at yet. But I'm really excited, I think, most broadly, not just for new things that weren't possible before, but as a much lower friction way to implement tool use.

“There's certain things that it's not very good at yet. But I'm really excited, I think, most broadly, not just for new things that weren't possible before, but as a much lower friction way to implement tool use.”
Speaker
Erik Schluntz
Publisher
Latent Space

14 / belief

I mean, this, so I'd say that a lot of previous agent work built sort of these very hard coded and rigid workflows where the model is sort of pushed through certain flows of steps. And I think to some extent, you know, that's needed with smaller models and models that are less smart.

“I mean, this, so I'd say that a lot of previous agent work built sort of these very hard coded and rigid workflows where the model is sort of pushed through certain flows of steps. And I think to some extent, you know, that's needed with smaller models and models that are less smart.”
Speaker
Erik Schluntz
Publisher
Latent Space

15 / belief

One is like the, I'd say maybe like doing the easier solution rather than the hard solution. And I'd say the second one, I think what you're talking about is like the lazy model is like when the model says like dot, dot, dot, code remains the same.

“One is like the, I'd say maybe like doing the easier solution rather than the hard solution. And I'd say the second one, I think what you're talking about is like the lazy model is like when the model says like dot, dot, dot, code remains the same.”
Speaker
Erik Schluntz
Publisher
Latent Space

16 / belief

From the day-to-day job. But I think one of the most interesting things about SWE-Bench is that all these other benchmarks are usually just isolated puzzles, and you're starting from scratch.

“From the day-to-day job. But I think one of the most interesting things about SWE-Bench is that all these other benchmarks are usually just isolated puzzles, and you're starting from scratch.”
Speaker
Erik Schluntz
Publisher
Latent Space

18 / belief

I think Eider, they have a really good blog where they explore some of these different methods for editing files, and they post results about them, which I think is interesting.

“I think Eider, they have a really good blog where they explore some of these different methods for editing files, and they post results about them, which I think is interesting.”
Speaker
Erik Schluntz
Publisher
Latent Space

20 / belief

I think the really interesting question for me, for all the startups out there, is this kind of divergence between the benchmarks and what real customers will want.

“I think the really interesting question for me, for all the startups out there, is this kind of divergence between the benchmarks and what real customers will want.”
Speaker
Erik Schluntz
Publisher
Latent Space

21 / evaluation

The question is, are those fairly inaccessible or are they just impossible because of the descriptions? But I think certainly some of the tasks, especially the ones that the human graders reviewed as like taking longer than four hours are extremely difficult.

“The question is, are those fairly inaccessible or are they just impossible because of the descriptions? But I think certainly some of the tasks, especially the ones that the human graders reviewed as like taking longer than four hours are extremely difficult.”
Speaker
Erik Schluntz
Publisher
Latent Space

22 / evaluation

The other thing I was going to say is that SWE-Bench is certainly hard to implement and expensive to run because each task, you have to parse, you know, a lot of the repo to understand where to put your code.

“The other thing I was going to say is that SWE-Bench is certainly hard to implement and expensive to run because each task, you have to parse, you know, a lot of the repo to understand where to put your code.”
Speaker
Erik Schluntz
Publisher
Latent Space

23 / recommendation

I think for agents in general, like having a planning step at the beginning, one, just having that plan will improve performance on the downstream task just because it's kind of like a bigger chain of thought, but also it's just such a better UX.

“I think for agents in general, like having a planning step at the beginning, one, just having that plan will improve performance on the downstream task just because it's kind of like a bigger chain of thought, but also it's just such a better UX.”
Speaker
Erik Schluntz
Publisher
Latent Space

24 / preference

You know, the classic idea is you can have a plug that can fit either way, and that's dangerous, or you can make it asymmetric so that it can't fit this way, it has to go like this, and that's a better tool because you can't use it the wrong way.

“You know, the classic idea is you can have a plug that can fit either way, and that's dangerous, or you can make it asymmetric so that it can't fit this way, it has to go like this, and that's a better tool because you can't use it the wrong way.”
Speaker
Erik Schluntz
Publisher
Latent Space

25 / preference

I think another one is like, if you have a real coding agent, you don't want to have it start on a task and like spin its wheels for hours because you gave it a bad prompt.

“I think another one is like, if you have a real coding agent, you don't want to have it start on a task and like spin its wheels for hours because you gave it a bad prompt.”
Speaker
Erik Schluntz
Publisher
Latent Space

27 / recommendation

I would say that as people are building systems around agents, I think the more you can separate out the different kinds of work the agent needs to do, the better you can tailor a prompt for that task. And I think that also creates a lot of like, for instance, if you were trying to make an agent that could both solve hard programming tasks, and it could just write quick test files for something that someone else had already made, the best way to do those two tasks might be very different prompts.

“I would say that as people are building systems around agents, I think the more you can separate out the different kinds of work the agent needs to do, the better you can tailor a prompt for that task. And I think that also creates a lot of like, for instance, if you were trying to make an agent that could both solve hard programming tasks, and it could just write quick test files for something that someone else had already made, the best way to do those two tasks might be very different prompts.”
Speaker
Erik Schluntz
Publisher
Latent Space

28 / commitment

We actually got acquired about six months ago, but I had left Cobalt about a year ago now, because I was starting to get a lot more excited about AI.

“We actually got acquired about six months ago, but I had left Cobalt about a year ago now, because I was starting to get a lot more excited about AI.”
Speaker
Erik Schluntz
Publisher
Latent Space
Search evidence