High Signal Podcasts Evidence ledger
Method
Browse
← Back to evidence

Evidence receipt / evaluation

Published · transcript-backed

Al Chang: evaluation

10 Jul 2023 Latent Space Code Interpreter == GPT 4.5 (w/ Simon Willison, Alex Volkov, Aravind Srinivas, Alex Graveley, et al.)

“I mean, the idea is basically that it goes down this line of thinking and it can't kind of back up, right? So it has to sort of finish its thought and then once it's finished its thought, it's like, oh wait, this doesn't work.”

— Al Chang

Source trail

Everything needed to verify it.

Speaker
Al Chang
Attribution
Verified speaker
Claim type
evaluation
Recorded
10 Jul 2023
Publisher
Latent Space

Transcript context

…libraries. Sometimes it gets there, sometimes it gives up. And when it gives up, you can coach it. You can say, Hey, try, instead of using a regular expression, try splitting it into five lines and, and taking this approach just like you would with the coding intern, right? If you've got an intern who gets stuck, you might go, Hey, have you thought about this option instead? But yeah, so there's definitely an art to coaching it, but the more time you spend with it, the better you get it at coaching it into finding the right solution. And, and it kind of makes sense, right? I mean, the idea is basically that it goes down this line of thinking and it can't kind of back up, right? So it has to sort of finish its thought and then once it's finished its thought, it's like, oh wait, this doesn't work. And then now that it has that in the context window, it can look back on it again and then think about it again, and then try it again, right? And so sometimes it unstick itself. Sometimes it will go, oh, this clearly doesn't work. I should try something else. But it'll often get it. Sometimes it gets into loops. You see it basically trying the same tweak over and over again, and that's the point where sometimes it will solve itself. It will figure it out itself. Other times it gets into a bit of a loop. It might try the same fix multiple times, and then you hit the stop button and you need to prompt it in a different direction to try and get a, a solution a different way. And you can kind of…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.

Search evidence