Shawn Wang: evaluation

Transcript context

…Yeah, exactly. Well, let's do a couple of things in parallel. First thing I'm going to do is I am just going to make a box over here and I'm going to say kitchen timer right in the middle of the box. And this is going to be the only prompt that I'm going to, I'm going to give it. Okay. Just going to click make real and just the, the kitchen timer box. As you see with these multimodal prompting, like someone will draw a calculator, like in a lot of complexity and say, you know, it makes this real and sure enough, you get back like a really complex full calculator. But if you did the same thing and you just said empty box, but just the word calculator, it would give you back the same thing is that it knows what a calculator looks like and it knows how it works and all that. So next let's also give it just the user interface, like without the state chart, we'll leave the state chart out, but we'll do just the user interface. And then we'll do just the state chart, you know, and say, Hey, make this real. And then we'll do both the state chart and the UI. So we have four different prompts with four potential different results based on, you know, variations of the same, same input. So first off our kitchen timer, where all we did was we, we sent it a box with the word kitchen timer. It has, I don't know what this box is for, but we have a time we have start, stop and reset. So I can double click in, I can click start. It doesn't do anything. Oh, what is this? Oh, whoa. If this, okay, well, if the numbers there, yeah, then it'll, it'll stop. If I stop it, it stops. I can start it. It'll keep going again. Okay. And I can reset it. And there we go. The only weird thing is that it works. Yeah. It has a, a number input field for the number of seconds that I can, I can type out. But yeah. You know what? In a pinch, in a pinch, I'll take it. If I really needed just to count 60 seconds or something. Next we have, or the result where the input was just my drawing of a kitchen timer. I didn't tell it it was a kitchen timer. I didn't send it the words kitchen timer and I didn't tell it how it should work, but it did produce something that kind of looks the same. Let's see if it works. So I'm going to click minute, second, start, reset. No. So unfortunately it did not make any working UI, although it did, you know, put the buttons in the right place or something like that. Maybe it over focuses on the UI because you told it, you just, that's all you gave it. Yeah. Yeah. I mean, there is in the prompt kind of language around, like use what you know about the way that applications work in order to sort of fill in the blanks here in terms of the behavior and all that. But let's go to the next one. This one is where we only sent it the state chart. There's also something in the prompt that says like, if it's red, it's not part of the UI. Like if it's red, then like treat that as an annotation rather than a thing that you should, should actually make. So this time it actually looks a lot like the previous one. But it does have these minute, second buttons. Oh, weird. It has plus and minus minute, seconds, and it also has this like stop state written at the bottom. So there's four buttons, you know, minus minute, minus second, plus minute, plus second, and then there's start and reset. So does it work? I can add a minute. I can also subtract a minute. All right.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.