Andrew Lee: evaluation

Transcript context

…Yeah, I think the mantra has stayed the same and we've rebuilt basically everything. I was thinking about this earlier, and even the pieces where I'm like, Oh, this has stayed the same. Actually, no, I've been totally rebuilt. So from a product perspective, the product is actually very different now. When we launched this thing in October, it was all focused on a workflow automation. We thought, Hey, it would be really cool if people could come in, describe a workflow, and we'd run the workflow for you. But basically, the feedback that we got right out of the gate was, Hey, once I've given this agent all of my context and hooks it up to all my stuff, I don't want it just running my workflows. I also just want to be able to talk to it synchronously too. And so it's not just a workflow automation tool anymore. Now it is a very general purpose agent. It's great for doing workflows, but it's also great for doing other types of stuff. So that required basically a total rebuild of the product experience and has developed a lot of technology behind it. So as an example, in a workflow automation tool in the previous iteration of the product, You basically had a main agent that you'd talk to for a brief period to set up your workflow. Once you were done setting up your workflow, you basically stopped talking to that agent. The chats were relatively short, and then our system would kick off runs of what we call the task agent on a periodic basis when events happened. Every agent was a pretty short thing, and you could do pretty simple context engineering to make that work. In a world where you want this general-purpose agent that you can talk to synchronously and run these automations Product experience people want is just one big linear chat where everything is in one chat. It works really well from a product experience, but the engineering gets really complicated because you can't just have an infinitely long chat history that you feed into the LLM. Even if you could, it'd be really, really expensive and you wouldn't want every automation to have to send in 10 million tokens from all the previous runs. And so we had to kind of totally rethink the way context engineering works. and say, what if, instead of the history being the thing we send to the LLM, what if the history is in the file system? What if the files are the agent? Marc Andreessen had a thing about this, and I think people figured this out, but we made the switch kind of in November where we said, really, okay, what we need is a file system. that has your history. Then we need the thing that we actually send to the LLM to be just hints at what's in the file system and what things you need to read to get the work done. That way, we can scale the agent up, including the number of chat messages that were sent. There's a lot of stuff we'll scale up in the future, but basically, you can scale from what fits in the context window to what just fits in the file system. the number of chat messages that were sent. There's a lot of stuff we'll scale up in the future, but basically, you can scale from what fits in the context window to what just fits in the file system. You can fit a lot more stuff in the file system. There's a bunch of other stuff we've rebuilt. Another big thing we've rebuilt is around computer use. When we launched, initially, computer use was this add-on. You could have a Linux machine. Actually, initially, it was a Windows machine, then it was a Linux machine tacked on. It really was an afterthought. You'd use it for certain things. It worked okay, but most things you do with agent didn't touch it. At this point, computer use is the absolute core of the product. Basically, everything you do is running shell commands, touching a file system, touching a database. We have a very tightly integrated browser use experience now where every agent has a headless VM and a browser VM that persist state across runs and allow you to do lots of really cool stuff. And it's sort of in the critical path. So now, if our computer use goes down, everything goes down versus before it was kind of this afterthought. We've also rethought the way our integrations work. So this is something I think for a product experience that maybe doesn't look terribly different to folks, but the basic architecture of how we plug in other systems to the Tasklet agents has been completely rethought, basically to allow the agent to have sort of more control and management of like, those connections. A simple example of a product experience that's improved is you used to not be able to connect multiple instances of the same type of thing. You couldn't have three different Gmail accounts connected to an agent, and now you can. The base architecture there has been rebuilt. I'd say basically every line of code has probably been touched in the last six months, and most of our fundamental assumptions were thrown out. The product still can do many of the same things, but hopefully much better now. Yeah, that's cool. So literally, you can't think of anything that has survived the last six months.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.