Flo Crivello: evaluation

Transcript context

…AVL and black trees because I was like, oh my god. You guys remember your training. You know? This is this is the moment we use those specking things. Because everybody in school is always like, when do you really use those specking things? Like, we're using AVL and black trees, baby. Right? Right? Black trees, they're called. It's so what is described, if you just do the naive implementation, you get that really nice emerging property for free, which is you have all of those linked context buckets that contain one another. But every context packet always contains just one of the context packet. And so you end up with, like, this Russian doll of sorts of context packets. Like, if you if you wanna open if you wanna get to the bottom, takes a very long time if you wanna you know? So what you do instead is that you have context packets contain multiple context packets. Okay? And and and and and so you you end up with a tree because, basically, what you want is you want the topmost context packet not to contain the last 100. You want it to contain the first context packet and the last context packet. Right? And so you end up with it's a self balancing tree. Those these two algorithms. Those the AVL tree, and then there's the red black tree, which are algorithms that are used to balance a tree. So instead of having one long line, you want to minimize the height of the tree such that, like, going to the bottom takes as few jumps as possible. Okay? Technically, AVL is the best is is the is is the demonstrably best way to balance a tree because it it leads to, like, the lowest height. A red black tree is better because it takes into account the cost to balance the tree, which is it's expensive because you need to regenerate a lot of your context packets, and you you you you miss your cash when you do that. It's it's very expensive. So red black trees the canonical implementation of red black trees is on binary trees, which is just a tree where each node's got two children. We went for a it's called a centering tree. It's literally just each node's got a 100 nodes. So the the node below that's gonna have, like, 10,000 nodes. Right? And so literally with two jumps, you can have 10,000 context buckets, and you can access, like, all the context in the universe. And that leads to some really surprising behaviors because you're literally no more than two LLM calls away from being able to access 10,000 context buckets, each one of which contains 200,000 tokens. Right? So you add, like, 2,000,000,000 tokens here of context in in two LLM calls. And and that is what leads to, like, those really surprising behaviors from those AI agents where you ask them any question and they remember everything perfectly all the time, and that's awesome. That was that was one really big thing we had we had we had to figure out. I hope, please, you know, like, copy us. ny question and they remember everything perfectly all the time, and that's awesome. That was that was one really big thing we had we had we had to figure out. I hope, please, you know, like, copy us. Like, I will just we just don't have the time to publish, but, you know, this is this is we don't mean for this to be secret. Like, I think this is a really powerful technique, and I I have been surprised to not see more communication about it. Just for calibration, how many tokens are you finding businesses have? Like, when you say with these two levels, can get to 2,000,000,000 tokens. I don't have a great intuition for Does that cover a 20 person team? Does however that's been in business for five years, what's kind of the heuristic for team size and length of history that translates into how many tokens? You can probably do the math, but, like, 2,000,000,000 is we should ask Cloud, but it's it's probably bigger than most libraries. So I think we we have few customers who whose knowledge base, whose memory is bigger than that. Most of the time when you get started, you consume, like for, like, a a team of 20, you know, the the hydration components. So it's like, again, it's this moment where, like, you connect your Notion, you connect your Slack, and then we crawl everything. We look at all of that stuff. For a team of 20, that's gonna tend to consume 5,000,000 tokens at most, you know, three to 5,000,000 tokens. And that's with, like, a team of 20 and, like, years of Slack history. So, like, we literally take all of your Slack history. That takes a while and actually here, surprisingly, the bottleneck is not even the LLMs. It's the Slack APIs. Like, it's not happy about you, like, crawling traffic. Experienced that. Yes. I'm sure. The right way to do it, if you're doing it personally, is to export your workspace archive. We can't ask users to do that, so we just crawl crawl the entire history. So no. I mean, like, 2,000,000,000 is is a lot of…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.