Speaker unverified: belief

Transcript context

…g is typically rag pipelines for agents or for a more sophisticated agentic memory for agents. That's when you typically see vector search start to enter the conversation when it comes into an agentic architecture in some way. Specifically, what you're talking about here is chunking and chunk size tends to be more in that rag pipeline use case. I've got some external data. I've got some proprietary data. I want to inject that proprietary data into my LLM of choice without having to go through an expensive fine-tuning or training process because the LLMs are trained on public data. They're not trained on my proprietary data. So, how do I inject that in a safe way that still gives me some data fidelity so that I'm not I'm not giving away my proprietary data, but I can still get it to focus on my business problems instead of every LLM knows who the last 30 Oscar winners were. And that's not super helpful when I'm trying to get an agent to help me figure out what my knock should do next or my my IT ticketing should do next. So, that's the use case. So if you've got a bunch of documentation that someone who is a human professional in that world, you now need to figure out how big of the how big that should the chunks be that I place in the individual documents and what's the fidelity of that. And when you're the application developer trying to figure this out, there's this trade-off that you're looking at. So if I make my chunks too small, let's say to the fidelity of a sentence, if I make my I potentially lose the context in which that sentence appears. Just if you just read one sentence from a novel, doesn't tell you a whole lot. But if you then also read the paragraph or the page, it tells you a lot more about what the context of that sentence is. So that's an argument for, well, maybe I should make my chunks bigger. There comes a point though where if my chunks are too big, my storage costs start to go up and my retrieval quality starts to go down because if I've got three pages, let's say, and I'm trying to get to the fidelity of a sentence, I now have too much information and I lose the fidelity of that sentence. So, what's the right answer? I I've been doing this a long time, Nathan. The answer is always it depends, right? So you have to go through as a developer you have to go through this iterative cycle like I chunk to a certain size I run some tests what's my retrieval quality okay now I try it again I try a different chunk size I do some tests I see what my retrieval quality is and I have to do that three or four times that for my use case to try to find out what the right balance is between chunk size cost and retrieval quality. So, one of the reasons we bought Voyage is because like those guys like their side hustle is teaching people at Stanford how to how to build LLMs. Those guys are all geniuses. And they came up with this new concept last summer called contextualized chunking. And we've got a blog article on this that we can potentially link through here. But the idea here is that if left to your own devices and figuring out your own iterations of chunk sizes, there comes a point in a graph where if le on this that we can potentially link through here. But the idea here is that if left to your own devices and figuring out your own iterations of chunk sizes, there comes a point in a graph where if you've got retrieval quality on your x on your y-axis and you've got chunk size on your x-axis. With a traditional embedding model, there comes a point where at lower chunk sizes, you have zero context. So the retrieval quality is low and it it builds, but then at some point it flattens and then it starts to degrade if your chunk sizes become too big. And that's why you have to go through these iterations to find the right combination. But what if you didn't? What if instead of sending that all as one big text blob? What if instead you send it as two? What if you sent the fidelity of the sentence you actually want and then as a second string, you send the size, you send the other contextual information. It's where the name contextualized chunking comes from. And we will figure out for you what the right chunk size is for that combination. and we will give you an a vector array that comes back that balances those things for you. When you flip that script, it turns out you can get better retrieval quality with a smaller chunk size, which otherwise is not possible if you're doing it the traditional way. You the way to get better chunk way to get better retrieval quality was with higher chunk size. But if you use this contextualized chunking, we released version three of that last summer. We just released four of that in the last six weeks. you can flip that script. Now, there still are use cases where you need to control the chunk size for different things, but if that's something that you wanted to not worry about or have to learn about, and the way I think about this is more developers will build AI agents in the next three years than did in the last 3 years. And in order to make that possible, we have to lower the learning curve. We have to make it easier for the developer ecosystem to learn how to do this. So if you don't want to have to learn how to go through the iterations of figuring out what the right balance of chunk size and retrieval quality is, instead if you could use contextualized chunking, we'll figure it out for you and you still get good retrieval quality out of it. So that's that's one of the benefits that Voyage offers that no other embedding model out on the market offers, that ease of use feature that gets you better retrieval quality. I definitely appreciate not having to worry about it, but I do want to learn a little more about it. It It sounds like this happens in sort of a testfree way at the level of the developer. So like I don't have to bring a bunch of evalu. >> So how at the level of kind of principles like how is it working in the background? And you said also I'm getting one vector back, right? So I pass in the sort of the chunk that I think I would really want to be able to zero in on and context and that gets converted into a single vector back that represents both of those things in I guess some sort of superp position. Yes, that's exactly how it works. So you get back one array of floats exactly how you converted into a single vector back that represents both of those things in I guess some sort of superp position. Yes, that's exactly how it works. So you get back one array of floats exactly how you did before. It's just now you don't have to go through the iterations of figuring out the size of the chunks you need. And similarly, there's another voyage feature that all the voyage all the voyage embedding models have called retroa reasoning because the other place you have to make a decision about storage cost versus retrieval quality is with number of dimensions. So what do I mean by that? Everybody knows what two dimensions is if you've taken high school level algebra, right? XY, right? But in an embedding space, you tend to have at least 256 dimensions and sometimes as high as 2048. And the more dimensions you have, so each dimension is represented by one of those floats in that array of floats. The more dimensions you have, the richer your embedding space is and the better retrieval quality you get. But at higher dimensions, that doesn't come for free. you've got a storing 256 floats takes up less space on disk and in the index and memory than it would for 2048. So you again have to go through this iteration of what's the right number of dimensions for my use case given my what my storage costs might be. All the voyage models have a feature in them called betroka reasoning. It comes from the Russian nesting dolls. If you think about how Russian nesting dolls work, right? You've got you've got one doll and you open it up and there's another one exactly the same but smaller a smaller fidelity inside and then you keep doing that and over and over. So what what the how the how the voyage models work is when we generate let's suppose you did at 1024 let's suppose you wanted 1024 dimensions you run some tests now you want to try 512 with a traditional model you have to run your entire corpus of data through a second time at 512 but with voyage models you don't have to do that when you've run it through at 1024 vectors those floats they're ordered So if you want to try 512, you just lop off the last 512 and that's now you immediately can begin testing on the remaining 512. So again, it doesn't completely solve the problem of figuring out what the right combination of retrieval quality and storage space is, but it helps you get to the answer faster. So these are there when you take some of these things and sum there we've got three or four of these kind of features that make it easier to use and help you get to your final answer more quickly and the idea there is to give you time back in your day that you can work on your business logic instead of figuring out the plumbing. >> Yeah, that's cool. I'm a huge fan of matura anything. Uh when does this stuff become necessary? So for me, I'm basically a business of one and I try to be an early adopter of everything that I can and I do have a pretty well working I call it deep context but basically a retrieval system that allows my agent to go into kind of all of my history from the last five years essentially which is emails and Slack messages and everything I publish online and DMs across all…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.