Simon Hørup Eskildsen: prediction

Transcript context

…Did you say it grows by feature? So for five to 30 is by the number of, like, what’s the, what’s the Scaling factor scale? It scales by the number of articles that you embed. It does, but what I meant by that is like five grand for like all of the other, like the Heroku, dinos, Postgres, like all the other, and this then storage is 30. Yeah. And then like 30 grand for one feature. Right. Which is like, what other articles are related to this one. Um, so it was just too much right to, to power everything. Their budget would’ve been maybe a few thousand dollars, which still would’ve been a lot. And so we put it in a bucket of, okay, we’re gonna do that later. We’ll wait, we will wait for the cost to come down. And that haunted me. I couldn’t stop thinking about it. I was like, okay, there’s clearly some latent demand here. If the cost had been a 10th, we would’ve shipped it and. This was really the only data point that I had. Right. I didn’t, I, I didn’t, I didn’t go out and talk to anyone else. It was just so I started reading Right. I couldn’t, I couldn’t help myself. Like I didn’t know what like a vector index is. I, I generally barely do about how to generate the vectors. There was a lot of hype about, this is a early 2023. There was a lot of hype about vector databases. There were raising a lot of money and it’s like, I really didn’t know anything about it. It’s like, you know, trying these little models, fine tuning them. Like I was just trying to get sort of a lay of the land. So I just sat down. I have this. A GitHub repository called Napkin Math. And on napkin math, there’s just, um, rows of like, oh, this is how much bandwidth. Like this is how many, you know, you can do 25 gigabytes per second on average to dram. You can do, you know, five gigabytes per second of rights to an SSD, blah blah. All of these numbers, right? And S3, how many you could do per, how much bandwidth can you drive per connection? I was just sitting down, I was like, why hasn’t anyone build a database where you just put everything on O storage and then you puff it into NVME when you use the data and you puff it into dram if you’re, if you’re querying it alive, it’s just like, this seems fairly obvious and you, the only real downside to that is that if you go all in on o storage, every right will take a couple hundred milliseconds of latency, but from there it’s really all upside, right? You do the first go, it takes half a second. And it sort of occurred to me as like, well. The architecture is really good for that. It’s really good for AB storage, it’s really good for nvm ESSD. It’s, well, you just couldn’t have done that 10 years ago. Back to what we were talking about before. You really have to build a database where you have as few round trips as possible, right? This is how CPUs work today. It’s how NVM E SSDs work. It’s how as, um, as three works that you want to have a very large amount of outstanding requests, right? Like basically go to S3, do like that thousand requests to ask for data in one round trip. Wait for that. Get that, like, make a new decision. Do it again, and try to do that maybe a maximum of three times. asically go to S3, do like that thousand requests to ask for data in one round trip. Wait for that. Get that, like, make a new decision. Do it again, and try to do that maybe a maximum of three times. But no databases were designed that way within NVME as is ds. You can drive like within, you know, within a very low multiple of DRAM bandwidth if you use it that way. And same with S3, right? You can fully max out the network card, which generally is not maxed out. You get very, like, very, very good bandwidth. And, but no one had built a database like that. So I was like, okay, well can’t you just, you know, take all the vectors right? And plot them in the proverbial coordinate system. Get the clusters, put a file on S3 called clusters, do json, and then put another file for every cluster, you know, cluster one, do js O cluster two, do js ON you know that like it’s two round trips, right? So you get the clusters, you find the closest clusters, and then you download the cluster files like the, the closest end. And you could do this in two round trips.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.