Nathan Labenz: evaluation

Transcript context

…And search is, you know, when you saw that search was 90% of your bill, you know, a, a lot of estimates think maybe 10 to 30% of the overall inference market is going to be search. And everyone thinks inference is going to be huge. And so I think the investors and enterprises are just now realizing how much they need search and what a big part it has to play in the world to come. So on search, one of the most interesting questions I think is on search engine optimization, which has in the last, you know, 2 decades been this enormous consumer marketing growth area. And there are a lot of questions because a lot of the web pages that you see on the web are marketing pages, which are built for SEO. And a lot of them are repetitive. They actually repeat other people's content. They paraphrase, We've had an industry the last two decades of billions of dollars being spent on SEO content. And one of the questions that I have is how does this kind of like more semantic based search end up changing what the SEO people will do? Because I, I often feel like you almost, you're almost kind of trying to prompt inject the LLM which is running the search and you're trying to get in there and hack it so that your, you know, your page goes up. So how does this work? Is is it an adversarial process with, you know, between the search engine provider and the SEO? I mean, I guess it's always been a little bit of adversarial in that people always try to get to 1st place on keyword search. But I, I wonder if the SEO folks are also reading AI research and that is something I don't know. But one of the interesting things that happened recently, again, another research article from Google, is that large language models remember better if you actually put the same information twice in the context, it kind of makes sense because they're going to look backwards as they're, as the context is learning. And so if it appears twice, they're kind of more likely to reinforce it. So I think the number of sites that are actually going to repeat key messages is going to grow because that repeated message is more likely to be picked up in an LLM answer if it is served by, you know, search either vector or keyword in. And so then you're right, there is there is going to be an escalation then of of looking at duplicates, near duplicates, semantic duplicates, rephrasing in order to make sure that the context stay as efficient and and unbiased.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.