Alessio Fanelli: preference

Transcript context

…So I think this again gets to the human process orchestration. It's one thing to do like a single monolithic push to create a training data set like that or an evaluation corpus. But I think it's another to have a repeatable process. And a lot of that realistically is pretty unsexy, like people management work. So that's probably a big part of it. So we have these four wars of AI framework, the data quality war, we kind of touched on a little bit now. About RAG, that's like the other battlefield, RAG and context sizes and kind of like all these different things. You work in a space that has a couple of different things. One, temporality of data is important because every quarter there's new data and like the new data usually overrides the previous one. So you cannot just like do semantic search and hope that you get the latest one. And then you have obviously very structured numbers thing that are very important to the token level. Like, you know, 50% gross margins and 30% gross margins are very different, but you know, this organization is not that different. Any thoughts on like how to build a system to handle all of that as much as you can share, of course? Yeah, absolutely. So I think this again, rather than having open ended retrieval, open ended reasoning, our approach is to decompose the problem into multiple different subsystems that have specific goals. And so, I mean, temporality is a great example. When you think about time, I mean, just look at all of the libraries for managing calendars. Time is kind of at the intersection of language and math. And this is one of the places where, without taking specific technical measures to ensure that you get high quality narrative overlays of statistics that are changing over time and have a description of how a PE multiple is increasing or decreasing, and like a retrieval system that is aware of the time, sort of the time intent of the user query, right? So if I'm asking something about breaking news, that's going to be very different than if I'm looking for a thematic account of the past 18 months in Fed interest rate policy. You have to have retrieval systems that are, to your point, like if I just look for something that is a nearest neighbor without any of that temporal or other qualitative metadata overlay, you're just going to get a kind of a bag of facts. I think that that is explicitly not helpful, because the worst failure state for these systems is that they are wrong in a convincing way. And so I think, at least presently, you have to have subsystems that are aware of the semantics of the documents, or aware of the semantics of the intent behind the question, and then have multiple, we have multiple evaluation steps. Once you have the generated outputs, we assess it multiple different ways to know, is this a factual statement given the sort of content that's been retrieved?…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.