Eilon Reshef: evaluation

Transcript context

…So, what I'm hearing is don't assume you can just outsource all your AI magic model building to the foundational model companies, you need to have your own AI expertise, ML expertise? Yeah, or even if you end up outsourcing the core work, at least you have to have the expertise to understand what is doable, what is not doable, what's the right way to approach it, what's the input you get to the LLM, is this going to be good quality or bad quality? Even if you just take the product management aspect, if the LLM gives you something that is 90% accurate, or I don't know, people are going to think is good, the product's going to look different than if it's 50% good. So, just the way you even think about it, the way you... I think Figma calls their AI feature like First Draft, which is a term I like because they realize it's not best, it's not great, but it's a good first draft. So, if you know what it is, it's easier not just to name, but how conceptualize, how to build a workflow around it, and what to train users to assume for it, and I think there is an expertise there that comes on top of LLMs, even if you just use LLMs, and you can't afford or you don't want to go deeper. For folks that want to do this at their company, what are the functions that you have that help you do this/skills of people you hire that you think are important?…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.