Hamel Husain: belief

Transcript context

…Do you have thoughts on what Statsig is going to do at OpenAI? Is there anything there that's interesting? That was a big deal, a huge acquisition. A- B test company people are like, "A-B test, the future." Thoughts? Just to add to the previous question a little bit, why is there this debate, A-B testing versus evals? I think, fundamentally, evals is... People are trying to wrap their head around how to improve their applications and fundamentally need to do... Data science is useful in products. Looking at data, doing data analytics. There's many different suite of tools, and you don't need to invent anything new. Sure, you don't need necessarily the whole breadth of data science, and it looks slightly different, just slightly, with LLMs. Your tactics might be different, so really what it is is using analytic tools to understand your product. Now, people say the word "Evals," trying to carve out this new thing, and saying evals and then A-B testing, but if you zoom out, it's the same data science as before, and I think that's what's causing the confusion is, "Hey, we need data science thinking," and AI product is helpful to have that thinking in AI products like it is in any product is my take on that. That's a really good take, I think just the word "Evals" triggers people now.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.