Zvi Mowshowitz: belief

Transcript context

…y niche. It is also exactly the stuff that if you're listening to this show, you came for. First, just how big a jump Fable is, measured against a number I put on the record before the model came out. I looked back at my prediction from the beginning of the year in the, I think it's, um, gosh, it's the folks that make the AI village that did this, um, this little forecasting competition. Last year for calibration, I made the top 5%, and I consider the results to have been validated by the fact that Ryan Greenblatt and Ajay Kotra were number two and three respectively. So the fact that they beat me, you know, validates the methodology. But okay, did it again this year. For frontier math, I came in above average, uh, above the median, giving it something like, I think I said sixty-three percent for tier four of frontier math. And Fable is twenty-five points ahead of that in the high eighties already in June. And obviously they have, they had this model trained, you know, I guess, I don't know if Mithos preview is exactly the same score, but- But raw capability isn't what unsettled Zvi. It was a behavior in Vending Bench, the simulated little business economics eval, and specifically what the model appeared to understand about its own behavior while it was doing it. things and fraying around the edges. Uh, I think that Venbench was actually the most worrisome sign in the model card. Mm-hmm.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.