Shawn Wang: evaluation

Transcript context

…Yeah, definitely. And like when you do geology, the time scale is just not the same. There is like one specific place in France where you can see rocks that are 1 billion years old and like the sheer scale of this is huge. Yeah, that's what I loved about geology, that the scale is completely different and it makes us see the rest of the world in perspective, I guess. We are like a blink in the length of time of the earth. But a very significant blink. So you went from large monoliths to large language models. I don't know how to make that transition there. And yeah, so like maybe could you describe your journey into Hugging Face? Obviously, I think you're like our second or third person from Hugging Face on the podcast and it's like the definitional sort of open AI company, maybe the real open AI. Yeah, I did. So at the end of my PhD, I realized that I did not want to stay in academia. And I actually got contacted by Meta because they wanted to offer me an internship. And I was like, wow, I can do an internship during my PhD. Where do I want to do an internship? And so I applied to Hugging Face. Thank you Meta for like opening this door for me. And I actually was hired to work on pre-trained graph transformers. And we train foundational graph transformer models. And it was a very interesting project. But it was a bit hard to accomplish with the resources we had at the time. We tried it for three months, gave it three months more. So the first three months were my internship, three months more were my first three months at Hugging Face. And then we dropped it. We still left a lot of artifacts about graph machine learning that people can use. But we stopped trying to basically compete with Google on this specific topic. And then we had a team which was doing model, like literally LLM training at the time. And we made a list of the different topics. And one topic which people were not interested in was actually evaluation. So I spent a month just reading all the papers I could about evaluation. And I discovered that it was very interesting. And so we started setting up our own internal evaluation suite, which later became LightEval. And once Tom saw that we were interested in evaluation, he sent us the leaderboard, which was a completely different initiative at the time. And then we basically became a small team doing evaluation and leaderboards at Hugging Face. I'm saying we because I'm including Nathan Habib, who is the engineer working with me on evaluation and leaderboards at Hugging Face.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.