High Signal Podcasts Evidence ledger
Method
Browse
How the evidence works

Public evidence record

Sarah Saab

Published podcast speaker

Claims
11
Episodes
1
Shows
1
Named items
0

Claim ledger

What Sarah said.

11 transcript-backed records

01 / belief

I think people working at the very frontier of state of the art models, I think, do believe that, that that evals need to be rigorous and robust and humans have to be in the loop. But I think we are also at a very, very early stage of sort of break it and apologize later, where I think a big swathe of our industry doesn't yet think that that sort of human mediated evaluation is going to be important and perhaps will get in the way of innovation.

“I think people working at the very frontier of state of the art models, I think, do believe that, that that evals need to be rigorous and robust and humans have to be in the loop. But I think we are also at a very, very early stage of sort of break it and apologize later, where I think a big swathe of our industry doesn't yet think that that sort of human mediated evaluation is going to be important and perhaps will get in the way of innovation.”
Speaker
Sarah Saab
Publisher
Machine Learning Street Talk

02 / belief

The first misconception is that the work is sort of grinding or boring, which actually in our experience when it comes to providing either training or evaluation or fine tuning data for SOTA models, it's not.

“The first misconception is that the work is sort of grinding or boring, which actually in our experience when it comes to providing either training or evaluation or fine tuning data for SOTA models, it's not.”
Speaker
Sarah Saab
Publisher
Machine Learning Street Talk

03 / belief

There's this image of it, I think, in the industry of being seedy. And I think that's not helped by some of the history of how, you know, data has been extracted from human beings whether or not they they know that their data is being used.

“There's this image of it, I think, in the industry of being seedy. And I think that's not helped by some of the history of how, you know, data has been extracted from human beings whether or not they they know that their data is being used.”
Speaker
Sarah Saab
Publisher
Machine Learning Street Talk

04 / belief

I'm I'm I'm really sort of vibing with the way you described that. And I think the very first thing we need to do is try to get perspective into the mix through representativeness.

“I'm I'm I'm really sort of vibing with the way you described that. And I think the very first thing we need to do is try to get perspective into the mix through representativeness.”
Speaker
Sarah Saab
Publisher
Machine Learning Street Talk

06 / belief

You know, humans live in society and they tend to share their cultural beliefs with their tribes. And I think that's why being able to stratify the data you gather for evaluation from people, I think is quite powerful actually.

“You know, humans live in society and they tend to share their cultural beliefs with their tribes. And I think that's why being able to stratify the data you gather for evaluation from people, I think is quite powerful actually.”
Speaker
Sarah Saab
Publisher
Machine Learning Street Talk

07 / belief

Although we are getting much better at the cross functional collaboration between researchers, academics, industry people, public bodies, and it's just such a very interesting point in, I think, the history of science and the future of science.

“Although we are getting much better at the cross functional collaboration between researchers, academics, industry people, public bodies, and it's just such a very interesting point in, I think, the history of science and the future of science.”
Speaker
Sarah Saab
Publisher
Machine Learning Street Talk

08 / belief

You know, those very, very enduring problems of what it means to be a person and a thinker are suddenly impossible to ignore as we're doing software releases and model deployments. I think yeah, that that's super unexpected for me.

“You know, those very, very enduring problems of what it means to be a person and a thinker are suddenly impossible to ignore as we're doing software releases and model deployments. I think yeah, that that's super unexpected for me.”
Speaker
Sarah Saab
Publisher
Machine Learning Street Talk

09 / belief

We're all trying to deliver stuff and there's a, you know, accelerating sort of, you know, hot industry around us, and the idea of waiting on a human to tell us if a thing worked feels counterintuitive, and I think our approach to that is stick a really well treated, verified, diversely demographic human behind an API, essentially, and make sure that the structures and infrastructure are there to ensure that human can go fast, understand instructions, and give you something akin to deterministic human in the loop behaviors.

“We're all trying to deliver stuff and there's a, you know, accelerating sort of, you know, hot industry around us, and the idea of waiting on a human to tell us if a thing worked feels counterintuitive, and I think our approach to that is stick a really well treated, verified, diversely demographic human behind an API, essentially, and make sure that the structures and infrastructure are there to ensure that human can go fast, understand instructions, and give you something akin to deterministic human in the loop behaviors.”
Speaker
Sarah Saab
Publisher
Machine Learning Street Talk

10 / evaluation

So I was talking to Claude about the halting problem a few days ago, and I was like, well, do you mean the algorithm won't end, or do you mean someone won't unplug the computer? And Claude goes, no, I mean, the algorithm won't end, you know, if we cannot confirm or deny whether the algorithm would end, and I just I just thought to myself, that seems kind of nonsensical to me because, you know, the universe will end.

“So I was talking to Claude about the halting problem a few days ago, and I was like, well, do you mean the algorithm won't end, or do you mean someone won't unplug the computer? And Claude goes, no, I mean, the algorithm won't end, you know, if we cannot confirm or deny whether the algorithm would end, and I just I just thought to myself, that seems kind of nonsensical to me because, you know, the universe will end.”
Speaker
Sarah Saab
Publisher
Machine Learning Street Talk

11 / preference

I love the word orchestration because it has the root word orchestra, which is kind of this beautiful, collaborative, you know, symphonic word, and I think that to me is the, you know, correcting a 5 year old over and over again, or correcting an 18 year old or a 25 year old about life, right, over and over again.

“I love the word orchestration because it has the root word orchestra, which is kind of this beautiful, collaborative, you know, symphonic word, and I think that to me is the, you know, correcting a 5 year old over and over again, or correcting an 18 year old or a 25 year old about life, right, over and over again.”
Speaker
Sarah Saab
Publisher
Machine Learning Street Talk
Search evidence