High Signal Podcasts Evidence ledger
Method
Browse
How the evidence works

Public evidence record

Michael Royzen

Published podcast speaker

Claims
15
Episodes
1
Shows
1
Named items
3

Books, apps, and tools

The evidenced stack.

Browse the grouped index →

tool / uses

BERT

“I used the standard BERT and also Longformer, which came out around the same time. And Longformer was interesting because it had a much bigger context window than those models at the time, like BERT, all of the first gen encoder only models, they only had a context window of 512 tokens and it's fixed. There's none of this alibi or ROPE that we have now where we can basically massage it to be longer. They're fixed, 512 absolute encodings. Longformer at the time was the only way that you can fit, say, like a sequence length or ask a question about like 4,000 tokens worth of text.”

Latent Space · 3 Nov 2023

Evidence receipt · Source ↗

tool / uses

Longformer

“I used the standard BERT and also Longformer, which came out around the same time. And Longformer was interesting because it had a much bigger context window than those models at the time, like BERT, all of the first gen encoder only models, they only had a context window of 512 tokens and it's fixed. There's none of this alibi or ROPE that we have now where we can basically massage it to be longer. They're fixed, 512 absolute encodings. Longformer at the time was the only way that you can fit, say, like a sequence length or ask a question about like 4,000 tokens worth of text.”

Latent Space · 3 Nov 2023

Evidence receipt · Source ↗

other / likes

Nvidia

“And we start working with Nvidia, which is great. And something that I love about Nvidia, by the way, is that after that intro, we got matched with like a dedicated team.”

Latent Space · 3 Nov 2023

Evidence receipt · Source ↗

Claim ledger

What Michael said.

4 transcript-backed records

01 / belief

Yeah, sorry. So we filtered Common Crawl just by the top, I think, 10,000, just to limit this, because obviously there's this massive long tail of small sites that are really cool, actually.

“Yeah, sorry. So we filtered Common Crawl just by the top, I think, 10,000, just to limit this, because obviously there's this massive long tail of small sites that are really cool, actually.”
Speaker
Michael Royzen
Publisher
Latent Space

03 / belief

My take is that it's much easier having been on both sides of that coin now, it's much easier to stay obsessed every single day when the genesis of your startup is something that really spoke to you in an incredibly meaningful way beyond just being some insight that you've noticed.

“My take is that it's much easier having been on both sides of that coin now, it's much easier to stay obsessed every single day when the genesis of your startup is something that really spoke to you in an incredibly meaningful way beyond just being some insight that you've noticed.”
Speaker
Michael Royzen
Publisher
Latent Space
Search evidence