Public evidence record
Published podcast speaker
Claim ledger
8 transcript-backed records
01 / belief
“Like they kind of occupy the same purposes and you call them, it does something on the system and, and that’s done. I think that in pre-training there’s just an enormous amount.”
02 / belief
“I think at this point a lot of people are familiar with the term of inference. Like funnily enough, like I went from, you know, inference being like a really niche topic to being something that’s like discussed on like normal people’s Twitter feeds.”
03 / commitment
“We will see before the end of the year an agent that is capable of running for longer than 24 hours with like self consistency the entire time.”
04 / evaluation
“Because there’s usually some, this is usually like a constant, you, you know, the SLA that you need to hit and then like you try and find the lowest cost version that hits all of these constraints.”
05 / prediction
“On the decode side because you’re doing a full Passover, all the weights and the entire sequence, every time you do a decode step and you’re, you don’t have the quadratic computation of KV cache, it’s usually memory bound because you’re retrieving a linear amount of memory and you’re doing a linear amount of compute as opposed to prefill where you retrieve a linear amount of memory and then use a quadratic.”
06 / observation
“Uh, the common models like Deeplearning recommendation model, which came outta meta and the wide and deep model, which was used or was released by Google were very accelerated by GPUs using, you know, the fast HBM on the chips, especially to do, you know, vector lookups.”
07 / observation
“Hey, you know, how do you program something in Cuda and run it? And then, and then we built, you know, like when Deep Learning was getting big, we built, we built Torch and, and, but so recently the amount of like layers that are added to that developer stack has just exploded because AI has become ubiquitous.”
08 / preference
“Yeah. And I’ll, I’ll go back and I have like a little crappy logging software I use and there’s just times where it wants to, like, I’m gonna go deep on research and it’ll, I eat up 80,000 tokens go on another go on another, yeah.”