app / likes
Spark
“Um, the only thing I like out of, out of Codex is the, is like Spark and like yeah.”
Public evidence record
Shawn Wang
Host · Latent Space
- Claims
- 500
- Episodes
- 64
- Shows
- 1
- Named items
- 20
Books, apps, and tools
The evidenced stack.
app / uses
Bank of America
“Yeah. I got, I got the tool. Uh, what, like, I hate, I use Bank of America. I hate bank, I hate the app. Mm-hmm. I hate the web. All banking websites just horrible.”
other / likes
Wiki approach
“I like the, the Wiki approach. Uh, my, I’m actually like, uh, you know, obviously I spent some my time at cognition, which, uh, you, you know very well.”
person / uses
Ted Chiang
“You were very excited because I read Ted Chiang over the holidays and I was very inspired by this short story called Understand, which apparently is, like, pretty old.”
app / uses
Overcast
“I used to use Overcast. So it would just link to the Overcast page.”
other / uses
Snip
“By the way, we use the snip count as a proxy for popularity, right? Because we have download counts, but for example, platforms like Spotify re-host our MP3 file.”
other / recommends
The Peel
“I think I strongly recommend Jack Bridger's Scaling DevTools, as well as Turner Novak's The Peel.”
other / recommends
Scaling DevTools
“I think I strongly recommend Jack Bridger's Scaling DevTools, as well as Turner Novak's The Peel.”
app / uses
AI News
“So that's what basically I use AI News for. I have a lot of prompts and a lot of steps and a lot of criteria and O1 just kind of checks through each kind of systematically.”
tool / built
Bolt.new
“It's funny because I built on top of the fork of Bolt.new that already has the multi LLM thing.”
other / uses
XML
“I use XML in other models as well, and it's just a really nice way to make sure that the thing that ends is tied to the thing that starts. That's the only way to do code fences where you're pretty sure example one start, example one end, that is one cohesive unit.”
app / uses
DevIn
“I used to tell people go to the DevIn demo and look at the four things that they offer and say each of those things is a startup.”
Claim ledger
What Shawn said.
500 transcript-backed records
01 / belief
Whereas now I think you can with the agents that have learned through the h- through, through time, through all the traces.
“Whereas now I think you can with the agents that have learned through the h- through, through time, through all the traces.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
02 / belief
Specifically on the hardware-software boundary part, it’s, it’s something I think about of our education system, in the States, but also maybe just in generally.
“Specifically on the hardware-software boundary part, it’s, it’s something I think about of our education system, in the States, but also maybe just in generally.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
03 / prediction
‘Cause it’s a statistical thing, but as long I don’t know if regulators understand that, you cannot extrapolate from a single incident, but we do because that’s all we have to go on.
“‘Cause it’s a statistical thing, but as long I don’t know if regulators understand that, you cannot extrapolate from a single incident, but we do because that’s all we have to go on.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
04 / belief
Uh, and, and, and I think that that leads you down a path of like complete domain specificity.
“Uh, and, and, and I think that that leads you down a path of like complete domain specificity.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
05 / belief
Whereas I think the model labs tend to just trust the model and, and be minimalist about it.
“Whereas I think the model labs tend to just trust the model and, and be minimalist about it.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
06 / belief
I, I think there’s just a kind of different stages when, when you talk about the world, one wanting more model companies, I talked think about like the neo labs.
“I, I think there’s just a kind of different stages when, when you talk about the world, one wanting more model companies, I talked think about like the neo labs.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
07 / belief
To, to the end customers and like, well, if they didn’t have you, they would’ve to hire in house and they’re not gonna hire in house so they have you. And like, I think that’s like a reasonable, like very robust to any whatever trends and, and discoveries that people make in, in the engineering layer.
“To, to the end customers and like, well, if they didn’t have you, they would’ve to hire in house and they’re not gonna hire in house so they have you. And like, I think that’s like a reasonable, like very robust to any whatever trends and, and discoveries that people make in, in the engineering layer.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
08 / belief
I, I, I think there, there’s just a sheer amount of like, like un scalability that like is wrangling people’s sensibilities right now.
“I, I, I think there, there’s just a sheer amount of like, like un scalability that like is wrangling people’s sensibilities right now.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
09 / belief
Comparatively, like, I think you should make, Hey, I guess while, while that’s going on, it’s not that bad to be a capabilities explorer on just the $200 a month plan from Cloud Code or from OpenAI.
“Comparatively, like, I think you should make, Hey, I guess while, while that’s going on, it’s not that bad to be a capabilities explorer on just the $200 a month plan from Cloud Code or from OpenAI.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
10 / belief
Um, and obviously I’m involved in cognition and also cursors doing, doing, uh, a lot of own model training. And I think that that is some part of the, what I’ve been calling the agent lab playbook, where you start off with the state of the art models from, uh, from the big labs and you, uh, specialize for your domain.
“Um, and obviously I’m involved in cognition and also cursors doing, doing, uh, a lot of own model training. And I think that that is some part of the, what I’ve been calling the agent lab playbook, where you start off with the state of the art models from, uh, from the big labs and you, uh, specialize for your domain.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
11 / belief
I think that, um, for it to, for the market structure to, to significantly change there would be, there needs to be significant change in like the economics or like the, the brand building or like the, the, the, the value propositions of the, of the companies involved and I.
“I think that, um, for it to, for the market structure to, to significantly change there would be, there needs to be significant change in like the economics or like the, the brand building or like the, the, the, the value propositions of the, of the companies involved and I.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
12 / belief
Of Braintrust where he, and he, I mean, you know, he has a good cross section of all the top AI companies and he says market share of open source is 5% and going down. Um, I think that’s changed.
“Of Braintrust where he, and he, I mean, you know, he has a good cross section of all the top AI companies and he says market share of open source is 5% and going down. Um, I think that’s changed.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
13 / belief
Everyone says like, yeah, we should always, always do this. And honestly like, I think the infrastructure for that is becoming easier with, um, like thinking machines tinker thing as well as primary like, uh, lab stuff.
“Everyone says like, yeah, we should always, always do this. And honestly like, I think the infrastructure for that is becoming easier with, um, like thinking machines tinker thing as well as primary like, uh, lab stuff.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
14 / belief
I, I, to me, the, the, the other thing was the coding one, um, which obviously I, I have now come full 360 on, but I think like.
“I, I, to me, the, the, the other thing was the coding one, um, which obviously I, I have now come full 360 on, but I think like.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
15 / evaluation
I think, uh, I don’t know people say this, but I, I, I don’t think they try it hard.
“I think, uh, I don’t know people say this, but I, I, I don’t think they try it hard.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
16 / recommendation
Yeah, like I would actually advise and topic to tune down the marketing because also it’s, it is just a very good model and you don’t have to make so many marketing claims around it.
“Yeah, like I would actually advise and topic to tune down the marketing because also it’s, it is just a very good model and you don’t have to make so many marketing claims around it.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
17 / commitment
Philosophically aligned on like, we will just enable access everywhere and we don’t know what you, what will come out of it.
“Philosophically aligned on like, we will just enable access everywhere and we don’t know what you, what will come out of it.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
18 / evaluation
Uh, so I think like basically what we’re talking about is the vertical versus horizontal, uh, debate in, in AI startups.
“Uh, so I think like basically what we’re talking about is the vertical versus horizontal, uh, debate in, in AI startups.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
19 / recommendation
Beyond that, I think honestly there is like, so I, I come from the sensibility of, I think everything that you are trying to do for agents experience now, which is the term that Matt Bowman and Nullify is trying to coin, is the same thing that you should have been doing for developer experience.
“Beyond that, I think honestly there is like, so I, I come from the sensibility of, I think everything that you are trying to do for agents experience now, which is the term that Matt Bowman and Nullify is trying to coin, is the same thing that you should have been doing for developer experience.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
20 / prediction
Um, I do think, like, uh, it is interesting that, uh, for a while I was, I was considering the theory that models capped out at two, 2 trillion, and I think that’s proving to be wrong.
“Um, I do think, like, uh, it is interesting that, uh, for a while I was, I was considering the theory that models capped out at two, 2 trillion, and I think that’s proving to be wrong.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
21 / prediction
I think the personalization turn that is coming, um, will be big. And I don’t know what that looks like because like basically we’re kind of, we feel kind of tapped out on the memory side of things.
“I think the personalization turn that is coming, um, will be big. And I don’t know what that looks like because like basically we’re kind of, we feel kind of tapped out on the memory side of things.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
22 / evaluation
Minimal in, in a sense of like, the worst you do is you just get hired into one of these labs anyway. So I, I think the, the market for people who just do things and try things and try to execute in like a competent way, even if like it doesn’t work out commercially, even if it just wasn’t that great anyway.
“Minimal in, in a sense of like, the worst you do is you just get hired into one of these labs anyway. So I, I think the, the market for people who just do things and try things and try to execute in like a competent way, even if like it doesn’t work out commercially, even if it just wasn’t that great anyway.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
23 / preference
Um, the only thing I like out of, out of Codex is the, is like Spark and like yeah.
“Um, the only thing I like out of, out of Codex is the, is like Spark and like yeah.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
24 / belief
Uh, so like I, I think, you know, our conversation like today has like really, uh, oh, I guess opened my eyes to a lot.
“Uh, so like I, I think, you know, our conversation like today has like really, uh, oh, I guess opened my eyes to a lot.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
25 / belief
Cool. Um, I think that that was all the questions I had. I said I, I have one sort of a bonus thing if you, if you wanna indulge in, uh, some Bing history.
“Cool. Um, I think that that was all the questions I had. I said I, I have one sort of a bonus thing if you, if you wanna indulge in, uh, some Bing history.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
26 / uncertainty
I, I don’t know, I don’t know, uh, you know, it’s, uh, people va-variously refer you as like CEO or, or, uh, I don’t know what that, that, that said previous role at Microsoft was.
“I, I don’t know, I don’t know, uh, you know, it’s, uh, people va-variously refer you as like CEO or, or, uh, I don’t know what that, that, that said previous role at Microsoft was.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
27 / belief
I think obviously, you know, there’s a lot of, uh, ML infra at, at Shopify that people can, uh, dive into.
“I think obviously, you know, there’s a lot of, uh, ML infra at, at Shopify that people can, uh, dive into.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
28 / uncertainty
Do you guys use stack diffs? I don’t know if, uh, that’s a, like, a merge queue stack diff type of thing.
“Do you guys use stack diffs? I don’t know if, uh, that’s a, like, a merge queue stack diff type of thing.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
29 / belief
I think I can, I can screen share, and then we can kind of go through some of the shocking stats that maybe, maybe put some numbers to what exactly is going on.
“I think I can, I can screen share, and then we can kind of go through some of the shocking stats that maybe, maybe put some numbers to what exactly is going on.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
30 / belief
I’m familiar with these people for the LLM, you know, autoregressive stack. But the other interesting category of these optimizers is also the diffusion people, whereas like Fel and, you know, uh, Pruna recently has come up a lot as well, which I think is like really underappreciated, uh, at least by myself, because I, I thought, oh, all the workload would be LLMs, but actually there’s a lot of diffusion as well.
“I’m familiar with these people for the LLM, you know, autoregressive stack. But the other interesting category of these optimizers is also the diffusion people, whereas like Fel and, you know, uh, Pruna recently has come up a lot as well, which I think is like really underappreciated, uh, at least by myself, because I, I thought, oh, all the workload would be LLMs, but actually there’s a lot of diffusion as well.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
31 / evaluation
Uh, one of the reasons I reached out was because you started promoting more sort of internal tooling, uh, primarily Tangle, but also a lot of people have seen and adopted Tobi’s QMD, uh, and obviously, I think, uh, Shopify has always been sort of leading in terms of, uh, engineering.
“Uh, one of the reasons I reached out was because you started promoting more sort of internal tooling, uh, primarily Tangle, but also a lot of people have seen and adopted Tobi’s QMD, uh, and obviously, I think, uh, Shopify has always been sort of leading in terms of, uh, engineering.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
32 / evaluation
Like I think for example, right, like in the audio kind, kind of use cases, the SSMs ef-effectively have unbounded context length because they, they just have to operate on like the most, the sliding window of the most recent stuff.
“Like I think for example, right, like in the audio kind, kind of use cases, the SSMs ef-effectively have unbounded context length because they, they just have to operate on like the most, the sliding window of the most recent stuff.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
33 / evaluation
Then the other thing that you mentioned, which also raised my eyebrows, was content-based caching, which you mentioned is, is, um, you know, is ve-very much, uh, um, a sort of efficiency measure about, uh, you know, just like recalculation only on, on sort of content addressing Which I think makes sense.
“Then the other thing that you mentioned, which also raised my eyebrows, was content-based caching, which you mentioned is, is, um, you know, is ve-very much, uh, um, a sort of efficiency measure about, uh, you know, just like recalculation only on, on sort of content addressing Which I think makes sense.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
34 / uncertainty
Uh, you, you, you just have a lot of comments. Uh, you, you, uh, I don’t know where you wanna start.
“Uh, you, you, you just have a lot of comments. Uh, you, you, uh, I don’t know where you wanna start.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
35 / uncertainty
I’m also calling back to there’s this like levels of EGI I don’t know if Opening Eye is still talking about this, but they used to talk about five levels of EGI and one of it was like, oh, it’s like an intern coding software patient.
“I’m also calling back to there’s this like levels of EGI I don’t know if Opening Eye is still talking about this, but they used to talk about five levels of EGI and one of it was like, oh, it’s like an intern coding software patient.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
36 / belief
I think like the only argument I have against this is basically scale testing, which obviously the larger pieces of software like Linux, MySQL, he calls up even the Datadog and Temporals and then maybe security testing where Yes.
“I think like the only argument I have against this is basically scale testing, which obviously the larger pieces of software like Linux, MySQL, he calls up even the Datadog and Temporals and then maybe security testing where Yes.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
37 / uncertainty
You make a pipe errors versus the standard, standard out. I don’t know. Okay. Whatever.
“You make a pipe errors versus the standard, standard out. I don’t know. Okay. Whatever.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
38 / uncertainty
There’s a lot of customization, but there’ll be multiple unicorns just doing this as a service. I don’t know.
“There’s a lot of customization, but there’ll be multiple unicorns just doing this as a service. I don’t know.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
39 / belief
All the coding review agent, it can merge autonomously. I think that’s something that a lot of people aren’t comfortable with.
“All the coding review agent, it can merge autonomously. I think that’s something that a lot of people aren’t comfortable with.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
40 / uncertainty
I was briefly part of the, that, that world is it salt? I don’t know. It’s actually really hard for humans to agree on what revenue is.
“I was briefly part of the, that, that world is it salt? I don’t know. It’s actually really hard for humans to agree on what revenue is.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
41 / belief
I had to go do a crash course in Beam and Elixir, and I think most people are not operating at that scale of concurrency where you need that.
“I had to go do a crash course in Beam and Elixir, and I think most people are not operating at that scale of concurrency where you need that.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
42 / belief
I had a mental reaction when you said don’t put the agent in a box. So I think you should put it in a box.
“I had a mental reaction when you said don’t put the agent in a box. So I think you should put it in a box.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
43 / belief
I think obviously you can spend the whole day going through some of these, but I do think that some of these have a lot of care or some of this you might wanna tell people, Hey, take this, but, make it your own.
“I think obviously you can spend the whole day going through some of these, but I do think that some of these have a lot of care or some of this you might wanna tell people, Hey, take this, but, make it your own.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
44 / commitment
Exactly. That’s these are my road roadmaps for a e wherever people hiring engineers, I will go.
“Exactly. That’s these are my road roadmaps for a e wherever people hiring engineers, I will go.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
45 / belief
Coming back to the final point I wanted to make is, yeah I think that there, there are multiple other, like you guys are working on this, but this is a pattern that every other company out there should adopt.
“Coming back to the final point I wanted to make is, yeah I think that there, there are multiple other, like you guys are working on this, but this is a pattern that every other company out there should adopt.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
46 / belief
I was gonna go back to your, I think the demo videos that you guys had was pretty illustrative.
“I was gonna go back to your, I think the demo videos that you guys had was pretty illustrative.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
47 / belief
I think one of the concerns I have with that kind of stuff is you think you’re making the right call by making, it’s persisted for all time across everything.
“I think one of the concerns I have with that kind of stuff is you think you’re making the right call by making, it’s persisted for all time across everything.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
48 / belief
Let’s introduce Symphony. I think we’ve been mentioning it every now and then. Elixir.
“Let’s introduce Symphony. I think we’ve been mentioning it every now and then. Elixir.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
49 / evaluation
It’s almost like you’re, it’s like a ratchet. It’s like you’re forcing build time discipline, because if you don’t, it’ll just grow and grow.
“It’s almost like you’re, it’s like a ratchet. It’s like you’re forcing build time discipline, because if you don’t, it’ll just grow and grow.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
50 / belief
I think Sun is also too polite to point out that, both like Google’s genie, demos as well as world Labs is marble, do not have interactive worlds.
“I think Sun is also too polite to point out that, both like Google’s genie, demos as well as world Labs is marble, do not have interactive worlds.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
51 / uncertainty
Like, which it’s hard to tell because you put out the end results, but we don’t know the inputs that go into it.
“Like, which it’s hard to tell because you put out the end results, but we don’t know the inputs that go into it.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
52 / belief
I think for your standards it is, for me it feels like vision is, is. I’ll leave that to the big labs graphics.
“I think for your standards it is, for me it feels like vision is, is. I’ll leave that to the big labs graphics.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
53 / belief
I honestly, do think that if people deeply understand everything we just covered, they will see what’s coming. I think you guys have, made some, a really significant contribution here.
“I honestly, do think that if people deeply understand everything we just covered, they will see what’s coming. I think you guys have, made some, a really significant contribution here.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
54 / preference
You could have just said no audio And audio in my mind has a lot of recursion, whereas in video you can just do recasting and that’s much computationally much simpler.
“You could have just said no audio And audio in my mind has a lot of recursion, whereas in video you can just do recasting and that’s much computationally much simpler.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
55 / preference
I will also say, one, my favorite story, books or biographies ever is, creativity Inc.
“I will also say, one, my favorite story, books or biographies ever is, creativity Inc.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
56 / evaluation
That you guys do. But, I, I could see, I could see that I think the, the human intent is something that people are not even used to because we’re so used to static worlds or, worlds that just don’t react, or, I don’t know.
“That you guys do. But, I, I could see, I could see that I think the, the human intent is something that people are not even used to because we’re so used to static worlds or, worlds that just don’t react, or, I don’t know.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
57 / belief
I remember, I don’t think it was January, right? It was like new reps it was, it dropped during new reps and everyone in Europes was December of 25th, I think.
“I remember, I don’t think it was January, right? It was like new reps it was, it dropped during new reps and everyone in Europes was December of 25th, I think.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
58 / belief
I think actually OpenAI has gone away from the original four Oh. Vision of the Omni model.
“I think actually OpenAI has gone away from the original four Oh. Vision of the Omni model.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
59 / belief
I think what’s interesting is just this general theory of developing individual capabilities in different teams and then merging them.
“I think what’s interesting is just this general theory of developing individual capabilities in different teams and then merging them.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
60 / commitment
We started the our new science pod that focuses specifically on the air for science.
“We started the our new science pod that focuses specifically on the air for science.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
61 / preference
The key thing I think over maybe like the last year or so with VO and gr Imagine and all these things is joining voice with video, right?
“The key thing I think over maybe like the last year or so with VO and gr Imagine and all these things is joining voice with video, right?”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
62 / prediction
One of my theories is that because the proofs takes so long, it’s actually just a proxy for long horizon reasoning and coherence and planning.
“One of my theories is that because the proofs takes so long, it’s actually just a proxy for long horizon reasoning and coherence and planning.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
63 / belief
I think we’re gonna go right into a demo. Yeah. I just wanna make an observation that, uh, you, you, [00:02:00] you put discover first before build.
“I think we’re gonna go right into a demo. Yeah. I just wanna make an observation that, uh, you, you, [00:02:00] you put discover first before build.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
64 / uncertainty
Your partnerships, uh, I mean, I don’t know if you ever hit of partnerships, but this is gonna be a bonanza for anyone on to do deals.
“Your partnerships, uh, I mean, I don’t know if you ever hit of partnerships, but this is gonna be a bonanza for anyone on to do deals.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
65 / recommendation
Mentions personal use of Bank of America.
“Yeah. I got, I got the tool. Uh, what, like, I hate, I use Bank of America. I hate bank, I hate the app. Mm-hmm. I hate the web. All banking websites just horrible.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
66 / preference
Kinda obsessed. I, I’ve been using it a lot, even for managing latent space. Like, uh, cowork helps me upload videos and like title things and like edit and everything.
“Kinda obsessed. I, I’ve been using it a lot, even for managing latent space. Like, uh, cowork helps me upload videos and like title things and like edit and everything.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
67 / belief
I think you guys do something different in planning, but, uh, I’ve been talking with Tariq who is on the cloud co team, and you guys are, he’s like, no, we just exposed planning.
“I think you guys do something different in planning, but, uh, I’ve been talking with Tariq who is on the cloud co team, and you guys are, he’s like, no, we just exposed planning.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
68 / belief
I, I think you still want, especially for people who are doing like high performance software or like very complex software, uh, you still want like, some view of the architecture.
“I, I think you still want, especially for people who are doing like high performance software or like very complex software, uh, you still want like, some view of the architecture.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
69 / belief
Uh, you know, look, my, my my, my view is like most things should be Tori by default, unless you really need the full power of electron, but.
“Uh, you know, look, my, my my, my view is like most things should be Tori by default, unless you really need the full power of electron, but.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
70 / belief
I mean, yeah, it’s, it’s weird how like, I guess usually I think safety and security is kind of like a boring word to, to engineers.
“I mean, yeah, it’s, it’s weird how like, I guess usually I think safety and security is kind of like a boring word to, to engineers.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
71 / belief
I think you’re, you’re kind of back on the Windows grind ‘cause you’re building out the Windows support.
“I think you’re, you’re kind of back on the Windows grind ‘cause you’re building out the Windows support.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
72 / belief
Like sometimes by, by default, some of the theis, I think even early called code, uh, we have to approve every single command.
“Like sometimes by, by default, some of the theis, I think even early called code, uh, we have to approve every single command.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
73 / uncertainty
Um, and I I, we have like Claude Coworks, it might be a GI, I don’t know if we, we have, uh, uploaded it yet, but one of the sessions was like a, like a Claude cowork thing.
“Um, and I I, we have like Claude Coworks, it might be a GI, I don’t know if we, we have, uh, uploaded it yet, but one of the sessions was like a, like a Claude cowork thing.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
74 / belief
Uh, but before we get into it, I, I think always wanna start with like a top level.
“Uh, but before we get into it, I, I think always wanna start with like a top level.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
75 / belief
I, I think you can actually do a better job of this. And I, this, this is one shot at my, uh, conference website.
“I, I think you can actually do a better job of this. And I, this, this is one shot at my, uh, conference website.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
76 / commitment
Do, do you operate, like if there’s one cha there must have at least one champion who’s like, yes, I will put my career on, on, on the line for this.
“Do, do you operate, like if there’s one cha there must have at least one champion who’s like, yes, I will put my career on, on, on the line for this.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
77 / uncertainty
Uh, people are 10 x saying, everyone’s saying, you know, uh, maybe engineers are out of a job. I don’t know.
“Uh, people are 10 x saying, everyone’s saying, you know, uh, maybe engineers are out of a job. I don’t know.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
78 / belief
I think for me, like the, the, the learning is kind of like you, like all workloads are hybrid.
“I think for me, like the, the, the learning is kind of like you, like all workloads are hybrid.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
79 / belief
There’s, but like, I think people are trained to write a certain way in school and Yeah.
“There’s, but like, I think people are trained to write a certain way in school and Yeah.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
80 / belief
I would say, yeah, like RU had this tweet where like everyone was in SF from like 2021 to 2023.
“I would say, yeah, like RU had this tweet where like everyone was in SF from like 2021 to 2023.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
81 / belief
Um, I think one thing that’s like kind of, of the moment right now is people are asking, is there any SOL sort of upper bounds.
“Um, I think one thing that’s like kind of, of the moment right now is people are asking, is there any SOL sort of upper bounds.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
82 / belief
Like, so like that, that, that chart that you see is them estimating what the human equivalent replacement is. Um, I think the, I think actually Enro release a more recent chart.
“Like, so like that, that, that chart that you see is them estimating what the human equivalent replacement is. Um, I think the, I think actually Enro release a more recent chart.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
83 / evaluation
Like, I don’t know if, I don’t know if I can say that, but like, you know, um, I think what my point kind of is, is that there’s, like, I look at slopes of the scaling laws and like, this slope is not working, man.
“Like, I don’t know if, I don’t know if I can say that, but like, you know, um, I think what my point kind of is, is that there’s, like, I look at slopes of the scaling laws and like, this slope is not working, man.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
84 / prediction
I think you, you had [00:01:00] cloud agents before, but this was like, you give cursor a computer, right?
“I think you, you had [00:01:00] cloud agents before, but this was like, you give cursor a computer, right?”
- Speaker
- Shawn Wang
- Episode
- Cursor's Third Era: Cloud Agents
- Publisher
- Latent Space
85 / evaluation
Yeah I would say, let’s call it a year ago the models weren’t even good enough to do any of this stuff.
“Yeah I would say, let’s call it a year ago the models weren’t even good enough to do any of this stuff.”
- Speaker
- Shawn Wang
- Episode
- Cursor's Third Era: Cloud Agents
- Publisher
- Latent Space
86 / prediction
So obviously has a research team. And my thesis is like you just, every agent lab is going to have a router because you’re going to be asked like, what’s what.
“So obviously has a research team. And my thesis is like you just, every agent lab is going to have a router because you’re going to be asked like, what’s what.”
- Speaker
- Shawn Wang
- Episode
- Cursor's Third Era: Cloud Agents
- Publisher
- Latent Space
87 / evaluation
Because Open AI is Codex app doesn’t have a file editor, like it has file viewer, but isn’t a file editor.
“Because Open AI is Codex app doesn’t have a file editor, like it has file viewer, but isn’t a file editor.”
- Speaker
- Shawn Wang
- Episode
- Cursor's Third Era: Cloud Agents
- Publisher
- Latent Space
88 / recommendation
Sometimes the, I don’t know if there’s a, this probably, you guys probably figured this out already, but since I, you need like a mute button.
“Sometimes the, I don’t know if there’s a, this probably, you guys probably figured this out already, but since I, you need like a mute button.”
- Speaker
- Shawn Wang
- Episode
- Cursor's Third Era: Cloud Agents
- Publisher
- Latent Space
89 / preference
I think that one’s interesting for Datadog. ‘cause Datadog wants to own that site.
“I think that one’s interesting for Datadog. ‘cause Datadog wants to own that site.”
- Speaker
- Shawn Wang
- Episode
- Cursor's Third Era: Cloud Agents
- Publisher
- Latent Space
90 / recommendation
You decide from me based on all the other people you know better than me. And I think every agent lab should basically end up doing this because that actually gives you extra power because you like people stop carrying or having loyalty with one lab.
“You decide from me based on all the other people you know better than me. And I think every agent lab should basically end up doing this because that actually gives you extra power because you like people stop carrying or having loyalty with one lab.”
- Speaker
- Shawn Wang
- Episode
- Cursor's Third Era: Cloud Agents
- Publisher
- Latent Space
91 / evaluation
There’s all these like really basic questions that no one stops to answer for people because everyone’s just like too busy launching.
“There’s all these like really basic questions that no one stops to answer for people because everyone’s just like too busy launching.”
- Speaker
- Shawn Wang
- Episode
- Cursor's Third Era: Cloud Agents
- Publisher
- Latent Space
92 / evaluation
Yeah. They want to be like we own your logs and give us our, some part of the, [00:17:00] self-healing software that everyone wants.
“Yeah. They want to be like we own your logs and give us our, some part of the, [00:17:00] self-healing software that everyone wants.”
- Speaker
- Shawn Wang
- Episode
- Cursor's Third Era: Cloud Agents
- Publisher
- Latent Space
93 / preference
That’s an interesting one because what I actually do want, like from a manna and open crawl, whatever, is like I want to be able to log in with my credentials to the thing, but not actually store it in any like secret store, whatever.
“That’s an interesting one because what I actually do want, like from a manna and open crawl, whatever, is like I want to be able to log in with my credentials to the thing, but not actually store it in any like secret store, whatever.”
- Speaker
- Shawn Wang
- Episode
- Cursor's Third Era: Cloud Agents
- Publisher
- Latent Space
94 / recommendation
Likes Wiki approach.
“I like the, the Wiki approach. Uh, my, I’m actually like, uh, you know, obviously I spent some my time at cognition, which, uh, you, you know very well.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
95 / belief
I mean, I think, I think like, you know, do do what some companies are doing, you know, I’m not saying it’s my situation exactly, but like give them equity and like Uhhuh it should probably would be worth more, uh, just like sort of helping them out.
“I mean, I think, I think like, you know, do do what some companies are doing, you know, I’m not saying it’s my situation exactly, but like give them equity and like Uhhuh it should probably would be worth more, uh, just like sort of helping them out.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
96 / belief
I think I, I think people take these eval numbers in pretty charts for granted, but No, there, I mean, there’s, there’s lots of really smart people at work during all this.
“I think I, I think people take these eval numbers in pretty charts for granted, but No, there, I mean, there’s, there’s lots of really smart people at work during all this.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
97 / belief
You know, what, what, what, what should I, what should I do? I think there used to be TechCrunch Disrupt.
“You know, what, what, what, what should I, what should I do? I think there used to be TechCrunch Disrupt.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
98 / belief
I think, you know, so one of the things that stability ai, uh, made an impression on me was like, well, you know, and at least now we can remake Game of Throne Season eight, and I can, you know, uh, like, like it was meant to be not, uh, not rushed.
“I think, you know, so one of the things that stability ai, uh, made an impression on me was like, well, you know, and at least now we can remake Game of Throne Season eight, and I can, you know, uh, like, like it was meant to be not, uh, not rushed.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
99 / belief
I do think like, uh, a lot of the last year’s, 2025 story was the rise of coding agents and I think [00:29:00] 2026 story is definitely knowledge work agents.
“I do think like, uh, a lot of the last year’s, 2025 story was the rise of coding agents and I think [00:29:00] 2026 story is definitely knowledge work agents.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
100 / commitment
Uh, and now I know you, you cut off his 8:00 PM I will try to get AI news out before 8:00 PM so I can help him.
“Uh, and now I know you, you cut off his 8:00 PM I will try to get AI news out before 8:00 PM so I can help him.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
101 / evaluation
I love, I love pushing back. I think that. That is what a lot of technology consultants love to hear this sort of thing, right?
“I love, I love pushing back. I think that. That is what a lot of technology consultants love to hear this sort of thing, right?”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
102 / belief
I think the let’s, let’s sort of reset on like, what was the original work that you guys did for Verified, which I think was pretty substantial.
“I think the let’s, let’s sort of reset on like, what was the original work that you guys did for Verified, which I think was pretty substantial.”
- Speaker
- Shawn Wang
- Episode
- ⚡️The End of SWE-Bench Verified — Mia Glaese & Olivia Watkins, OpenAI Frontier Evals & Human Data
- Publisher
- Latent Space
103 / commitment
I will note that you guys had a trend in verifying benchmarks. ‘cause I just recently saw, I think Quinn had a HLE verified for humanities.
“I will note that you guys had a trend in verifying benchmarks. ‘cause I just recently saw, I think Quinn had a HLE verified for humanities.”
- Speaker
- Shawn Wang
- Episode
- ⚡️The End of SWE-Bench Verified — Mia Glaese & Olivia Watkins, OpenAI Frontier Evals & Human Data
- Publisher
- Latent Space
104 / evaluation
I, I think you guys have, you know, really made a lot of progress and I think taking a lot of industry leadership for C Bench verified and, and now moving on to C Orange Pro.
“I, I think you guys have, you know, really made a lot of progress and I think taking a lot of industry leadership for C Bench verified and, and now moving on to C Orange Pro.”
- Speaker
- Shawn Wang
- Episode
- ⚡️The End of SWE-Bench Verified — Mia Glaese & Olivia Watkins, OpenAI Frontier Evals & Human Data
- Publisher
- Latent Space
105 / belief
Maybe decade, but, uh, uh, just like I, I think, you know, I, I just going back to the discussion about how a GI would just kind of consume everything.
“Maybe decade, but, uh, uh, just like I, I think, you know, I, I just going back to the discussion about how a GI would just kind of consume everything.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
106 / belief
I, I mean, to me, like, you know, philanthropic, like building on cloud code, I think, uh, it makes sense to me the, the real.
“I, I mean, to me, like, you know, philanthropic, like building on cloud code, I think, uh, it makes sense to me the, the real.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
107 / belief
I, I think that, uh, you know, speaking about ad there’s, there’s a really like interesting tie in that obviously you guys are hit on, which is like these sort, this sort of like America first movement or like sort of re industrialized here.
“I, I think that, uh, you know, speaking about ad there’s, there’s a really like interesting tie in that obviously you guys are hit on, which is like these sort, this sort of like America first movement or like sort of re industrialized here.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
108 / belief
My, my, uh, sort of, uh, thesis, which we’re not gonna have to go into here is actually I think a, um, what I’ve been calling Agent Labs, which are [00:54:00] people who build on top of, uh, all the other models.
“My, my, uh, sort of, uh, thesis, which we’re not gonna have to go into here is actually I think a, um, what I’ve been calling Agent Labs, which are [00:54:00] people who build on top of, uh, all the other models.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
109 / preference
I think like it’s important to hear you guys’ perspective because the rest of us are just kind of looking at headlines and not knowing how to make sense of any of this.
“I think like it’s important to hear you guys’ perspective because the rest of us are just kind of looking at headlines and not knowing how to make sense of any of this.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
110 / belief
Uh, cool. Uh, I think. There, there’s more general, like broad questions, but like, I guess what, what do you, uh, wish you were asked more in, in, in general, like, you know, like you, you have such a broad scope.
“Uh, cool. Uh, I think. There, there’s more general, like broad questions, but like, I guess what, what do you, uh, wish you were asked more in, in, in general, like, you know, like you, you have such a broad scope.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
111 / belief
I think, uh, one thing that, uh, anti-gravity from, from Google also did was like, just come out the gate to very, very strong multimodal, including videos, and that’s the highest bandwidth communication prompt that you can give to the model, which is fantastic.
“I think, uh, one thing that, uh, anti-gravity from, from Google also did was like, just come out the gate to very, very strong multimodal, including videos, and that’s the highest bandwidth communication prompt that you can give to the model, which is fantastic.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
112 / uncertainty
I don’t know if you have any predictions that you, that you like to keep, you know, like, uh, one, one way to do this is you have your tests whenever a new model comes out that you run, uh, what’s something that you’re, you’re not quite happy with yet.
“I don’t know if you have any predictions that you, that you like to keep, you know, like, uh, one, one way to do this is you have your tests whenever a new model comes out that you run, uh, what’s something that you’re, you’re not quite happy with yet.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
113 / belief
Uh, I think one of your, your favorite examples was you can put a low resource language in the context and it just learns.
“Uh, I think one of your, your favorite examples was you can put a low resource language in the context and it just learns.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
114 / uncertainty
Like one of my, uh, so I interviewed ETA who was on, who was on that team. Uh, and he was like, yeah, I, I don’t know how they work.
“Like one of my, uh, so I interviewed ETA who was on, who was on that team. Uh, and he was like, yeah, I, I don’t know how they work.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
115 / belief
Uh, w w while we’re on this topic, you know, I think there’s a lot of, um, uh, this, the concept of precision at all is weird when we’re sampling, you know, uh, we just, at the end of this, we’re going to have all these like chips that I’ll do like very good math.
“Uh, w w while we’re on this topic, you know, I think there’s a lot of, um, uh, this, the concept of precision at all is weird when we’re sampling, you know, uh, we just, at the end of this, we’re going to have all these like chips that I’ll do like very good math.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
116 / evaluation
I think motion, you know, I still want to shout out, I think Gemini, still the only native video understanding model that’s out there.
“I think motion, you know, I still want to shout out, I think Gemini, still the only native video understanding model that’s out there.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
117 / recommendation
Mentions personal use of Ted Chiang.
“You were very excited because I read Ted Chiang over the holidays and I was very inspired by this short story called Understand, which apparently is, like, pretty old.”
- Speaker
- Shawn Wang
- Episode
- The First Mechanistic Interpretability Frontier Lab — Myra Deng & Mark Bissell of Goodfire AI
- Publisher
- Latent Space
118 / belief
I think for me, one way in which I think about world models is just like this, like, having this consistent model of the world where everything that you generate operates within the rules of that world.
“I think for me, one way in which I think about world models is just like this, like, having this consistent model of the world where everything that you generate operates within the rules of that world.”
- Speaker
- Shawn Wang
- Episode
- The First Mechanistic Interpretability Frontier Lab — Myra Deng & Mark Bissell of Goodfire AI
- Publisher
- Latent Space
119 / uncertainty
Uh, I don’t know if you’ve, you’ve any conversations with him about anything in particular on AI building and AI infra.
“Uh, I don’t know if you’ve, you’ve any conversations with him about anything in particular on AI building and AI infra.”
- Speaker
- Shawn Wang
- Episode
- Artificial Analysis: Independent LLM Evals as a Service — with George Cameron and Micah-Hill Smith
- Publisher
- Latent Space
120 / uncertainty
Uh, so that’s the, that’s the last of the proprietary art numbers, I guess. I don’t know how you sort of classify all these.
“Uh, so that’s the, that’s the last of the proprietary art numbers, I guess. I don’t know how you sort of classify all these.”
- Speaker
- Shawn Wang
- Episode
- Artificial Analysis: Independent LLM Evals as a Service — with George Cameron and Micah-Hill Smith
- Publisher
- Latent Space
121 / uncertainty
I don’t know. Maybe it might be though. You put it like a JSON field, say, say confidence and maybe it spits out something.
“I don’t know. Maybe it might be though. You put it like a JSON field, say, say confidence and maybe it spits out something.”
- Speaker
- Shawn Wang
- Episode
- Artificial Analysis: Independent LLM Evals as a Service — with George Cameron and Micah-Hill Smith
- Publisher
- Latent Space
122 / uncertainty
Would that be the other way around in a normal capability environments? I don’t know.
“Would that be the other way around in a normal capability environments? I don’t know.”
- Speaker
- Shawn Wang
- Episode
- Artificial Analysis: Independent LLM Evals as a Service — with George Cameron and Micah-Hill Smith
- Publisher
- Latent Space
123 / belief
I think the next point in history that I have for you is AI Grant that you guys decided to join and move here.
“I think the next point in history that I have for you is AI Grant that you guys decided to join and move here.”
- Speaker
- Shawn Wang
- Episode
- Artificial Analysis: Independent LLM Evals as a Service — with George Cameron and Micah-Hill Smith
- Publisher
- Latent Space
124 / uncertainty
We don’t know for a fact that it’s like eight runs and then with the LM judge on top.
“We don’t know for a fact that it’s like eight runs and then with the LM judge on top.”
- Speaker
- Shawn Wang
- Episode
- Artificial Analysis: Independent LLM Evals as a Service — with George Cameron and Micah-Hill Smith
- Publisher
- Latent Space
125 / belief
I’m like, this is going to be our main thing. When I first called you, I think you hadn’t decided on starting a company yet.
“I’m like, this is going to be our main thing. When I first called you, I think you hadn’t decided on starting a company yet.”
- Speaker
- Shawn Wang
- Episode
- Artificial Analysis: Independent LLM Evals as a Service — with George Cameron and Micah-Hill Smith
- Publisher
- Latent Space
126 / uncertainty
There’s a little bit of debate over the accuracy of TileBench. I don’t know if you’re clued in to what’s going on.
“There’s a little bit of debate over the accuracy of TileBench. I don’t know if you’re clued in to what’s going on.”
- Speaker
- Shawn Wang
- Episode
- Artificial Analysis: Independent LLM Evals as a Service — with George Cameron and Micah-Hill Smith
- Publisher
- Latent Space
127 / belief
I think one of the openness questions from this year was people messing with the license.
“I think one of the openness questions from this year was people messing with the license.”
- Speaker
- Shawn Wang
- Episode
- Artificial Analysis: Independent LLM Evals as a Service — with George Cameron and Micah-Hill Smith
- Publisher
- Latent Space
128 / uncertainty
All these nuances, like, once you dig into benchmarks, you’re like, I don’t know how anyone believes the numbers on all these things.
“All these nuances, like, once you dig into benchmarks, you’re like, I don’t know how anyone believes the numbers on all these things.”
- Speaker
- Shawn Wang
- Episode
- Artificial Analysis: Independent LLM Evals as a Service — with George Cameron and Micah-Hill Smith
- Publisher
- Latent Space
129 / belief
I think there’s a general field of calibration as well, like the confidence in your answer versus the rightness of the answer.
“I think there’s a general field of calibration as well, like the confidence in your answer versus the rightness of the answer.”
- Speaker
- Shawn Wang
- Episode
- Artificial Analysis: Independent LLM Evals as a Service — with George Cameron and Micah-Hill Smith
- Publisher
- Latent Space
130 / belief
I did give you shit for missing fireworks, and how do you have a model benchmarking thing without fireworks? But you had together, you had perplexity, and I think we just started chatting there.
“I did give you shit for missing fireworks, and how do you have a model benchmarking thing without fireworks? But you had together, you had perplexity, and I think we just started chatting there.”
- Speaker
- Shawn Wang
- Episode
- Artificial Analysis: Independent LLM Evals as a Service — with George Cameron and Micah-Hill Smith
- Publisher
- Latent Space
131 / uncertainty
I don’t know if you have anything to add there. Or we could just go right into showing people the benchmark and like looking around and asking questions about it.
“I don’t know if you have anything to add there. Or we could just go right into showing people the benchmark and like looking around and asking questions about it.”
- Speaker
- Shawn Wang
- Episode
- Artificial Analysis: Independent LLM Evals as a Service — with George Cameron and Micah-Hill Smith
- Publisher
- Latent Space
132 / observation
Let’s talk about TechStack behind that. But okay, I’m going to rewind all the way to when you guys started this project.
“Let’s talk about TechStack behind that. But okay, I’m going to rewind all the way to when you guys started this project.”
- Speaker
- Shawn Wang
- Episode
- Artificial Analysis: Independent LLM Evals as a Service — with George Cameron and Micah-Hill Smith
- Publisher
- Latent Space
133 / belief
Let’s pick on hardware efficiency since you also have, you also track hardware stuff. And I think the general assertion or the message is that the efficiency from next gen Nvidia chips is actually not 4X.
“Let’s pick on hardware efficiency since you also have, you also track hardware stuff. And I think the general assertion or the message is that the efficiency from next gen Nvidia chips is actually not 4X.”
- Speaker
- Shawn Wang
- Episode
- Artificial Analysis: Independent LLM Evals as a Service — with George Cameron and Micah-Hill Smith
- Publisher
- Latent Space
134 / belief
I think to some extent, I’m mixed opinion on that one because to some extent, your target audience is not people in AI Grants who are obviously at the frontier.
“I think to some extent, I’m mixed opinion on that one because to some extent, your target audience is not people in AI Grants who are obviously at the frontier.”
- Speaker
- Shawn Wang
- Episode
- Artificial Analysis: Independent LLM Evals as a Service — with George Cameron and Micah-Hill Smith
- Publisher
- Latent Space
135 / preference
I mean, I’m a fan of things that, truths that don’t change because you can build and plan for that.
“I mean, I’m a fan of things that, truths that don’t change because you can build and plan for that.”
- Speaker
- Shawn Wang
- Episode
- Artificial Analysis: Independent LLM Evals as a Service — with George Cameron and Micah-Hill Smith
- Publisher
- Latent Space
136 / preference
What I love talking to people like you who sit across the ecosystem is, well, I have theories about what people want, but you have data and that’s obviously more relevant.
“What I love talking to people like you who sit across the ecosystem is, well, I have theories about what people want, but you have data and that’s obviously more relevant.”
- Speaker
- Shawn Wang
- Episode
- Artificial Analysis: Independent LLM Evals as a Service — with George Cameron and Micah-Hill Smith
- Publisher
- Latent Space
137 / prediction
One way I was thinking about kicking on this conversation is we will likely release this right after CoreWeave IPO.
“One way I was thinking about kicking on this conversation is we will likely release this right after CoreWeave IPO.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
138 / belief
Uh, I think a lot of people obviously very tied to the VS code paradigm and the extensions that come along with it.
“Uh, I think a lot of people obviously very tied to the VS code paradigm and the extensions that come along with it.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
139 / belief
I think there's an overall, also, my sense is that Anthropic is going hard after developers in the way that other labs are not.
“I think there's an overall, also, my sense is that Anthropic is going hard after developers in the way that other labs are not.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
140 / recommendation
So abusing the trust system is like setting up a trust system creates the damage from the trust system. And so I actually want to encourage people to try out MCP Inspector because all you got to do is actually just look at the traffic.
“So abusing the trust system is like setting up a trust system creates the damage from the trust system. And so I actually want to encourage people to try out MCP Inspector because all you got to do is actually just look at the traffic.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
141 / evaluation
Totally. And I think that is partially why it made your launch successful because you launch with a sufficiently spanning set of here's examples and then people just copy paste and expand from there.
“Totally. And I think that is partially why it made your launch successful because you launch with a sufficiently spanning set of here's examples and then people just copy paste and expand from there.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
142 / evaluation
I think once you enable two way and once you enable client server to be the same and delegation of work to another MCP server, it's definitely more agentic than not.
“I think once you enable two way and once you enable client server to be the same and delegation of work to another MCP server, it's definitely more agentic than not.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
143 / recommendation
Like they're, they're not normal servers in the, in the sense that they, they wrap some API and it's just easier to interact with those than to work at the APIs. And so I'll, I'll highlight the, the memory one first, just because like, I think there are, there are a few memory startups, but actually you don't need them if you just use this one.
“Like they're, they're not normal servers in the, in the sense that they, they wrap some API and it's just easier to interact with those than to work at the APIs. And so I'll, I'll highlight the, the memory one first, just because like, I think there are, there are a few memory startups, but actually you don't need them if you just use this one.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
144 / evaluation
You know, I think you, you maybe have a published some research that says like, actually sometimes to get, to get the model working the right way, you have to do multi-step prompting or jailbreaking to, to, to behave the way that you want.
“You know, I think you, you maybe have a published some research that says like, actually sometimes to get, to get the model working the right way, you have to do multi-step prompting or jailbreaking to, to, to behave the way that you want.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
145 / belief
Yeah. And I think the reason I have to be here in SF is because I make friends with people who know things and are smarter than me, and we do go for chats and they're nice enough to share some stuff.
“Yeah. And I think the reason I have to be here in SF is because I make friends with people who know things and are smarter than me, and we do go for chats and they're nice enough to share some stuff.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
146 / belief
I don't, I think that's just a more general policy of no matter what they have at the top tier, they always want to have smaller versions of that in the, in the lower tiers.
“I don't, I think that's just a more general policy of no matter what they have at the top tier, they always want to have smaller versions of that in the, in the lower tiers.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
147 / belief
We were talking on, on the car about Decagon and Sierra, obviously Brett, Brett Taylor is founder of Sierra. And yeah, it seems like there's just this, these layers of agents that'll like, I think you just look at like the income statement or like the, the org chart of any large scaled company and you start picking them off one by one.
“We were talking on, on the car about Decagon and Sierra, obviously Brett, Brett Taylor is founder of Sierra. And yeah, it seems like there's just this, these layers of agents that'll like, I think you just look at like the income statement or like the, the org chart of any large scaled company and you start picking them off one by one.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
148 / belief
I mean it's, it's, it's interesting. Like, I, I think the natural winner of that should be sourcegraph.
“I mean it's, it's, it's interesting. Like, I, I think the natural winner of that should be sourcegraph.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
149 / belief
In one of the episodes that we did was I think he estimated that open source model usage in work in enterprises is that like 5% and going down.
“In one of the episodes that we did was I think he estimated that open source model usage in work in enterprises is that like 5% and going down.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
150 / belief
I, I interviewed him in 2023 and I think he may have been the first person on our podcast to like, probably be a GBT rapper.
“I, I interviewed him in 2023 and I think he may have been the first person on our podcast to like, probably be a GBT rapper.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
151 / commitment
I'm happy to be wrong in public, you know, I think that's how you learn the most, right?
“I'm happy to be wrong in public, you know, I think that's how you learn the most, right?”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
152 / preference
Yeah, but I don't regard, I don't have, I don't use, put, put that in my mental framework of things like I come at this much more as a content creator or market analyst of like, yeah, it, it really does matter to me what has part of market fit because.
“Yeah, but I don't regard, I don't have, I don't use, put, put that in my mental framework of things like I come at this much more as a content creator or market analyst of like, yeah, it, it really does matter to me what has part of market fit because.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
153 / evaluation
Like I, I, to some extent, I think the only reason you and I are talking about it is that they, both of them have reported like ridiculous numbers.
“Like I, I, to some extent, I think the only reason you and I are talking about it is that they, both of them have reported like ridiculous numbers.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
154 / evaluation
This is an actual tipping point. And I think I like as people who are like, our function as podcasters and industry analysts is to raise the bar or focus attention on things that you think matter.
“This is an actual tipping point. And I think I like as people who are like, our function as podcasters and industry analysts is to raise the bar or focus attention on things that you think matter.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
155 / prediction
I think literally the, like the information just published a, like, this is something that they're worried about, which is when operator or whoever accesses your website on behalf of you, how does it indicate that it's not you, but it's, it's an agent of you.
“I think literally the, like the information just published a, like, this is something that they're worried about, which is when operator or whoever accesses your website on behalf of you, how does it indicate that it's not you, but it's, it's an agent of you.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
156 / evaluation
I was gonna say this, so I have a list [00:24:00] of like two years ago we, I wrote the Anatomy of autonomy posts where it was like the, the first, like what's going on in agents and, and and, and, and what is actually making money. Because I think there's a lot of gen I skeptics out there.
“I was gonna say this, so I have a list [00:24:00] of like two years ago we, I wrote the Anatomy of autonomy posts where it was like the, the first, like what's going on in agents and, and and, and, and what is actually making money. Because I think there's a lot of gen I skeptics out there.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
157 / evaluation
I mean, again, I I, I don't like this label of how Fast Open Source caught up because it's really how Fast Deepsea caught up.
“I mean, again, I I, I don't like this label of how Fast Open Source caught up because it's really how Fast Deepsea caught up.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
158 / belief
I should also thank Sean Puri, who I've chatted with back and forth, who's been, I guess, getting me in touch with your people. But also, I think like, just giving us a lot of context, because obviously, My First Million joined you guys, and they've been chatting with you guys a lot.
“I should also thank Sean Puri, who I've chatted with back and forth, who's been, I guess, getting me in touch with your people. But also, I think like, just giving us a lot of context, because obviously, My First Million joined you guys, and they've been chatting with you guys a lot.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
159 / commitment
Yep. And I think that the modifier for like, we will pay you, we'll pay somebody and maybe it's you, is like 30 to 60.
“Yep. And I think that the modifier for like, we will pay you, we'll pay somebody and maybe it's you, is like 30 to 60.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
160 / belief
You are kind of a marketplace, but he has a third angle, which is the model providers, and he lets them compete. And I think that sort of Chai three-way marketplace maybe makes a lot of sense.
“You are kind of a marketplace, but he has a third angle, which is the model providers, and he lets them compete. And I think that sort of Chai three-way marketplace maybe makes a lot of sense.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
161 / belief
I'm not an investor in B, but I'm just a friend. And I think you should be able to speak freely of your opinions regardless.
“I'm not an investor in B, but I'm just a friend. And I think you should be able to speak freely of your opinions regardless.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
162 / belief
Um, and I think like, if there was any real threat to like Photoshop or Canva, it's this thing.
“Um, and I think like, if there was any real threat to like Photoshop or Canva, it's this thing.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
163 / belief
Uh, I'm not trying to sway anyone, but yeah, I think like something I struggled with, with this conviction, you said you pursue things to conviction, but like you start out not knowing anything.
“Uh, I'm not trying to sway anyone, but yeah, I think like something I struggled with, with this conviction, you said you pursue things to conviction, but like you start out not knowing anything.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
164 / commitment
I think the, the problem, like, so even though at my conferences, GraphRag is super popular and people are getting knowledge, graph religion, and I will say like, it's getting space, getting traction in two areas, conversation memory, and then also just rag in general, like the, the, the document data.
“I think the, the problem, like, so even though at my conferences, GraphRag is super popular and people are getting knowledge, graph religion, and I will say like, it's getting space, getting traction in two areas, conversation memory, and then also just rag in general, like the, the, the document data.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
165 / preference
You know, like, because people, we know premature engineering is the root of all evil.
“You know, like, because people, we know premature engineering is the root of all evil.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
166 / belief
Sure. Well, I think, you know, reflecting on the finance thing for a bit. So I, I was, used to be a trader, uh, sell side and buy side.
“Sure. Well, I think, you know, reflecting on the finance thing for a bit. So I, I was, used to be a trader, uh, sell side and buy side.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
167 / uncertainty
Um, and the cost is not, it's not cheap, but I mean, we're in New York city right now and, uh, I don't know, I paid $8 for a coffee this morning, so, uh, the coffee is cheaper in Zurich than the New York city.
“Um, and the cost is not, it's not cheap, but I mean, we're in New York city right now and, uh, I don't know, I paid $8 for a coffee this morning, so, uh, the coffee is cheaper in Zurich than the New York city.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
168 / belief
Nice. So I really had to talk about it also because I think people interested in AI want to think about like, how can we, we're an AI podcast, we have to talk about the AI podcast app.
“Nice. So I really had to talk about it also because I think people interested in AI want to think about like, how can we, we're an AI podcast, we have to talk about the AI podcast app.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
169 / recommendation
Mentions personal use of Snip.
“By the way, we use the snip count as a proxy for popularity, right? Because we have download counts, but for example, platforms like Spotify re-host our MP3 file.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
170 / belief
I would say that one observation I've made about podcasting, this is the general state of the market.
“I would say that one observation I've made about podcasting, this is the general state of the market.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
171 / preference
They can correct me, right? So first of all, I think the main job is for it to be a podcast listening app.
“They can correct me, right? So first of all, I think the main job is for it to be a podcast listening app.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
172 / preference
A lot of the reason that people like Latent Space is it takes an idiot like me who could be doing a lot more with my life, making a lot more money, having a real job somewhere else. I just choose to do this because I like it.
“A lot of the reason that people like Latent Space is it takes an idiot like me who could be doing a lot more with my life, making a lot more money, having a real job somewhere else. I just choose to do this because I like it.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
173 / preference
I would say we host our podcast on Substack, but they're not very serious about their podcasting tools.
“I would say we host our podcast on Substack, but they're not very serious about their podcasting tools.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
174 / evaluation
I would say that, you know, obviously I'm a power user of all these tools. You have done a better job than Descript.
“I would say that, you know, obviously I'm a power user of all these tools. You have done a better job than Descript.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
175 / preference
Is this something I've been thinking about as kind of like AI that's not in your face? Because right now, you know, we like to say like, oh, use Notion has Notion AI.
“Is this something I've been thinking about as kind of like AI that's not in your face? Because right now, you know, we like to say like, oh, use Notion has Notion AI.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
176 / evaluation
I mean, on my side, I, I think I watched only like half of the talks. Cause I was running around and I think people saw me like towards the end, I was kind of collapsing.
“I mean, on my side, I, I think I watched only like half of the talks. Cause I was running around and I think people saw me like towards the end, I was kind of collapsing.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
177 / preference
I would say like a lot of people rely on Anki, Anki notes like flashcards and all that to do that.
“I would say like a lot of people rely on Anki, Anki notes like flashcards and all that to do that.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
178 / evaluation
I don't know. I think that's, that's me being a slow adopter. No, no. I mean, that's.
“I don't know. I think that's, that's me being a slow adopter. No, no. I mean, that's.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
179 / prediction
But like the real features come when you actually turn on the AI stuff. And so the reason I got snipped, because I got fed up with Overcast not implementing any AI features at all.
“But like the real features come when you actually turn on the AI stuff. And so the reason I got snipped, because I got fed up with Overcast not implementing any AI features at all.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
180 / recommendation
Mentions personal use of Overcast.
“I used to use Overcast. So it would just link to the Overcast page.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
181 / belief
I think that we're basically convening today to talk about the new API. So perhaps you guys want to just kick off.
“I think that we're basically convening today to talk about the new API. So perhaps you guys want to just kick off.”
- Speaker
- Shawn Wang
- Episode
- ⚡️The new OpenAI Agents Platform
- Publisher
- Latent Space
182 / prediction
Yeah, it's a it's really, I think, a really nice to have. But all I'll say is that my friend Corey Quinn says that anything that can be used as a database will be used as a database.
“Yeah, it's a it's really, I think, a really nice to have. But all I'll say is that my friend Corey Quinn says that anything that can be used as a database will be used as a database.”
- Speaker
- Shawn Wang
- Episode
- ⚡️The new OpenAI Agents Platform
- Publisher
- Latent Space
183 / evaluation
There's a lot of interest. I think Pokemon really is a good agent benchmark, to be honest.
“There's a lot of interest. I think Pokemon really is a good agent benchmark, to be honest.”
- Speaker
- Shawn Wang
- Episode
- ⚡️The new OpenAI Agents Platform
- Publisher
- Latent Space
184 / preference
Probably if it's an open source stack, I would say like a lot of the AI engineers that I talk to want to own this part of the stack.
“Probably if it's an open source stack, I would say like a lot of the AI engineers that I talk to want to own this part of the stack.”
- Speaker
- Shawn Wang
- Episode
- ⚡️The new OpenAI Agents Platform
- Publisher
- Latent Space
185 / evaluation
I think that Swarm has really popularized the handoff technique, which I thought was like, you know, really, really interesting for sort of a multi-agent.
“I think that Swarm has really popularized the handoff technique, which I thought was like, you know, really, really interesting for sort of a multi-agent.”
- Speaker
- Shawn Wang
- Episode
- ⚡️The new OpenAI Agents Platform
- Publisher
- Latent Space
186 / commitment
And assistance API, we've, uh, has a target sunset date of first half of 2020. So this is kind of like, in my mind, there was a kind of very poetic mirroring of the API with the models.
“And assistance API, we've, uh, has a target sunset date of first half of 2020. So this is kind of like, in my mind, there was a kind of very poetic mirroring of the API with the models.”
- Speaker
- Shawn Wang
- Episode
- ⚡️The new OpenAI Agents Platform
- Publisher
- Latent Space
187 / evaluation
Yeah, yeah, uh, for sure. And then one thing on the, on like the breadth, you know, I think a lot of the deep research, open deep research implementations have this sort of hyper parameter about, you know, how deep they're searching and how wide they're searching.
“Yeah, yeah, uh, for sure. And then one thing on the, on like the breadth, you know, I think a lot of the deep research, open deep research implementations have this sort of hyper parameter about, you know, how deep they're searching and how wide they're searching.”
- Speaker
- Shawn Wang
- Episode
- ⚡️The new OpenAI Agents Platform
- Publisher
- Latent Space
188 / belief
I think that there's a movement towards more solo founders in the Valley more generally, but people who are hearing this for the first time have no idea.
“I think that there's a movement towards more solo founders in the Valley more generally, but people who are hearing this for the first time have no idea.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
189 / belief
I think the government itself should embrace AI more to do more sort of human-friendly form filling.
“I think the government itself should embrace AI more to do more sort of human-friendly form filling.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
190 / recommendation
The only solution I know, which is to kind of keep a kind of warm pool of servers around, which is expensive, but maybe not so expensive because it's just CPUs.
“The only solution I know, which is to kind of keep a kind of warm pool of servers around, which is expensive, but maybe not so expensive because it's just CPUs.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
191 / uncertainty
I think I'm just trying to make sure that I'm not making false predictions. Because, like, you can predict that things will be better generically, but how?
“I think I'm just trying to make sure that I'm not making false predictions. Because, like, you can predict that things will be better generically, but how?”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
192 / preference
That's the only. I think that's the only way to do it. Yeah. Like, unless Chrome has some special API for you.
“That's the only. I think that's the only way to do it. Yeah. Like, unless Chrome has some special API for you.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
193 / evaluation
Where he basically observed that the browser is turning the operating system into a poorly debugged set of device drivers, because most of the apps are moved from the OS to the browser.
“Where he basically observed that the browser is turning the operating system into a poorly debugged set of device drivers, because most of the apps are moved from the OS to the browser.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
194 / evaluation
Just because like you're one of the, you know, best performing, I think, LLM tool companies that have started up in the last couple of years.
“Just because like you're one of the, you know, best performing, I think, LLM tool companies that have started up in the last couple of years.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
195 / prediction
There's always the question of the, are you a point solution or are you the sort of all in one? And I think the point solutions tend to win quickly, but then the only ones have a very tight cohesive experience.
“There's always the question of the, are you a point solution or are you the sort of all in one? And I think the point solutions tend to win quickly, but then the only ones have a very tight cohesive experience.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
196 / commitment
This is one of those things where I forgot to mention in my intro that I'm an investor in Browserbase. And I remember that when you pitched to me, like a lot of the stuff that we have today, we like wasn't on the original conversation.
“This is one of those things where I forgot to mention in my intro that I'm an investor in Browserbase. And I remember that when you pitched to me, like a lot of the stuff that we have today, we like wasn't on the original conversation.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
197 / preference
Because that's the only way that you can sort of, like, once it's reliable enough, obviously.
“Because that's the only way that you can sort of, like, once it's reliable enough, obviously.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
198 / recommendation
Recommends Scaling DevTools. Recommends The Peel.
“I think I strongly recommend Jack Bridger's Scaling DevTools, as well as Turner Novak's The Peel.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
199 / belief
I think a little bit of implementation and a little bit of vision, like kind of 50 50.
“I think a little bit of implementation and a little bit of vision, like kind of 50 50.”
- Speaker
- Shawn Wang
- Episode
- The Inventors of Deep Research
- Publisher
- Latent Space
200 / belief
Like, we think we know, like, you know, how to prompt them better, but engineering with them, I think also very, very unknown.
“Like, we think we know, like, you know, how to prompt them better, but engineering with them, I think also very, very unknown.”
- Speaker
- Shawn Wang
- Episode
- The Inventors of Deep Research
- Publisher
- Latent Space
201 / belief
Comparing and then, you know, covering the EU and then, yeah, like I said, like going into the meat production and then it'll also, what's nice is it kind of reasons over like why are there differences? And I think what's really cool here is like, it's, it's showing that there's like a difference in philosophy between how the U.
“Comparing and then, you know, covering the EU and then, yeah, like I said, like going into the meat production and then it'll also, what's nice is it kind of reasons over like why are there differences? And I think what's really cool here is like, it's, it's showing that there's like a difference in philosophy between how the U.”
- Speaker
- Shawn Wang
- Episode
- The Inventors of Deep Research
- Publisher
- Latent Space
202 / belief
I think that, you know, we were talking to you about like your top tips for using deep research.
“I think that, you know, we were talking to you about like your top tips for using deep research.”
- Speaker
- Shawn Wang
- Episode
- The Inventors of Deep Research
- Publisher
- Latent Space
203 / evaluation
I would say I was very keen on, I think even at the end of last year, people were already saying it was one of the most exciting agents that was coming out of Google.
“I would say I was very keen on, I think even at the end of last year, people were already saying it was one of the most exciting agents that was coming out of Google.”
- Speaker
- Shawn Wang
- Episode
- The Inventors of Deep Research
- Publisher
- Latent Space
204 / evaluation
I'm calling this out because everyone is like, oh my God, it takes hours for, it does hours of work autonomously for me.
“I'm calling this out because everyone is like, oh my God, it takes hours for, it does hours of work autonomously for me.”
- Speaker
- Shawn Wang
- Episode
- The Inventors of Deep Research
- Publisher
- Latent Space
205 / evaluation
That was super counterintuitive for us. So actually, the first time I realized that, what you're saying is when I was talking to Jason Calacanis and he was like, do you actually just make the answer in 10 seconds and just make me wait for the balance?
“That was super counterintuitive for us. So actually, the first time I realized that, what you're saying is when I was talking to Jason Calacanis and he was like, do you actually just make the answer in 10 seconds and just make me wait for the balance?”
- Speaker
- Shawn Wang
- Episode
- The Inventors of Deep Research
- Publisher
- Latent Space
206 / evaluation
Yep. Because I think at that point, like, users will just drop off. Nope. But what's been surprising is, like, that's not the case at all.
“Yep. Because I think at that point, like, users will just drop off. Nope. But what's been surprising is, like, that's not the case at all.”
- Speaker
- Shawn Wang
- Episode
- The Inventors of Deep Research
- Publisher
- Latent Space
207 / belief
One thing I just, I just observe is that I think the early Google days had this interesting mix of PM and engineer, which I think you are, you didn't, you didn't wait for PM to tell you these are my, this is my PRD.
“One thing I just, I just observe is that I think the early Google days had this interesting mix of PM and engineer, which I think you are, you didn't, you didn't wait for PM to tell you these are my, this is my PRD.”
- Speaker
- Shawn Wang
- Episode
- The AI Architect — Bret Taylor
- Publisher
- Latent Space
208 / belief
What are the shapes of graphs that people should know? So the way that I would phrase this is I think Anthropic did a very good public service and also kind of surprisingly influential blog post, I would say, when they wrote Building Effective Agents.
“What are the shapes of graphs that people should know? So the way that I would phrase this is I think Anthropic did a very good public service and also kind of surprisingly influential blog post, I would say, when they wrote Building Effective Agents.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
209 / belief
I think the one thing that I would say like yours, you know, as a library, you don't have that much control of it over the infrastructure.
“I think the one thing that I would say like yours, you know, as a library, you don't have that much control of it over the infrastructure.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
210 / belief
Obviously, on Latent Space, we don't really discuss that because I think AGI is kind of this hand wavy concept that isn't super relevant.
“Obviously, on Latent Space, we don't really discuss that because I think AGI is kind of this hand wavy concept that isn't super relevant.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
211 / belief
I think there is always appeal. Every developer eventually gets graph religion and goes, oh, yeah, everything's a graph.
“I think there is always appeal. Every developer eventually gets graph religion and goes, oh, yeah, everything's a graph.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
212 / belief
I mean, it's, it's growing now. And I think anyway, everyone's taking it seriously.
“I mean, it's, it's growing now. And I think anyway, everyone's taking it seriously.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
213 / belief
I've often wondered, I think there should be some startup or something that does automated data insights.
“I've often wondered, I think there should be some startup or something that does automated data insights.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
214 / belief
I think the other big project that you were, you were involved with was just Cloud 3.
“I think the other big project that you were, you were involved with was just Cloud 3.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
215 / uncertainty
I don't know how big the prompt distribution needs to be because you are literally catering to everyone.
“I don't know how big the prompt distribution needs to be because you are literally catering to everyone.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
216 / belief
Oh, I'll also say that I think one takeaway I got from your, this conversation is that ChatGPT will have to integrate a lot more with my life.
“Oh, I'll also say that I think one takeaway I got from your, this conversation is that ChatGPT will have to integrate a lot more with my life.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
217 / evaluation
Drafting anything like I want to draft like copy for my conference that I'm running, like I'll put it there first and then I like, it'll just have the canvas up and I'll just say what I don't like about it and it changes.
“Drafting anything like I want to draft like copy for my conference that I'm running, like I'll put it there first and then I like, it'll just have the canvas up and I'll just say what I don't like about it and it changes.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
218 / evaluation
I think because it's maybe sold as like sort of writing help when really like it's kind of, it's the scratch pad.
“I think because it's maybe sold as like sort of writing help when really like it's kind of, it's the scratch pad.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
219 / preference
You know, how could something this big work in like an industrial space and in the things that we're doing, you know, it's a really exciting time for us. And it's just, you know, it's a lot of work, but what I really like about working in digital space is the, you know, the visual space is always the best place to stay.
“You know, how could something this big work in like an industrial space and in the things that we're doing, you know, it's a really exciting time for us. And it's just, you know, it's a lot of work, but what I really like about working in digital space is the, you know, the visual space is always the best place to stay.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
220 / recommendation
Mentions personal use of AI News.
“So that's what basically I use AI News for. I have a lot of prompts and a lot of steps and a lot of criteria and O1 just kind of checks through each kind of systematically.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
221 / belief
I think there's a lot of inference insights that we can get from that, as well as human psychology insights, kind of a weird blend of the two.
“I think there's a lot of inference insights that we can get from that, as well as human psychology insights, kind of a weird blend of the two.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
222 / belief
I think one problem I have, and this is a broader products question maybe, is that the ELOs apply to the whole user population.
“I think one problem I have, and this is a broader products question maybe, is that the ELOs apply to the whole user population.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
223 / uncertainty
I don't know that they do tree search. They never said they do. It's implied. Yes.
“I don't know that they do tree search. They never said they do. It's implied. Yes.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
224 / preference
I think we use at the top, spreading out to different experts and then at the bottom with rejection sampling, choosing from different paths.
“I think we use at the top, spreading out to different experts and then at the bottom with rejection sampling, choosing from different paths.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
225 / uncertainty
The web is increasingly closing to the bots and the scrapers, Twitter, Reddit, Quora, Stack Overflow. I don't know what else.
“The web is increasingly closing to the bots and the scrapers, Twitter, Reddit, Quora, Stack Overflow. I don't know what else.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
226 / belief
Basically, I think dating, uh, research, like I just wanted to like spell out more things, like just the big verticals.
“Basically, I think dating, uh, research, like I just wanted to like spell out more things, like just the big verticals.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
227 / belief
I would say if you were to take the exit being O1 for search, literally, you really need to prioritize search trajectories, like almost maybe paying a bunch of grad students to go research things.
“I would say if you were to take the exit being O1 for search, literally, you really need to prioritize search trajectories, like almost maybe paying a bunch of grad students to go research things.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
228 / belief
I think people who are in the know, know that Comfy is like the tool for image generation and now other multimodality stuff.
“I think people who are in the know, know that Comfy is like the tool for image generation and now other multimodality stuff.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
229 / evaluation
I think the naming as well matters. It seemed like a branch off of the main, main tree of development.
“I think the naming as well matters. It seemed like a branch off of the main, main tree of development.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
230 / evaluation
Small and yeah. Because it used to be latent diffusion models and then they trained it up.
“Small and yeah. Because it used to be latent diffusion models and then they trained it up.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
231 / commitment
Uh, I, yeah, I, you know, I think We will go over the individual, uh, talks in a separate episode.
“Uh, I, yeah, I, you know, I think We will go over the individual, uh, talks in a separate episode.”
- Speaker
- Shawn Wang
- Episode
- Latent.Space 2024 Year in Review
- Publisher
- Latent Space
232 / uncertainty
We don't know what the small model size is, but yeah, it's probably in the double digits or maybe single digits, but probably double digits.
“We don't know what the small model size is, but yeah, it's probably in the double digits or maybe single digits, but probably double digits.”
- Speaker
- Shawn Wang
- Episode
- Latent.Space 2024 Year in Review
- Publisher
- Latent Space
233 / belief
I think the collective terminology has been inference time, and I think that makes sense because test time, calling it test, meaning, has a very pre trained bias, meaning that the only reason for running inference at all is to test your model.
“I think the collective terminology has been inference time, and I think that makes sense because test time, calling it test, meaning, has a very pre trained bias, meaning that the only reason for running inference at all is to test your model.”
- Speaker
- Shawn Wang
- Episode
- Latent.Space 2024 Year in Review
- Publisher
- Latent Space
234 / evaluation
As someone who also does charts, XAI is continually snubbed because they don't work well with the benchmarking people.
“As someone who also does charts, XAI is continually snubbed because they don't work well with the benchmarking people.”
- Speaker
- Shawn Wang
- Episode
- Latent.Space 2024 Year in Review
- Publisher
- Latent Space
235 / belief
Everyone knew, like, F1 we had a preview at the Fireworks HQ, and then [01:37:00] I think some other labs did it, but I think R1 and QWQ, Quill, from the Quent team, Both Alibaba affiliated, I think, are the leading contenders on that front end.
“Everyone knew, like, F1 we had a preview at the Fireworks HQ, and then [01:37:00] I think some other labs did it, but I think R1 and QWQ, Quill, from the Quent team, Both Alibaba affiliated, I think, are the leading contenders on that front end.”
- Speaker
- Shawn Wang
- Episode
- Latent.Space 2024 Year in Review
- Publisher
- Latent Space
236 / belief
Interestingly, I think the same person responsible for artifacts and canvas, Karina, officially left Anthropic after this to join OpenAI on the rare reverse moves.
“Interestingly, I think the same person responsible for artifacts and canvas, Karina, officially left Anthropic after this to join OpenAI on the rare reverse moves.”
- Speaker
- Shawn Wang
- Episode
- Latent.Space 2024 Year in Review
- Publisher
- Latent Space
237 / belief
I think even, maybe if you ask them, like, they wouldn't necessarily draw a straight line.
“I think even, maybe if you ask them, like, they wouldn't necessarily draw a straight line.”
- Speaker
- Shawn Wang
- Episode
- Latent.Space 2024 Year in Review
- Publisher
- Latent Space
238 / belief
I think every, I mean, there should be, just like every consumer product is going to have a, going to eventually want a gateway, you know, for, for managing their requests and ops tool, you know, that kind of stuff, um, code interpreter for maybe not exposing the code, but executing code under the hood for sure.
“I think every, I mean, there should be, just like every consumer product is going to have a, going to eventually want a gateway, you know, for, for managing their requests and ops tool, you know, that kind of stuff, um, code interpreter for maybe not exposing the code, but executing code under the hood for sure.”
- Speaker
- Shawn Wang
- Episode
- Latent.Space 2024 Year in Review
- Publisher
- Latent Space
239 / belief
Another discovery, I think, um, Ilya actually worked on a previous version called GPT 0 in 2021.
“Another discovery, I think, um, Ilya actually worked on a previous version called GPT 0 in 2021.”
- Speaker
- Shawn Wang
- Episode
- Latent.Space 2024 Year in Review
- Publisher
- Latent Space
240 / belief
Uh, I think that, I think you've been, uh, the, the, a dream partner to, to build Lanespace with.
“Uh, I think that, I think you've been, uh, the, the, a dream partner to, to build Lanespace with.”
- Speaker
- Shawn Wang
- Episode
- Latent.Space 2024 Year in Review
- Publisher
- Latent Space
241 / belief
I think that is where, uh, At least one hour of software experience to go, then it will be another few years [01:33:00] for that to happen in real life outside of the screen.
“I think that is where, uh, At least one hour of software experience to go, then it will be another few years [01:33:00] for that to happen in real life outside of the screen.”
- Speaker
- Shawn Wang
- Episode
- Latent.Space 2024 Year in Review
- Publisher
- Latent Space
242 / belief
People have actually, I would, I would say I've been surprised by how well received that was.
“People have actually, I would, I would say I've been surprised by how well received that was.”
- Speaker
- Shawn Wang
- Episode
- Latent.Space 2024 Year in Review
- Publisher
- Latent Space
243 / belief
NVIDIA, I think we continue to talk about, I think I was at the Taiwanese trade show, Comtex, and saw him signing, you know, You know, women body [01:30:00] parts. And I think that was maybe a sign of the times, maybe a sign that things have peaked, but things are clearly not peaked because they continued going.
“NVIDIA, I think we continue to talk about, I think I was at the Taiwanese trade show, Comtex, and saw him signing, you know, You know, women body [01:30:00] parts. And I think that was maybe a sign of the times, maybe a sign that things have peaked, but things are clearly not peaked because they continued going.”
- Speaker
- Shawn Wang
- Episode
- Latent.Space 2024 Year in Review
- Publisher
- Latent Space
244 / belief
I think if your, if your, your job is a, at least AI content creator or VC or, you know, someone who, whose job it is to stay on, stay on top of things, you should already be spending like a thousand dollars a month on, on stuff.
“I think if your, if your, your job is a, at least AI content creator or VC or, you know, someone who, whose job it is to stay on, stay on top of things, you should already be spending like a thousand dollars a month on, on stuff.”
- Speaker
- Shawn Wang
- Episode
- Latent.Space 2024 Year in Review
- Publisher
- Latent Space
245 / belief
Um, and that one, I would say Risa is obviously a star, but she's been on every episode, every podcast, but Isamah, I think, you know, actually being the guy who worked on the audio model, being able to talk to him, I think was, was a great gift for us.
“Um, and that one, I would say Risa is obviously a star, but she's been on every episode, every podcast, but Isamah, I think, you know, actually being the guy who worked on the audio model, being able to talk to him, I think was, was a great gift for us.”
- Speaker
- Shawn Wang
- Episode
- Latent.Space 2024 Year in Review
- Publisher
- Latent Space
246 / belief
I think like what we have today is not secure enough because it's like normal security when like this is literally a state level interest.
“I think like what we have today is not secure enough because it's like normal security when like this is literally a state level interest.”
- Speaker
- Shawn Wang
- Episode
- Latent.Space 2024 Year in Review
- Publisher
- Latent Space
247 / belief
I think we can also use this as part of a general theme of the safety wing of OpenAI leaving.
“I think we can also use this as part of a general theme of the safety wing of OpenAI leaving.”
- Speaker
- Shawn Wang
- Episode
- Latent.Space 2024 Year in Review
- Publisher
- Latent Space
248 / belief
Uh, I think, um, I have been asked my opinion on this before, and I said, I think I said it on a podcast, which is like, the main layer that you need is [01:38:00] the separate off roles, so that you don't assume it's a human, um, doing these things.
“Uh, I think, um, I have been asked my opinion on this before, and I said, I think I said it on a podcast, which is like, the main layer that you need is [01:38:00] the separate off roles, so that you don't assume it's a human, um, doing these things.”
- Speaker
- Shawn Wang
- Episode
- Latent.Space 2024 Year in Review
- Publisher
- Latent Space
249 / evaluation
I think the quote that I highlighted in AI News was that it is the best, like Blackwell is the best selling series.
“I think the quote that I highlighted in AI News was that it is the best, like Blackwell is the best selling series.”
- Speaker
- Shawn Wang
- Episode
- Latent.Space 2024 Year in Review
- Publisher
- Latent Space
250 / belief
Ilya saying the word agentic on stage at Eurips, it's a big deal. Satya, I think also saying that a lot these days.
“Ilya saying the word agentic on stage at Eurips, it's a big deal. Satya, I think also saying that a lot these days.”
- Speaker
- Shawn Wang
- Episode
- Latent.Space 2024 Year in Review
- Publisher
- Latent Space
251 / recommendation
Bash transcription, I would say Whisper, is something that you should be using on a as much as possible.
“Bash transcription, I would say Whisper, is something that you should be using on a as much as possible.”
- Speaker
- Shawn Wang
- Episode
- Latent.Space 2024 Year in Review
- Publisher
- Latent Space
252 / prediction
So likewise at Meta, likewise at OpenAI, likewise at the other labs as well. So like the GPU ultra rich are going to keep doing that because I think partially it's an article of faith now that you just need it.
“So likewise at Meta, likewise at OpenAI, likewise at the other labs as well. So like the GPU ultra rich are going to keep doing that because I think partially it's an article of faith now that you just need it.”
- Speaker
- Shawn Wang
- Episode
- Latent.Space 2024 Year in Review
- Publisher
- Latent Space
253 / recommendation
Oh yeah, magically going to close the gap between the closed source and open source. So basically I think my advice to people is keep track of the slow cooking of benchmark language because the labs that are not that frontier will keep measuring themselves on last year's benchmarks and then the labs that are actually frontier will Tell you about [01:09:00] benchmarks you've never heard of and you'll be like, Oh, like, okay, there's, there's new, there's new territory to, to, to go on.
“Oh yeah, magically going to close the gap between the closed source and open source. So basically I think my advice to people is keep track of the slow cooking of benchmark language because the labs that are not that frontier will keep measuring themselves on last year's benchmarks and then the labs that are actually frontier will Tell you about [01:09:00] benchmarks you've never heard of and you'll be like, Oh, like, okay, there's, there's new, there's new territory to, to, to go on.”
- Speaker
- Shawn Wang
- Episode
- Latent.Space 2024 Year in Review
- Publisher
- Latent Space
254 / uncertainty
Because I think there's a lot of doubt or questions about where ML engineering stops and AI engineering starts.
“Because I think there's a lot of doubt or questions about where ML engineering stops and AI engineering starts.”
- Speaker
- Shawn Wang
- Episode
- Latent.Space 2024 Year in Review
- Publisher
- Latent Space
255 / evaluation
I think the Researchers that I talked with at NeurIPS were kind of positive on this because basically you need private test [00:14:00] sets to prevent contamination.
“I think the Researchers that I talked with at NeurIPS were kind of positive on this because basically you need private test [00:14:00] sets to prevent contamination.”
- Speaker
- Shawn Wang
- Episode
- Latent.Space 2024 Year in Review
- Publisher
- Latent Space
256 / belief
Hey, and today we are delighted to be, I think, the first podcast in the new Codeium office.
“Hey, and today we are delighted to be, I think, the first podcast in the new Codeium office.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
257 / observation
I want to double click on one thing just because you brought it up and it's like a rare thing to touch on VPO sales we don't get to actually we talk to pretty early stage founders mostly they don't usually have a pretty built out sales function advice what kind of sales works in this kind of field what didn't work anything you can share with other founders?
“I want to double click on one thing just because you brought it up and it's like a rare thing to touch on VPO sales we don't get to actually we talk to pretty early stage founders mostly they don't usually have a pretty built out sales function advice what kind of sales works in this kind of field what didn't work anything you can share with other founders?”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
258 / recommendation
he had like three dollar signs in the title I was like I can't do that so it's either building AI for the enterprise and then I also said the worst the most dangerous thing an AI startup can do is build for other AI startups which I think both of you will co-sign and I think basically the main thesis which I really liked was like go slow to go fast like here's the if you actually build for like security, compliance, personalization, usage analytics, latency budgets, and scale from the start then you're going to pay that cost now but eventually it's going to pay off in the long run and this is the actual insight you cannot do this later like if you build the easy thing first as an MVP it's like just ship it with whatever's easy to do and then you tack on the enterprise ready.
“he had like three dollar signs in the title I was like I can't do that so it's either building AI for the enterprise and then I also said the worst the most dangerous thing an AI startup can do is build for other AI startups which I think both of you will co-sign and I think basically the main thesis which I really liked was like go slow to go fast like here's the if you actually build for like security, compliance, personalization, usage analytics, latency budgets, and scale from the start then you're going to pay that cost now but eventually it's going to pay off in the long run and this is the actual insight you cannot do this later like if you build the easy thing first as an MVP it's like just ship it with whatever's easy to do and then you tack on the enterprise ready.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
259 / evaluation
This one was more like the money one which you know it's funny because I think developers are like quite uninterested in money.
“This one was more like the money one which you know it's funny because I think developers are like quite uninterested in money.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
260 / prediction
Because vision models tend to just consume the whole image, if you are able to sort of focus on images based on the query, I think that, like, can get you a lot of extra performance.
“Because vision models tend to just consume the whole image, if you are able to sort of focus on images based on the query, I think that, like, can get you a lot of extra performance.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
261 / belief
Yours was more of a rename. I think a slightly different direction as well. And then we can talk about both.
“Yours was more of a rename. I think a slightly different direction as well. And then we can talk about both.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
262 / prediction
I noticed that, you know, I think the promise of general purpose agents has kind of died.
“I noticed that, you know, I think the promise of general purpose agents has kind of died.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
263 / commitment
It's funny because I built on top of the fork of Bolt.new that already has the multi LLM thing.
“It's funny because I built on top of the fork of Bolt.new that already has the multi LLM thing.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
264 / prediction
I think in theory, we should be able to like run tests because you can run the full backend.
“I think in theory, we should be able to like run tests because you can run the full backend.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
265 / recommendation
talk about, but also kind of what we went through in our, in our call was I have PMF now, what is, is kind of what I've been saying. And so like, I think the first answer is hire a data scientist because we have to sort of figure out like from our data that you're now sitting on a ton of different customers and we don't really know the different customer segments.
“talk about, but also kind of what we went through in our, in our call was I have PMF now, what is, is kind of what I've been saying. And so like, I think the first answer is hire a data scientist because we have to sort of figure out like from our data that you're now sitting on a ton of different customers and we don't really know the different customer segments.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
266 / belief
We can cut over to computer use if we're okay with moving on to topics on this, if anything else. I think we're good.
“We can cut over to computer use if we're okay with moving on to topics on this, if anything else. I think we're good.”
- Speaker
- Shawn Wang
- Episode
- The new Claude 3.5 Sonnet, Computer Use, and Building SOTA Agents — with Erik Schluntz, Anthropic
- Publisher
- Latent Space
267 / belief
We're not going to talk about the training for that because that's still confidential. But I think Anthropic's done a really good job, like applying the model to different things.
“We're not going to talk about the training for that because that's still confidential. But I think Anthropic's done a really good job, like applying the model to different things.”
- Speaker
- Shawn Wang
- Episode
- The new Claude 3.5 Sonnet, Computer Use, and Building SOTA Agents — with Erik Schluntz, Anthropic
- Publisher
- Latent Space
268 / belief
XML is good for everyone, not just Cloud. Cloud was just the first one to popularize it, I think.
“XML is good for everyone, not just Cloud. Cloud was just the first one to popularize it, I think.”
- Speaker
- Shawn Wang
- Episode
- The new Claude 3.5 Sonnet, Computer Use, and Building SOTA Agents — with Erik Schluntz, Anthropic
- Publisher
- Latent Space
269 / belief
Like, you're kind of skeptical about self-driving as a business. So I want to double click on this a little bit, because I mean, I think that shouldn't be taken away.
“Like, you're kind of skeptical about self-driving as a business. So I want to double click on this a little bit, because I mean, I think that shouldn't be taken away.”
- Speaker
- Shawn Wang
- Episode
- The new Claude 3.5 Sonnet, Computer Use, and Building SOTA Agents — with Erik Schluntz, Anthropic
- Publisher
- Latent Space
270 / belief
E2B is a close friend of ours that Alessio has led around in, but also I think there's others where they're focusing on snapshotting memory so that it can do time travel for debugging.
“E2B is a close friend of ours that Alessio has led around in, but also I think there's others where they're focusing on snapshotting memory so that it can do time travel for debugging.”
- Speaker
- Shawn Wang
- Episode
- The new Claude 3.5 Sonnet, Computer Use, and Building SOTA Agents — with Erik Schluntz, Anthropic
- Publisher
- Latent Space
271 / belief
For people who don't know, who maybe haven't dived into SWE-Bench, I think the general perception is they're like tasks that a software engineer could do.
“For people who don't know, who maybe haven't dived into SWE-Bench, I think the general perception is they're like tasks that a software engineer could do.”
- Speaker
- Shawn Wang
- Episode
- The new Claude 3.5 Sonnet, Computer Use, and Building SOTA Agents — with Erik Schluntz, Anthropic
- Publisher
- Latent Space
272 / disagreement
I think we'll go into suite agent in a little bit, but I kind of reject the fact that, you know, you need to choose one prompt and like have your whole performance be predicated on that one prompt.
“I think we'll go into suite agent in a little bit, but I kind of reject the fact that, you know, you need to choose one prompt and like have your whole performance be predicated on that one prompt.”
- Speaker
- Shawn Wang
- Episode
- The new Claude 3.5 Sonnet, Computer Use, and Building SOTA Agents — with Erik Schluntz, Anthropic
- Publisher
- Latent Space
273 / recommendation
Mentions personal use of XML.
“I use XML in other models as well, and it's just a really nice way to make sure that the thing that ends is tied to the thing that starts. That's the only way to do code fences where you're pretty sure example one start, example one end, that is one cohesive unit.”
- Speaker
- Shawn Wang
- Episode
- The new Claude 3.5 Sonnet, Computer Use, and Building SOTA Agents — with Erik Schluntz, Anthropic
- Publisher
- Latent Space
274 / belief
You ran the PyTorch team at Meta for a number of years and we previously had Sumit Chintal on and I think we were just all very interested in the history of GenEI.
“You ran the PyTorch team at Meta for a number of years and we previously had Sumit Chintal on and I think we were just all very interested in the history of GenEI.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
275 / belief
At some point, we'll show on the YouTube, the image that Ray, I think, has been working on with all the different modalities that you offer.
“At some point, we'll show on the YouTube, the image that Ray, I think, has been working on with all the different modalities that you offer.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
276 / evaluation
Noam and basically everyone on the Strawberry team was very insistent that what they did for reinforcement learning, chain of thought, cannot be replicated by a whole bunch of open source model calls. Do you think that that is wrong?
“Noam and basically everyone on the Strawberry team was very insistent that what they did for reinforcement learning, chain of thought, cannot be replicated by a whole bunch of open source model calls. Do you think that that is wrong?”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
277 / recommendation
I think I want to restate the value proposition of Fireworks for people who are comparing you versus a raw GPU provider like a RunPod or Lambda or anything like those, which is like you create the developer experience layer and you also make it easily scalable or serverless or as an endpoint.
“I think I want to restate the value proposition of Fireworks for people who are comparing you versus a raw GPU provider like a RunPod or Lambda or anything like those, which is like you create the developer experience layer and you also make it easily scalable or serverless or as an endpoint.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
278 / evaluation
And then, do you see any more opportunity on the... You know, I think you made a big splash with 1,000 tokens per second.
“And then, do you see any more opportunity on the... You know, I think you made a big splash with 1,000 tokens per second.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
279 / evaluation
Cosign was doing well on SweetBench, but they didn't want to leak those results. So that's why you don't see O1 preview on SweetBench, because they don't submit their reasoning choices.
“Cosign was doing well on SweetBench, but they didn't want to leak those results. So that's why you don't see O1 preview on SweetBench, because they don't submit their reasoning choices.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
280 / prediction
You get short-term gains by having specialists, domain specialists, and then someone just needs to train like a 10x bigger model on 10x more inference, 10x more data, 10x more model perhaps, whatever the current scaling law is. And then it supersedes all the individual models because of some generalized intelligence slash world knowledge.
“You get short-term gains by having specialists, domain specialists, and then someone just needs to train like a 10x bigger model on 10x more inference, 10x more data, 10x more model perhaps, whatever the current scaling law is. And then it supersedes all the individual models because of some generalized intelligence slash world knowledge.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
281 / evaluation
Basically it's just like the meta version of whatever Hugging Face offers, you know, or TensorRT, or BLM, or whatever the open source opportunity is. But to me, it's not clear that just because Meta open sources Lama, that the rest of LamaStack will be adopted.
“Basically it's just like the meta version of whatever Hugging Face offers, you know, or TensorRT, or BLM, or whatever the open source opportunity is. But to me, it's not clear that just because Meta open sources Lama, that the rest of LamaStack will be adopted.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
282 / evaluation
The LoRa thing is interesting because I think you also, the reason people add additional costs to it, it's not because they feel like charging people.
“The LoRa thing is interesting because I think you also, the reason people add additional costs to it, it's not because they feel like charging people.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
283 / commitment
But I think you guys are one of the reasons I agreed to advise you. Because I think when you first met me, I was kind of dubious.
“But I think you guys are one of the reasons I agreed to advise you. Because I think when you first met me, I was kind of dubious.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
284 / observation
The reason is because I have another assistant that also is recording and trying to come up with facts about me.
“The reason is because I have another assistant that also is recording and trying to come up with facts about me.”
- Speaker
- Shawn Wang
- Episode
- Agents @ Work: Lindy.ai
- Publisher
- Latent Space
285 / belief
You've been in our orbit a few times. I think you spoke at our Latent Space anniversary.
“You've been in our orbit a few times. I think you spoke at our Latent Space anniversary.”
- Speaker
- Shawn Wang
- Episode
- Agents @ Work: Lindy.ai
- Publisher
- Latent Space
286 / prediction
I think you're helping. Like, you're paving the road to AGI.
“I think you're helping. Like, you're paving the road to AGI.”
- Speaker
- Shawn Wang
- Episode
- Agents @ Work: Lindy.ai
- Publisher
- Latent Space
287 / evaluation
Because if you go from one building to two buildings, congrats, you're now remote from the other building.
“Because if you go from one building to two buildings, congrats, you're now remote from the other building.”
- Speaker
- Shawn Wang
- Episode
- Agents @ Work: Lindy.ai
- Publisher
- Latent Space
288 / disagreement
There's an inconsistency in your statements because you simultaneously believe that AGI is very soon and you also say stakes are low.
“There's an inconsistency in your statements because you simultaneously believe that AGI is very soon and you also say stakes are low.”
- Speaker
- Shawn Wang
- Episode
- Agents @ Work: Lindy.ai
- Publisher
- Latent Space
289 / evaluation
The framing can be different if you were, so I think tinkerers has this connotation of not serious or like small.
“The framing can be different if you were, so I think tinkerers has this connotation of not serious or like small.”
- Speaker
- Shawn Wang
- Episode
- Agents @ Work: Lindy.ai
- Publisher
- Latent Space
290 / evaluation
I think that's good. I'll maybe carve out that I think the UK has done really well.
“I think that's good. I'll maybe carve out that I think the UK has done really well.”
- Speaker
- Shawn Wang
- Episode
- Agents @ Work: Lindy.ai
- Publisher
- Latent Space
291 / evaluation
This is something I think about for AI engineering as well, which is the big labs want you to hand over everything in the prompts, and only code of English, and then the smaller brains, the GPU pours, always want to write more code to make things more deterministic and reliable and controllable.
“This is something I think about for AI engineering as well, which is the big labs want you to hand over everything in the prompts, and only code of English, and then the smaller brains, the GPU pours, always want to write more code to make things more deterministic and reliable and controllable.”
- Speaker
- Shawn Wang
- Episode
- Agents @ Work: Lindy.ai
- Publisher
- Latent Space
292 / recommendation
One thing that just occurred to me, so I'm a big fan of virtual mailboxes. I recommend that everybody have a virtual mailbox.
“One thing that just occurred to me, so I'm a big fan of virtual mailboxes. I recommend that everybody have a virtual mailbox.”
- Speaker
- Shawn Wang
- Episode
- Agents @ Work: Lindy.ai
- Publisher
- Latent Space
293 / evaluation
that are starting to provide integrations as a service, right? I used to work in an integrations company.
“that are starting to provide integrations as a service, right? I used to work in an integrations company.”
- Speaker
- Shawn Wang
- Episode
- Agents @ Work: Dust.tt
- Publisher
- Latent Space
294 / evaluation
There's all these other companies that are like, we will do the integrations for you.
“There's all these other companies that are like, we will do the integrations for you.”
- Speaker
- Shawn Wang
- Episode
- Agents @ Work: Dust.tt
- Publisher
- Latent Space
295 / preference
Anyway, but like you want creators who are empowered by a bunch of agents, Dust agents to do all this stuff because then ultimately it's just the brand, the curation.
“Anyway, but like you want creators who are empowered by a bunch of agents, Dust agents to do all this stuff because then ultimately it's just the brand, the curation.”
- Speaker
- Shawn Wang
- Episode
- Agents @ Work: Dust.tt
- Publisher
- Latent Space
296 / evaluation
It's funny because in some ways, the model labs are competing for you, right? You don't have to do any effort.
“It's funny because in some ways, the model labs are competing for you, right? You don't have to do any effort.”
- Speaker
- Shawn Wang
- Episode
- Agents @ Work: Dust.tt
- Publisher
- Latent Space
297 / commitment
We build our brand in one specific niche. And in future, if we want to choose to spin off platforms for other things, we can because we have that brand.
“We build our brand in one specific niche. And in future, if we want to choose to spin off platforms for other things, we can because we have that brand.”
- Speaker
- Shawn Wang
- Episode
- Agents @ Work: Dust.tt
- Publisher
- Latent Space
298 / belief
Anastasios, I actually saw you, I think at last year's NeurIPS. You were presenting a paper, which I don't really super understand, but it was some theory paper about how your method was very dominating over other sort of search methods.
“Anastasios, I actually saw you, I think at last year's NeurIPS. You were presenting a paper, which I don't really super understand, but it was some theory paper about how your method was very dominating over other sort of search methods.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
299 / evaluation
I think my fundamental philosophical doubt is, does the router model have to be at least as smart as the smartest model?
“I think my fundamental philosophical doubt is, does the router model have to be at least as smart as the smartest model?”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
300 / preference
I love the notebooks you guys publish. Actually really good just for learning statistics.
“I love the notebooks you guys publish. Actually really good just for learning statistics.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
301 / evaluation
The classic one for human preference evaluation is humans demonstrably prefer longer contexts or longer outputs, which is actually something that we don't necessarily want. You guys, I think maybe two months ago put out some length control studies.
“The classic one for human preference evaluation is humans demonstrably prefer longer contexts or longer outputs, which is actually something that we don't necessarily want. You guys, I think maybe two months ago put out some length control studies.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
302 / recommendation
Blast from the past. It's always interesting how NeurIPS and all these academic conferences are sort of six months behind what people are actually doing, but conformal risk control, I would recommend people check it out.
“Blast from the past. It's always interesting how NeurIPS and all these academic conferences are sort of six months behind what people are actually doing, but conformal risk control, I would recommend people check it out.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
303 / belief
I would say OpenAI has a pretty similar person, Andrew Mason, I think his name is.
“I would say OpenAI has a pretty similar person, Andrew Mason, I think his name is.”
- Speaker
- Shawn Wang
- Episode
- How NotebookLM Was Made
- Publisher
- Latent Space
304 / belief
I noticed that you say no a lot, I think, or you try to ship one thing and that there's different about you than maybe other PMs or other teams that try to ship, but they're like, Oh, here are all the knobs.
“I noticed that you say no a lot, I think, or you try to ship one thing and that there's different about you than maybe other PMs or other teams that try to ship, but they're like, Oh, here are all the knobs.”
- Speaker
- Shawn Wang
- Episode
- How NotebookLM Was Made
- Publisher
- Latent Space
305 / belief
I'm trying to figure out what you nailed compared to others. And I think that the way that you treat your, the AI is like a little bit different than a lot of the builders I talked to.
“I'm trying to figure out what you nailed compared to others. And I think that the way that you treat your, the AI is like a little bit different than a lot of the builders I talked to.”
- Speaker
- Shawn Wang
- Episode
- How NotebookLM Was Made
- Publisher
- Latent Space
306 / evaluation
Top P top cake. It doesn't matter. I'll just put it in the docs and you figure it out.
“Top P top cake. It doesn't matter. I'll just put it in the docs and you figure it out.”
- Speaker
- Shawn Wang
- Episode
- How NotebookLM Was Made
- Publisher
- Latent Space
307 / uncertainty
Yeah, I think Steven Johnson, I think he's on your team. I don't know what his role is.
“Yeah, I think Steven Johnson, I think he's on your team. I don't know what his role is.”
- Speaker
- Shawn Wang
- Episode
- How NotebookLM Was Made
- Publisher
- Latent Space
308 / uncertainty
I would love, like, a, you know, like when you play Baldur's Gate, like, you roll, you roll like 17 on Charisma and like, it's like what race they are. I don't know.
“I would love, like, a, you know, like when you play Baldur's Gate, like, you roll, you roll like 17 on Charisma and like, it's like what race they are. I don't know.”
- Speaker
- Shawn Wang
- Episode
- How NotebookLM Was Made
- Publisher
- Latent Space
309 / belief
I actually said, I think you saw this. I think that Notebook LLM was kind of like the ChatGPT moment for Google.
“I actually said, I think you saw this. I think that Notebook LLM was kind of like the ChatGPT moment for Google.”
- Speaker
- Shawn Wang
- Episode
- How NotebookLM Was Made
- Publisher
- Latent Space
310 / belief
You know, I think a lot of it is competent PMing and engineering, but also just, you know, it's interesting how a lot of these projects are always like low key research previews for you.
“You know, I think a lot of it is competent PMing and engineering, but also just, you know, it's interesting how a lot of these projects are always like low key research previews for you.”
- Speaker
- Shawn Wang
- Episode
- How NotebookLM Was Made
- Publisher
- Latent Space
311 / belief
I would say one of the toughest AI engineering disciplines out there because even their API doesn't do interruptions that well, to be honest.
“I would say one of the toughest AI engineering disciplines out there because even their API doesn't do interruptions that well, to be honest.”
- Speaker
- Shawn Wang
- Episode
- How NotebookLM Was Made
- Publisher
- Latent Space
312 / belief
I think sometimes the difficulty is because we're dealing with non-deterministic models, sometimes you just got a bad roll of the dice and it's always on the distribution that you could get something bad.
“I think sometimes the difficulty is because we're dealing with non-deterministic models, sometimes you just got a bad roll of the dice and it's always on the distribution that you could get something bad.”
- Speaker
- Shawn Wang
- Episode
- How NotebookLM Was Made
- Publisher
- Latent Space
313 / uncertainty
I'm not sure that the Singapore government is aware that safety sometimes is a bad word in some AI circles because their work is associated with censorship.
“I'm not sure that the Singapore government is aware that safety sometimes is a bad word in some AI circles because their work is associated with censorship.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
314 / belief
I think many Singaporeans are, and that's kind of my pitch on the AI engineer side.
“I think many Singaporeans are, and that's kind of my pitch on the AI engineer side.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
315 / belief
The 30-year war, in my mind, is the kind of scale of operation that we did that leads me to speak English today.
“The 30-year war, in my mind, is the kind of scale of operation that we did that leads me to speak English today.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
316 / uncertainty
I don't know how we get them over the hump of going into industry and being successful engineers and I fear that we're going to create a whole bunch of certificates that don't mean anything.
“I don't know how we get them over the hump of going into industry and being successful engineers and I fear that we're going to create a whole bunch of certificates that don't mean anything.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
317 / uncertainty
I think there's a key term of sovereign AI as well that's kind of going around. I don't know what level this is at.
“I think there's a key term of sovereign AI as well that's kind of going around. I don't know what level this is at.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
318 / belief
I think one of your agencies had a part to do with that, and I'm bringing my own conference as well to host alongside.
“I think one of your agencies had a part to do with that, and I'm bringing my own conference as well to host alongside.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
319 / preference
It's not even about the pivoting. They might just train from the start, but the point is that they can take a foundation model that is capable of anything and actually fashion it into a useful product at the end of it.
“It's not even about the pivoting. They might just train from the start, but the point is that they can take a foundation model that is capable of anything and actually fashion it into a useful product at the end of it.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
320 / evaluation
I will say I've been dealing with EDB a little bit from my conference, and they've been extremely responsive and it's been nice to see, because I never get to see this out of government, nice to see that as someone that wants to bring a foreign business into Singapore, they're kind of rolling on the welcome mat.
“I will say I've been dealing with EDB a little bit from my conference, and they've been extremely responsive and it's been nice to see, because I never get to see this out of government, nice to see that as someone that wants to bring a foreign business into Singapore, they're kind of rolling on the welcome mat.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
321 / preference
I don't have a point apart from which to say, I hope that people who are looking to enter Singapore, don't have that preconception that we are hard to deal with because we're very eager, I think, is my perception.
“I don't have a point apart from which to say, I hope that people who are looking to enter Singapore, don't have that preconception that we are hard to deal with because we're very eager, I think, is my perception.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
322 / preference
I don't know what the right answer is, but I will say that my perception is a lot of the Bay Area, San Francisco is on the, let everything be unregulated as possible.
“I don't know what the right answer is, but I will say that my perception is a lot of the Bay Area, San Francisco is on the, let everything be unregulated as possible.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
323 / belief
Sometimes it's not even just that the ideas are not worth anything. It's almost just the persistence and execution that I think you do very well.
“Sometimes it's not even just that the ideas are not worth anything. It's almost just the persistence and execution that I think you do very well.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
324 / belief
I've had the joy of watching you build this company a little bit, and I think you're one of the top founders I've ever met, so it's just great to sit down with you and learn a little bit.
“I've had the joy of watching you build this company a little bit, and I think you're one of the top founders I've ever met, so it's just great to sit down with you and learn a little bit.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
325 / uncertainty
You want to sprinkle LLMs into a database. Because we know how to scale systems, we don't know how to scale agents that are quite hard to be reliable.
“You want to sprinkle LLMs into a database. Because we know how to scale systems, we don't know how to scale agents that are quite hard to be reliable.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
326 / belief
You can also do evals and all the other stuff. So there's a lot of tooling, and I think you saw something or you just had the self-belief where I didn't, or you saw something that was missing still, even in that space from DIY no-code Google Sheets to custom tool, they were first movers.
“You can also do evals and all the other stuff. So there's a lot of tooling, and I think you saw something or you just had the self-belief where I didn't, or you saw something that was missing still, even in that space from DIY no-code Google Sheets to custom tool, they were first movers.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
327 / commitment
I will mention one thing that I'm trying to figure out. We obliquely mentioned the GPU inference market.
“I will mention one thing that I'm trying to figure out. We obliquely mentioned the GPU inference market.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
328 / uncertainty
I don't know if it's quotable, but you said you had something from Vercel, from Malte.
“I don't know if it's quotable, but you said you had something from Vercel, from Malte.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
329 / belief
One other part of the triangle that I drew that you disagree with, and I thought that was very insightful, was fine-tuning. So I had all these overlapping circles, and I think you agreed with most of them, and I was like, at the center of it all, because you need logging from Ops, and then you need a gateway, and then you need a database with a framework, or whatever, was fine-tuning.
“One other part of the triangle that I drew that you disagree with, and I thought that was very insightful, was fine-tuning. So I had all these overlapping circles, and I think you agreed with most of them, and I was like, at the center of it all, because you need logging from Ops, and then you need a gateway, and then you need a database with a framework, or whatever, was fine-tuning.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
330 / belief
While we're in the name-dropping and doing shout-outs, I think a lot of people in the San Francisco startup scene know Alana, and most people won't.
“While we're in the name-dropping and doing shout-outs, I think a lot of people in the San Francisco startup scene know Alana, and most people won't.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
331 / belief
I think there's a lot of marketing from SingleStore that makes sense, but there's a lot of doubt in people's minds.
“I think there's a lot of marketing from SingleStore that makes sense, but there's a lot of doubt in people's minds.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
332 / belief
I think there's a lot of value in being the chosen tool for an industry because then you just get a lot of community patience for figuring stuff out.
“I think there's a lot of value in being the chosen tool for an industry because then you just get a lot of community patience for figuring stuff out.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
333 / belief
I think that you are very perceptive in your mental modeling of me, because I do disagree 15-25%.
“I think that you are very perceptive in your mental modeling of me, because I do disagree 15-25%.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
334 / belief
I think being so early in single store also taught you, among all these engineering lessons, you also learned a lot of business lessons that you took with you into Impira.
“I think being so early in single store also taught you, among all these engineering lessons, you also learned a lot of business lessons that you took with you into Impira.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
335 / belief
I have a question on this market because I think after Impera, there's a cohort of new Imperas coming out.
“I have a question on this market because I think after Impera, there's a cohort of new Imperas coming out.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
336 / uncertainty
Yeah, yeah, I'm a little bit scared to fine tune especially for vision, because I don't know what I don't know for stuff like vision, right?
“Yeah, yeah, I'm a little bit scared to fine tune especially for vision, because I don't know what I don't know for stuff like vision, right?”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
337 / belief
Like, you know, I think, I think, before we recorded, we discussed a little bit about the sensitivities around basically calling random store owners and putting, putting like an AI on them.
“Like, you know, I think, I think, before we recorded, we discussed a little bit about the sensitivities around basically calling random store owners and putting, putting like an AI on them.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
338 / uncertainty
Maybe they're going to run L1 for five hours instead of five minutes, and then suddenly it works. So, I don't know.
“Maybe they're going to run L1 for five hours instead of five minutes, and then suddenly it works. So, I don't know.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
339 / belief
I think one, one people, one issue that people have with the voice so far as, as released in advanced voice mode is the refusals.
“I think one, one people, one issue that people have with the voice so far as, as released in advanced voice mode is the refusals.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
340 / belief
Like, you know, like I think for me, like, WebSockets, you just receive a bunch of events.
“Like, you know, like I think for me, like, WebSockets, you just receive a bunch of events.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
341 / belief
Yeah, I have more questions on, a couple questions on voice, and then also, like, your call to action, like, what you want feedback on, right? So, I think we should spend a bit more time on voice, because I feel like that's, like, the big splash thing.
“Yeah, I have more questions on, a couple questions on voice, and then also, like, your call to action, like, what you want feedback on, right? So, I think we should spend a bit more time on voice, because I feel like that's, like, the big splash thing.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
342 / belief
You guys had a very inspiring model spec. I think Joanne worked on that. Where you said, like, yeah, we don't want to overly refuse all the time.
“You guys had a very inspiring model spec. I think Joanne worked on that. Where you said, like, yeah, we don't want to overly refuse all the time.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
343 / uncertainty
Like, I always want to eval the eval. I don't know if that ever came up. Like, sometimes the evals themselves are wrong, and there's no way for me to tell you.
“Like, I always want to eval the eval. I don't know if that ever came up. Like, sometimes the evals themselves are wrong, and there's no way for me to tell you.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
344 / uncertainty
I don't know. I don't know if I got an invite. No. I can't just talk to you. Yeah, like, and then there was some speculation around October 1st.
“I don't know. I don't know if I got an invite. No. I can't just talk to you. Yeah, like, and then there was some speculation around October 1st.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
345 / belief
Awesome. I think that's our time. Thank you so much, guys. Yeah, thank you so much.
“Awesome. I think that's our time. Thank you so much, guys. Yeah, thank you so much.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
346 / prediction
I mean, I'm aware of them, but I think I'm excited to see how all distillation works. That's something that we've been doing like, I don't know, I've been like doing it between our models for a while And I've seen really good results like I've done back in a day like from GPT 4 to GPT 3.
“I mean, I'm aware of them, but I think I'm excited to see how all distillation works. That's something that we've been doing like, I don't know, I've been like doing it between our models for a while And I've seen really good results like I've done back in a day like from GPT 4 to GPT 3.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
347 / evaluation
I'm pretty science-based, like, you know, but probably the most like spiritual woo-woo thing about me is I don't think that would lead to consciousness or AGI just because like there's something in- there's a soul, you know?
“I'm pretty science-based, like, you know, but probably the most like spiritual woo-woo thing about me is I don't think that would lead to consciousness or AGI just because like there's something in- there's a soul, you know?”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
348 / prediction
I think that's the main learning I have from Devin. They cracked that. Actually, there was no foundational planning breakthrough.
“I think that's the main learning I have from Devin. They cracked that. Actually, there was no foundational planning breakthrough.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
349 / recommendation
The way I put this is you should not be a prompt engineer because it is the goal of the big labs to put you out of a job.
“The way I put this is you should not be a prompt engineer because it is the goal of the big labs to put you out of a job.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
350 / evaluation
I think OpenAI was just like, look, this is a thing now. We have to fix this. These students just rushed it.
“I think OpenAI was just like, look, this is a thing now. We have to fix this. These students just rushed it.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
351 / evaluation
You're one of like, you're maybe the first PhD thesis defense I've ever watched in like this AI world, because most people just publish single papers, but every paper of yours is a banger.
“You're one of like, you're maybe the first PhD thesis defense I've ever watched in like this AI world, because most people just publish single papers, but every paper of yours is a banger.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
352 / prediction
I actually always wanted to take like a selfie and go like, you know, POV, you're about to revolutionize the world of agents because we have two of the most awesome hiring agents in the house.
“I actually always wanted to take like a selfie and go like, you know, POV, you're about to revolutionize the world of agents because we have two of the most awesome hiring agents in the house.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
353 / evaluation
I will mostly agree and I'll slightly disagree in terms of this, which is like, whether designing for humans also overlaps with designing for AI. So Malte Ubo, who's the CTO of Vercel, who is creating basically JavaScript's competitor to LangChain, they're observing that basically, like if the API is easy to understand for humans, it's actually much easier to understand for LLMs, for example, because they're not overloaded functions.
“I will mostly agree and I'll slightly disagree in terms of this, which is like, whether designing for humans also overlaps with designing for AI. So Malte Ubo, who's the CTO of Vercel, who is creating basically JavaScript's competitor to LangChain, they're observing that basically, like if the API is easy to understand for humans, it's actually much easier to understand for LLMs, for example, because they're not overloaded functions.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
354 / evaluation
Like, I actually have most of my problems with AI news when the model thinks it knows more than it knows because it combines knowledge with intelligence.
“Like, I actually have most of my problems with AI news when the model thinks it knows more than it knows because it combines knowledge with intelligence.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
355 / evaluation
I actually might tweak my approach based on that, because I was trying to give bad examples of do not do this, and it still does it, and maybe that doesn't work.
“I actually might tweak my approach based on that, because I was trying to give bad examples of do not do this, and it still does it, and maybe that doesn't work.”
- Speaker
- Shawn Wang
- Episode
- The Ultimate Guide to Prompting
- Publisher
- Latent Space
356 / uncertainty
I guess the best way to go through this, you know, you picked out 58 techniques out of your, I don't know, 4,000 papers that you reviewed, maybe we just pick through a few of these that are special to you and discuss them a little bit.
“I guess the best way to go through this, you know, you picked out 58 techniques out of your, I don't know, 4,000 papers that you reviewed, maybe we just pick through a few of these that are special to you and discuss them a little bit.”
- Speaker
- Shawn Wang
- Episode
- The Ultimate Guide to Prompting
- Publisher
- Latent Space
357 / recommendation
Prompt Layer, Braintrust, PromptFu, and HumanLoop, I guess would be my top picks from that category of people. And there's probably others that I don't know about.
“Prompt Layer, Braintrust, PromptFu, and HumanLoop, I guess would be my top picks from that category of people. And there's probably others that I don't know about.”
- Speaker
- Shawn Wang
- Episode
- The Ultimate Guide to Prompting
- Publisher
- Latent Space
358 / preference
I think one problem for me for designing these things with cost awareness is the question of, well, okay, at the baseline, you can just use the same model for everything, but realistically you have a range of models, and actually you just want to sample all range.
“I think one problem for me for designing these things with cost awareness is the question of, well, okay, at the baseline, you can just use the same model for everything, but realistically you have a range of models, and actually you just want to sample all range.”
- Speaker
- Shawn Wang
- Episode
- The Ultimate Guide to Prompting
- Publisher
- Latent Space
359 / evaluation
Endorse all that. And I think getting things into structured output and doing those scoring is a very core part of AI engineering that we don't talk about enough.
“Endorse all that. And I think getting things into structured output and doing those scoring is a very core part of AI engineering that we don't talk about enough.”
- Speaker
- Shawn Wang
- Episode
- The Ultimate Guide to Prompting
- Publisher
- Latent Space
360 / recommendation
I'll call out two recent papers which people might want to look into, which is a Salesforce yesterday released a paper called Diversity Empowered Intelligence, which is a, I think a shot at the bow for scale AI.
“I'll call out two recent papers which people might want to look into, which is a Salesforce yesterday released a paper called Diversity Empowered Intelligence, which is a, I think a shot at the bow for scale AI.”
- Speaker
- Shawn Wang
- Episode
- The Ultimate Guide to Prompting
- Publisher
- Latent Space
361 / evaluation
You've done very well. And I think you've honestly done the community a service by reading all these papers so that we don't have to, because the joke is often that, you know, what is one prompt is like then inflated into like a 10 page PDF that's posted on archive.
“You've done very well. And I think you've honestly done the community a service by reading all these papers so that we don't have to, because the joke is often that, you know, what is one prompt is like then inflated into like a 10 page PDF that's posted on archive.”
- Speaker
- Shawn Wang
- Episode
- The Ultimate Guide to Prompting
- Publisher
- Latent Space
362 / recommendation
We just assume that people roughly know, but yeah, I think a dedicated episode directly on this, I think is something that's sorely needed. And then, you know, something I prompted Sander with is when I wrote about the rise of the AI engineer, it was actually a direct opposition to the rise of the prompt engineer, right?
“We just assume that people roughly know, but yeah, I think a dedicated episode directly on this, I think is something that's sorely needed. And then, you know, something I prompted Sander with is when I wrote about the rise of the AI engineer, it was actually a direct opposition to the rise of the prompt engineer, right?”
- Speaker
- Shawn Wang
- Episode
- The Ultimate Guide to Prompting
- Publisher
- Latent Space
363 / belief
I think the problem here comes from like, I think we understand how to do this in a normal ML context, but when you're trying to build AGI, the real world is everything.
“I think the problem here comes from like, I think we understand how to do this in a normal ML context, but when you're trying to build AGI, the real world is everything.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
364 / belief
Distill the knowledge of the benchmark into a model, and then obviously it's going to perform better on the benchmark. But I think what's less understood now is, um, you know, the sort of un gamable leaderboards, like the LMSys leaderboard, like some, it's also possible to game those things, and you can distill smaller models to do well on those.
“Distill the knowledge of the benchmark into a model, and then obviously it's going to perform better on the benchmark. But I think what's less understood now is, um, you know, the sort of un gamable leaderboards, like the LMSys leaderboard, like some, it's also possible to game those things, and you can distill smaller models to do well on those.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
365 / uncertainty
Is there any well known brand that People can link to, uh, you know, I know about like AI influencers, like on Instagram or AI wrappers, but I don't know about brand, uh, identities.
“Is there any well known brand that People can link to, uh, you know, I know about like AI influencers, like on Instagram or AI wrappers, but I don't know about brand, uh, identities.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
366 / belief
I'm sure it's very uncomfortable for him, but I think, I think he kind of embraces it.
“I'm sure it's very uncomfortable for him, but I think, I think he kind of embraces it.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
367 / belief
I would say to bring people up to speed as well in like very recent developments.
“I would say to bring people up to speed as well in like very recent developments.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
368 / prediction
So when you, when you quantize things, obviously you're going to lose precision because you just have less bits to store information in.
“So when you, when you quantize things, obviously you're going to lose precision because you just have less bits to store information in.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
369 / evaluation
I don't know what to make about it because I don't think it's adopted seriously by the large labs.
“I don't know what to make about it because I don't think it's adopted seriously by the large labs.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
370 / preference
I regret to say that I don't think my chat history is used as much these days, because I'm using Cursor, the native AI IDE.
“I regret to say that I don't think my chat history is used as much these days, because I'm using Cursor, the native AI IDE.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
371 / recommendation
Specifically because you brought up spreadsheets, I want to share my personal experience because I think Google has done a really good job that people don't know about, which is if you use Google Sheets, Gemini is integrated inside of Google Sheets and it helps you write formulas.
“Specifically because you brought up spreadsheets, I want to share my personal experience because I think Google has done a really good job that people don't know about, which is if you use Google Sheets, Gemini is integrated inside of Google Sheets and it helps you write formulas.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
372 / belief
I think one of the decisions before you as a CEO is how much you have like the house model be like the one true thing, and then how much you spend time working on customer models.
“I think one of the decisions before you as a CEO is how much you have like the house model be like the one true thing, and then how much you spend time working on customer models.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
373 / belief
The decision to fork VS Code, I think, was controversial. You guys started as a VS Code extension.
“The decision to fork VS Code, I think, was controversial. You guys started as a VS Code extension.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
374 / belief
I'll briefly comment here. So this is the standard approach I would say most, uh, code tooling startups are pursuing.
“I'll briefly comment here. So this is the standard approach I would say most, uh, code tooling startups are pursuing.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
375 / prediction
I think that translation between natural language, English versus code, and back and forth, I think is actually a really ripe source of synthetic data.
“I think that translation between natural language, English versus code, and back and forth, I think is actually a really ripe source of synthetic data.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
376 / belief
I hope that I sort of editorialized the title a little bit, and I know you were slightly uncomfortable with it, but you just own it anyway. I think you're very good at the hot takes.
“I hope that I sort of editorialized the title a little bit, and I know you were slightly uncomfortable with it, but you just own it anyway. I think you're very good at the hot takes.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
377 / uncertainty
I was actually, we just did an interview with Yi Tay from Reka, I don't know if you're familiar with his work, but also another encoder-decoder bet, and one of his arguments was actually people kind of over-index on the decoder-only GPT-3 type paradigm.
“I was actually, we just did an interview with Yi Tay from Reka, I don't know if you're familiar with his work, but also another encoder-decoder bet, and one of his arguments was actually people kind of over-index on the decoder-only GPT-3 type paradigm.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
378 / belief
People consider Streamlit and Gradio to be the state of the art, but I think there's so much to improve, and having what you call web foundations and web fundamentals at the core of it, I think, would be really helpful.
“People consider Streamlit and Gradio to be the state of the art, but I think there's so much to improve, and having what you call web foundations and web fundamentals at the core of it, I think, would be really helpful.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
379 / belief
I want to talk about more about things from the paper, but I think we're still in this sort of demo section.
“I want to talk about more about things from the paper, but I think we're still in this sort of demo section.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
380 / prediction
It's weird because like, I guess with segment anything too, it's like 4D because you solve time, you know, you started with 3D and now you're solving the 4D.
“It's weird because like, I guess with segment anything too, it's like 4D because you solve time, you know, you started with 3D and now you're solving the 4D.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
381 / recommendation
Mentions personal use of DevIn.
“I used to tell people go to the DevIn demo and look at the four things that they offer and say each of those things is a startup.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
382 / uncertainty
I don't know, you're way better at prompting than me, so I wanted to capture how you prompted as well.
“I don't know, you're way better at prompting than me, so I wanted to capture how you prompted as well.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
383 / belief
Nemotron is worth looking at if you're interested in synthetic data. Multimodal labeling, I think, has happened a lot.
“Nemotron is worth looking at if you're interested in synthetic data. Multimodal labeling, I think, has happened a lot.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
384 / belief
Open source is synonymous with local, private, and all the good things that people want. And I think their vision of even running this stuff on CPUs at a very, very fast speed by just being extremely cracked, I think is very understated.
“Open source is synonymous with local, private, and all the good things that people want. And I think their vision of even running this stuff on CPUs at a very, very fast speed by just being extremely cracked, I think is very understated.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
385 / uncertainty
I don't know if we Okay, do you want to pick a multiple voices, emotion we also have Chinese language.
“I don't know if we Okay, do you want to pick a multiple voices, emotion we also have Chinese language.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
386 / uncertainty
I'm excited for OpenAI phone. I don't know if you would buy an OpenAI phone. I mean, I'm very locked into the iOS ecosystem.
“I'm excited for OpenAI phone. I don't know if you would buy an OpenAI phone. I mean, I'm very locked into the iOS ecosystem.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
387 / belief
I have to be careful what I say here, but I think because we still want to respect their work.
“I have to be careful what I say here, but I think because we still want to respect their work.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
388 / uncertainty
Hey, this is our long-awaited one-on-one episode. I don't know how long ago the previous one was.
“Hey, this is our long-awaited one-on-one episode. I don't know how long ago the previous one was.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
389 / belief
I think Google or Meta should pay $1 billion for Noam alone. The purchase price for a Character is $1 billion, which is super underpriced.
“I think Google or Meta should pay $1 billion for Noam alone. The purchase price for a Character is $1 billion, which is super underpriced.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
390 / belief
I think we can move on from accents. It can do accents. We get that. I was impressed by the New Zealand versus Australia.
“I think we can move on from accents. It can do accents. We get that. I was impressed by the New Zealand versus Australia.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
391 / belief
People really enjoyed it. It's just really, I think our travel schedules have been really difficult to get this stuff together.
“People really enjoyed it. It's just really, I think our travel schedules have been really difficult to get this stuff together.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
392 / belief
We just didn't have the paper beforehand. But I think, like, when I call Lama3, the synthetic data model is you have the license for it, but then you also have the roadmap, the recipe, because it's in the paper.
“We just didn't have the paper beforehand. But I think, like, when I call Lama3, the synthetic data model is you have the license for it, but then you also have the roadmap, the recipe, because it's in the paper.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
393 / belief
Yeah. I think that the danger with the lawsuit, because this lawsuit is very public.
“Yeah. I think that the danger with the lawsuit, because this lawsuit is very public.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
394 / belief
I think LLM Ops or whatever you call this, AI Engineer Ops, the Ops layer on top of the LLM layer might follow the same evolution path as the ML Ops layer.
“I think LLM Ops or whatever you call this, AI Engineer Ops, the Ops layer on top of the LLM layer might follow the same evolution path as the ML Ops layer.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
395 / belief
To me, I think a lot of people were trying to puzzle out the strategic moves between OpenAI and Apple here because Apple is in a very good position to commoditize OpenAI.
“To me, I think a lot of people were trying to puzzle out the strategic moves between OpenAI and Apple here because Apple is in a very good position to commoditize OpenAI.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
396 / belief
Some things don't change. But on top of that, I think one of the reasons I put vector databases inside of these wars is in order to grow, the vector databases have to become more frameworks.
“Some things don't change. But on top of that, I think one of the reasons I put vector databases inside of these wars is in order to grow, the vector databases have to become more frameworks.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
397 / belief
I think we're the only ones to coordinate for the paper release for the big launch, the 4.
“I think we're the only ones to coordinate for the paper release for the big launch, the 4.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
398 / commitment
I will call out, you know, they have some interesting experimentations with Mamba and Mistral NEMO is actually on the efficiency frontier chart that I drew that is still relevant.
“I will call out, you know, they have some interesting experimentations with Mamba and Mistral NEMO is actually on the efficiency frontier chart that I drew that is still relevant.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
399 / belief
I think that what RAG Ops Wars are to me, like the tooling around the ecosystem.
“I think that what RAG Ops Wars are to me, like the tooling around the ecosystem.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
400 / belief
I think the opposition has increased enough that it's not going to be a real concern for people.
“I think the opposition has increased enough that it's not going to be a real concern for people.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
401 / belief
Laughing. I think we had, we have a simple laughing one. This one, this one, you got it.
“Laughing. I think we had, we have a simple laughing one. This one, this one, you got it.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
402 / belief
Exactly. So, it's like, I think... If you take it more seriously than ChatGPT, you'll win.
“Exactly. So, it's like, I think... If you take it more seriously than ChatGPT, you'll win.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
403 / preference
I think I want to leave some discussion time open for miscellaneous trends that are happening in the industry that don't exactly fit in the four words or are a layer above the four words.
“I think I want to leave some discussion time open for miscellaneous trends that are happening in the industry that don't exactly fit in the four words or are a layer above the four words.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
404 / evaluation
Whenever image generation is concerned, obviously because of the Gemini issue, it is very tricky for large companies to release that.
“Whenever image generation is concerned, obviously because of the Gemini issue, it is very tricky for large companies to release that.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
405 / preference
At the same time, I believe that people should be able to make their own decisions about all these deals.
“At the same time, I believe that people should be able to make their own decisions about all these deals.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
406 / evaluation
These are the same things to the model. That is a huge, huge win for interpretability, because up to now, we were only doing interpretability on toy models, like a few million parameters, a model of Go or chess or whatever.
“These are the same things to the model. That is a huge, huge win for interpretability, because up to now, we were only doing interpretability on toy models, like a few million parameters, a model of Go or chess or whatever.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
407 / recommendation
We should basically, when we're getting to the point where we're over-training models 100 times past Chinchilla ratio to optimize for inference, the next thing is actually like, hey, let's stop using so much memory when training because we're going to quantize it anyway for inference.
“We should basically, when we're getting to the point where we're over-training models 100 times past Chinchilla ratio to optimize for inference, the next thing is actually like, hey, let's stop using so much memory when training because we're going to quantize it anyway for inference.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
408 / prediction
They will be building on architectures like Mobile LM and Small LM, which basically innovate in terms of shared weights and shared matrices for small models so that you just optimize the amount of file size and memory that you take up. And I think just general trend on device models, the only way that intelligence too cheap to meter happens is everything happens on device.
“They will be building on architectures like Mobile LM and Small LM, which basically innovate in terms of shared weights and shared matrices for small models so that you just optimize the amount of file size and memory that you take up. And I think just general trend on device models, the only way that intelligence too cheap to meter happens is everything happens on device.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
409 / uncertainty
Because I think that one of the burning rumors is, I don't know, nothing from Scott, I haven't talked to him at all about this, even though he's very friendly.
“Because I think that one of the burning rumors is, I don't know, nothing from Scott, I haven't talked to him at all about this, even though he's very friendly.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
410 / commitment
I think there are 10 directions that I outlined below, but we'll talk about that later.
“I think there are 10 directions that I outlined below, but we'll talk about that later.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
411 / belief
I would say the diffusion people have actually started to swing back to pixel level and probably that will presage the language people also moving towards, you know, 1 million vocabulary and then, you know, whatever the natural limit is for character level.
“I would say the diffusion people have actually started to swing back to pixel level and probably that will presage the language people also moving towards, you know, 1 million vocabulary and then, you know, whatever the natural limit is for character level.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
412 / uncertainty
We still need to have, at the architecture level, some kind of variable inference length thing that lets you actually think in latent space, like you're talking about. I don't know if there's any papers that you're thinking about.
“We still need to have, at the architecture level, some kind of variable inference length thing that lets you actually think in latent space, like you're talking about. I don't know if there's any papers that you're thinking about.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
413 / belief
One of our previous guests, Brian Bischoff, is also asking about like, how do we think about evals for practical things like confidence estimation, structured output, you know, stuff like that.
“One of our previous guests, Brian Bischoff, is also asking about like, how do we think about evals for practical things like confidence estimation, structured output, you know, stuff like that.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
414 / evaluation
We love just Meta's commitment to open source and, you know, you do what you need to do to make it work for your organization.
“We love just Meta's commitment to open source and, you know, you do what you need to do to make it work for your organization.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
415 / evaluation
The other thing is RLHF came from the alignment community. And I think there's a lot of conception that maybe it's due to safety concerns, but I feel like it's really over the past two, three years expanded to just this produces a better model period, even if you don't really are not that concerned about existential risk.
“The other thing is RLHF came from the alignment community. And I think there's a lot of conception that maybe it's due to safety concerns, but I feel like it's really over the past two, three years expanded to just this produces a better model period, even if you don't really are not that concerned about existential risk.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
416 / prediction
To me, you know, I'm not exactly sure what you guys did, but like, I feel like when people say synthetic data, there needs to be different categories of synthetic data now, because I think there's so many different usage of this thing.
“To me, you know, I'm not exactly sure what you guys did, but like, I feel like when people say synthetic data, there needs to be different categories of synthetic data now, because I think there's so many different usage of this thing.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
417 / evaluation
We're basically saying that, I think that one of the lessons from AlphaGo is that people thought that human interest in Go would be diminished because computers are better than humans.
“We're basically saying that, I think that one of the lessons from AlphaGo is that people thought that human interest in Go would be diminished because computers are better than humans.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
418 / evaluation
I don't know how to describe, you've done so much work in a very short amount of time at Meta, but you were most notably leading Llama 2 and now today we're also coordinating on the release of Llama 3.
“I don't know how to describe, you've done so much work in a very short amount of time at Meta, but you were most notably leading Llama 2 and now today we're also coordinating on the release of Llama 3.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
419 / evaluation
Because I didn't know that, I don't know how much to believe, you know, like there's a lot of these kinds of papers where it makes a lot of noise, but it doesn't actually pan out.
“Because I didn't know that, I don't know how much to believe, you know, like there's a lot of these kinds of papers where it makes a lot of noise, but it doesn't actually pan out.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
420 / evaluation
I think, yeah, very underrated, very underrated, this sort of PhD with industry expertise, because you're also publishing papers the whole time.
“I think, yeah, very underrated, very underrated, this sort of PhD with industry expertise, because you're also publishing papers the whole time.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
421 / evaluation
Well, I think your progress into NLP was like really strong, because like the first thing you worked on at Meta was Bloom.
“Well, I think your progress into NLP was like really strong, because like the first thing you worked on at Meta was Bloom.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
422 / belief
I'll throw in one more, which is, I think the sample of the chatbot arena data is actually out there.
“I'll throw in one more, which is, I think the sample of the chatbot arena data is actually out there.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
423 / belief
I think the only other, before this, it was kind of like ML perf, that's the other big leaderboard that I can think about where, obviously, maybe AlexNet, specific competitions, specific benchmarks, but not something that aggregates across all the other benchmarks.
“I think the only other, before this, it was kind of like ML perf, that's the other big leaderboard that I can think about where, obviously, maybe AlexNet, specific competitions, specific benchmarks, but not something that aggregates across all the other benchmarks.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
424 / recommendation
You just gave me an idea that Goodreads should be a data set because these are all novels that are commentaries about the contents of the novel.
“You just gave me an idea that Goodreads should be a data set because these are all novels that are commentaries about the contents of the novel.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
425 / evaluation
I think obviously MMLU Pro is the top one, just because that's the top number that a lot of people report.
“I think obviously MMLU Pro is the top one, just because that's the top number that a lot of people report.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
426 / evaluation
Yeah, so I really like this concept of evaluation. So actually, yeah, I think there's typically what I always say is like sort of 25 is random chance, 50 is average human, 75 is expert human, 90 is you're cheating.
“Yeah, so I really like this concept of evaluation. So actually, yeah, I think there's typically what I always say is like sort of 25 is random chance, 50 is average human, 75 is expert human, 90 is you're cheating.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
427 / evaluation
Obviously, I think you're like our second or third person from Hugging Face on the podcast and it's like the definitional sort of open AI company, maybe the real open AI.
“Obviously, I think you're like our second or third person from Hugging Face on the podcast and it's like the definitional sort of open AI company, maybe the real open AI.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
428 / belief
I think. Okay. The last part that makes me uncomfortable about MOE debate is actually it's related to another paper that you wrote about the efficiency misnomer, in the sense that, like, now people are trying to make the debate all about the active parameters rather than total parameters.
“I think. Okay. The last part that makes me uncomfortable about MOE debate is actually it's related to another paper that you wrote about the efficiency misnomer, in the sense that, like, now people are trying to make the debate all about the active parameters rather than total parameters.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
429 / belief
I imagine it's sensitive. But also in my mind this is now the most And this is the most valuable on the job training in the world.
“I imagine it's sensitive. But also in my mind this is now the most And this is the most valuable on the job training in the world.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
430 / belief
We can make this the ending conversation, which is, I think you're an inspiration to a lot of other people who want to follow your career path, and, you know, I'm really glad that we got the chance to, like, walk through your career a bit.
“We can make this the ending conversation, which is, I think you're an inspiration to a lot of other people who want to follow your career path, and, you know, I'm really glad that we got the chance to, like, walk through your career a bit.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
431 / belief
I think in retrospect, you understand now that this is a very valuable experience. And I think now, today, it will be much more competitive to get the job that you got, whereas you didn't, you know, two years ago, you didn't have to try that hard to get it.
“I think in retrospect, you understand now that this is a very valuable experience. And I think now, today, it will be much more competitive to get the job that you got, whereas you didn't, you know, two years ago, you didn't have to try that hard to get it.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
432 / commitment
I will put myself on the other, so your mirror image, which is like, long context is good for prototyping, but any production system will just move to RAG.
“I will put myself on the other, so your mirror image, which is like, long context is good for prototyping, but any production system will just move to RAG.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
433 / commitment
Like how do you, like, from the start of like, you haven't trained anything yet and you're about to kick off the run, like, are you able to like call your shots and say, we will beat GP 3.
“Like how do you, like, from the start of like, you haven't trained anything yet and you're about to kick off the run, like, are you able to like call your shots and say, we will beat GP 3.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
434 / uncertainty
I don't know if you have a view on it, but it's just, like, it should be known to be hard.
“I don't know if you have a view on it, but it's just, like, it should be known to be hard.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
435 / uncertainty
I don't, unfortunately, I don't know enough to push back on this, but on the surface of it, it seems to make sense.
“I don't, unfortunately, I don't know enough to push back on this, but on the surface of it, it seems to make sense.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
436 / uncertainty
I don't know if you know, the post money valuation, or the pre money valuation is public.
“I don't know if you know, the post money valuation, or the pre money valuation is public.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
437 / belief
I think you addressed this in your post, but the temptation would have been just to run everything on TPUs.
“I think you addressed this in your post, but the temptation would have been just to run everything on TPUs.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
438 / uncertainty
I, I, I don't know if you have commentary on whether this is obvious to you or this is the, this is the way, or they'll just be, they'll coexist.
“I, I, I don't know if you have commentary on whether this is obvious to you or this is the, this is the way, or they'll just be, they'll coexist.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
439 / belief
You got a whole bunch of GPUs. I think you disclosed somewhere, but I don't remember the exact number.
“You got a whole bunch of GPUs. I think you disclosed somewhere, but I don't remember the exact number.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
440 / uncertainty
I don't know how much, how many evals you want to do, right? Like, I do think Andre Capalti mentioned that.
“I don't know how much, how many evals you want to do, right? Like, I do think Andre Capalti mentioned that.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
441 / belief
Funding, for startups, incubation, holding academic conferences, I think iClear next year is going to be in Singapore, so people come here and get exposed to it.
“Funding, for startups, incubation, holding academic conferences, I think iClear next year is going to be in Singapore, so people come here and get exposed to it.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
442 / belief
I think always the way to go. Is it like a hundred experts, a thousand experts , for some reason the, the community settled on eight.
“I think always the way to go. Is it like a hundred experts, a thousand experts , for some reason the, the community settled on eight.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
443 / uncertainty
Sorry, I, I should have used that word. So like, you know, I don't know if you have any commentary on like ra, deep seek Snowflake Quinn all these proliferation of Moe e models that seem to all be spars op cycle because, you know, you, you were advisor on, on the spars op cycling paper.
“Sorry, I, I should have used that word. So like, you know, I don't know if you have any commentary on like ra, deep seek Snowflake Quinn all these proliferation of Moe e models that seem to all be spars op cycle because, you know, you, you were advisor on, on the spars op cycling paper.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
444 / belief
Mm-Hmm. and I put OS pretty low because I think you have a good base model and then you upcycle it and it bumps you a little bit.
“Mm-Hmm. and I put OS pretty low because I think you have a good base model and then you upcycle it and it bumps you a little bit.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
445 / belief
I think that you navigate the meta very well, and part of my goal here is to also isolate how you think about the meta for other people to reflect on, because I think obviously you do it very well.
“I think that you navigate the meta very well, and part of my goal here is to also isolate how you think about the meta for other people to reflect on, because I think obviously you do it very well.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
446 / commitment
Because I feel like, I'll pick on Palm 2 and Emergence, because I really want to make sure I tell those stories.
“Because I feel like, I'll pick on Palm 2 and Emergence, because I really want to make sure I tell those stories.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
447 / evaluation
Like, just because you re parameterize some benchmarks in evals and, you know, make it linear, doesn't mean emergence is completely gone.
“Like, just because you re parameterize some benchmarks in evals and, you know, make it linear, doesn't mean emergence is completely gone.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
448 / evaluation
Like, or So it's a very interesting observation where like, most efficiency work is just busy work, or like, it's work at a small scale that doesn't, that just ignores the fact that like, this thing doesn't scale, because you haven't scaled it.
“Like, or So it's a very interesting observation where like, most efficiency work is just busy work, or like, it's work at a small scale that doesn't, that just ignores the fact that like, this thing doesn't scale, because you haven't scaled it.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
449 / preference
Rules like that, mantras that you've developed for yourself where you're like, okay, I must do this. So for example, recently for me, I've been trying to run my life on calendar for a long time, and I found that the only way that I work is I write things down on pen and paper, and I cross them off individually.
“Rules like that, mantras that you've developed for yourself where you're like, okay, I must do this. So for example, recently for me, I've been trying to run my life on calendar for a long time, and I found that the only way that I work is I write things down on pen and paper, and I cross them off individually.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
450 / uncertainty
You got the NIC health checks, GPU health check, this space health check, Docker D message. I don't know what that is.
“You got the NIC health checks, GPU health check, this space health check, Docker D message. I don't know what that is.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
451 / uncertainty
I don't know if, John, you had something that you were sort of burning to ask or.
“I don't know if, John, you had something that you were sort of burning to ask or.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
452 / belief
I think people will find creative use cases. And like Jon said, I think the multimodality examples will naturally lend themselves to long context.
“I think people will find creative use cases. And like Jon said, I think the multimodality examples will naturally lend themselves to long context.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
453 / uncertainty
Got it. And I don't know about like what you would highlight so far as a post-acquisition, but the most recent news is that you guys released DBRX.
“Got it. And I don't know about like what you would highlight so far as a post-acquisition, but the most recent news is that you guys released DBRX.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
454 / belief
I think that's basically our sort of recap of our discussion based on Imbue's releases today.
“I think that's basically our sort of recap of our discussion based on Imbue's releases today.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
455 / evaluation
I think a lot of people are exploring that and I think every now and then people get a bout of knowledge graph religion and then it kind of doesn't work out.
“I think a lot of people are exploring that and I think every now and then people get a bout of knowledge graph religion and then it kind of doesn't work out.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
456 / uncertainty
I don't know if that's useful or not, but I mean, that's useful to me as a, as someone who might be working with them.
“I don't know if that's useful or not, but I mean, that's useful to me as a, as someone who might be working with them.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
457 / belief
I think what's unusual is like, I think Shutterstock's doing multiple deals in multiple labs.
“I think what's unusual is like, I think Shutterstock's doing multiple deals in multiple labs.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
458 / recommendation
Yeah, I mean, like Kubernetes, like that point about Kubernetes, I will say as a former AWS employee, like it seems like it would be ideal for imbue to at some point make it more abstracted or agnostic because you're going to want to, you know, replicate your setup.
“Yeah, I mean, like Kubernetes, like that point about Kubernetes, I will say as a former AWS employee, like it seems like it would be ideal for imbue to at some point make it more abstracted or agnostic because you're going to want to, you know, replicate your setup.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
459 / evaluation
One thing I'll, one thing I'll mention quickly is that a lot of the stuff that you mentioned is typically not part of the normal interview loop. It's actually really hard to interview for because this is the stuff that you polish out in, as you go into production, the coding interviews are typically about the happy path.
“One thing I'll, one thing I'll mention quickly is that a lot of the stuff that you mentioned is typically not part of the normal interview loop. It's actually really hard to interview for because this is the stuff that you polish out in, as you go into production, the coding interviews are typically about the happy path.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
460 / evaluation
I have some appreciation, I think when you had me on your podcast, I was still working at Temporal and that was like a nice Framework, if you live within Temporal's boundaries, you can pretend that all those faults don't exist, and you can, you can code in a sort of very fault tolerant way.
“I have some appreciation, I think when you had me on your podcast, I was still working at Temporal and that was like a nice Framework, if you live within Temporal's boundaries, you can pretend that all those faults don't exist, and you can, you can code in a sort of very fault tolerant way.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
461 / belief
I, I think that people raise this idea of fallbacks to models, but I don't think it's, I don't, I don't see it practiced very much.
“I, I think that people raise this idea of fallbacks to models, but I don't think it's, I don't, I don't see it practiced very much.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
462 / belief
I forgot to mention that you also run Ink and Switch, which is one of the leading research labs, in my mind, of the tools for thought productivity space, , whatever people mentioned there, or maybe future of programming even, a little bit of that.
“I forgot to mention that you also run Ink and Switch, which is one of the leading research labs, in my mind, of the tools for thought productivity space, , whatever people mentioned there, or maybe future of programming even, a little bit of that.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
463 / belief
I mean , I, I selected as one of my keynotes, Justine Tunney, working at LlamaFall in Mozilla, because I think there's a lot of people interested in that stuff.
“I mean , I, I selected as one of my keynotes, Justine Tunney, working at LlamaFall in Mozilla, because I think there's a lot of people interested in that stuff.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
464 / prediction
You even shared your job description, your reading list, and your interview loop. So, , if anyone's looking to hire AI engineers, I expect this to be the definitive piece and definitive podcast covering it.
“You even shared your job description, your reading list, and your interview loop. So, , if anyone's looking to hire AI engineers, I expect this to be the definitive piece and definitive podcast covering it.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
465 / evaluation
If some AI engineer would not know, I don't know what, , I don't know where we would stoop to, to call something required knowledge, , or you're not part of the cool kids club.
“If some AI engineer would not know, I don't know what, , I don't know where we would stoop to, to call something required knowledge, , or you're not part of the cool kids club.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
466 / evaluation
The ML first mindset, I think, is something that I struggle with as well, because the errors, when they do happen, are bad.
“The ML first mindset, I think, is something that I struggle with as well, because the errors, when they do happen, are bad.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
467 / evaluation
I, I do often say that I think AI engineering is about 90 percent software engineering with like the, the 10 percent of like really strong really differentiated AI engineering.
“I, I do often say that I think AI engineering is about 90 percent software engineering with like the, the 10 percent of like really strong really differentiated AI engineering.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
468 / recommendation
You know, use Mistral to rephrase some existing part of your data sets, generate more tokens, anything like that, or any other form of synthetic data that you choose to mention? I think you also mentioned the large world model paper, right?
“You know, use Mistral to rephrase some existing part of your data sets, generate more tokens, anything like that, or any other form of synthetic data that you choose to mention? I think you also mentioned the large world model paper, right?”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
469 / evaluation
One thing I see in the recent papers that have been coming out is this sort of concept of multi-stage training data. And if you're doing full fine tuning, maybe the move or the answer is don't train 500 billion tokens on just code, because then yeah, it's going to massively overfit to just code.
“One thing I see in the recent papers that have been coming out is this sort of concept of multi-stage training data. And if you're doing full fine tuning, maybe the move or the answer is don't train 500 billion tokens on just code, because then yeah, it's going to massively overfit to just code.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
470 / evaluation
Yeah, I think, you know, the one thing that makes this sort of generative AI era very different from the sort of data science-y type era is that it is very non-deterministic and it's hard to control.
“Yeah, I think, you know, the one thing that makes this sort of generative AI era very different from the sort of data science-y type era is that it is very non-deterministic and it's hard to control.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
471 / belief
You know, people always trying to define the difference between ML and AI. And I think in AI, we definitely care a lot more about out of domain generalization and that's all under the umbrella of learning, but it is a very specific kind of learning.
“You know, people always trying to define the difference between ML and AI. And I think in AI, we definitely care a lot more about out of domain generalization and that's all under the umbrella of learning, but it is a very specific kind of learning.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
472 / belief
I think being on Twitter and looking at all these new headlines is really helpful, but then it only gets you a very surface level understanding.
“I think being on Twitter and looking at all these new headlines is really helpful, but then it only gets you a very surface level understanding.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
473 / belief
I think there's definitely a few of us ex-finance people moving into tech and then finding ourselves gravitating towards data and AI.
“I think there's definitely a few of us ex-finance people moving into tech and then finding ourselves gravitating towards data and AI.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
474 / observation
Or is there a mode where people are knowingly playing a game? Because you told me that you make more money when someone believes they're talking directly to the creator.
“Or is there a mode where people are knowingly playing a game? Because you told me that you make more money when someone believes they're talking directly to the creator.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
475 / uncertainty
I don't know what is in the comparison set for you, like human, you know, like, or whatever scale has skill spellbook.
“I don't know what is in the comparison set for you, like human, you know, like, or whatever scale has skill spellbook.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
476 / belief
I screwed up in my intro saying that you're an agency and I realized immediately, I immediately regretted that saying, you're a SaaS tool.
“I screwed up in my intro saying that you're an agency and I realized immediately, I immediately regretted that saying, you're a SaaS tool.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
477 / belief
I think there's also a rise in interest in recording devices where you're effectively recording your entire day and summarizing them.
“I think there's also a rise in interest in recording devices where you're effectively recording your entire day and summarizing them.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
478 / belief
We're pretty sex-positive, I think, but feel free to say what you think you can say.
“We're pretty sex-positive, I think, but feel free to say what you think you can say.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
479 / belief
Perfect Well, thank you so much This is I think you're very naturally incentivized by Growing community and giving your thought and insight to the rest of us.
“Perfect Well, thank you so much This is I think you're very naturally incentivized by Growing community and giving your thought and insight to the rest of us.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
480 / belief
I think you're down in Menlo. And so maybe you're a little bit higher quality of life than the rest of us in SF.
“I think you're down in Menlo. And so maybe you're a little bit higher quality of life than the rest of us in SF.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
481 / prediction
You're very much known for sort of cognitive architectures, and I think, like, a lot of the AI research has been focused on simulating the mind, or simulating consciousness, maybe.
“You're very much known for sort of cognitive architectures, and I think, like, a lot of the AI research has been focused on simulating the mind, or simulating consciousness, maybe.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
482 / evaluation
It's just retrieval. And here it's like, The home of generative AI, this, whatever hyperstition is in my mind, like this is actually pushing the edge of what generative and creativity in AI means.
“It's just retrieval. And here it's like, The home of generative AI, this, whatever hyperstition is in my mind, like this is actually pushing the edge of what generative and creativity in AI means.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
483 / evaluation
When you say you need a certain set of tools for people to sort of invent things from first principles Devin is the agent that I think has been able to utilize its tools very effectively.
“When you say you need a certain set of tools for people to sort of invent things from first principles Devin is the agent that I think has been able to utilize its tools very effectively.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
484 / preference
I like the analogy of white walkers, because they're effectively reanimated from our corpses.
“I like the analogy of white walkers, because they're effectively reanimated from our corpses.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
485 / belief
I would say that most people would be familiar with Instructor from your talks and your tweets and all that.
“I would say that most people would be familiar with Instructor from your talks and your tweets and all that.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
486 / uncertainty
Jason, you are extremely famous, so I don't know what I'm going to do introducing you, but you're one of the Waterloo clan.
“Jason, you are extremely famous, so I don't know what I'm going to do introducing you, but you're one of the Waterloo clan.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
487 / evaluation
I think one of the agents loopholes or one of the things that is a real barrier for agents is LLMs really like to get stuck into a lane.
“I think one of the agents loopholes or one of the things that is a real barrier for agents is LLMs really like to get stuck into a lane.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
488 / belief
In fact, I think you're doing an offsite and we're actually organizing our biggest AI UX meetup around whenever she's in town in San Francisco.
“In fact, I think you're doing an offsite and we're actually organizing our biggest AI UX meetup around whenever she's in town in San Francisco.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
489 / belief
I think you're about to start us on like GPT-3 and how that changed things for you.
“I think you're about to start us on like GPT-3 and how that changed things for you.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
490 / uncertainty
Yep. And then just to recap as well, like the models you were using back then were like, I don't know, would they like BERT type stuff or T5 or I don't know what timeframe we're talking about here.
“Yep. And then just to recap as well, like the models you were using back then were like, I don't know, would they like BERT type stuff or T5 or I don't know what timeframe we're talking about here.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
491 / belief
I think the interesting insight that we got from talking to David Luan, who is CEO of multimodality has effectively two different flavors.
“I think the interesting insight that we got from talking to David Luan, who is CEO of multimodality has effectively two different flavors.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
492 / uncertainty
You know, the fun thing you can do with a credit system, which is data for data, basically you can give people more credits if they give data back to you. I don't know if you've already done that.
“You know, the fun thing you can do with a credit system, which is data for data, basically you can give people more credits if they give data back to you. I don't know if you've already done that.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
493 / evaluation
Because I just didn't find them reliable because they just hallucinated their own uncertainty.
“Because I just didn't find them reliable because they just hallucinated their own uncertainty.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
494 / evaluation
Basically, I think I'm very just impressed by how first principles, your ideas around what the workflow is. And I think that's why you're not as reliant on like the LLM improving, because it's actually just about improving the workflow that you would recommend to people.
“Basically, I think I'm very just impressed by how first principles, your ideas around what the workflow is. And I think that's why you're not as reliant on like the LLM improving, because it's actually just about improving the workflow that you would recommend to people.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
495 / prediction
Now you just match the other two guys. And so that puts An insane amount of pressure on what gpt5 is going to be because it's just going to have like the only option it has now because all the other models are multimodal all the other models are long context all the other models have perfect recall gpt5 has to match everything and do more to to not be a flop
“Now you just match the other two guys. And so that puts An insane amount of pressure on what gpt5 is going to be because it's just going to have like the only option it has now because all the other models are multimodal all the other models are long context all the other models have perfect recall gpt5 has to match everything and do more to to not be a flop”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
496 / belief
I think maybe the only new thing was this Reddit deal with Google for like a 60 million dollar deal just ahead of their IPO, very conveniently turning Reddit into a AI data company.
“I think maybe the only new thing was this Reddit deal with Google for like a 60 million dollar deal just ahead of their IPO, very conveniently turning Reddit into a AI data company.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
497 / recommendation
Likes AI Breakdown.
“We love AI Breakdown as one of the best examples Daily podcasts to keep up on AI news, so we were especially excited to be back on Watch out and take”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
498 / belief
I went and looked at when Meta went from being a VR company to an AI company. And I think the stock I'm trying to look up the details now.
“I went and looked at when Meta went from being a VR company to an AI company. And I think the stock I'm trying to look up the details now.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
499 / belief
We definitely did see some leaks on GPT 4. 5, as I think a lot of people reported and I'm not sure if you covered it.
“We definitely did see some leaks on GPT 4. 5, as I think a lot of people reported and I'm not sure if you covered it.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space
500 / belief
Wars are fought around limited resources. And I think probably the, you know, the most limited resource is talent, but the talent expresses itself in a number of areas.
“Wars are fought around limited resources. And I think probably the, you know, the most limited resource is talent, but the talent expresses itself in a number of areas.”
- Speaker
- Shawn Wang
- Publisher
- Latent Space