Public evidence record
Published podcast speaker
Claim ledger
21 transcript-backed records
01 / belief
“I think, you know, LLMs aren't perfect. And I think there's a lot of discussion around the pitfalls of using LLMs to evaluate themselves.”
02 / belief
“I mean, I think buried in ModelIO is some stuff around like few-shot example selectors that I think is really powerful.”
03 / belief
“I think JSONformer and stuff like that are still really interesting for like local models, for sure.”
04 / prediction
“I expect that it will probably get better over time as everything in this field.”
05 / belief
“Cause I think like the best thing is you stumble upon a really good idea and you build something really awesome.”
06 / belief
“Again, like I think the main thing that even I find valuable about Langsmith is just like the debugging aspect of it.”
07 / belief
“I was, yeah, I was a bit surprised to see that as well, but I think there's generally a lot of interest in agents and it's also really hard to get them to work.”
08 / belief
“Like, I don't think, like I think like there's, which I know is very hypocritical to say.”
09 / belief
“I think there's also like, you know, like if there is, if there's just more options available, like prices are going to go down.”
10 / belief
“Like one, like, you know, if we think about what it takes to build a really reliable, like context-aware reasoning application, there's probably a bunch of different nodes that are doing a bunch of different things.”
11 / belief
“I think a few places where we've started to see this be like one of the main things is the AI simulation that came out.”
12 / belief
“I think another example of this, I mean, Langsmith, which we can maybe talk about was like very kind of like, I think we worked on that for like three or four months before announcing it kind of like publicly, two months maybe before giving it to kind of like anyone in beta.”
13 / belief
“Yeah, we'll probably go. And yeah, we'll do more of that because I think that's definitely part of the application of a chain or agent is you start with a default one, then you improve it over time.”
14 / belief
“I was looking it up for the podcast and the first tweet was on, I think October 24th.”
15 / evaluation
“I think maybe orchestration is a little bit more than 5%, but I like agree that those are like really big pain points that get exacerbated when you have these complex chains and agents where you can't really see what's going on inside of them.”
16 / evaluation
“Yeah, I mean, I think that's, you know, you talk about like, again, this whole space is just changing so fast, but you talk about something that could like really change how, because like, you know, a lot of lang chain is kind of like a data orchestration tool in some sense.”
17 / recommendation
“To reduce hallucinations, we did a webinar on like evaluating RAG this past week. And I think there's this great project called RAGOS that evaluates four different things across two different spectrums.”
18 / recommendation
“You probably start with like a zero-shot prompt. But I think that's a really powerful one that's probably just talked about less because you don't need it right off the bat.”
19 / evaluation
“I definitely think probably the main ones, like basically the LLM. So the reason I think the debugging in Lancsmith and debugging in general is so needed for these LLM apps is that if you're building, like, again, let's think about like what we want people to build in with LangChain.”
20 / preference
“We still think there's, it's just really application specific. So we prioritized instead, making it easy for people to write custom evaluators and then run them client side and then upload the results so that they can manually inspect them because I think manual inspection is still a pretty big part of evaluation for better or worse.”
21 / evaluation
“I don't know if those are like underappreciated though, because I think a lot of people talk about text splitting as being a hard part, and it is a really important part of creating these retrieval applications.”