Andrew Lee: evaluation

Transcript context

…Brave Search API gives AI agents a fast, independent search index for research, RAG pipelines, images, places, and fewer hallucinations. Get $5 in free credits at https://brave.com/search/api/?mtm_campaign=q2-26-cognitive-revolution Sequence handles the full revenue workflow for complex pricing, from quoting and metering to invoicing, revenue recognition, and collections. Book a public demo at https://sequencehq.com and use code COGNISM in the source field to save 20% off year one Yeah, I think the overall, you know, approach of always better than models, I think that is, I totally agree. That is like totally held up. When we started working on Tasket, we were using four, Claude four. And that was able, that actually could get you a long ways, and it worked pretty well. 4.5 was a big unlock. So 4.5 started to work much better to do doing computer use, and it could just like manage sort of navigating through the various different connections and tools enablement process and stuff much better. And that, you know, came out very early in the lifetime of Task that was like, I think, a big bonus for us. And the cost reductions around Opus were huge for us. in, I want to say, December when that happened, they dropped. So initially, we could really only have people on Sonnet, and then Opus dropped from 15 to 5, and that was a big unlock for us as well. I think 4.6 was a solid incremental change. It made it nicer for, again, for computer use, which has become increasingly important for us, both headless and headful, as well as CodeGen, and that enabled our Instant Apps feature, which I think we'll probably talk about today, which is a very cool feature. 4.7, we actually haven't rolled out yet. I think 4.7 has been much better in certain areas. It's much better in code and one-shotting long projects. But for the types of... iterative knowledge work that we support, it doesn't actually seem like a huge boost and it's a lot more expensive. The tokenizer changes they made increased our costs for like 30% and costs are huge for us because we essentially pass them on to users. We actually opted not to ship 4.7 as a default recommended model. We are going to ship it, but it's an advanced user option if people want to and we're going to note that, hey, this actually costs a lot more. So I think the progress there has been great. The reason we started on Anthropic and I've been so Anthropic focused for so long was just that the basic core of our agent, like the ability for it to navigate through a discovery process of connections and activate the right tool in your agents and then manage its context the way we manage its context, just the core inner workings requires kind of a base level of intelligence that the other models just couldn't do. You basically couldn't use the same harness with them and have it expected to work well. That has changed, which is really exciting for us. It's kind of scary to be like, we're totally dependent on this one vendor, even though they're great. Drop down models are amazing, don't get me wrong. Supply chain risks are everywhere if you take that approach.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.