Simon Willison: commitment

Transcript context

…so it has movie pie, but also Pi ffm, so ffm Python bindings for sure. Okay. Might be using those instead of calling out, shelling out to a process. In that case, So I want to talk about the data analysis thing because it is so good at it. It is so good. And that, that actually gave me a little bit of an existential crisis a few weeks ago. Well, well because, so my day job, my, my, my principal project, I ru, I run this open source project called Dataset, which is all about building tools to help people interrogate their data. And it's built on top of SQLite a web application. It was, it's originally targeted data journalism to help journalists find stories and data. And I started messing around with code interpreter and it did everything on my roadmap for the next two years, just out of the box, which was both extremely exciting as a journalist and kind kind of like, wow, okay, so what's my software for? If this thing does it all already. So I've had to dramatically like, pivot the work that I'm doing to say, okay, well dataset plus large language models needs to be better than code interpreter. Cuz dataset without large large language models, code interpreter basically does everything already, which is, you know, it was an interesting moment. models needs to be better than code interpreter. Cuz dataset without large large language models, code interpreter basically does everything already, which is, you know, it was an interesting moment. But yeah, so the project that I tried this on was a, a few months ago there was this story where a Whole Foods in San Francisco shut down because there were so many like police reports and, and, and calls about, about, about crime and all of that kinda stuff. Yeah, yeah, yeah. So I was reading those stories and they were saying it had a thousand calls from this Whole Foods in a year and a half and thinking, yeah, but supermarkets have crime is a thousand calls in a year and a half, actually notable or not. And so I thought, okay, you know, I'll try out this code Decept thing and see if I can get an answer. I found this CSV file of every call to the police in San San Francisco from 2018 to today. So I think it was 250,000 phone calls that had been logged. And each one says, well, the location it came from and the category of the report and all of that kind and, and when it happened. And so I tried to upload that to code interpreter and it said No cause it's too big. So I zipped it and uploaded the zip file and it just kicked straight into action. It said, okay, I understand this's a CSV file of these incident reports. These are the columns, that kind of stuff. And so then I said, okay, well there's the, the, the location I care about is this latitude and longitude. I figured out latitude and longitude of this Whole Foods. And then I picked another supermarket of a similar size that was like, Ha a mile and half away and got its latitude and longitude. And I said to it, and this is all just English typing. I said, figure out the number of calls within 500 meters of this point, and then, and then compare them with the number of calls within 500 meters of this other point. And do me a plot over time. I literally just said, do me a plot. Over time it didn't say what kind of plot, and that was enough. It was like, okay, well if I'm gonna do everything within the distance, I need to use the have assigned formula for latitude, longitude, distances. So I'll define a Python function that does have assigned distance calculations. And then I'll use that to filter the data in this 250,000 rows down to just the ones within 500 meters at this point, at this point. And then I'll look at those per month, calculate those numbers and plot those on the comparative chart. So it gave me a chart with a line for the Safeway that was the, the Safeway, and a line for the Whole Foods comparing the two in one place. And this was after, I think I uploaded the file and I typed in a single prompt and it did everything based off of that. I watched it, it churned away, it tried different things. It, and it outputs this chart. And the chart answered my question, right? The answer is yes. This Whole Foods was getting a lot more calls than the equivalent size Safeway a couple of miles away, so, so the reporting that that, you know, a thousand calls in a year and a half is not normal for a supermarket, but oh my God. And then on top of all of that, at the end, I said, you know what?…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.