Ethan Smith: evaluation

Transcript context

…A couple more questions I want to get through and then I'll let you go. One is, I've noticed this proliferation of startups that are using GPT-3 basically to generate pages and to win SEO through just autogenerated AI driven pages. Do you think this is an effective strategy for a startup? Do you recommend it? Do you think it's long term, going to last? How do you think about AI and SEO? I mostly don't like it and don't recommend it, but there are exceptions. So a couple things on this. The first thing is the content should be good and it should be useful. And we've played around with GPT-3. So GPT-3 is basically generating sentences that are similar to other sentences that have been written by somebody else. So GPT-3 takes common crawl and very large data sets. So a large snapshot of the whole web, so tons of sentences, and then it trains a model to generate new sentences that are similar to other sentences that have been written. So they're grammatically correct, they're spelled correctly. Sometimes they sound like they are written by a person, sometimes they're not. But a lot of times they are. A lot of times it's indistinguishable from a human. The problem is that you want the sentence to be factual. You want there to be underlying wisdom in the sentence. So I'll give you an example. We were playing around GPT-3 to generate a description of a product, of a soap. And GPT-3 said the soap was oxide free, but it was not oxide free. But GPT-3 doesn't know that. And some other thing was oxide free on the web. And so it just said it was oxide free. So it made this false claim. But if you're a user, you would never know that. It was written well, it was spelled correctly, it was grammatically correct and it was just factually inaccurate. And so to have content be useful and to use AI in a useful way, there needs to be underlying wisdom that the AI is communicating. So if you have structured data, then it can be useful. So let's say that you're writing an article about a basketball game. If you have structured data about each thing that happened in the basketball game, somebody scored this point at this particular time, and then this happens, something else. You generate sentences that are factual, they're based on actual information or knowledge, and the sentences are therefore useful. With Power is actually an interesting example. So they're extracting structure from clinical trial information and generating sentence based on that structured data. So in that sense it's useful, but most AI generated content is just sentences that are similar to other sentences that have been written. And so in general, I recommend against AI. An example would be, how should I save for my retirement? If I have an article about that and it's just not truthful and I'm making decisions about my retirement based on this article, the GPT-3 wrote, that's really bad. Or if I have cancer and I'm researching cancer options and this AI content suggests some treatment that is not a good treatment, that's a big problem. And so I think we want to be really careful about how we use AI for content generation. Where AI is really useful is actually all the other stuff that I talked about. So if you think about the workflow of creating content, you start with what should I be writing about? What are the subtopics? AI is really useful is actually all the other stuff that I talked about. So if you think about the workflow of creating content, you start with what should I be writing about? What are the subtopics? So like 300 different keywords, what are the sub themes? What's that outline? How is it performing? All of these things can be done really well with AI. So AI can extract structure, they can help me decide which topic to write by going through these thousands of keywords and clustering them and looking for overlaps and things like that. It could tell me what my topical authority is. It could sort them. It could go through the keywords and find these subtopic themes. It could do all of that really well. And then a human can write a piece of content. So given all that information, human can write a piece of content, they're a domain expert, they have wisdom, and they can write an article based on that structure. That's a great application of AI. What's not a great application of AI is writing an article suggesting cancer treatments based on no underlying wisdom.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.