Jonathan Becker: evaluation

Transcript context

…Yeah. So I think from a testing point of view, let's say I was running ads on Meta, beneath the campaign level when I create a campaign, the structure that my testing might take would be that I would have an audience, a single audience at the ad set level, and then I would have two nearly identical creatives within that ad set. The only thing being different across those two creative assets is a single variable. So it might be the copy, it might be an image or whatnot. That allows us to isolate a lot of variables and really test into one singular change across two creative assets. There's a lot of nuance to this. So sometimes the ad-serving algorithms, even when we set up a test structurally in that way, we'll serve one ad a different number of impressions than the other ad, in which case we then have to say, what is a leveling factor that allows us to look at these two ads equally, even though one received dramatically more impressions than the other? This becomes where it becomes subjective in terms of how you want to determine success. But a good example is looking at the click-through rate, which is essentially a ratio or a metric like impressions until conversion, which is a leveling metric that allows us to determine, even though in a scenario where two ads facing the same audience within the same campaign received different numbers of impressions, we can still measure the efficiency or effectiveness of one ad over the other using metrics like that. And so from a testing point of view, I think that this is one way that we might look at trying to assess ad performance so that we can gather learnings, send that back to a creative team and say, hey, it turns out that when this copy is used at this stage of the funnel, it converts 50% more frequently than this other copy. So let's now take that copy, use it as our base copy and challenge it with a different type of copy and see if we can continually iterate and refine. So that's a very practical example of how ad creative testing might work on a channel like Meta. You asked for specific examples where we've seen an unlock. There's two that come to mind. Several years ago we realized that highly produced ads from brand teams, and there's nothing wrong with brand teams, we work with them all the time. They do amazing work. I think we're just trying to work as a singular unit as opposed to being fragmented. But a lot of the brand guidelines of different companies would end up yielding these highly polished assets. And when you launch those on Instagram back in the day or something like that, what we found is that they would always underperform next to user generated assets. So a brand that essentially has an influencer in front of it that says, I tried this product, I love it, I'm filming this ad from my iPhone. Look, check it out. Here's the product, here's me, I'm better off for it. Whatever it is that they're talking about. it that says, I tried this product, I love it, I'm filming this ad from my iPhone. Look, check it out. Here's the product, here's me, I'm better off for it. Whatever it is that they're talking about. The unpolished iPhone, mobile phone creative, suddenly we realized massively outperformed these other channels because there was an authenticity to it. And rather than the brands themselves saying, hey, trust us, our product is great, here's a third party essentially validating what is so great about these brands, basically. So I can't speak to specific clients because I'm not allowed to talk about the work that we've done in large part, but that would be one example. And another example is we worked with a furniture company several years ago, which scaled dramatically, but they were having difficulty early on thinking through how to scale social ads. And so paid search worked really well for them. Social ads, again, they had these highly produced styled rooms and one of the owners had their dog in the office all the time, and so they have these models sitting on furniture or whatever it was, and they would take these beautiful styled shots of their furniture in a room and one of the art directors was like, put the dog on the couch and let's take some photos of dogs on furniture, which made it more playful and approachable. That one change resulted in a total unlock of their performance on paid social channels. It would double or triple the ROAS that we were seeing type thing. So it's these minute differences that you can test into that ultimately can drive performance. That reminds me of Airbnb. The photography team found, and actually the photography team was initially including people in the photos of listings, and that was the [inaudible 00:42:16] images of listing photos. And it turns out when there's no people in the listing photo, much higher conversion. And the theory basically people don't want to see other random strangers in a place they're going to sleep, and that was not expected.…

Stored transcript either side of the excerpt. The highlighted words are the published quote; the surrounding text is unedited source, never generated.