MetricWorks launches Polaris incremental measurement to rescue mobile ads

When iOS 14 arrives soon, it will drop mobile game and app advertisers into the darkness. The loss of the data associated with the Identifier for Advertisers (IDFA) could cripple targeted advertising, as Apple moves to elevate user privacy on its mobile devices.

And when the lights go dark, MetricWorks hopes to have a flashlight that restores marketing visibility to mobile gaming and app advertisers in a way that complies with Apple’s privacy requirements. Singular recently reported that 68.5% of users say they will opt-out of IDFA targeting in a recent poll. And with Apple’s new rules, granular “last touch” measurement, which tries to measure the last thing that causes people to take an action in a game or app, will no longer be viable.

This is where MetricWorks hopes to shine, as it can provide all of the lost granular data using a concept dubbed “incrementality” without the use of IDFA data. MetricWorks believes that being a drop-in replacement for “last touch data” should not be the only reason to move to an incremental measurement platform.

CEO Brian Krebs said in an interview with GamesBeat that incrementality experiments will provide the answer.

“An even stronger reason to move away from last touch is it does not represent marketing effectiveness,” Krebs said. “We’ve proven through nine months of back-testing and experimentation with the largest mobile advertisers in the world, that ‘last touch’ measurement is more or less randomly assigning installs and value. It does not care about the truth.”

Above: Mobile marketers are confused by Apple’s IDFA change, based on a November survey by AppsFlyer.

Image Credit: AppsFlyer

MetricWorks believes that launching Polaris will bring a new top-down incremental measurement platform that values the entire user-journey to conversion, or actions such as downloading an app. This solution does not depend on bottom-up “last touch” data or device IDs (IDFA or Google’s GAID) in any way. It uses aggregated daily data and blends two techniques, econometrics and automated experimentation, to deliver truth in measurement. Together, these techniques make it easy for marketers to realize marketing effectiveness across their campaigns, regardless of IDFA retirement, Krebs said.

MetricWorks Polaris helps restore the capability to test how well a marketing campaign is performing. It makes incremental measurement real and accessible to mobile marketers with the same KPIs they use today for campaign optimization. The turnkey SaaS solution also plugs directly into the marketer’s BI systems through APIs similar to those of their measurement partners today.

“There’s a lot of doom and gloom. So we’re trying to step away from that,” said Krebs. “It’s important to highlight the potential of new approaches and to rethink, not just measurement, but the ecosystem and how the different pieces work together in the post-IDFA world.  Whatever tools are created need to be easy to adopt and succeed with daily.  I believe we’ve delivered that in spades.”

Why this matters

Above: Apple will require opt-in from the user for data tracking.

Image Credit: Apple

Any mobile gaming studio with limited resources will need to spend a lot of time optimizing SKAdNetwork. But MetricWorks believes that SKAdNetwork is like a feeble candle in the dark. And it carries some risk of deluding unaware marketers. MetricWorks has a powerful flashlight in Polaris, which requires miniscule client effort.

“We’re trying to do some pretty bleeding edge things,” said MetricWorks’ Chris Hoyt in an interview with GamesBeat. “With iOS 14, and post-IDFA, our system really becomes the only access to granular level measurement people are going to have. Being able to provide daily granular incremental measurement data for mobile advertisers, and all the KPIs they’re accustomed to like revenue and retention, is what we do.”

MetricWorks said that Polaris emerged from an extensive and successful beta with mobile gaming leaders. Its key innovations include:

• A an easy process with low-client effort
• Proprietary granular daily models
• Daily incrementality outputs
• Higher accuracy through proprietary hybrid models (econometrics and ground truth experiments)
• A translation layer that extracts existing key performance indicators (KPIs) from the econometric models for easy use

“What it does is automate the design and execution of the ground truth experiments to test your traffic and understand the incrementality,” Krebs said. “We firmly believe that incrementality is the way to go. It’s just difficult for most mobile gaming companies to do on their own and that’s the problem MetricWorks solves.”

Those are big words we’ve used so far, and we know from an AppsFlyer survey that many brand advertisers haven’t fully grasped the significance of the IDFA change, which could cost the advertising, game, and app industries billions of dollars. But if you want to read through this story and try to understand what Krebs is trying to say, then it might just save your company a bundle of money.

We’ll explain that “last touch” more in a second.

Apple is offering to replace the precise real-time IDFA user data with the SKAdNetwork, a framework for measuring mobile ads with obfuscations that will protect user privacy and deliver results 24 hours after an ad runs. It takes away the function of measuring the effect of ads from third-party measurement firms.

Apple’s SKAd Network

SKAd Network measures the results of advertising, but not in real time. It provides a report that doesn’t have user identifiers. When someone clicks on an ad and opens the store, the network will say which ad network, publisher, and campaign number led to the click. It will be sent after 24 hours, and it will not have device or IDFA information.

Any mobile gaming studio that has limited resources will have to spend a lot of time optimizing SKAdNetwork. But MetricWorks believes that SKAd Network is like a feeble candle in the dark. And it carries some risk of deluding unaware marketers. MetricWorks Polaris can address this.

“What it does is automates the design and execution of the pseudo experiments to test your traffic and understand the incrementality,” Krebs said. “We feel that it’s not only a timely thing to be looking at, with this big inflection point that the deprecation of IDFA is causing throughout the industry. But quite honestly, we feel the timeframe mentality has long been here. We believe incrementality is the way to go. It’s just difficult.”

But here’s where we need to back up and explain incrementality. Bear with us, as this is going to be an elaborate but hopefully clear explanation of a scientific way of running experiments with marketing campaigns.

Incrementality explained

Above: Marketing depends on incrementality measurements.

Image Credit: MetricWorks

Marketing is only truly effective when it is incremental. Incrementality is the lift, in terms of any goal, you get from an ad that is above all other media spending plus organic demand. If certain media spend is cannibalizing organic lift or overlapping with other media, the true value of a particular campaign is murky, as you don’t see what is complicating the picture when it comes to ad spending.

With something like SKAd Network, it’s easy to fall into the trap of relying on the “last touch,” or the last ad viewed in a given time period before a user takes an action like downloading a game, Krebs said. It’s easy to assume that the ad was responsible for the action. But that’s not necessarily true if you’re not measuring things correctly.

“The only way that you can understand your marketing effectiveness, which is the whole purpose of measurement in the first place, is understanding your incrementality,” Krebs said. “The question becomes, ‘What if you completely stopped all your spend on one of your media sources tomorrow?’ And the answer you often hear is, ‘My overall daily installs across my game road app would decrease by the number of installs currently being attributed to that media source.’”

He added, “The only way to actually measure ground truth or empirical incrementality is through a well-designed empirical experiment. And that’s exactly what we provide, as a core offering along with econometric models to fill in the gaps.”

How incrementality experiments work

Above: Last Touch has poor incrementality results.

Image Credit: MetricWorks

The key to a marketing experiment is that you need a control. If you have two equivalent markets, and you stop spending on ads in one market, you can measure the incremental effect by looking at the difference in downloads between the control market and the one where you stopped spending.

“The concept is about a system that can actually identify truth through the experimentation,” Krebs said. “And then from that truth, derive metrics for you to make decisions that are grounded in truth. The reality is that the best measurement tools that are available today with ‘last touch’ are almost never representing truth.”

As an example to explain how incrementality experiments work. Imagine you have a campaign running in Germany along with several other countries. “Last touch” attribution is assigning about $1,000 in day 7 revenue per day to Germany for that campaign. That is, the last ad measured in Germany gets the credit for creating about $1,000 in revenue seven days into a campaign.

Above: MetricWorks demos how to do ad experiments the right way.

Image Credit: MetricWorks

The ideal way to measure incrementality would be to pause Germany on that campaign and compare the decrease in German Game World day 7 revenue with a parallel universe in which there is no spending pause. That parallel universe is called a control. Since we only have one universe, we need to create a clone or control of Germany. How do we do this?

By using a model to combine many other countries into a weighted average, MetricWorks creates a synthetic control that closely matches Germany (see diagram above). The synthetic control, which is the weighted average of Game World day 7 revenue in France, United Kingdom, and Italy, matches quite well with Germany for an extended period of time. We can trust it to approximate the parallel universe in which the pause never happened.

Since you can only observe what actually happened in Germany after the pause, you need a synthetic control to understand what would’ve happened in Germany if you hadn’t paused.

If you pause the spending in Germany but  keep it going in the other three countries, you’ll see the difference in the loss of installs. The difference between Germany and the synthetic control after the pause is actually $400 per day instead of $1,000. So that’s the true incremental day 7 revenue. Based on the average daily media spend of $4,000, you may have thought that day 7 return on ad spend was 25% whenit was actually only 10%. The day 7 returns are $600 per day less than expected. That’s over $200,000 per year.

“If incrementality is saying that a certain media source is good, performance wise, but ‘last touch’ is saying that it’s performing poorly, you’re not going to be spending there,” Krebs said. “There’s a lot of untapped opportunities that you’re just completely going to waste in terms of opportunity costs. On the other side, if incrementality is saying that this media is just performing very poorly but ‘last touch’ is saying that it is performing well, you’re spending your budget very ineffectively.”

Econometrics to the rescue

Above: Econometrics can help you understand incrementality.

Image Credit: MetricWorks

That’s a hard-earned truth. Mobile advertisers can only run so many experiments since there is an opportunity cost to running any experiment. The key is generating experiments with the highest information gain with the lowest opportunity cost. This is what MetricWorks Polaris does automatically for marketers. While experiments provide incrementality ground truth, the scope is limited to the traffic tested in the experiment. Marketers can use another technique called econometrics (built into Polaris) to get a more comprehensive understanding of incrementality across all traffic.

Econometrics relies on quantifying historical correlations. Just because Game World revenue consistently increases on days when you spend more on a certain media source doesn’t necessarily mean the uptick was caused by that media source. It could be a coincidence.

“There’s some uncertainty and we actually quantify that uncertainty for you. It’s not a black box model,” Krebs said. “But at least you know, that in these areas of traffic that you’ve already tested, it’s absolute ground truth incrementality. And that has reverberating effects throughout the rest of the econometric model actually in the traffic that you didn’t even test.”

Combining incrementality and econometrics

Above: After IDFA, multiple measurement techniques won’t work.

Image Credit: MetricWorks

“We’re going to try to beat the launch of iOS 14.5, if at all possible. So people know that a solution is available to them,” Krebs said. “Companies are going to be freaking out right now trying to figure out what they’re going to do. We want to give them hope via Polaris.”

MetricWorks automates every step from designing experiments and calculating ground truth to training econometric models and deriving results. Incrementality is delivered in the form of the KPIs you rely on today including return on ad spend so marketing decision-making processes don’t need to change.

“It’s the only product of its kind. The standout really is in the application of techniques that aren’t normally applied to mobile marketing,” Krebs said. “We’re trying to get the world to look at measurement completely differently than in the past. What we’re doing through econometrics, and through quasi experimentation using synthetic controls, is offering people a fresh look at measurement. More importantly, we get to understand marketing effectiveness. The only way to do that is through incrementality.”

He added, “The alternative to this would be that only a handful of large gaming companies would be able to afford to build something like this. So with Polaris, we’re offering this low effort, ready-to-use approach to the broader market. It very much is taking it from academics to real-world implementation.”