Paywall AB Testing for Optimizing In-App Revenue

Paywall AB testing is a necessary strategy for any app publisher wanting to optimize revenue. In this article we’ll provide an introduction to paywall testing by covering the following topics:

• The paywall as an engine for growth & retention
• Why you should be testing your paywall
• How to conduct a paywall AB test

The Paywall as an Engine for Growth & Retention

The paywall screen is how apps make money through IAPs and subscriptions. In fact, most app publishers spend very little time updating their paywall let alone optimizing it.

The paywall can be one of the essential elements contributing to your success in growth and retention this year. Let’s take look:

The Paywall and Growth

Focusing on app growth usually means a combination of efforts including:

• Converting existing audience to your mobile app via tactics such as smart app banners and email marketing
• Improving discoverability via keyword tuning and other App Store Optimization (ASO) techniques
• Paid acquisition through channels such as Apple Search Ads or other cost-per-install placements

It’s important that new users see your paywall during their first session. In fact, 70-80% of subscription starts occur on the day of install (D0) according to data from AppsFlyer and Redbox Mobile.

Considering this, it’s important to think of your paywall as not one monolithic object that all users flow through regardless of origin. That would be like having a single landing page on your web site regardless of traffic source.

Conceptually, you want different paywall instances aligned to each acquisition source:

• Organic Discovery -> App Install -> Paywall A
• Email Campaign -> App Install -> Paywall B
• Apple Search Ad -> App Install -> Paywall C

To optimize in-app revenue, your paywall needs to be operated like a growth marketing asset, not a static coded screen.

The Paywall and Retention

Most app publishers use push notifications for app engagement. Push can be a helpful way to improve D7 retention, but you should also consider what happens once the user is in the app.

It may not be obvious, but the paywall isn’t just important for growth. It’s important for retention as well especially for subscription apps. Consider that most app subscriptions are auto-renewable. The desired state for app publishers is for a user to be a paying subscriber on an annual plan with auto-renew turned on.

However, subscription customers can end up in other states as well. The paywall can help you message users in context of those other states to nudge them to that desired place.

Here are a few example of retention paywall instances tied to user context:

• Monthly Plan Subscriber -> Upgrade Candidate -> Annual Offer Paywall
• Active sub w/ Auto-Renew off -> Voluntary Churn Risk -> Stay Offer Paywall
• User is a Former Sub -> Winback Opportunity -> Winback Offer Paywall

The key to effective retention paywalls is to power your experience with individual-level subscriber data. This means your external marketing channels, the in-app experience, and everything in-between should all have the same context about a user’s subscription journey. This way, the experience including marketing assets will be personalized and relevant.

Why you should conduct paywall AB testing

Yes, you should be conducting paywall AB testing. The exception to the rule is if your app doesn’t have much usage. If this is you, focus on top-of-the-funnel activities such as ASO keyword tuning.

If you do have steady app traffic, you almost certainly will be able to improve in-app revenue via one or more of the following:

• Generating more one time purchases
• Improving subscription trial starts
• Improving free-to-paid conversion
• Reducing subscription churn
• Boosting customer LTV (lifetime value)

Testing a Single Paywall

If your app has only one paywall, you can benefit from paywall AB testing. Most apps convert at low enough rates that much upside exists. Here are a few guidelines:

Sadly, it’s not uncommon to see freemium app with free-to-paid conversion rates of less than 1%. With testing, you can boost conversion to 5-15% or more.

For apps with a hard paywall, conversion rates depend on whether you offer a free trial or not. It’s not uncommon to see trial conversions of 5% or more. With testing, you can boost free trial conversion rates to 30-50% or more.

No matter what your conversion rate is, your paywall can be improved. The only question is by how much.

Just realize that the one-size-fits-all paywall won’t be perfectly aligned with each and every user. For fully optimized revenue, you’ll need to move to multiple, segmented paywalls.

Testing with Multiple, Segmented Paywalls

As we’ve explained, to optimize in-app revenue it’s important to have different paywalls focused on different acquisition channels and user context. To say this another way, you should have a paywall segmentation strategy.

With paywall segmentation, you and an opportunity to conduct paywall AB testing that is focused on improving the specific metric that a certain paywall is responsible for.

For instance, you could run a test of the paywall presented to users acquired via Apple Search Ads to tightly focus on improvement to that channel. For instance, you could focus on improving your ROAS (Return on Advertiser Spend - defined as the total revenue generated divided by total ad spend).

Similarly, you can test variants of your Annual Offer to see which does a better job of converting existing monthly subs to your annual plan.

To prioritize your testing strategy, focus on the key metrics you’re trying to improve first. Also look for the opportunities that have a lot of room for improvement so you can get some quick wins.

How to Conduct Paywall AB Testing

If you’re convinced your app is a good candidate for paywall AB testing, let’s take a look at how you actually run a test. We’ll cover the three key elements to help you get started with your first test:

• Choosing an AB testing tool
• Selecting variants & allocating traffic
• Determining the winning paywall

Choosing an AB Testing Tool

There are different tools and techniques for running an AB test. For the purpose of this article, which pertains to mobile app paywall AB testing, you have the following options:

1. Use a tool designed for mobile paywall testing (e.g. Nami)
2. Use a mobile backend to feed different data to your app (e.g. Firebase)
3. Adapt a general purpose testing tool (e.g. Optimizely)

There are pros and cons to each of these approaches. If you’re already using a general purpose tool, something like Optimizely might make sense. The downside, is it’s not particularly mobile-first and certainly wasn’t built to focus on testing paywalls.

A solution like Firebase is closer to mobile-first, but is not as friendly for the marketing folks on your team that are used to great campaign management systems.

In our humble opinion, Nami is the easiest way to ship a mobile paywall and yes, conduct paywall AB tests. In fact, no code is needed to design the paywall or run the test. Our powerful campaign engine helps you with paywall segmentation so can focus the right test on the right outcome.

Determining which paywall elements to test

Now that you’ve chosen a tool, it’s time to figure out what you want to test. As we’ve discussed you may have multiple paywalls each playing a different role in your growth and retention strategy.

This means the elements you will want to test really depends on the paywall and it’s purpose. Here are just a few things you might test based upon common mobile app paywall designs:

• Free trial duration on subscription plans
• Impact of including a lifetime in-app purchase
• Marketing copy for describing app’s benefits
• Call-to-action button text
• Which plans and how many to choose from
• Paywall layout and design asthetics

Remember, the element you want to change needs to align with the metric you want to influence. For example, if you want to drive more users into choosing the annual plan the possible elements to test may narrow to:

• Providing a more attract free trial
• Including a featured product badge to call out the annual plan
• Paywall layout and design asthetics, as relating to plans

Your test can include multiple changes to the paywall or a single change. There are pros and cons to each. Multiple changes may help you make progress faster while a single change may help you gain better intuition from each test.

Selecting Variants & Allocating Traffic

Now that you’ve chosen what you want to test, it’s time to create your variant. If you're using a no code tool like Nami, it’s easy to create the variant for paywall ab testing.

As a best practice, variant A should be the paywall that is already active in your app so you have a baseline to start with. Variant B contains the changes you want to test.

Next, you need to decide how to allocate traffic to to your variants. It could be as simple as a 50%/50% split or 80%/20% split. The specific granularity of traffic allocation will depend on the tool you’re using. Nami allows any whole number so long as the total equals 100%.

Some considerations when deciding how to allocate traffic:

• If you have a data baseline with variant A, you may want to push more traffic to variant B to get an answer to your test more quickly.
• If you are at scale you may be able to determine a winner with less traffic going to variant B. That’s great, since if it performs poorly you’ll be glad you didn’t send more traffic than necessary to find that out.

Determining the Winner

Your test is running and results are coming in. You’re probably eager to find out the winner. However, just because Paywall B is converting better, doesn’t mean it’s the winner. It might be, but to be sure you need to reach statistical significance.

A statistical significant result is one that has not occurred by random chance. This is why it’s important not to look at those early results which may appear like a conclusion when in fact there’s not yet enough yet to reach such a conclusion.

You might be wondering how long the test will take to reach statistical significance. The answer is somewhat frustrating: it depends. It depends how much traffic each variant is receiving as well as how many conversions you’ve received against that traffic.

You’re looking to exceed a 90% or higher confidence level before taking comfort that the results of the test are known. If you’re interested in the statistics, check out this guide.

Amongst all this statistics talk, there is good news! Most AB testing tools will do the math for you and crown a winner once statistical significance is reached with high confidence.

A Quick Note on Multi-Variate Testing (MVT)

In this article we’ve talked all about paywall AB testing. The AB naturally means we’re testing one thing against another. This limits the test to just two paywalls.

Marketers who have performed testing in other contexts will likely be familiar with multi-variant testing (MVT). While it may sound appealing to test more things at once (in this case, more paywall variations), it’s not for everyone.

MVT requires much more traffic to deliver a statistically significant result. Given how anemic most mobile app conversion rates are, it’s our recommendation that you stick to AB testing even with a lot of traffic.

There’s so much opportunity to improve without the complexity of MVT. Once you’re chasing single percentage points of optimization, you can consider introducing MVT into your testing arsenal.

Final Thoughts

Paywall AB testing is an essential for any app publisher wanting to optimize in-app revenue. In fact, paywall AB testing can boost conversion rates by 2-3X or more through straightforward changes.

At Nami, we’re focused on helping app publishers thrive in the app economy. We’re doing this by turning the paywall into a full-fledged marketing asset. Paywall AB testing is just one of the capabilities available as part of our solution. If you’re interested in getting a demo or learning more, we’re happy to help.