fabernovel loader

Sep 5, 2016 | 7 min read


A/B Testing: taking your users' behavior into account

Corentin Huard

Project Manager

Il y a plus de 1,5 million d’applications sur le Play Store et un nombre similaire sur l’App Store. Il est donc nécessaire de comprendre ce qu’attendent vos utilisateurs, de leur offrir un service qu’ils veulent utiliser, et de vous différencier par rapport aux autres acteurs du domaine pour gagner en popularité. La première minute passée sur une application est déterminante pour savoir si l’utilisateur va la garder ou pas.

There are more than 1.5 million applications on the Play Store and almost as much on the App Store. Thus, if you want to earn popularity, it is necessary to make an effort to understand your users’ expectations and to offer them the app they really want to use. The first minute spent on an application is decisive and has a huge influence on whether the user will keep it or not.

Being aware of these points, you decided to modify the design of your app taking good care of your UX in order to increase your conversion rate. You spent hours working on the appearance of this button, thinking about a new title, and finally you were satisfied; your users could do nothing but follow the nice path you had prepared for them through your app. Yet nothing happened as planned. Not only did your conversion rate decrease, but worse, users seemed to flee your application!

The conclusion is simple: you did not manage to answer your users’ needs, and the reason is obvious: it is almost impossible to predict if a solution will be efficient or not without testing it. Unfortunately, stuck on your project details, it is hard to have a broader view and to have your work exposed to the user and his habits, but you should.

Acquiring substantial audience requires a lot of resources, therefore it is essential to know the good moves that will help you win the loyalty of your users and understand their behavior. So, you need to test your new features and the changes you want to set up. A solution exists: A/B Testing.

What is A/B Testing?

As its name suggests, this solution consists in creating and displaying one variation (or several) of the current version of your app and to assign it to a panel of users, which will be your test sample, in order to know if it pleases.

This method allows you to test empirically all the items one can encounter in your app, and it is useful everywhere your user has to perform an action. The version which provides the higher conversion rate “wins”. Your conversion rate is related to the objectives of your application or the feature/functionality you are currently testing:

  • subscription
  • placing items in a cart
  • a simple click

Why should you do A/B Testing?

A/B Testing is an easy way to test quickly several alternatives. When you ask yourself whether a version is better than another one, you cannot choose randomly and it is risky to impose one to your users without knowing their possible reactions. Doing A/B Testing will allow you to try new ideas on a limited panel and to take into account your users’ actions when they face them in order to decide whether to proceed or not with those ideas.

Moreover, this necessity to test new ideas is recurring, since it will always be interesting to challenge what you have once developed. Thus, setting up a tool which will allow you to do this quickly (most of the time without needing to submit your app again) and using it on a regular basis is a terrific asset. Whatever the change you make is, it can always result in a consequent modification of the user’s behavior and so deserves to be tested.

You could choose to develop several variants of your application and decide from your back-end which version you should display to which users. But this process would be heavy and not flexible when an A/B Testing tool would allow you to test various changes, easily.

A/B Testing solutions can also help you to:

  • see how you can raise your conversion rate
  • gather data
  • know your audience better

To finish convincing you, you should know that all the big shots in this field (Facebook, Google, Instagram,…) are using it continuously to improve the experience they offer.

A few rules

Now that you understand better how interesting A/B Testing can be, here are a few rules that are important to follow to use this method and be the most efficient possible.

Rule #1: Have a test sample large enough

If you want a result easily understandable you need to have a consequent enough user base. Also, the panel that will see the variant needs to be representative.

Rule #2: Plan your test

As in every project, you need a good timeline to monitor your tests. Fix the time they should last and the date at which they should begin.

Clarify the goals you want to reach (your conversion rate, increase the use of a feature, a view number,…) and estimate them.

Keep in mind that your main goal is to make progress in the long term and that mutation may not happen immediately (in the mobile world, we are used to seeing things evolve really fast, but it is sometimes necessary to wait in order to observe modifications in your users’ behavior).

Rule#3: Be clear on the items you want to test

Ask yourself the right questions, make assumptions (based on the data you gather), define what will be the items tested and produce all the variants you want to test.

In order to have a better grasp on what your users desire, what they try to do or would like to do within your application, some tools exist that analyse the user experience throughout all his navigation (IBM Tealeaf, UserReplay,…). Or you can simply meet your users, interview and observe them.

Rule#4: Define your users panel

You will need to define well the target you will aim at with your variants (for example tablet users that live in France). You should also know well your analytics in order to have a good view on the current situation and to compare it with its alternative. Most of the A/B Testing solutions provide you with these data.

Rule#5: Start simple

Start by testing simple stuff in your application (a color, a title) and test only one thing at a time (you can lead several tests abreast but each one of them must concern only one variable). It will help you understand which change really makes a difference.

Test the items that truly make sense (those that can change everything, not a vague hidden paragraph).

Rule#6: Choose the version you should keep:

Most of the time, A/B Testing tools provide you with analytics and statistics you can use to decide which variant you should keep. The statistical confidence index is a good indicator to help you in this choice.

Items to test

You can truly test everything within your app with A/B Testing. The most interesting items are basically those which leads (directly or not) the user into performing an action. Here is a non-exhaustive list of what you can test:

  • titles and subtitles
  • paragraphs
  • catch phrases
  • call to action texts
  • buttons
  • images
  • navigation
  • push
  • payment gateways (which provider)
  • pricing
  • store description

A/B Testing tools

Name User-Friendly Easy set up Dedicated dashboard Visual Editor Free Website
Taplytics yes yes yes yes no https://taplytics.com/
Apptimize yes yes yes yes no https://apptimize.com/
Leanplum yes yes yes yes no https://leanplum.com/
Optimizely yes yes yes yes no https://optimizely.com/mobile/
Mixpanel yes yes yes yes no https://mixpanel.com/mobile-ab-testing
Azetone yes yes yes yes no https://www.azetone.com/
Splitforce medium medium yes no no https://splitforce.com/
Kii medium medium yes no no https://en.kii.com/platform/features/ab-testing/
Clutch.io no medium no no yes http://clutchio.github.io/ab-testing.html
Switchboards no no no no yes https://github.com/KeepSafe/Switchboard
Abba no no no no yes https://github.com/maccman/abba

And some others like AB Tasty, VWO, Fliptest, Vanity, Tune, SWRVE that we have not tested yet.

How can you choose the right tool?

As you may have noticed, there are multiple tools that offer various functions and characteristics. In order to help you in your choice think about the features you need and ask yourself the question of the budget you are ready to allocate to this solution. Return on investment can be massive especially when you compare it with that of some traffic acquisition solutions.

You can use these criterias to take a decision:

  • is it compatible with all the platforms you use?
  • is it easy to set up (installation, definition of variables,…)?
  • is it quick (do you need to submit your application again)?
  • does it come with a dedicated interface to visualize the ongoing tests or to create new ones?
  • does it come with a visual editor (which allow you to make quick changes, without writing code and without having scheduled in advance which item should vary)?
  • is it possible to use segmentation?
  • does it offer additional features (analytics, push, heatmaps,…)?
  • who will use it in your team (product manager, marketing, developer)?
  • does it allow collaborative work?
  • does the development team ensure assistance?

Some of the solutions presented in the previous table (most of the free solutions) are less user friendly than others, nevertheless they have the advantage of being open source and fully customizable, and they can be used as a base on which you can build your personal A/B Testing tool.


Today, in order to distinguish yourself in the mobile application world you have no choice but making efforts to understand well your client’s experience, to take good care of your navigation and to offer to your user what they came for. For this purpose a tool such as A/B Testing can be a real advantage. It can become almost mandatory if you wish to improve continuously the service you propose and increase your conversion rate.

Applidium can help you to implement one of the solution presented above into your mobile application project or even help you to develop your own tool.

logo business unit


150 talents to face technological challenges of digital transformation

next read