A/B Testing for Apps and Websites: What It Is and How to Run One
A/B testing for apps and websites: what it is, when it works, how many visitors you need, and how to run a test that leads to a clear decision.
In this article
A/B testing means showing two versions of the same page, screen or flow to different groups of real users at the same time, then letting one metric decide which version performs better. For apps and websites, it is the most direct way to settle a design debate with evidence instead of opinion.
See our concept and A/B testing service.
What A/B Testing Is, in Plain Terms
Version A is usually what you have now. Version B changes one thing: a headline, a checkout step, the order of a sign-up form, or a whole redesigned page. Traffic is split at random, so both groups are alike, and you compare the result on the metric you chose before the test started.
A/B testing tells you which version wins. It does not tell you why. That is why good teams pair it with usability testing or interviews: the test measures behaviour, the research explains it.
What You Can A/B Test on an App or Website
| What to test | Example | Metric to watch |
|---|---|---|
| Copy and headlines | Benefit-led headline vs feature list | Click-through to the next step |
| Forms and sign-up | 5 fields vs 3 fields plus later profile completion | Completed sign-ups |
| Checkout flow | One-page checkout vs step-by-step | Completed orders |
| Onboarding | Tutorial screens vs learning in context | Day-7 retention |
| Pricing and offer display | Monthly vs annual shown first | Plan chosen, revenue per user |
| Full redesign | New site vs current site | Conversion, average order, task success |
How to Run an A/B Test, Step by Step
- Write one hypothesis. "Moving delivery cost to the product page will reduce checkout drop-off." One change, one expected effect.
- Pick one primary metric. Decide it before launch. Track a few guardrail metrics, such as refunds or support tickets, so a "win" does not hide a problem.
- Estimate the sample size. Your current conversion rate and the smallest change worth detecting decide how many visitors each version needs. A free sample size calculator is enough.
- Split traffic at random. Same time period, same audience, 50/50 unless there is a reason to send less traffic to the riskier version.
- Run full weeks. Include weekdays, weekends and payday, because buying behaviour in Indonesia shifts around them. Do not stop the test the moment B looks ahead.
- Read the result, then ask why. A clear winner gets shipped. A flat result is also useful: it tells you the change did not matter, so the next test should be bolder.
When A/B Testing Is the Wrong Tool
- Low traffic. If a page gets a few hundred visits a month, a test can run for months without a clear answer. Moderated research with 5 to 8 people finds the big problems faster.
- Ideas that are not built yet. Building two full versions just to test them is expensive. Test mockups or prototypes first.
- Questions about motivation. "Why do people abandon the cart?" is a research question. A/B testing can only check the fix you come up with.
Real Examples from Our Projects
Testing concepts before building. An adult FMCG brand (under NDA) wanted a gamified tips site with scratch-to-reveal tips and guest stories. Before development, we ran 10 remote concept sessions comparing 3 concepts on mockups. The finding shaped the build: a reveal mechanic sparks curiosity once, and people return when what is revealed is new, useful or personal. That is the kind of insight an A/B test on a live site cannot give you.
A/B testing a full redesign. For an adult lifestyle e-commerce brand (under NDA), we worked over three years and 3 studies. We tested and interviewed 157 people, used machine learning clustering to define 5 customer archetypes, redesigned the site around them, and then ran an A/B test of the new site against the old one in Jakarta and Bali. The research made the redesign strong; the A/B test proved it before launch.
The pattern in both projects is the same: research narrows the options, testing confirms the choice.
How to Start
Pick the one page or flow where a better result would matter most this quarter, usually sign-up, checkout or a key landing page. Check how much traffic it gets, then decide whether to start with concept testing on prototypes or a live A/B test. If you want help designing the test and reading the results, talk to us.
Frequently Asked Questions
What is A/B testing?
A/B testing is an experiment where two versions of a page, screen or flow are shown to different groups of real users at the same time, and one metric, such as sign-ups or purchases, decides which version performs better.
How long should an A/B test run?
Long enough to reach the sample size you planned, and at least one or two full weeks so weekday and weekend behaviour are both included. Stopping early because one version looks ahead is the most common mistake.
How much traffic do I need for A/B testing?
It depends on your current conversion rate and the size of change you want to detect. Small sites often need weeks to detect small changes, so they test bigger differences or use moderated research instead.
What is the difference between A/B testing and concept testing?
Concept testing shows mockups or prototypes to a small group before anything is built, to learn which idea is worth building and why. A/B testing compares live versions with real traffic to measure which one performs better.
Can you A/B test a full redesign?
Yes. Send part of the traffic to the new site and part to the old one, and compare the same metrics. It works best when the redesign is already shaped by research, so the test confirms a strong option.
Sources
Project figures from rekursiv case studies
- Concept-testing a gamified tips site Adult FMCG brand · Under NDA
- From segmentation to a tested redesign Adult lifestyle e-commerce · Under NDA
