Effective A/B Testing (Part 3)

If Part 1 was about making sure your tests are set up for significance, and Part 2 was about knowing whether your test will be reliable, this one is about the thinking behind it all. Because without the right mindset, even good tests lead nowhere.

1. Run as many tests as possible

This is where most teams start. And to be fair, it’s not a bad place to begin.

We told ourselves things like “even failed tests are learnings” and “you’ve got to start somewhere.” So we started… And kept starting.

The problem? Most of our tests didn’t reach significance. Many never finished. Some delivered no usable insight at all. You can’t learn from a test that tells you nothing. Wins and losses matter. Inconclusive results don’t.

That’s when we realized volume alone wasn’t enough.

2. Run as many significant tests as possible

Our first evolution came when we introduced planning. That may sound basic, but hear me out. 

We began checking for sample size requirements, estimating minimum detectable effects, and choosing higher-traffic pages (all the things we go through in part 2).

We were now planning each test and testing more reliably, but still not always effectively.

Some of the things we were testing were too small to matter. The tests were clean, but the impact was minimal. We were testing for the sake of it, not for the sake of results.

3. Get as many significant winners as possible

This was the real breakthrough.

We stopped testing ideas just because we could. Instead, we started asking harder questions before running a test:

• Do we believe this could meaningfully improve the experience?
• Is the potential lift worth the effort?
• Is there a strategic reason to run this?

The result? Fewer tests, but more winners. When we became more selective, the success rate went up. Not because our luck changed, but because our focus did.

“You won’t win every test. But you’ll win more often if you only run the ones that are worth it.”

4. Maximize total conversion rate lift

This is where we’re heading now.

Instead of aiming for more winners, we’re aiming for more impact. That means fewer small tweaks and more bold changes. Grouping multiple updates into one variant. Focusing less on isolating effects and more on growing outcomes. This is often referred to as leap tests

Sometimes that means learning less from a test. But it often means gaining more in real business value.

If you are a true A/B-testing purist, this goes against a lot of advice, but if we just adhere to the theory and isolate everything, and don’t have quite the traffic to run isolated experiments in 2 weeks, then grouping together can be a great way to gain quick wins.

A mindset shift is what turns testing into optimization

Here’s how our goals changed, and what that meant for results:

When we changed what we were aiming for, everything else followed. Better ideas, faster learnings, and more confident decisions. But even with the right mindset, there are traps that can derail your testing before the results come in...

The silent killers of A/B testing programs

Even with the right mindset, it’s easy to fall into patterns that quietly sabotage your results. These are two that nearly stalled our progress.

Testing too many variants

The logic makes sense: more variants mean more learning. Right?

Not exactly. Every new variant splits your traffic. If you don’t have very high traffic volume, this makes it harder to reach significance, or even to get usable results at all.

We ran multi-variant tests with A, B and C variants where nothing crossed the finish line. Valuable time was lost, and we were left with weak signals and no direction.

“Just because you can test it doesn’t mean you should.”

Only include variants if you truly believe they could win. Otherwise, you’re just watering down your traffic.

Being too cautious

This one surprised us. We thought small changes were smart. Safer. Easier to test.

But tiny changes mean tiny effects. And tiny effects require massive sample sizes to prove.

We were running clean, careful tests that couldn’t give us a clear signal unless we waited for months. And even when they worked, the payoff was minor.

Eventually, we started testing bigger ideas. Layout changes. Content rewrites. New flows.

“It’s better to run one bold test with clear results than ten cautious ones that tell you nothing.”

ℹ️ The larger the effect you're trying to detect, the smaller your required sample size. So not only do bigger changes have a better chance of improving results, they also help you reach conclusions faster. That’s why we often group multiple changes into one variant early on. Once we see an uplift, we can isolate later if we need to.

The takeaway

Your mindset sets the tone for everything else. If your goal is just to run tests, you’ll find ways to stay busy. If your goal is to drive meaningful improvement, your tests will change, and so will your results.

The best testing programs are built on:

• A mindset focused on impact, not activity
• A habit of choosing high-potential ideas over easy ones
• The discipline to test fewer things, but do it well
  
  

Coming up in part 4

In Part 4, we’ll dive into when to stop an A/B test, and when to let it run. We’ll talk about the danger of peeking too soon, the temptation to keep pushing for significance, and what happens when you keep going "just one more week."

Want to get notified when the next post is live?

Yes, notify me!