How to end experiments quicker#

There are two ways of ending an experiment sooner:

• By lowering the confidence level.

• Setting up a time limit.

Lowering the confidence level#

In a regular setup, every experiment needs to reach statistical significance to end. This basically means that the test needs to be run by a certain number of people to achieve results that are mathematically reliable.

Reaching statistical significance means that the winner variant will, most likely, reflect the same test results on a bigger audience. This statistical significance allows a certain margin of error, because achieving a result with 0% of error is basically impossible. This margin could also be understood as the confidence level, and it can be tweaked to get results quicker, but with a downside. If you dial down the confidence level will make your test need less people to complete, but will also mean that you’ll have less confidence on the replicability of results.

For altering the confidence level, click on your user, go to Settings > Auto-settings

The confidence level is automatically setted at 90%, but you can go down to 80% and up to 99%. If you have more than one site, you’ll be able to set a different confidence level for each one. We recommend using different percentages of confidence until you find the sweet spot that works the best for your traffic on each site.

Setting up a time frame#

If you go back to Setting> Auto-settings, you’ll see that, under Confidence Level, you can also select the time limit you want to set for each website you are testing on.

It will be automatically set on 30 days, but you can go down to 14 days, and up to 180 days. When an experiment reaches the time limit, it will end no matter if it has reached or not statistical significance. If the experiment reaches statistical significance before reaching the time limit, it will also end there.