A/B Testing & Experiments

A/B testing (also called split testing) is a method of comparing two or more versions of a web page to determine which one performs better. You show different versions to different visitors simultaneously and measure which version produces more of the outcome you want — such as form submissions, button clicks, or purchases.

The key advantage over just "changing your page and seeing what happens" is that A/B testing separates the effect of the change from natural variation in traffic over time. Both versions run at the same time, so external factors affect both equally.

A/B testing compares complete page versions against each other. Variant A has one headline, Variant B has a different headline — and you compare the whole experience.

Multivariate testing tests combinations of multiple changes simultaneously to isolate the effect of each individual element.

For most businesses, A/B testing is more practical because it requires significantly less traffic to reach statistical significance. Iteratist supports A/B/n testing — meaning you can run a control plus multiple variants simultaneously.

There is no hard limit on the number of variants per experiment. However, in practice we recommend limiting tests to 2–4 variants (control plus 1–3 variants). The more variants you add, the more traffic each one receives per day is divided, meaning it takes longer to reach statistical significance for each one. With 4 variants at 25% each, your test will take roughly four times as long as a simple A/B split.

A strong hypothesis follows this structure: "If we change [specific element] to [specific change], then [metric] will improve because [reason based on insight]."

For example: "If we change the headline from 'AI-powered resume rewriting' to 'Get past ATS filters in 60 seconds', then form submissions will increase because it focuses on the specific problem the visitor is trying to solve rather than the technology."

The hypothesis matters in Iteratist because it feeds the AI advice engine — the more specific your hypothesis, the more targeted the AI recommendations will be.

The control is the original, unchanged version of your page — the baseline that all other variants are measured against. Every experiment in Iteratist has exactly one control. It receives traffic just like any other variant, and its conversion rate becomes the benchmark that determines whether a challenger variant is performing better or worse.

Run tests for at least two full weeks regardless of traffic volume, and don't stop early just because one variant appears to be winning. Short tests are vulnerable to day-of-week effects — Monday traffic often converts very differently to Friday traffic — and early results can be misleading due to small sample sizes.

For statistical reliability, you need each variant to have at least 100 conversions, ideally 200+. Iteratist shows you the current confidence level in real time so you know when you have enough data. A winner is only declared when confidence reaches 95%.

Yes. You can pause a running experiment at any time and resume it later. While paused, no new visitors are assigned to variants — they see the default version of your page as if no test were running. All data collected before the pause is retained, and data collection resumes from where it left off when you relaunch the experiment.

Note: pausing and resuming a test over an extended period can introduce noise into your results, as visitor behaviour may shift between sessions. Where possible, run tests continuously to completion.

Iteratist tracks conversions when a visitor reaches a specific URL — your thank-you page, order confirmation page, or any destination that a converted visitor reaches. You enter this Goal URL when setting up the experiment, and the tracking snippet detects it automatically with no additional code needed.

You can also label the metric type for reporting clarity: CTA button click, form submission, scroll depth (50% or 75%), 30 seconds on page, or page view.

Technically yes, but we recommend against it unless the experiments are testing completely non-overlapping elements of the page. Running two simultaneous tests on the same page creates an interaction effect — Variant A of test one combined with Variant B of test two may behave differently to any combination in isolation, making your results difficult to interpret. Run one test at a time on any given page for clean, actionable data.

When you end an experiment, traffic allocation stops and all visitors see the default version of your page. Your experiment data and results are preserved permanently in the platform — you can review them at any time. The next step is to implement the winning variant's changes permanently in your actual website code or CMS, then start a new test to keep improving.