Blog · Measurement

CRO testing: how to run a test programme for clients that holds up

CRO testing only works when a page has the traffic to answer the question, the test is planned before it starts, and the result is read once. This guide starts with the traffic maths, then covers the backlog, the brief, running and reading tests honestly, and how to report them to a client.

What is CRO testing?

CRO testing is the controlled part of conversion rate optimisation: showing different versions of a page or step to randomly split groups of visitors at the same time, and measuring which one converts better. The most common form is an A/B test, with one control and one variant. Multivariate tests change several elements at once and need much more traffic.

The guides that rank for this topic, such as Mouseflow’s and AWA Digital’s, explain what a test is and how to run one. This guide adds the step that comes before them: whether a client’s site has the traffic to run a programme at all, and how many tests a year it can support. It then covers running the programme for a client, from the backlog to the monthly report.

Can the site support a testing programme?

This is the first question, and the one most programmes skip. A test needs enough visitors in each variant to tell a real difference from noise, and the number depends on three things: the page’s conversion rate, the smallest lift worth detecting and the traffic the page gets.

The smallest lift worth detecting, often called the minimum detectable effect, matters most. Halving it roughly quadruples the visitors a test needs. The table shows one example page, with 5,000 visitors a week and a 3% conversion rate.

How long one test takes, by the lift you want to detect
Smallest lift to detect Visitors per variant Weeks per test Tests a year on this page
5% 207,938 84 0
10% 53,211 22 2
15% 24,193 10 5
20% 13,914 6 8
30% 6,455 3 17
50% 2,518 2 26
Example page: 5,000 visitors a week, all in the test, 3% conversion rate, two variants split 50/50, 95% confidence two-sided, 80% power. Worked out with the same formula as the A/B test sample size calculator; weeks are rounded up to whole weeks, and tests a year are 52 divided by weeks per test, rounded down.

The arithmetic changes the conversation with a client. On this page, a test designed to detect a 10% lift needs 53,211 visitors in each variant, or 106,422 in total. At 5,000 visitors a week, that is 149 days of traffic, rounded up to 22 whole weeks, so the page can support two such tests a year. Designed to detect a 20% lift, the same page needs 6 weeks per test and can support eight a year.

Work it out for your client’s page

How many tests a year can this page support?

Enter the page’s weekly traffic and conversion rate. The defaults are an example, not a benchmark.

Visitors needed per variant53,2110 when the rate or lift is 0, or the target rate would reach 100%
Weeks per test, in whole weeks22Two variants; days of traffic rounded up to whole weeks
Tests a year on this page252 / weeks per test, rounded down; 0 means one test would run longer than a year
Two variants split 50/50, 95% confidence two-sided, 80% power: the same formula and rounding as the A/B test sample size calculator, which also handles more variants, uneven splits and one-sided tests.

If only half the visitors enter the test, the run time roughly doubles: at a 50% share, the 10% example above goes from 22 weeks to 43. For the full calculation with more than two variants, uneven splits or a one-sided test, use the A/B test sample size calculator. To find the page’s current rate first, the conversion rate calculator does that, and conversion rate benchmarks by industry explains why a site’s own rate is a better starting point than an industry average.

What to do when the traffic is not there

Many small and medium sites cannot support a full testing programme on any single page. That is a finding, not a failure, and it is better said at the start than discovered after six months of inconclusive tests.

  • Test bigger changes. A new offer, a shorter form or a different page structure is more likely to produce a large difference than a button colour.
  • Test higher in the funnel. A step with a higher conversion rate, such as clicks to the pricing page, needs fewer visitors than the final purchase.
  • Pool similar pages. The same change to every product page or every service page can be tested across all of them at once.
  • Do not test at all. Make the changes the research supports and compare before and after, and tell the client that this is not a controlled test.

The test programme cycle

A testing programme is a loop, not a list of tests. Each test starts from research and ends with a result that feeds the next round of research.

The test programme cycle
  1. 1ResearchAnalytics, recordings, surveys and sales calls show where visitors drop and why.
  2. 2BacklogEvery idea written as a hypothesis, scored and ranked.
  3. 3BriefOne page per test: change, metric, sample size, run time, decision rule.
  4. 4Build and QAVariants built, checked on real devices, tracking tested.
  5. 5RunWhole weeks, no stopping early, a sample ratio check.
  6. 6Read and reportOne read at the end; ship, drop or retest; the client report.

The cycle is the same for a large ecommerce site running several tests a month and a small lead-generation site running a handful a year. What changes is how many tests are in flight and how much research each one needs.

Research: where test ideas come from

Good test ideas come from evidence about where visitors drop and why. Ideas from opinion alone, such as a redesign someone likes, tend to produce small or random differences.

  • Analytics. The steps with the largest drop, the devices and sources that convert worst, and the pages with high traffic and low conversion. Check the tracking first; the GA4 audit guide covers how.
  • Session recordings and heatmaps. Where people hesitate, rage-click, scroll past the offer or abandon a form field.
  • On-page surveys. One question on the page that loses most visitors, such as what stopped them today.
  • Sales and support. The questions prospects ask before buying, and the objections that lose deals.
  • Form analytics. Which fields people start and abandon.

Write each finding down with its source. A test idea that cannot point to evidence goes to the bottom of the backlog.

The backlog and how to prioritise it

Every idea goes into one backlog, written as a hypothesis: because of this evidence, changing this will improve that metric. The backlog is the client’s, not the agency’s, and it survives a change of supplier.

Several scoring frameworks exist for ranking ideas. Most score some mix of expected impact, confidence in the evidence and effort. The exact framework matters less than scoring consistently and adding one factor most frameworks leave out: whether the page can run the test in a reasonable time.

  1. Evidence. How strong is the research behind the idea? Several sources pointing the same way score higher than one.
  2. Reach. How many visitors a week see the page or step? The calculator above turns this into weeks per test.
  3. Size of change. Is the change bold enough to plausibly produce a difference the page can detect?
  4. Effort. Design, build and QA time, including any developer work.

Ideas that score high on evidence but fail on reach are often still worth doing. They become changes to ship and measure before and after, not tests.

Write the brief before you build

Every test gets a one-page brief, agreed with the client before anything is built. The brief is what stops a test being reinterpreted after the result is in.

The one-page test brief
Field What goes in it Example
Hypothesis Because of evidence X, changing Y will improve Z Because 40% of form starts stop at the phone field, making it optional will raise form completions
Evidence Where the idea came from, with links Form analytics export, five session recordings, two sales call notes
Page and audience Where the test runs and who sees it The quote page, all devices, new and returning visitors
Primary metric The one number that decides the test Completed quote requests per visitor
Guardrail metrics Numbers that must not get worse Qualified leads per request, page load time
Sample size and run time From the calculator, in whole weeks As many whole weeks as the calculator gives for this page
Decision rule Written before launch Ship if significant at 95% and guardrails hold; otherwise keep the control
Owner and dates Who builds, who reads, launch and end dates Set when the test is scheduled
A template, not a standard. The example row is hypothetical.

The decision rule is the field that matters most. Written before launch, it settles in advance what result means ship, what means keep the control, and what happens if the guardrail metrics get worse. Written after, it tends to bend towards whatever the data shows.

Choose one primary metric

A test with five success metrics will usually find one that improved by chance. Choose one primary metric that matches the business goal, such as completed quote requests or purchases (the guide to the marketing metrics worth tracking covers which numbers tie to revenue), and treat the others as guardrails or context. Clicks and scroll depth are useful for diagnosing why a test won or lost, not for deciding it.

Build and check the test

Most broken tests are broken at the build stage: a variant that does not render on one browser, a tracking event that fires twice, a redirect that drops the campaign parameters. Check before launch, not after.

  • Devices and browsers. Check every variant on the devices that carry most of the page’s traffic, using real phones where possible.
  • Tracking. Complete the primary conversion in each variant and confirm it is recorded once, against the right variant.
  • Speed and flicker. If the testing tool changes the page after it loads, check that visitors do not see the original before the variant appears.
  • Campaign parameters. If the test redirects to a different URL, check that UTM parameters and click IDs survive the redirect.

Google’s guidance on website testing sets four rules for tests that might be crawled. Do not cloak, meaning show Googlebot different content from visitors; Google says that is against its spam policies whether you are testing or not. Use rel=canonical on alternate test URLs, pointing to the original. Use 302 (temporary) redirects, not 301s, when sending visitors to a variant URL. And run the experiment only as long as necessary, then remove the test URLs and scripts; Google says that running an experiment for an unnecessarily long time may be treated as an attempt to deceive search engines.

Run it without peeking

Once a test starts, the hardest part is leaving it alone. Testing dashboards show a running significance figure, and the temptation is to stop as soon as the variant looks like a winner.

Evan Miller’s How Not To Run an A/B Test explains why that breaks the result. Standard significance figures assume the sample size was fixed in advance. In his worst-case example, checking for significance after every observation and stopping at the first significant result produced a false positive 26.1% of the time, against the 5% the dashboard implied. He also shows that peeking ten times turns what looks like 1% significance into an actual 5%. His advice: decide the sample size in advance and do not believe the significance figure until the test has run its course.

  • Run whole weeks. Visitors behave differently on weekdays and weekends; a test that ends on a Wednesday over-weights some days.
  • Do not stop early on a good day. Look at the dashboard for faults, not for the result.
  • Do not extend a test to reach significance. Running longer because the result is close is the same problem in reverse.
  • Do not change the test while it runs. Editing a variant, the split or the audience mid-test means starting again.

Some testing tools use sequential or Bayesian statistics designed for continuous monitoring. If a client’s tool does, read its documentation on when a result can be acted on, and follow that rule rather than the fixed-sample one.

Check the traffic split

If a test is set to split traffic 50/50 and the variants end up with noticeably different numbers of visitors, something is wrong. This is called a sample ratio mismatch. Microsoft’s experimentation team describes it as a symptom of quality problems such as faulty bucketing, redirects in one variant or bots being filtered unevenly, and says every A/B test at Microsoft must pass a sample ratio check before its effects are analysed. The same article reports that about 6% of Microsoft’s A/B tests had one, and that about 10% of a certain kind of LinkedIn test used to.

Glancing at the two counts is not enough, because whether a gap matters depends on the sample size. The check is a chi-square test of the observed visitor counts against the split you configured; many testing tools run it automatically. If a test fails it, find the cause before trusting the result.

Read the result once

When the planned run time ends, read the result once against the decision rule in the brief. The A/B test significance calculator checks the final numbers if the testing tool’s own figures are unclear.

  • A clear winner. The primary metric improved, the result is significant at the agreed level and the guardrails held. Ship it, remove the test code and record the result.
  • A clear loser. The variant did worse. Keep the control and record what was learned; a loss is evidence about what visitors want.
  • No detectable difference. The test could not tell the variants apart at the planned size. That does not prove they are the same; it means any difference is smaller than the test was designed to detect. Keep the simpler or cheaper version, or test a bolder change.

Be careful with segments

After a test, it is tempting to slice the result by device, source or new and returning visitors until one segment shows a win. Each slice is another chance of a false positive, and each has fewer visitors than the test was sized for. If one segment looks different, treat it as an idea for the next test, sized for that segment, not as a result.

Shipped wins tend to look smaller later

A winning variant’s measured lift is an estimate, and the tests that cross the significance line are disproportionately the ones whose estimate was high by chance. Expect the effect after shipping to be smaller than the test showed, and say so in the report rather than projecting the full lift onto annual revenue.

When not to run a test

Not every change needs a test, and some changes should not wait for one.

Test it, or just ship it?
Can the page reach the sample size in eight weeks or less?
Yes
Is the change risky, costly to build, or something people disagree about?
Yes
Test it. This is what a test is for: a real question, enough traffic to answer it, and a decision that depends on the answer.
No
Ship it, and watch the numbers. A fix for a broken form or a confusing label does not need a test; make the change and compare the following weeks with the weeks before.
No
Is there a bolder version of the change, or a busier page or step with a higher rate?
Yes
Test that instead. A bigger change needs fewer visitors to detect, and a step with a higher conversion rate needs fewer too.
No
Do not test. Make the changes the research supports, measure before and after, and say plainly to the client that this is not a controlled test.
Eight weeks is a working limit, not a rule. Longer tests run into seasonal changes and Google’s advice to run experiments only as long as necessary.

Broken things should be fixed, not tested: a form that errors on one browser, a checkout step that does not load, a price that is wrong. Changes required by law or policy, such as consent banners or accessibility fixes, should also go live without waiting for a test.

CRO testing tools after Google Optimize

Google Optimize, which many small sites used as a free testing tool, is no longer available as of 30 September 2023, and Google says any experiments still running on that date ended. Google’s help page lists AB Tasty, Optimizely and VWO as A/B testing providers it was collaborating with on Google Analytics integrations.

When choosing a tool for a client, the questions are practical rather than about features.

  • Who owns the account? The client, on their own payment method, as with every other tool.
  • How does it change the page? Client-side tools edit the page in the browser and can cause flicker; server-side testing avoids that but needs developer time.
  • What statistics does it use? Fixed-sample, sequential or Bayesian, and what that means for when you can act on a result.
  • Does it check the traffic split? A built-in sample ratio check catches broken tests early.
  • Does it connect to the client’s analytics? So test variants can be analysed alongside the rest of the site’s data.
  • What does it cost at the client’s traffic? Many tools price by visitors tested, so check the price at the traffic the programme needs.

Reporting tests to a client

A testing programme lives or dies on how it is reported. A client who only hears about winners will expect every test to win; a client who sees the whole log understands what the programme is buying.

  • Tests completed this period. Each with its hypothesis, primary metric, result and decision.
  • Tests running. Start date, planned end date and any faults found.
  • What was shipped. Winning variants now live, and anything changed without a test.
  • What was learned. One or two sentences on what the results say about the client’s visitors, including from losses.
  • What is next. The top of the backlog and why.

Report the result as the test measured it: the lift on the primary metric, with the confidence level, not a projected annual revenue figure. If a revenue estimate helps the client, label it as an estimate and use a conservative version of the lift, for the reason given above.

The same principles apply to all client reporting; the PPC reporting guide covers how to build a report a client will read, and most of it carries over to testing.

What a programme looks like month to month

A testing programme for a client usually settles into a rhythm. The exact numbers depend on the traffic maths above, so agree them only after doing it.

  • First month. Tracking audit, research, the first backlog and the traffic maths for the main pages. Possibly no tests at all, which is worth saying in advance.
  • Each month after. One or more tests running, briefs written for the next ones, research topped up, a monthly report.
  • Each quarter. A review of the whole test log: what has been learned, which pages are worth testing, and whether the programme should continue as tests or switch to changes measured before and after.

What a client should not be promised is a number of winning tests or a percentage lift. The programme can commit to the process: the research, the number of tests the traffic allows, honest reading and reporting. Whether the tests win depends on what the research finds.

Common CRO testing mistakes

Most testing programmes that disappoint fail in one of a few predictable ways.

  • Testing pages without the traffic. Months of inconclusive tests on a page that could never detect the lift.
  • Stopping on a good day. The result looks significant at the moment someone checks, and is a false positive.
  • Testing tiny changes. Button colours and single words on low-traffic pages, where only large differences can be detected.
  • No brief. The success metric and decision rule chosen after the result is in.
  • Broken tracking. A variant that records conversions twice, or not at all, produces a confident wrong answer.
  • Hunting through segments. Slicing a flat result until one segment wins.
  • Leaving tests running. Test code and variant URLs left on the site after the test ended, against Google’s advice.
  • Reporting only winners. The client learns to expect every test to win, and stops trusting the programme when they do not.

The CRO testing checklist

The checklist follows the programme. The first group decides whether a programme makes sense at all.

The CRO testing checklist
Before you start the programme
Before each test
While it runs
After it ends
Ticks are saved in your browser.

If the tracking itself needs work before any testing can start, my analytics service covers that set-up and audit.

The short version

Before promising a client a CRO testing programme, work out how many tests their pages can support: at 5,000 visitors a week and a 3% conversion rate, detecting a 10% lift takes 22 weeks per test. Test bold changes on busy pages, write a brief and decision rule before launch, run whole weeks without stopping early, check the traffic split, and report losses as well as wins.

Operator note

The most useful thing I can tell a client at the start of a testing programme is how many tests their pages can actually run. It is often fewer than they hoped, and it changes which ideas are worth testing.

A loss reported plainly does more for the programme than a win reported loudly. It shows the tests are honest.

Frequently asked

What is CRO testing?
Showing different versions of a page or step to randomly split groups of visitors at the same time and measuring which converts better. The most common form is an A/B test, with one control and one variant.
How much traffic do you need for CRO testing?
It depends on the conversion rate and the smallest lift worth detecting. At 5,000 visitors a week and a 3% rate, detecting a 10% lift takes 22 weeks per test with two variants; detecting a 20% lift takes 6 weeks.
How long should an A/B test run?
As long as the sample size needs, decided before launch and rounded up to whole weeks, so every day of the week is represented equally. Do not stop it early because it looks significant, and do not extend it to reach significance.
What is peeking in A/B testing?
Checking a running test and stopping it when the result looks significant. Standard significance figures assume a fixed sample size, so stopping early raises the false positive rate well above the level the dashboard shows.
What is a sample ratio mismatch?
When the number of visitors in each variant differs significantly from the split you set, such as 50/50. It points to a fault in the test, so find the cause before trusting the result.
Do A/B tests hurt SEO?
Not if run as Google advises: no cloaking, rel=canonical on alternate test URLs, 302 rather than 301 redirects, and removing the test as soon as it ends.
What replaced Google Optimize?
Google Optimize ended on 30 September 2023. Google lists AB Tasty, Optimizely and VWO as A/B testing providers it was working with on Google Analytics integrations.
Related reading
Conversion rate benchmarks by industry, and the denominator problem GA4 audit: a checklist for agencies, in the order that saves data PPC reporting: the KPIs that matter, a one-page client report and Google’s rules
Working on this yourself?

I have 3 client slots open for Q4 2026. If this raised a question about your own situation, or you want help with it, send it over.

Say hello