CRO testing: how to run a test programme for clients that holds up
CRO testing only works when a page has the traffic to answer the question, the test is planned before it starts, and the result is read once. This guide starts with the traffic maths, then covers the backlog, the brief, running and reading tests honestly, and how to report them to a client.
What is CRO testing?
CRO testing is the controlled part of conversion rate optimisation: showing different versions of a page or step to randomly split groups of visitors at the same time, and measuring which one converts better. The most common form is an A/B test, with one control and one variant. Multivariate tests change several elements at once and need much more traffic.
The guides that rank for this topic, such as Mouseflow’s and AWA Digital’s, explain what a test is and how to run one. This guide adds the step that comes before them: whether a client’s site has the traffic to run a programme at all, and how many tests a year it can support. It then covers running the programme for a client, from the backlog to the monthly report.
Can the site support a testing programme?
This is the first question, and the one most programmes skip. A test needs enough visitors in each variant to tell a real difference from noise, and the number depends on three things: the page’s conversion rate, the smallest lift worth detecting and the traffic the page gets.
The smallest lift worth detecting, often called the minimum detectable effect, matters most. Halving it roughly quadruples the visitors a test needs. The table shows one example page, with 5,000 visitors a week and a 3% conversion rate.
| Smallest lift to detect | Visitors per variant | Weeks per test | Tests a year on this page |
|---|---|---|---|
| 5% | 207,938 | 84 | 0 |
| 10% | 53,211 | 22 | 2 |
| 15% | 24,193 | 10 | 5 |
| 20% | 13,914 | 6 | 8 |
| 30% | 6,455 | 3 | 17 |
| 50% | 2,518 | 2 | 26 |
The arithmetic changes the conversation with a client. On this page, a test designed to detect a 10% lift needs 53,211 visitors in each variant, or 106,422 in total. At 5,000 visitors a week, that is 149 days of traffic, rounded up to 22 whole weeks, so the page can support two such tests a year. Designed to detect a 20% lift, the same page needs 6 weeks per test and can support eight a year.
Work it out for your client’s page
Enter the page’s weekly traffic and conversion rate. The defaults are an example, not a benchmark.
If only half the visitors enter the test, the run time roughly doubles: at a 50% share, the 10% example above goes from 22 weeks to 43. For the full calculation with more than two variants, uneven splits or a one-sided test, use the A/B test sample size calculator. To find the page’s current rate first, the conversion rate calculator does that, and conversion rate benchmarks by industry explains why a site’s own rate is a better starting point than an industry average.
What to do when the traffic is not there
Many small and medium sites cannot support a full testing programme on any single page. That is a finding, not a failure, and it is better said at the start than discovered after six months of inconclusive tests.
- Test bigger changes. A new offer, a shorter form or a different page structure is more likely to produce a large difference than a button colour.
- Test higher in the funnel. A step with a higher conversion rate, such as clicks to the pricing page, needs fewer visitors than the final purchase.
- Pool similar pages. The same change to every product page or every service page can be tested across all of them at once.
- Do not test at all. Make the changes the research supports and compare before and after, and tell the client that this is not a controlled test.
The test programme cycle
A testing programme is a loop, not a list of tests. Each test starts from research and ends with a result that feeds the next round of research.
- 1ResearchAnalytics, recordings, surveys and sales calls show where visitors drop and why.
- 2BacklogEvery idea written as a hypothesis, scored and ranked.
- 3BriefOne page per test: change, metric, sample size, run time, decision rule.
- 4Build and QAVariants built, checked on real devices, tracking tested.
- 5RunWhole weeks, no stopping early, a sample ratio check.
- 6Read and reportOne read at the end; ship, drop or retest; the client report.
The cycle is the same for a large ecommerce site running several tests a month and a small lead-generation site running a handful a year. What changes is how many tests are in flight and how much research each one needs.
Research: where test ideas come from
Good test ideas come from evidence about where visitors drop and why. Ideas from opinion alone, such as a redesign someone likes, tend to produce small or random differences.
- Analytics. The steps with the largest drop, the devices and sources that convert worst, and the pages with high traffic and low conversion. Check the tracking first; the GA4 audit guide covers how.
- Session recordings and heatmaps. Where people hesitate, rage-click, scroll past the offer or abandon a form field.
- On-page surveys. One question on the page that loses most visitors, such as what stopped them today.
- Sales and support. The questions prospects ask before buying, and the objections that lose deals.
- Form analytics. Which fields people start and abandon.
Write each finding down with its source. A test idea that cannot point to evidence goes to the bottom of the backlog.
The backlog and how to prioritise it
Every idea goes into one backlog, written as a hypothesis: because of this evidence, changing this will improve that metric. The backlog is the client’s, not the agency’s, and it survives a change of supplier.
Several scoring frameworks exist for ranking ideas. Most score some mix of expected impact, confidence in the evidence and effort. The exact framework matters less than scoring consistently and adding one factor most frameworks leave out: whether the page can run the test in a reasonable time.
- Evidence. How strong is the research behind the idea? Several sources pointing the same way score higher than one.
- Reach. How many visitors a week see the page or step? The calculator above turns this into weeks per test.
- Size of change. Is the change bold enough to plausibly produce a difference the page can detect?
- Effort. Design, build and QA time, including any developer work.
Ideas that score high on evidence but fail on reach are often still worth doing. They become changes to ship and measure before and after, not tests.
Write the brief before you build
Every test gets a one-page brief, agreed with the client before anything is built. The brief is what stops a test being reinterpreted after the result is in.
| Field | What goes in it | Example |
|---|---|---|
| Hypothesis | Because of evidence X, changing Y will improve Z | Because 40% of form starts stop at the phone field, making it optional will raise form completions |
| Evidence | Where the idea came from, with links | Form analytics export, five session recordings, two sales call notes |
| Page and audience | Where the test runs and who sees it | The quote page, all devices, new and returning visitors |
| Primary metric | The one number that decides the test | Completed quote requests per visitor |
| Guardrail metrics | Numbers that must not get worse | Qualified leads per request, page load time |
| Sample size and run time | From the calculator, in whole weeks | As many whole weeks as the calculator gives for this page |
| Decision rule | Written before launch | Ship if significant at 95% and guardrails hold; otherwise keep the control |
| Owner and dates | Who builds, who reads, launch and end dates | Set when the test is scheduled |
The decision rule is the field that matters most. Written before launch, it settles in advance what result means ship, what means keep the control, and what happens if the guardrail metrics get worse. Written after, it tends to bend towards whatever the data shows.
Choose one primary metric
A test with five success metrics will usually find one that improved by chance. Choose one primary metric that matches the business goal, such as completed quote requests or purchases (the guide to the marketing metrics worth tracking covers which numbers tie to revenue), and treat the others as guardrails or context. Clicks and scroll depth are useful for diagnosing why a test won or lost, not for deciding it.
Build and check the test
Most broken tests are broken at the build stage: a variant that does not render on one browser, a tracking event that fires twice, a redirect that drops the campaign parameters. Check before launch, not after.
- Devices and browsers. Check every variant on the devices that carry most of the page’s traffic, using real phones where possible.
- Tracking. Complete the primary conversion in each variant and confirm it is recorded once, against the right variant.
- Speed and flicker. If the testing tool changes the page after it loads, check that visitors do not see the original before the variant appears.
- Campaign parameters. If the test redirects to a different URL, check that UTM parameters and click IDs survive the redirect.
Keep the test safe for search
Google’s guidance on website testing sets four rules for tests that might be crawled. Do not cloak, meaning show Googlebot different content from visitors; Google says that is against its spam policies whether you are testing or not. Use rel=canonical on alternate test URLs, pointing to the original. Use 302 (temporary) redirects, not 301s, when sending visitors to a variant URL. And run the experiment only as long as necessary, then remove the test URLs and scripts; Google says that running an experiment for an unnecessarily long time may be treated as an attempt to deceive search engines.
Run it without peeking
Once a test starts, the hardest part is leaving it alone. Testing dashboards show a running significance figure, and the temptation is to stop as soon as the variant looks like a winner.
Evan Miller’s How Not To Run an A/B Test explains why that breaks the result. Standard significance figures assume the sample size was fixed in advance. In his worst-case example, checking for significance after every observation and stopping at the first significant result produced a false positive 26.1% of the time, against the 5% the dashboard implied. He also shows that peeking ten times turns what looks like 1% significance into an actual 5%. His advice: decide the sample size in advance and do not believe the significance figure until the test has run its course.
- Run whole weeks. Visitors behave differently on weekdays and weekends; a test that ends on a Wednesday over-weights some days.
- Do not stop early on a good day. Look at the dashboard for faults, not for the result.
- Do not extend a test to reach significance. Running longer because the result is close is the same problem in reverse.
- Do not change the test while it runs. Editing a variant, the split or the audience mid-test means starting again.
Some testing tools use sequential or Bayesian statistics designed for continuous monitoring. If a client’s tool does, read its documentation on when a result can be acted on, and follow that rule rather than the fixed-sample one.
Check the traffic split
If a test is set to split traffic 50/50 and the variants end up with noticeably different numbers of visitors, something is wrong. This is called a sample ratio mismatch. Microsoft’s experimentation team describes it as a symptom of quality problems such as faulty bucketing, redirects in one variant or bots being filtered unevenly, and says every A/B test at Microsoft must pass a sample ratio check before its effects are analysed. The same article reports that about 6% of Microsoft’s A/B tests had one, and that about 10% of a certain kind of LinkedIn test used to.
Glancing at the two counts is not enough, because whether a gap matters depends on the sample size. The check is a chi-square test of the observed visitor counts against the split you configured; many testing tools run it automatically. If a test fails it, find the cause before trusting the result.
Read the result once
When the planned run time ends, read the result once against the decision rule in the brief. The A/B test significance calculator checks the final numbers if the testing tool’s own figures are unclear.
- A clear winner. The primary metric improved, the result is significant at the agreed level and the guardrails held. Ship it, remove the test code and record the result.
- A clear loser. The variant did worse. Keep the control and record what was learned; a loss is evidence about what visitors want.
- No detectable difference. The test could not tell the variants apart at the planned size. That does not prove they are the same; it means any difference is smaller than the test was designed to detect. Keep the simpler or cheaper version, or test a bolder change.
Be careful with segments
After a test, it is tempting to slice the result by device, source or new and returning visitors until one segment shows a win. Each slice is another chance of a false positive, and each has fewer visitors than the test was sized for. If one segment looks different, treat it as an idea for the next test, sized for that segment, not as a result.
Shipped wins tend to look smaller later
A winning variant’s measured lift is an estimate, and the tests that cross the significance line are disproportionately the ones whose estimate was high by chance. Expect the effect after shipping to be smaller than the test showed, and say so in the report rather than projecting the full lift onto annual revenue.
When not to run a test
Not every change needs a test, and some changes should not wait for one.
Yes
Yes
No
No
Yes
No
Broken things should be fixed, not tested: a form that errors on one browser, a checkout step that does not load, a price that is wrong. Changes required by law or policy, such as consent banners or accessibility fixes, should also go live without waiting for a test.
CRO testing tools after Google Optimize
Google Optimize, which many small sites used as a free testing tool, is no longer available as of 30 September 2023, and Google says any experiments still running on that date ended. Google’s help page lists AB Tasty, Optimizely and VWO as A/B testing providers it was collaborating with on Google Analytics integrations.
When choosing a tool for a client, the questions are practical rather than about features.
- Who owns the account? The client, on their own payment method, as with every other tool.
- How does it change the page? Client-side tools edit the page in the browser and can cause flicker; server-side testing avoids that but needs developer time.
- What statistics does it use? Fixed-sample, sequential or Bayesian, and what that means for when you can act on a result.
- Does it check the traffic split? A built-in sample ratio check catches broken tests early.
- Does it connect to the client’s analytics? So test variants can be analysed alongside the rest of the site’s data.
- What does it cost at the client’s traffic? Many tools price by visitors tested, so check the price at the traffic the programme needs.
Reporting tests to a client
A testing programme lives or dies on how it is reported. A client who only hears about winners will expect every test to win; a client who sees the whole log understands what the programme is buying.
- Tests completed this period. Each with its hypothesis, primary metric, result and decision.
- Tests running. Start date, planned end date and any faults found.
- What was shipped. Winning variants now live, and anything changed without a test.
- What was learned. One or two sentences on what the results say about the client’s visitors, including from losses.
- What is next. The top of the backlog and why.
Report the result as the test measured it: the lift on the primary metric, with the confidence level, not a projected annual revenue figure. If a revenue estimate helps the client, label it as an estimate and use a conservative version of the lift, for the reason given above.
The same principles apply to all client reporting; the PPC reporting guide covers how to build a report a client will read, and most of it carries over to testing.
What a programme looks like month to month
A testing programme for a client usually settles into a rhythm. The exact numbers depend on the traffic maths above, so agree them only after doing it.
- First month. Tracking audit, research, the first backlog and the traffic maths for the main pages. Possibly no tests at all, which is worth saying in advance.
- Each month after. One or more tests running, briefs written for the next ones, research topped up, a monthly report.
- Each quarter. A review of the whole test log: what has been learned, which pages are worth testing, and whether the programme should continue as tests or switch to changes measured before and after.
What a client should not be promised is a number of winning tests or a percentage lift. The programme can commit to the process: the research, the number of tests the traffic allows, honest reading and reporting. Whether the tests win depends on what the research finds.
Common CRO testing mistakes
Most testing programmes that disappoint fail in one of a few predictable ways.
- Testing pages without the traffic. Months of inconclusive tests on a page that could never detect the lift.
- Stopping on a good day. The result looks significant at the moment someone checks, and is a false positive.
- Testing tiny changes. Button colours and single words on low-traffic pages, where only large differences can be detected.
- No brief. The success metric and decision rule chosen after the result is in.
- Broken tracking. A variant that records conversions twice, or not at all, produces a confident wrong answer.
- Hunting through segments. Slicing a flat result until one segment wins.
- Leaving tests running. Test code and variant URLs left on the site after the test ended, against Google’s advice.
- Reporting only winners. The client learns to expect every test to win, and stops trusting the programme when they do not.
The CRO testing checklist
The checklist follows the programme. The first group decides whether a programme makes sense at all.
If the tracking itself needs work before any testing can start, my analytics service covers that set-up and audit.
Before promising a client a CRO testing programme, work out how many tests their pages can support: at 5,000 visitors a week and a 3% conversion rate, detecting a 10% lift takes 22 weeks per test. Test bold changes on busy pages, write a brief and decision rule before launch, run whole weeks without stopping early, check the traffic split, and report losses as well as wins.
The most useful thing I can tell a client at the start of a testing programme is how many tests their pages can actually run. It is often fewer than they hoped, and it changes which ideas are worth testing.
A loss reported plainly does more for the programme than a win reported loudly. It shows the tests are honest.
Frequently asked
What is CRO testing?
How much traffic do you need for CRO testing?
How long should an A/B test run?
What is peeking in A/B testing?
What is a sample ratio mismatch?
Do A/B tests hurt SEO?
What replaced Google Optimize?
I have 3 client slots open for Q4 2026. If this raised a question about your own situation, or you want help with it, send it over.
Say hello