Skip to content

Growth Dept Email is open to brands by invitation. Ask for an account

GROWTH DEPT.

Holdout tests

Keep a share of your profiles out of marketing for a while, and compare what they buy with everyone else, to see what your email and texts add.

About 2 minutes to read

What a holdout test tells you

Attribution gives a message credit when someone ordered after opening or clicking it. Some of those people would have ordered anyway. A holdout test answers the bigger question: how much more do people buy because they get your marketing? It keeps a random share of your profiles out of every campaign and flow message on the channels you choose, then compares how often each group converts.

Set one up

  1. Go to Analytics then Experiments and open the Holdout tests tab.
  2. Choose Create test and name it.
  3. Set the share of profiles to hold out, from 0.01% to 50%.
  4. Choose the channels: email, text message or both.
  5. Choose the conversion metric. It starts as your account’s, usually Placed Order.
  6. Save it as a draft, then choose Start test when you are ready.
The Create holdout test window with a name, a 10% holdout, both channels ticked and the conversion metric
A new test holding out 10% of profiles from email and texts.
Note. One test runs at a time. Starting it takes effect straight away for every campaign and flow message that goes out after that.

Who is held out

Each profile lands in a group by a fixed calculation on the test and the profile, so a person stays in the same group for the whole test, including people who join your list after it starts. Held-out people are skipped when a campaign or flow message would go to them on a test channel.

Order confirmations and other transactional messages still go to everyone, and so do test sends.

Read the results

Profiles
Everyone in the account, split by group.
Converters
People in the group with at least one conversion event since the test started.
Conversion rate
Converters divided by profiles in the group.
Value
What those conversions were worth, in your account's currency.

The line under the table compares the two conversion rates: how much more, or less, often people who received marketing converted than the held-out group. Until someone in the held-out group converts, there is nothing to compare, and it says so.

A running holdout test with profiles, converters, conversion rate and value for each group
A running test on the made-up Harbor & Pine demo account. The demo's numbers are illustrative: its held-out profiles were never really held out.

How big, and for how long

A bigger holdout gives a clearer answer sooner, and costs more, because those people miss your offers while it runs. The form starts at 10%. Run it long enough to cover your customers’ usual time between orders, so both groups have had the chance to buy.

The page does not test whether the difference is statistically significant. With small groups, a gap of a few percent can be chance; let the test run longer before acting on it.

End or delete a test

End test sends marketing to everyone again from that moment; the results stay on the page. A test that is not running can be deleted, with its results.

Related articles

  • Custom report types

    The five kinds of custom report, what each one answers, and how to run, schedule and export one.

  • How benchmarks compare your results

    Benchmarks rate each result against your own earlier periods. How the ratings work, and why there is no comparison with other brands.

  • Bot clicks and machine opens

    How we keep security scanners and privacy features from inflating your clicks, opens and attributed revenue.

Something wrong or missing? Email hello@growthdept.ai.