When you look at a campaign's results, it's tempting to credit the mail with every sale that came in afterward. But some of those customers would have bought anyway — mail or no mail. A holdout group is how LettrLabs measures the difference between "revenue that happened" and "revenue the mail actually caused."

A holdout group is a small, randomly selected slice of your target audience that LettrLabs deliberately does not mail, even though they matched your targeting criteria. Everyone else in the audience gets mailed as normal.

Because the holdout group looks statistically like the rest of your mailed audience, whatever revenue it generates on its own is a good stand-in for "what would have happened anyway" — the baseline. Comparing that baseline against the revenue from the group that was actually mailed tells you how much extra revenue the campaign caused. That extra amount is called incremental revenue, and the efficiency measure built from it is incremental ROAS, or iROAS — incremental revenue divided by what you spent on the campaign.

For example: if your holdout group generates $2,000 in revenue on its own and your mailed group generates $5,000, the mail's incremental lift is roughly $3,000 — that's the number iROAS is built from. (These figures are illustrative only, not a real campaign result.)

Holdout testing works the same way across every automation type LettrLabs supports — it's not limited to Shopify-triggered campaigns. You can set up a holdout group for a regular automation, for a Radius Mail campaign, for a HailTrace or storm-triggered automation, and even for a single, one-off order. The mechanism behind the scenes is the same in every case: a portion of the audience is held back from mailing so it can serve as your baseline.

Important: iROAS is only meaningful for campaigns where a real holdout group was actually set up. If your campaign didn't use a holdout, don't rely on its iROAS number to judge performance — use it for campaigns that did.

If no holdout was configured for a given order or campaign, there's no baseline to compare against — so an iROAS figure for that order either won't be shown, or shouldn't be trusted if you happen to see one attached to a rolled-up report. Before you make a decision based on iROAS, confirm that a holdout group was actually set up for that specific campaign. When it was, iROAS is one of the most reliable ways to see whether a campaign is truly paying for itself. When it wasn't, lean on other metrics instead.

A holdout group works best when it's large enough to be statistically meaningful. A tiny holdout — just a handful of addresses — can be swung wildly by one or two unrelated sales, which makes the resulting iROAS noisy rather than a true signal. The bigger and more representative your holdout group, the more you can trust the iROAS it produces.

Shopify Best Practices: Winback & Retargeting Campaigns