Incrementality
Using a holdout to test an affiliate partnership
Plan a partnership holdout with controlled assignment, a shared store outcome, an observation window and checks for cross-exposure.
A holdout compares the same store outcome for eligible people who can receive a specified partner activity with those withheld from it. Plan assignment before the placement runs. A convenient split into existing audience segments may reflect pre-existing differences; call it random only when assignment was random.
Specify the tested change
Name the partner, placement, audience, offer and period. To test a newsletter inclusion, for example, keep the store offer available on the same terms to both groups while only one receives the inclusion. If the offer changes too, the result measures the combined package unless the design separates the changes.
Confirm what the publisher can control. It may split an email audience or suppress a defined group from an app placement. A public article or shareable code may reach both groups.
Record the assignment unit and any route by which a person can enter both groups or encounter the withheld placement elsewhere. If reliable withholding is impossible, choose a different design and qualify the resulting claim.
Set the measurement plan
Choose one primary outcome, such as approved first-time-customer orders or contribution before affiliate-specific costs. Measure it from store records in both assigned groups, including orders with no affiliate credit. Fix the observation period, order-status cut-off, customer-status rule and currency. Decide in advance what gain would justify the partnership and whether the expected volume could distinguish that gain from ordinary variation.
Record the eligible population, assignment method, exposure each group may receive, concurrent promotions, analysis date and costs. Keep stock, price and other marketing as comparable as practical, and document differences that arise.
Key metrics and thresholds for evaluating holdout results
- Primary Outcome
- Approved first-time-customer orders
- Order Status Cut-off
- Orders must be approved by end of observation window
- Currency
- AUD (Australian Dollar)
- Costs to Deduct
- Commission, fixed placement fees, funded discounts
Analyse the assigned groups
Use each group’s assigned population as the denominator for its outcome rate. Compare rates, or use an analysis that accounts for unequal allocation; raw order totals can mislead when group sizes differ. Include everyone assigned under the agreed rule, even if some people did not open the placement. Report the estimated difference and uncertainty.
Before interpreting the result, check whether the placement ran, control exposure was limited as planned and the same checkout routes were measured in both groups. Cross-exposure weakens the contrast. Do not discard orders after seeing the result without a rule set beforehand.
For a financial decision, compare contribution before affiliate-specific costs across groups and then subtract the difference in commission, fixed placement payments and funded discounts. If the estimate is inconclusive, a more controllable placement, longer observation period or larger eligible audience may make the next test more informative. Limit the conclusion to the tested partner, audience, offer and period.
