Running a split campaign

How to Split Test Offers to Your House List (Without Customer Service
Nightmares)
SUMMARY:
Are you convinced a particular offer would increase your ROI dramatically,
but management thinks it’s chancy? Check out this new Case Study on Pete’s
Coffee & Tea. Last December they tested three different email discount
offers – with surprising results.
Our favorite part: see data on how adding the offer to every page of the
site that email clickthoughs see on their path (not just their initial
landing page) helped improve results. Yes, includes creative samples:

CHALLENGE
“We’re a highly branded site. The people that visit us for the most part
already know us,” says Brian Platter, General Manager Home Delivery, Peet’s
Coffee & Tea. “They look forward to our emails.”

Because coffee isn’t a “commodity business” — that is, consumers mainly
purchase based on brand and taste preferences rather than price — Peet’s
doesn’t often do discounting promos. “We only do it when we’re trying to
encourage people to do something different.”

But when Peet’s wanted to encourage customers to give Peet’s products as
gifts last December, the team decided to run a free shipping promotion.

In the past, the team had offered free shipping on orders of $50 or more
which is slightly higher than the site’s average order value of $40-$45.
“Fifty dollars is conservative,” says Platter. “The ‘powers that be’ worry
about giving away something and not getting enough in return.” The
ecommerce team suspected that if they dropped the required purchase minimum
conversions would rise high enough to make up for the potential drop in
average order per cart.

Platter talked fast and although he agreed to stick with the safer $50
minimum for all regular site visitors, he won management’s approval to split
the house email list into a series of test cells each with a different
discount minimum for the December promo campaign.

However, this could cause a customer service nightmare. What if customers
got one offer via email and then a completely different one when they
clicked to the site? How could Peet’s run three different test discounts via
email and a single discount offer online without confusing people?

CAMPAIGN
The team split the house list into three parts (link to sample creative
below):

#1. The bulk of the list was sent a control offer featuring the conservative
$50 threshold.

#2. 20,000 names were sent a $30 minimum offer (the team had considered $25
but they tested $30 instead to see if they could get consumers to purchase a
minimum of three pounds of coffee).

#3. 20,000 names were sent a zero minimum offer — ie. free shipping no
matter how low the purchase was.

To keep the test as clean as possible, the subject line for all three email
creatives read the same:
Peet’s Gifts for All, Free Shipping for You

The creative featured a photo on the top left of a package of coffee and a
package of chocolate covered cherries. Beside that, on the right, was a box
that included the text, “Give the gift of Peet’s and Get Free Shipping.”
Running through the middle of the page was a red banner that said, depending
on the offer received: “Free Shipping on Orders of $50 or more.,” “Free
Shipping on Orders of $30 or more” or “Free Shipping on all online orders.”

Below that was a series of gift ideas.

Key — because these were campaigns to current customers who already knew
the site, the team knew there was a good chance email respondents would surf
a bit instead of buying directly from a single landing page. So every page
of each customer’s experience had to repeat the exact same offer the
customer had received in their email.

Anyone who received a particular offer via email saw that same offer on both
the home page and on all subsequent product pages they surfed. “We were able
to have the home page and coffee page ‘magically’ match the offer on the
email,” explains Platter. This worked by designating a certain area of the
site as a “content slot.” The content in that slot could be changed
depending upon the source of the visitor.

While developing this landing page system for the email campaign, the team
began to wonder if the fact of the continual reinforcement of the offer on
every page of a site would itself result in higher conversion rates. So they
decided just for fun, to split regular (non-email-driven) site traffic in
half for a week. All home page visitors saw the $50 shipping offer but of
clickthroughs, half were shown a page that included the offer again, while
the other half were shown a page that did not include the offer.

In the meantime, the first email campaign with the split test was sent on a
Monday. By Wednesday the team had enough response data to determine which
offer was the winner. They used that data to send the entire house file the
winning offer as a “reminder” on Thursday.

Results
The winning promotion was the offer of free shipping with orders of $30 or
more. “I was surprised, because I was assuming that either the no-limit
would raise conversion so high that it would win, or that just keeping the
threshold high, which would keep the average order value high, would win. I
didn’t expect the middle,” says Platter.

–The control (the $50 threshold) converted at a rate of about 0.5%.

–The $30 threshold converted 14% better than the control, and the average
order value (AOV) of those who purchased at the $30 threshold was 94% of the
control’s AOV. “So the conversion lift more than outweighed the slight
reduction in AOV,” says Platter.

–The promotion with a zero dollar threshold had a conversion rate of 1%,
twice as high as the control, but the AOV was only 72% of the control.

Generally, Peet’s emails convert at a rate of between 0.2% and 1%. A
Father’s Day offer last year had a conversion rate of 0.5% on an offer of
free shipping with a $50 minimum purchase.

Platter was also surprised by the results from the split test for
non-email-traffic. Turns out visitors who saw the web offer repeated through
the site did *not* convert more than those who saw the offer only on the
home page. However, the former group’s AOV was 9.5% higher than the latter.
So, although folks didn’t convert from shoppers to buyers at a higher rate,
they did convert for a slightly higher dollar purchase.

“The key was how little effort it was to actually learn something and be
able to apply that learning on the fly,” says Platter. “We applied the
learning later that week for a nice surplus. And it took all of a couple
hours work.”

Useful links related to this story:
Creative samples from Peet’s
http://www.marketingsherpa.com/cs/peets/study.html

Offermatica, the testing and optimization company Peet’s uses:
http://offermatica.com/

Peet’s Coffee & Tea
http://www.peets.com/

___________________________________________________________
Win a BlackBerry device from O2 with Yahoo!. Enter now. http://www.yahoo.co.uk/blackberry

Leave a Reply

Your email address will not be published. Required fields are marked *