Current Issue
This Month's Print Issue

Follow Fast Company

We’ll come to you.

Fastcoworks Created for and commissioned by Adobe
Fastcoworks Created for and commissioned by Adobe
How A/B Testing Can Revolutionize Your Entire Company

The practice of A/B testing—comparing the performance of two versions of the same content—is essential to retail companies like Home Depot. Even small optimizations around buttons, product specs and descriptions, the credit card interface and other elements on sites can increase conversion rates.

But many old-school marketers still regard A/B testing as little more than window dressing.

At Adobe Summit in Las Vegas last week, Jin Huang, Home Depot’s senior manager for testing and optimization, explained four ways that front-end optimization can have profound effects on business fundamentals.

The key, she argued, is to think about testing and optimization more broadly. Rather than seeing it as small comparisons, keep in mind your company’s larger purpose.

"There has to be more to optimization than just one single objective," she said, "because the lifting of a single metric might not actually lift your final goal."

This is true whether that goal is increasing conversions, or something abstract, such as goodwill.

1) Supporting offline channels

For a company like Home Depot, which has both e-commerce and brick-and-mortar channels, testing can prevent real-life customer service disasters—the kind that are easy fodder for Twitter rants.

One example, Huang said, is the in-store pickup option on

If you live between two stores, the site uses your IP address to select a location for pickup.

"That means I have to guess which store to suggest to the customer," Huang said. "But sometimes the customer fails to realize the Home Depot we picked isn’t the one they usually go to."

By segmenting new customers based on telltale items in their cart, Home Depot was able to optimize the system’s choice for pickup. The test worked so well that mis-shipments quickly dropped by 50 percent.

2) Bringing in-store expertise online

For a lot of us, a trip to a Home Depot store is full of educational moments. You rely on employees in their trademark orange aprons to guide you to the right aisle and perhaps explain the particulars of electrical or plumbing parts. But what happens online?

Lots of e-commerce sites use basic manufacturer specifications, such as physical dimensions or color, to sort products. Home Depot adds another layer of information, Huang said. Take toilets.

"As it turns out, flushing power is the most important characteristic when a customer is toilet shopping," she said. "We were able to take our store experts’ knowledge to prioritize certain features on our product pages and A/B test the way we design them."

3) Evaluating vendors

Then there’s what Huang called the "shiny new thing" conundrum. Third-party vendors tout one solution after another to help you reach your goals: a product-photo widget that allows users to upload their own product images; an address picker that helps users write addresses tailored for the U.S. Postal Service; a "scan your credit card" function to facilitate online transactions. Which to pursue?

Home Depot’s digital marketers A/B tested the candidates. None of the three improved conversion. The company passed, avoiding an unwise investment.

4) Providing data for capital investments

Digital marketing involves an endless series of decisions: What product category should we make videos for? Which products deserve their own landing pages?

"When we did A/B testing, we were able to tell which categories we should make videos for," Huang said. "We were even able to tell which videos worked best, between vendor-made, Home Depot-made and [user-generated content]."

This can apply to supply-chain investments, too.

"It’s important for us to understand how shipping can change conversion rates," Huang said.

That’s because reduced shipping time does not have a linear relation to conversion. It’s more of a step function.

"There’s a big bump from five to four days, then another one from two day[s] to one day," Huang said. "This helps us inform the supply-chain team about the best strategy—not just how fast they should ship, but whether we should subsidize some shipping options to boost conversion."

By integrating A/B testing throughout the organization, you’re not only creating new data. You’re also creating new opportunities for more informed decision-making.

This article was created for and commissioned by Adobe, and the views expressed are their own.