Papa Fouche: The A/B Test: Inside the Technology That's Changing the Rules of Business

.ab_strike { color: #000000; font-size: 1.3em; font-weight: bold; margin-bottom: 5px; text-decoration: line-through; }

Illustration: Si Scott

You have to make choices.

Choose everything.

The online payment platform WePay designed its entire homepage through a testing process. “We did it as a contest,” CEO Bill Clerico says. “A few of our engineers built different homepages, and we just put them in rotation.” For two months, every user that came to WePay.com was randomly assigned a homepage, and at the end the numbers made the decision.

In the past, that exercise would have been impossible—and because it was impossible, the design would have emerged in a completely different way. Someone in the company, perhaps Clerico himself, would have wound up choosing a design. But with A/B testing, WePay didn’t have to make a decision. After all, if you can test everything, then simply choose all of the above and let the customers sort it out.

For that same reason, A/B increasingly makes meetings irrelevant. Where editors at a news site, for example, might have sat around a table for 15 minutes trying to decide on the best phrasing for an important headline, they can simply run all the proposed headlines and let the testing decide. Consensus, even democracy, has been replaced by pluralism—resolved by data.

The mantra of “choose everything” also becomes a way for companies to test out relationships with other companies—and in so doing becomes a powerful way for them to win new business and take on larger rivals. In 2011 a fund-raising site called GoFundMe was talking with WePay about the possibility of switching to its service from payment giant PayPal. GoFundMe CEO Brad Damphousse was open about his dissatisfaction with PayPal’s service; WePay responded, as startups usually do, by claiming that its product solved all the problems that plagued its larger competitor. “Of course we were skeptical and didn’t really believe them,” Damphousse recalls with a laugh.

But using A/B, WePay could present Damphousse with an irresistible proposition: Give us 10 percent of your traffic and test the results against PayPal in real time. It was an almost entirely risk-free way for the startup to prove itself, and it paid off. After Damphousse saw the data on the first morning, he switched half his traffic by the afternoon—and all of it by the next day.

Photo: Spencer Higgins

The person at the top makes the call.

Data makes the call.

Google insiders, and A/B enthusiasts more generally, have a derisive term to describe a decisionmaking system that fails to put data at its heart: HiPPO—”highest-paid person’s opinion.” As Google analytics expert Avinash Kaushik declares, “Most websites suck because HiPPOs create them.”

Tech circles are rife with stories of the clueless boss who almost killed a project because of a “mere opinion.” In Amazon’s early days, developer Greg Linden came up with the idea of giving personalized “impulse buy” recommendations to customers as they checked out, based on what was in their shopping cart. He made a demo for the new feature but was shot down. Linden bristled at the thought that the idea might not even be tested. “I was told I was forbidden to work on this any further. It should have stopped there.”

Instead Linden worked up an A/B test. It showed that Amazon stood to gain so much revenue from the feature that all arguments against it were instantly rendered null by the data. “I do know that in some organizations, challenging an SVP would be a fatal mistake, right or wrong,” Linden wrote in a blog post on the subject. But once he’d done an objective test, putting the idea in front of real customers, the higher-ups had to bend. Amazon’s culture wouldn’t allow otherwise.

Siroker recalls similar shifts during his time with the Obama campaign. “It started as a pretty political environment—where, as you can imagine, HiPPO syndrome reigned supreme. And I think over time people started to see the value in taking a step back and saying, ‘Well, here’s three things we should try. Let’s run an experiment and see what works. We don’t know.’”

This was the culture that he had come from at Google, what you might call a democracy of data. “Very early in Google’s inception,” Siroker explains, “if an engineer had an idea and had the data to back it up, it didn’t matter that they weren’t the VP of some business unit. They could make a case. And that’s the culture that Google believed in from the beginning.” Once adopted, that approach will beat the HiPPOs every time, he says. “A/B will empower a whole class of businesses to say, ‘We want to do it the way Google does it. We want to do it the way Amazon does it.’”

Says WePay’s Bill Clerico: “On Facebook, under the heading of Religious Views, my profile says: ‘In God we trust. All others, bring data.’”

Pages: 1 2 3 View All

You Might Like
Related Links by Contextly