Before You Freak Out About Pinterest and the Zappos Study...

Image courtesy Stock.Xchng

A lot of people have been buzzing today over this Bloomberg story both announcing Zappos’ launch of a recommendation engine based on Pinterest pins, and stating that–although Pinterest drives the highest number of shares compared to Facebook and Twitter–it also contributes the lowest revenue of the three.

Several publications picked up the thread this morning (Laura Hazard Owen of GigaOm wrote a story including some pointed criticism of the Zappos recommendation engine, while Business Insider posted this widely-shared piece), and the takeaway–that Pinterest drives low revenue for brands–has been widely circulated on the interwebs.

This is odd on several levels; why would Zappos tout a new recommendation engine on the one hand, while admitting it’s not effective at driving revenue on the other? But before we jump to conclusions, let’s take a look at the Bloomberg story and ask a few pointed questions.

1. What was the methodology of the study?

This is a tough one, because we don’t have the data or know the methodology of the Amazon/Zappos analysis. Some questions to consider:

• How many sales/users does this study represent?
• Do these numbers represent average order value per social platform, or average order value per share?
• For that matter, how was a share defined and calculated? Is a pin being equated to a tweet or a facebook wall post? What are the precise metrics being compared?
• What was the time frame of the study? A week? A month? A quarter? More?

2. What methodology was used for conversion attribution?

The way the conversions were attributed will have a dramatic impact on the results. Questions to consider:

• How was the data tracked? Was it tagged? If so, and if the analytics tool is using last-click attribution, the conversion rates could be (not saying they are, but could be) under counted because of the fact that people rarely see content and convert (i.e., buy) within a single browser session. Sometimes you see that sweet hat (shirt, painting, necklace, muscle car, mansion, whatever), then you go away, come back and buy weeks or even months later–on another device, even. (Note: this is why time frame of the study is so important). On the other hand, if you did an A/B test (Within a six-month period, Pinterest users spent an average of $X, non-Pinterest users spent an average of $Y), you could get a different view of the revenue generation value of Pinterest vs. non-Pinterest users.
• What is the content of the posts from Facebook and Twitter?  Is it just straight shares, or does it include coupon-related affiliate links? You need to know whether this was an apples-to-apples comparison to determine the relative revenue generation performance of the social platforms. No fair to compare a share to a coupon, whatever social network it comes from.

3. How accurate is the recommendation engine?

It’s too new and we have too little experience to have a real sense of this, but it stands to reason that the quality of the recommendation engine will affect conversion results.  Here’s an example from the GigaOm story: the fact that I pinned a black-and-white dress does not actually make me more likely to buy a men’s sky-blue tie. But maybe that’s just me.

4. How do people usually behave on these social networks?

Here’s where we need to think about socialgraphics; that is, how people actually use social networks. Some points to consider:

• Age of the Network. Facebook and Twitter have been around for several years. Pinterest is still relatively new, with a lot of new users who are still figuring it out.
• Behavior on the network. Twitter is a fast-moving 140 characters, the Facebook Timeline passes quickly but allows for more editorializing, yet Pinterest allows for a more permanent record of one’s preferences.  As a result, something I pin in September might influence your purchase in March. You cannot track that with last-click attribution, so Pinterest would not get credit for that sale. You have to ask: does that seem logical, given how high the sharing behavior is on Pinterest?  Again, we don’t have the data, but a starting question might be: do one or more social networks have the advantage when it comes to last-click attribution? That could explain why Twitter scored so high.  Is there another way to prove revenue impact across Pinterest, Facebook and Twitter? Does it tell us something different, or does it confirm what we found the first time? How does it compare to other studies, such as the research released earlier this year by Shopify?
• Relationships on the network. How close are the ties between Facebook friends, Twitter followers, Pinterest pinners? We are still very early on in our understanding of how people in a social graph are influenced by each other, relative to how people in an interest graph behave.

Finally, there are just too many unanswered questions to draw a conclusion about Pinterest based on the Amazon/Zappos data. But this story raises some really important issues about how we measure revenue impact of social media, because it highlights the analytical traps that we’re vulnerable to if we assume that all networks behave in the same way.

My advice, whether you’re concerned about this particular study or the revenue contribution from your own social media programs, is to use this example as a thought-starter to make sure that you are asking the right questions, drawing useful results and–of course–making informed decisions.

For more information about how brands are determining the revenue contribution of social media, please see The Social Media ROI Cookbook.

I’d love your thoughts. Any questions you have? Points I missed? Please tweet, comment or share however you prefer.