What the end of the Twitter-DataSift partnership means for customers


Photo: Bob Kelly, cc 2.0


Let’s get this out of the way right up front. DataSift is a client of mine at Altimeter Group. I am closely connected to Twitter via my role as a board member of the Big Boulder Initiative, of which Chris Moody, VP Data Strategy at Twitter, is chair. 

On Friday, Zach Hofer-Shall, head of Twitter’s ecosystem, published a post on the Gnip/Twitter blog entitled “Working Directly With the Twitter Data Ecosystem.” Shortly thereafter, Nick Halstead, CEO of DataSift, published a post (since updated) entitled “Twitter Ends its Partnership with DataSift – Firehose Access Expires on August 13, 2015“. The next day, Tim Barker, DataSift’s Chief Product Officer, added another take: “Data Licensing vs. Data Processing.”

In a nutshell, this means that Twitter has ended its reseller agreements with third parties, and, going forward, will be the sole distributor of Twitter data.

To be clear, this does not mean that Twitter is discontinuing firehose access; in the future, Twitter will control its own data relationships rather than licensing data through resellers. 

These posts have sparked a flurry of commentary across a wide spectrum, from support to vilification to philosophizing on the meaning of platforms, analytics and ecosystems. I’ve included links to a few at the end of this post.

It wasn’t a huge surprise for anyone watching Twitter go public, and subsequently disclose the revenues from the direct data business, to anticipate that Twitter might realize that this was an area ripe for a significant strategy shift. And it was a short hop from there to conclude that Datasift’s (and possibly other’s) days of reselling data received via the Twitter firehose might be numbered.

It also hasn’t been a surprise to see Twitter enhance its analytics, re-evaluate the short-and-long-term value of its data and announce strategic partnerships such as the ones with IBM and Dataminr as it seeks to build its partner strategy and revenue potential.

Meanwhile, DataSift has continued to execute on its own strategy, which includes broadening its data sources far beyond social data, announcing VEDO, its data categorization platform, and via its developing its privacy-first PYLON technology (see its announcement with Facebook on how they are providing privacy-safe topic data).

Long story short: no one was shocked at the news. But the reaction to it has been polarizing in the extreme. What seems to have fanned the flames are a few different dynamics:

  • The context of the announcement, including Twitter’s recent moves against Meerkat and a historically fraught relationship with its ecosystem
  • The fact that Gnip and Datasift were competitors before the acquisition, complicating DataSift’s relationship with Twitter
  • Datasift’s strong public reaction to the news; and to some extent, the timing of the announcement, which came late on a Friday; a classic—though ineffective—news burial tactic.

It also doesn’t help that Twitter has been (except for the original post) all but silent this past week. But that shouldn’t come as a surprise to anyone either. It’s a public company, and, as such, required to comply with regulatory requirements governing communications with the public. According to Nasdaq, Twitter is expected to report earnings on April 28. So they’re in quiet period, and, as will surprise no one, won’t talk about confidential negotiations between parties.

As a privately-held company, however, DataSift has more leeway to comment publicly. I’m not going to repeat their position here; it is clearly stated in several posts on the DataSift blog.

But none of this gets at the most important issue: the impact of this decision on customers and users of Twitter data. Here are a few constituencies to consider:

1. Companies that sell social technology

To gauge the impact, you need to consider how these companies gained access to Twitter data before now:

How they get Twitter data Impact Implications
Directly from Twitter, via firehose or public API No change None
From Gnip (and therefore now from Twitter) No change None
From third-party resellers such as DataSift Ends August 13, 2015 Must re-assess how they migrate from DataSift to Twitter

But this requires a bit of context.

Before the acquisition, there was a reason companies—social technology or enterprise—would select Gnip or DataSift (or, before its acquisition by Apple, Topsy) if they wanted direct access to Twitter data: they had different value propositions.

DataSift positioned itself as a platform in which the access to social and other types of data came with value-adds such as advanced filtering, enrichments, taxonomies, machine-learning and other sophisticated data processing capabilities to enable users to derive insights from the data.

Gnip, on the other hand, was a simpler, less expensive option: they offered enrichments, but the value proposition was historically more about simplicity and reliability than sophisticated data processing. This tended to be an easy calculation for a lot of social tech companies who wanted to add their own capabilities to the data.

So, speaking broadly, analytics or social technology companies (even brands) who could or wanted to handle raw data would have been better suited to Gnip. Those who wanted a more plug-and-play product that processed data consistently across multiple sources would more likely favor DataSift. Once Twitter acquired Gnip, it didn’t take a team of data scientists to conclude that Twitter had bigger plans for its data business, and that a lot of that development would happen under the waterline, as these things tend to do.

But that doesn’t eliminate the very real issue that migration is going to be highly disruptive for customers of DataSift.

But there is another data point that’s important to consider.

Unlike most industry shifts, it’s been very difficult to get any social analytics companies to talk on the record about this news. On background, some stated that they were never quite sure whether DataSift intended to be a partner or a competitor because they weren’t a pure reseller; the platform—with its ability to perform sentiment analysis, enrichments, and provide classifiers and taxonomies—pushed it uncomfortably for some into analytics territory.

Some said they’re concerned about Twitter’s plans as well. Now that Twitter has discontinued data licensing, what will it now monetize? Will they take more control of  or develop their own analytics? If not, what then?

This is unsettling for some in the social analytics community, who are also being buffeted by business intelligence and marketing/enterprise cloud companies (think Oracle, Salesforce, Adobe) eager to wrap social data into a broader insight offering. It’s a time of shifting strategies and shifting alliances.

2. End users of technology (brands and enterprise)

For the most part, end users of Twitter data don’t have much to worry about, unless they are current or potential DataSift customers and can’t (or don’t want to) ingest the firehose in its raw form. If they are, they’ll need to migrate to Twitter, and assess the extent to which Twitter is currently (or via roadmap) able and willing to provide the type of processing they need.

If enterprises get their Twitter and other social data from social analytics providers, they are more insulated from this news. The question I would ask is whether and how Twitter intends to normalize data from other social media platforms. Will users have a clear sightline across multiple social (and other) data sources? Will they analyze more than Twitter data? Will they handle that through partnerships (and if so, with whom?) The ability to normalize data across sources has been a clear value proposition for DataSift; less so from Gnip, especially since the acquisition by Twitter. And of course we can’t discount the fact that Twitter likely has more up its sleeve than its able to disclose right now.

3. Agencies, consultancies and other professional services firms

Agencies can be affected in any number of ways, based upon their social (and other) analytics strategy and business model. Those who offer their own bespoke analytics would do well to learn more about Twitter’s data roadmap; how much will be product and how much service. Those who use a mix of analytics tools would likely be less affected.

As for professional services firms, there is a tremendous amount of opportunity in custom analytics for enterprise. The challenge is that 1) data processing isn’t core to that business 2) developing custom analytics doesn’t scale well and 3) Twitter data, especially in the context of other social, enterprise and external data, is extremely complex. As a result, professional services firms will need to approach Twitter to better understand what the company will and won’t offer in the future and where the synergies lie. Either way, it’s going to be a delicate dance.

For all DataSift customers, Tim Barker’s post is a shot across the bow for Twitter; even if Twitter disagrees with his assessment of the impact of terminating the reseller agreement, it’s a starting point for customers to begin their conversations with Twitter and suss out what exactly a shift to a direct relationship might entail.

One other option is for customers to bring Twitter data into DataSift via a Gnip connector. DataSift is working on that now.

A few last thoughts

In the end, a lot is still unknown, and Twitter’s silence is enabling people to fill the void with speculation, which creates uncertainty and doubt among those whose businesses depend on to some extent on Twitter. But in my opinion, all of this is an inevitable if painful blip for both companies: DataSift will move on, and Twitter will continue to build out its data business, which will likely create even more uncertainty in the social data ecosystem for some time to come.

But, as one person who spoke off the record rather philosophically commented, “In a world where you’re dealing with third-party data, you can never be completely comfortable.”

 Other points of view




About susanetlinger

Industry Analyst at Altimeter Group
This entry was posted in Analytics, Big Boulder Initiative, DataSift, Facebook, Sentiment Analysis, Social Analytics, Social Data, Social media, Social media measurement, Twitter, Uncategorized. Bookmark the permalink.

2 Responses to What the end of the Twitter-DataSift partnership means for customers

  1. Pingback: What the End of the Twitter-DataSift Partnershi...

  2. Rob Key says:

    Nice review Susan. I will go on the record here and say that i think this announcement — long in the making — goes cross-current to what we thought at one time would happen: data aggregation would get consolidated. Instead it is getting increasingly fractured, which, in our view, means that analytics platforms/solutions (like ours at Converseon/Revealed Context) are now increasingly needed to bring help together the disparate streams, normalize them and apply consistent text analytics/sentiment analysis/custom classifiers to the data tuned to clients needs — with the goal of ensuring “one truth” across all the data and fully integrated into organizations. It has also become quite clear to us that the critical “last mile” of analytics is rapidly becoming the most important now. Many solutions can get you to the proverbial 20 yard line, but getting through the redzone to the the goal line requires custom classifiers, specific taxonomy training and more sophisticated relevancy segmentation to get the data to the point that it’s most useful, aligned to business KPIs, can be mainstreamed into organizations and leverage its predictive capabilities. In our view that this has to happen from the “client-out” perspective versus further up at the platform level/aggregation level. This demand for custom social analytics should help drive the creation for more marketplaces, API integration, etc. with key platforms Thanks for sharing your perspective.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s