Netflix represents the overlap of the two great traditions of creative accounting-- Hollywood and Silicon Valley

[I scheduled this a few days ago. Eventful days.]

Since writing this, I've run the story past a number of people who have worked with this kind of consumer data at large companies, and every one of them has had the same reaction. All of them suggested that Netflix was tracking these numbers; it just didn't want anyone to know what they said. I'm also coming around to that conclusion.

I have a feeling that Netflix's transparency is about to become a bigger part of this story.

Monday, October 20, 2014

The myth of a data-driven Netflix

There's been a lot of talk about Netflix stock this week (usually with words like "plummet"), but a big part of the story has largely gone unnoticed, probably in part because it involves statistics.

As mentioned before, some aspects of the Netflix narrative such as the company building and HBO type content library, are simply, factually incorrect. Others, while not blatantly wrong, are difficult to reconcile with the facts.

One of the accepted truths of the Netflix narrative is that CEO Reed Hastings is obsessed with data and everything the company does is data driven (for example "What little Netflix has also shared about its programming strategy is that its every decision is guided by data."). The evidence in support of this belief is largely limited to a model that Netflix crowd sourced a few years ago and to endless assertions from executives at the company that they do know what they are doing despite evidence to the contrary.

Of course, all 21st century corporations are relatively data-driven. The fact that Netflix has large data sets on customer behavior does not set it apart, nor does the fact that it has occasionally made use of that data. Furthermore, we have extensive evidence that the company often makes less use of certain data then do most other competitors.

On pertinent case in point, particularly for the SEC, is churn rates.

But Netflix disagrees. “With respect to various operational metrics, management has evolved its use of these metrics as the business has evolved,” it wrote the SEC in response. Because it is so easy to quit and then restart a Netflix subscription, it said, “the churn metric is a less reliable measure of business performance, specifically consumer acceptance of the service.”

This is problematic on any number of levels. In terms of marketing, pricing and long-term corporate strategy, having a complete picture of how long people stay and why they leave is huge. The only excuse for not reporting churn would be if you had such a detailed picture of who was leaving and why that this additional metric was redundant.

In other words, Hasting should have a good, data-supported explanation for a recent sudden loss of subscribers.

Netflix CEO Reed Hastings blamed the subscriber drop-off on a $1 price increase the company instituted back this spring.

"Our best sense is it's an effect of our price increase back in May," Hastings said Wednesday night in an interview with CNBC. "With a little bit higher prices, you get a little bit fewer subscribers. So that's our sense of it. But we can't be 100% sure. We had so much benefit from Orange in Q2 and the early Q3, but that's what we think."

Phrases you don't want to hear in these circumstances include "our best sense" and "that's what we think." They convey the impression of a CEO who was blindsided by a bad day at the NASDAQ.

When contemplating a price increase, well-run companies look at the impact on retention and on acquisition. When Netflix management said

[M]anagement believes that in a largely fixed-cost streaming world with ease of cancellation and subsequent rejoin, net additions provides the most meaningful insight into our business performance and consumer acceptance of our service. The churn metric is a less relevant and reliable measure of business performance, and does not accurately reflect consumer acceptance of our service.

They were basically saying that losing one customer and gaining another is the same as keeping the same customer. That's a dangerous approach under the best of circumstances but it can be deadly when trying to gauge the impact of a pricing change.

Just to be clear, for years analysts and the SEC have been asking for more data, or at least more detailed statistics and Netflix has been saying "trust us, the aggregate number are good enough." Now the company appears to have screwed up badly, and they've done it in pretty much exactly the way you would expect a company to screw up when it doesn't drill down into the data.