The myth of a data-driven Netflix

There's been a lot of talk about Netflix stock this week (usually with words like "plummet"), but a big part of the story has largely gone unnoticed, probably in part because it involves statistics.

As mentioned before, some aspects of the Netflix narrative such as the company building and HBO type content library, are simply, factually incorrect. Others, while not blatantly wrong, are difficult to reconcile with the facts.

One of the accepted truths of the Netflix narrative is that CEO Reed Hastings is obsessed with data and everything the company does is data driven (for example "What little Netflix has also shared about its programming strategy is that its every decision is guided by data."). The evidence in support of this belief is largely limited to a model that Netflix crowd sourced a few years ago and to endless assertions from executives at the company that they do know what they are doing despite evidence to the contrary.

Of course, all 21st century corporations are relatively data-driven. The fact that Netflix has large data sets on customer behavior does not set it apart, nor does the fact that it has occasionally made use of that data. Furthermore, we have extensive evidence that the company often makes less use of certain data then do most other competitors.

On pertinent case in point, particularly for the SEC, is churn rates.

But Netflix disagrees. “With respect to various operational metrics, management has evolved its use of these metrics as the business has evolved,” it wrote the SEC in response. Because it is so easy to quit and then restart a Netflix subscription, it said, “the churn metric is a less reliable measure of business performance, specifically consumer acceptance of the service.”

This is problematic on any number of levels. In terms of marketing, pricing and long-term corporate strategy, having a complete picture of how long people stay and why they leave is huge. The only excuse for not reporting churn would be if you had such a detailed picture of who was leaving and why that this additional metric was redundant.

In other words, Hasting should have a good, data-supported explanation for a recent sudden loss of subscribers.

Netflix CEO Reed Hastings blamed the subscriber drop-off on a $1 price increase the company instituted back this spring.

"Our best sense is it's an effect of our price increase back in May," Hastings said Wednesday night in an interview with CNBC. "With a little bit higher prices, you get a little bit fewer subscribers. So that's our sense of it. But we can't be 100% sure. We had so much benefit from Orange in Q2 and the early Q3, but that's what we think."

Phrases you don't want to hear in these circumstances include "our best sense" and "that's what we think." They convey the impression of a CEO who was blindsided by a bad day at the NASDAQ.

When contemplating a price increase, well-run companies look at the impact on retention and on acquisition. When Netflix management said

[M]anagement believes that in a largely fixed-cost streaming world with ease of cancellation and subsequent rejoin, net additions provides the most meaningful insight into our business performance and consumer acceptance of our service. The churn metric is a less relevant and reliable measure of business performance, and does not accurately reflect consumer acceptance of our service.

They were basically saying that losing one customer and gaining another is the same as keeping the same customer. That's a dangerous approach under the best of circumstances but it can be deadly when trying to gauge the impact of a pricing change.

Just to be clear, for years analysts and the SEC have been asking for more data, or at least more detailed statistics and Netflix has been saying "trust us, the aggregate number are good enough." Now the company appears to have screwed up badly, and they've done it in pretty much exactly the way you would expect a company to screw up when it doesn't drill down into the data.

p.s. I'm considering putting out a collection of business posts (something similar to the education reform e-book Things I Saw at the Counter-Reformation). Any thoughts or suggestions would be appreciated.

Netflix, the Emmys and the power of a happy narrative

I probably should have been more explicit about this but the Netflix thread* always consisted of two intertwined sub-threads: The first was a discussion of the business model and methods and how they fit in with the larger television industry; the second concerned the media narrative and the way journalists and pundits found ways to justify the story they wanted to tell.

Last night something big happened to both of those threads. For perhaps the first time in recent memory, the Emmys actually mattered.

To understand the impact on the first thread, you have to take a detailed look at the most likely business model for Netflix. Most likely here does not mean most commonly given. That would be that Netflix hoped to bring in enough additional subscribers to justify the hundred million dollars they spent on House of Cards. Even the numbers from Netflix don't seem to support that claim and as certain shrewd observers (such as Mark Rogowsky and Rocco Pendola), some of the numbers we get from Netflix are questionable.

The far more credible model was that the company was as interested in the prestige and PR value associated with the perception of having a critical and commercial hit. Thus if House of Cards brought in an additional $50 million through new subscribers and decreased churn and also provided the indirect benefits from reputation then the deal made financial sense.

At this point it's useful to step back and compare the way Netflix developed original programming to what we might call the HBO model, also practiced by FX, USA, TNT, and AMC. The HBO model normally starts by hiring a collection of respected B-list talents such as David Chase, Timothy Olyphant, or Louis CK and finding promising C-list talents such as James Gandolfini or Jon Hamm. This increases the risk of failure on the individual show level but it produces a good aggregate ROI and it makes you look like a genius when someone people have never heard of becomes the next big thing.

Netflix took largely the opposite approach with H of C, spending lavishly for hot property, an A-list star and and the director of such films as Seven, Fight Club, Zodiac, and The Social Network. As mentioned before, this expense is difficult to justify strictly in terms of subscribers and churn but it could make sense if the show generates sufficient publicity and has a substantial brand effect.

This is where the Emmy's come in. For a network show an Emmy is not that big of a deal. It can give a nudge to an on-the-bubble series but it generally doesn't translate to a big bump in absolute numbers. It is, however, a valuable branding tool and, more importantly, it's an indicator of branding success. Thus, when AMC was trying to rebrand itself a few years ago, that first award for Mad Men really was a big deal.

Under these circumstances, there's no way that Netflix is happy about the results of these weekend's awards. They wrote huge checks to get House of Cards and Arrested Development (without even managing to acquire all the IP rights). They plastered LA with "For Your Consideration" ads. For all that, they had a poor showing in the nominations and a worse showing on Sunday. After all this effort they got a couple of meaningless "Creative Arts" Emmys (casting and cinematography) and the second tier major, directing (which, given David Fincher's reputation, was almost a given).

Ideally, when a company has devoted this kind of resources to a goal and that goal doesn't pan out, the company will at least reexamine its strategy and tactics, but Netflix isn't really in a position to do that and one of the reasons for that inability to react involves the other thread.

Netflix stock is running very high at the moment and Reed Hastings, Theodore Sarandos and other high ranking people at the company have raked in huge amounts of money in large part because of a pervasive and overwhelmingly positive official narrative. The admission that the model needs work could easily end up costing these executives tens of billions of dollars.

More importantly, the journalists covering the story are working very hard to see to it that Hastings never has to make that admission.

Here's the Wall Street Journal:

If Hollywood wasn't already taking Netflix seriously, it is now.

The streaming video service scored a win at the TV industry's Emmy Awards on Sunday night as David Fincher took the best director prize for political drama "House of Cards." It marked the first victory in a major category for an online video distributor.

The Emmy win could boost Netflix's prestige in Hollywood as an outlet for high-quality original series and further encourage writers, producers and actors to consider Netflix projects at a time when competition for talent among TV networks is as fierce as ever.

You can argue that the Emmys don't matter, but you can't argue that Netflix's one win in a category no one cares about constituted a good night. (Unfairly or not, TV is not considered to be a director's medium. Fincher didn't even show up to collect the statue.) However, we have reached the point in the journalistic narrative process where response is not all that dependent on the nature of the stimulus. Anything short of complete disaster will prompt cheerful, bullish stories. That's what the narrative calls for.

p.s. I dictated much of this using my smartphone. I believe I got all of the homonyms but I apologize if some slipped by me.

* Among others:

Edging away from the genius hypothesis

Netflix can never be the next HBO

Curiously, agressively anti-social

Two quotes about Netflix, presented (almost) without comment

House of Cards -- either already in the black or seriously underperforming