Wednesday, August 22, 2018

The complicated statistics and conflicting objectives of video recommendation algorithms

What is the objective of the algorithm and how do we know if it's achieving that goal? Well, that depends. The things we would really like to know like impact on long-term customer loyalty and retention, particularly in the event of price increases and/or loss of popular content. Unfortunately, getting an answer to this question that does not entail waiting five or 10 years is extremely difficult.

I suspect the most popular alternative would be simply measuring how often customers accept the recommendation, but that metric is deeply problematic even under the best of circumstances and is almost worthless if approached naïvely.

A big part of the problem is that customers almost always walk into the door with some kind of mental queue, albeit often a vague and incomplete one. A recommendation algorithm that does not change that queue in a nontrivial way is a failure, a complete waste of time and money.

[I almost veered off into a discussion of how models and algorithms interact in this situation, but I'm trying to stay on topic and that's a subject that really needs a serious post or perhaps even thread of its own.]

Now we get to the next level of complexity. If you fail to change the queue you accomplish nothing, but if you do change the queue you still don't necessarily accomplish anything of business value. This is where we need to start getting specific about our objectives because there are some perfectly reasonable but contradictory choices to consider

If the sole concern is serving the customer (and serving the customer is never the sole concern) then the goal might be to get the viewers to watch and give a high ratings to shows they were previously unaware of. Another interesting related metric might be to look at before and after ratings, comparing how much they expected to enjoy a program with how much they actually did.

While great for viewers, this can very easily turn around and bite the company in the ass. For example, if Netflix gets viewers interested in classic cinema, they are likely to start migrating to other services with far better cinephile collections. Filmstruck is probably the best-known of these, but here in LA and possibly in your town as well, the public library offers a free streaming service which includes the Criterion Collection, a catalog which Netflix can't possibly compete against.

So chances are the business not only needs to change the queue; it needs to direct the viewers toward certain programs.

Before we pursue this any further, we need to remember that convincing a person to watch a movie or a TV show that he or she is unfamiliar with has never been easy and has grown far more difficult in recent years. For the first three or so decades of television as a mass medium (let's call it 1950 to 1980 just to have some round numbers), you basically had three networks to choose from. Some minor players popped up – – PBS, independent stations, a few abortive attempts at a fourth network – – but most people spent most of their time watching CBS, NBC, or ABC (usually in that order). Simply getting a show on the air guaranteed enough of an audience for word to get around.

If you want to build the IP value of a television show in 2018, the best way to do it is probably still going the route of a prime time network run and a wide syndication release, but that paradigm is clearly fading. Despite what you may have heard, no one has come up with anything close to replacing it. As a result, companies are either relying on established properties that achieved high name recognition status under the old paradigm or are desperately trying to prop up new properties     with dump trucks full of marketing and PR money. Under these circumstances, the suggestion that the viewership for the programming being produced by the streaming services is driven by recommendation engines should be taken with extreme skepticism.

Furthermore, at the risk of being cynical, it would probably be a good idea to approach any story about the role of recommendation algorithms in the world of online video with the assumptions that the sources for the stories have a strong incentive spin them a certain way and that there is a very good chance that those sources don't know what they're doing.

No comments:

Post a Comment