Netflix keeps the ratings for its shows a closely guarded secret, primarily to maintain a strategic advantage over competitors. If Amazon, for instance, knew that Netflix’s top two shows were, say, Fuller House and Stranger Things, that would certainly help inform the type of original programming Amazon would seek to develop. What’s more, Netflix’s chief content officer Ted Sarandos has said that releasing ratings data would be counterproductive because some content is released niche audiences.
“There is a very natural inclination to say, ‘Relative to this show, this show is a failure.'” Sarandos explained last year. “That puts a lot of creative pressure on the talent.”
In light of Netflix’s secrecy with respect to show ratings, third-party companies have come up with roundabout ways to figure out what Netflix’s massive 109 million subscribers tend to watch most often. Most recently, Nielsen began boasting about a new service that relies upon audio data to figure out what shows and episodes Netflix subscribers are watching.
Here’s how it all works. Nielsen’s technology is essentially an opt-in service that monitors audio when users are watching Netflix on their HDTV via a set-top box. The statistical problems with this approach are numerous. For starters, Nielsen’s technology — as an opt-in service — is prone to self-selection bias. Imagine, for example, that the demographic of users likely to agree to use Nielsen’s technology are sci-fi enthusiasts. This would skew Nielsen’s ratings towards programs like Stranger Things and perhaps downplay the popularity of crime thrillers like Mindhunter.
More importantly, Nielsen’s technology doesn’t incorporate any data from content watched via mobile devices like smartphones and tablets. Given how many hours of content people digest on mobile devices, Nielsen’s methodology is embarrassingly incomplete and flawed from the very start. To this point, Netflix just a few days ago revealed that 67% of subscribers tend to watch content outside of their living rooms, which is to say outside of the realm of set-top boxes.
The New York Times also adds:
Additionally, Nielsen’s data comes only from customers in the United States, and Netflix has made an aggressive play in international markets. Of its roughly 104 million paying subscribers, a little more than half live outside the United States, according to Netflix’s third-quarter earnings.
Earlier efforts to sleuth out streaming ratings have faced criticism. Last year, NBC executives released some Netflix ratings data from a company called Symphony Advanced Media, which similarly used audio fingerprinting technology from a panel of 15,000 people who downloaded a special app.
All in all, it’s no wonder that Sarandos has said that ratings estimates for Netflix shows are completely inaccurate.