Last week, M-Lab, a group that monitors select Internet network links, issued a report claiming interconnection disputes caused significant declines in consumer broadband speeds in 2013 and 2014.
This was not news. Everyone knew the disputes between Netflix and Comcast/Verizon/AT&T and others affected consumer speeds. We wrote about the controversy here, here, and here, and our “How the Net Works” report offered broader context.
The M-Lab study, “ISP Interconnection and Its Impact on Consumer Internet Performance,” however, does have some good new data. Although M-Lab says it doesn’t know who was “at fault,” advocates seized on the report as evidence of broadband provider mischief at big interconnection points.
But the M-Lab data actually show just the opposite. As you can see in the three graphs below, Comcast, Time Warner, Verizon, and to a lesser extent AT&T all show sharp drops in performance in May of 2013. Then network performance of all four networks at the three monitoring points in New York, Dallas, and Los Angeles all show sudden improvements in March of 2014.
The simultaneous drops and spikes for all four suggest these firms could not have been the cause. It would have required some sort of amazingly precise coordination among the four firms. Rather, the simultaneous action suggests the cause was some outside entity or event. Dan Rayburn of StreamingMedia agrees and offers very useful commentary on the M-Lab study here.
As it happens, Netflix was moving much of its content away from third-party content delivery networks (CDNs) in the spring of 2013 and onto its own OpenConnect platform, which used Cogent and Level 3 to connect to many broadband providers. The smaller cable firms, Cablevision and Cox, meanwhile, had agreed to connect to Netflix for free and unsurprisingly show no degradation.
Nine months of degraded performance ensued. Then, in the spring of 2014, Netflix agreed to connect directly to the networks of the major BSPs, first Comcast, then Verizon, then Time Warner and AT&T. General performance on most networks across the Internet improved. The M-Lab data actually reinforce what we had already pieced together: Netflix moving its traffic onto the networks of Cogent and Level 3 overwhelmed both the capacity of those networks and blew through their peering agreements with the broadband providers.
Network analyst George Ou offered his perspective of what had happened: “Cogent purchased on the order of 1 terabit of capacity from Comcast and got 4 terabits from Comcast. When I say ‘purchased’, I mean bartered. Then they turned around and sold 100 terabits of capacity to [their] own customers. That’s why Cogent customers are suffering slow performance. Then they demanded 8x the bandwidth from Comcast at no extra cost and when Comcast refused, they blamed their slow service on Comcast.”
When Netflix and Comcast finally agreed to connect in the late winter of 2014, it improved performance for everyone. Around 30% of all Netflix’s traffic (which itself can account for a third of all peak-hour U.S. traffic) had been removed from the oversold Cogent and Level 3 networks, thus opening up capacity for those backbones’ connections to the other broadband providers. Additional direct connections between Netflix and the other BSPs then improved things further.
A previous analysis by Peter Sevcik of NetForecast last summer showed this phenomenon of a rising tide after the Comcast/Netflix hook-up (see below). M-Lab thus confirmed what many had already revealed, a non-broadband provider cause of the slow down, exactly the opposite of the media interpretation of the M-Lab study.
But wait! As we were about to post this analysis, we got word via Dan Rayburn that Cogent — quite amazingly — just admitted that in February 2014 it instituted a new fast-lane / slow-lane quality-of-service regime on its network. M-Lab’s monitoring equipment was actually given fast-lane status! This is ironic on a number of levels — and we still need to figure out if this was coordinated or coincident with the Comcast-Netflix hookup — but it may help explain why the performance improvement across Cogent-BSP links as shown by M-Lab was even more sudden and dramatic than the Netflix improvement noticed elsewhere. And of course, it reinforces, yet again, the real source of the slow-down.
