Study: Comcast and Verizon connections to Cogent dropped below 0.5Mbps

Plenty of Comcast and Verizon customers know just how bad Internet service was on major ISPs during the months-long battle over who should pay to deliver Netflix traffic.

But now we have more numbers on the performance declines, thanks to a new report from the Measurement Lab Consortium (M-Lab). M-Lab hosts measuring equipment at Internet exchange points to analyze connections between network operators and has more than five years' worth of measurements. A report released today examines connections between consumer Internet service providers ("Access ISPs" in M-Lab parlance) and backbone operators ("Transit ISPs"), including the ones that sent traffic from Netflix to ISPs while the money fights were still going on.

Netflix eventually agreed to pay Comcast, Verizon, Time Warner Cable, and AT&T for direct connections to their networks, but until that happened there was severe degradation in links carrying traffic from Netflix and many other Web services to consumers. Connections were particularly bad between ISPs and Cogent, one of the backbone operators that Netflix paid to carry its traffic.

"Using Measurement Lab (M-Lab) data, and constraining our research to the United States, we observed sustained performance degradation experienced by customers of Access ISPs AT&T, Comcast, CenturyLink, Time Warner Cable, and Verizon when their traffic passed over interconnections with Transit ISPs Cogent Communications (Cogent), Level 3 Communications (Level 3), and XO Communications (XO)," researchers wrote. "In a large number of cases we observed similar patterns of performance degradation whenever and wherever specific pairs of Access/Transit ISPs interconnected. From this we conclude that ISP interconnection has a substantial impact on consumer internet performance—sometimes a severely negative impact—and that business relationships between ISPs, and not major technical problems, are at the root of the problems we observed."

M-Lab was founded by the New America Foundation's Open Technology Institute, the PlanetLab Consortium, Google, and academic researchers. M-Lab made its full dataset available online and encouraged further research.

New Yorkers suffered most

While M-Lab observed degraded performance across the country, the worst it found was in the New York City connections between Cogent and ISPs. Cogent exchanges traffic with ISPs without payment, but the ISPs demanded money when Cogent was sending more traffic than it received. The dispute continued until Netflix started delivering traffic to ISPs directly, taking the stress off Cogent's connections with ISPs.

Though Netflix was the biggest source of congestion, the tests did not involve Netflix—rather, M-Lab evaluates streams of data that aren't specific to any application. The tests are crowdsourced, with the data streams being "sent from the user’s device (laptop, mobile, or other client-side gadget) to the nearest M-Lab measurement point."

“The three degraded Access ISPs [Comcast, Time Warner Cable, and Verizon] failed to achieve median download throughputs above 4Mbps when connecting over Cogent in New York City for most of the period between Spring 2013 and March 2014,” M-Lab wrote. “While daily median download throughput overall hovered around 4Mbps, performance degradation was much worse during peak use hours. For much of the time between Spring 2013 and March 2014, download speeds during peak use hours remained well below 4Mbps. By January 2014, the download throughput rate during peak use hours for Comcast and Verizon traffic over Cogent’s network was less than 0.5Mbps, the minimum rate necessary for web browsing and email according to the FCC. Note that only between 2:00 AM and 1:00 PM were the three affected Access ISPs able to attain speeds above 4 Mbps across the Transit ISP Cogent.”

This chart shows how speeds between Cogent and the ISPs dropped off a cliff for several months, recovering only after Netflix began paying the ISPs for direct connections to their networks:

Even Netflix's monthly speed rankings never showed these ISPs delivering fewer than 0.5Mbps on average, but the M-Lab numbers are more granular.

"We can identify Access ISP/Transit ISP interconnections that appear to be under load, instead of just giving a rough number for the Access ISP's performance," M-Lab researcher Collin Anderson told Ars. "This is important because Netflix states that they had sought transit from multiple providers and were attempting to spread load across these links. From a measurement perspective this better identifies harm, but also shows how stressed certain Tier 1 providers were during that period in ways that Netflix numbers do not."

It didn't get quite as bad in other cities but the M-Lab report detailed performance problems in Dallas, Los Angeles, Seattle, Atlanta, Chicago, and Washington, DC.

Netflix has said that only Comcast, Time Warner Cable, AT&T, and Verizon demanded interconnection payments. Other ISPs let Netflix connect for free and, in some cases, host video storage boxes inside their networks to improve performance for customers.

According to M-Lab, performance remained strong on ISPs that weren't involved in the money disputes.

"In New York City, Access ISP Cablevision customers did not experience the same degradation symptoms as Comcast, Time Warner Cable, and Verizon customers when connecting across the Cogent network," M-Lab wrote. "Similarly, in Dallas and Los Angeles during the relevant time period, customers of Access ISP Cox did not experience a significant pattern or degree of degraded download throughput when connecting across Cogent sites."