All Americas network connectivity issues with Level3 have been resolved, also subscribers who have Comcast as an ISP will now connect to East Coast resources through a direct peering relationship with Comcast. We expect this direct peering relationship for West Coast subscribers on Comcast to go into production early next week.
The DevOps team's monitoring has concluded that packet loss levels to subscribers on various ISPs is once again at normal levels. Our network provider for East US will continue to work with Level3 to return then transit to normal levels while also continue to build out the new direct peering relationship with Comcast.
Posted Apr 18, 2019 - 15:45 UTC
Update
While over the past 48 hours most connections through Level3 have only had very intermittent packet loss or had already been routed through different egress providers. Our network provider has disabled direct peering with Level3 until they are able to ensure a higher quality connection within the Level3 network.
If you or your end users are still experiencing packet loss or other network issues in transit please report the issue to the support team. The end user public IP and a MTR report is always appreciated.
Posted Apr 18, 2019 - 13:18 UTC
Update
The DevOps team and our network providers continue to monitor and reroute connections that experience higher than normal levels of packet loss. Please report any specific issues to the support team for optimization.
Our network provider has also begun the process to begin peering directly with Comcast to further reduce our transit over Level3 to end users on Comcast. This is not currently in production but will be in the near future.
Posted Apr 15, 2019 - 14:03 UTC
Update
The DevOps team and our network providers are continuing to evaluate transit through Level3, some paths have been restored for egress through our East Coast POP. We will continue to monitor and optimize the paths through out the day.
Posted Apr 12, 2019 - 14:17 UTC
Monitoring
The DevOps team continues to monitor transit through Level3, packet loss continues to show clean connections through the alternative paths. Our DevOps and network providers will continue monitor our primary peering on the East Coast will move traffic back once it is determined to be in good quality again.
Posted Apr 11, 2019 - 16:30 UTC
Identified
In efforts to reduce packet loss, Level 3 egress has been rerouted through our West Coast POP. While latency is slightly higher, packet loss has been reduced greatly.
Posted Apr 11, 2019 - 16:18 UTC
Update
The DevOps Team and our network provider continue to investigate peering with Level3, so far packet loss is occurring once within Level3 transit. The teams continue to investigate alternative paths to improve performance.
Posted Apr 11, 2019 - 16:12 UTC
Investigating
The DevOps team is currently working with our network providers investigating observed packet loss to some locations within the United States. As with the April 9th issue we have specifically identified customers who traverse through Level 3 are experiencing packet loss.