Observer

Routing issues with routes including 66.109.7.162

I'm seeing lots of connectivity issues when connecting to any host with a route including 66.109.7.162.  According to whois, this is a Spectrum/Charter owned IP. Can someone investigate what's going on with those routes? A destination host that tries to route through 66.109.7.162 is 69.166.128.1. 

10 REPLIES 10
Observer

Re: Routing issues with routes including 66.109.7.162

Sharer

Re: Routing issues with routes including 66.109.7.162

May be a peering issue again... hard to tell without seeing the route.

Not uncommon to see congestion where things get handed off to someone like Level3/Century or Zayo in some regions.
Observer

Re: Routing issues with routes including 66.109.7.162

Here are a few sample traces. Everything at or after 66.109.7.162 is a dumpster fire. Not seeing any issues with routes not including 66.109.7.162.

 

1 * * *
2 cpe-174-111-066-065.triad.res.rr.com (174.111.66.65) 24.654 ms 34.521 ms 34.475 ms
3 24.28.254.96 (24.28.254.96) 22.571 ms 22.490 ms 22.194 ms
4 be34.drhmncev01r.southeast.rr.com (24.93.64.196) 24.656 ms 21.800 ms 22.585 ms
5 66.109.10.176 (66.109.10.176) 35.072 ms bu-ether15.asbnva1611w-bcr00.tbone.rr.com (66.109.6.80) 30.591 ms 66.109.10.176 (66.109.10.176) 35.280 ms
6 ae-2-0.c1.chi75.tbone.rr.com (66.109.3.25) 35.837 ms 34.916 ms bu-ether12.vinnva0510w-bcr00.tbone.rr.com (66.109.6.31) 34.963 ms
7 0.ae1.pr0.dca20.tbone.rr.com (66.109.6.167) 30.286 ms 0.ae0.pr0.dca20.tbone.rr.com (107.14.19.65) 29.569 ms 0.ae3.pr0.dca20.tbone.rr.com (107.14.19.160) 27.827 ms
8 * 66.109.7.162 (66.109.7.162) 547.728 ms *
9 * * *
10 * * *
11 * dr5-po32.ral.hostedsolutions.com (216.27.48.66) 37.240 ms *

 

 

1 * * *
2 cpe-174-111-066-065.triad.res.rr.com (174.111.66.65) 38.657 ms 39.881 ms 39.142 ms
3 24.28.254.96 (24.28.254.96) 26.484 ms 26.264 ms 26.624 ms
4 be34.drhmncev01r.southeast.rr.com (24.93.64.196) 28.206 ms 28.246 ms 30.723 ms
5 66.109.10.176 (66.109.10.176) 35.325 ms bu-ether15.asbnva1611w-bcr00.tbone.rr.com (66.109.6.80) 35.927 ms 30.384 ms
6 ae-2-0.c1.chi75.tbone.rr.com (66.109.3.25) 34.628 ms 39.864 ms 38.146 ms
7 0.ae3.pr0.dca20.tbone.rr.com (107.14.19.160) 28.309 ms 30.256 ms 27.104 ms
8 * * *
9 ae-0-11.bar1.Raleigh1.Level3.net (4.69.137.177) 28.200 ms * 33.251 ms
10 * * 4.71.162.30 (4.71.162.30) 33.326 ms
11 dr5-po32.ral.hostedsolutions.com (216.27.48.66) 33.886 ms 37.819 ms *

 

1 * * *
2 cpe-174-111-066-065.triad.res.rr.com (174.111.66.65) 134.848 ms 135.436 ms 135.198 ms
3 24.28.254.96 (24.28.254.96) 21.135 ms 22.082 ms 21.808 ms
4 be34.drhmncev01r.southeast.rr.com (24.93.64.196) 26.465 ms 24.603 ms 27.579 ms
5 bu-ether15.asbnva1611w-bcr00.tbone.rr.com (66.109.6.80) 31.459 ms 30.748 ms 66.109.10.176 (66.109.10.176) 29.979 ms
6 ae-2-0.c1.chi75.tbone.rr.com (66.109.3.25) 30.499 ms 35.071 ms 30.758 ms
7 0.ae0.pr0.dca20.tbone.rr.com (107.14.19.65) 28.922 ms 0.ae3.pr0.dca20.tbone.rr.com (107.14.19.160) 33.534 ms 0.ae0.pr0.dca20.tbone.rr.com (107.14.19.65) 68.301 ms
8 * 66.109.7.162 (66.109.7.162) 539.639 ms 635.064 ms
9 ae-0-11.bar1.Raleigh1.Level3.net (4.69.137.177) 37.035 ms * 30.639 ms
10 4.71.162.30 (4.71.162.30) 30.145 ms 39.287 ms 34.047 ms11 dr5-po32.ral.hostedsolutions.com (216.27.48.66) 35.883 ms * *

Observer

Re: Routing issues with routes including 66.109.7.162

Here's a comparable trace from a different Spectrum account originating from about 100 miles east of here. Uses a different peer to Level 3 and all is well there:

 

1 * * *
2 174.111.103.77 (174.111.103.77) 30.362 ms 30.776 ms 30.945 ms
3 cpe-024-025-041-232.ec.res.rr.com (24.25.41.232) 14.321 ms 14.519 ms 14.969 ms
4 be31.chrcnctr01r.southeast.rr.com (24.93.64.186) 26.369 ms 26.546 ms 26.818 ms
5 bu-ether14.atlngamq46w-bcr00.tbone.rr.com (66.109.6.82) 29.226 ms 26.502 ms 30.812 ms
6 66.109.5.125 (66.109.5.125) 23.576 ms 19.866 ms 19.542 ms
7 ae-10.edge4.Atlanta2.Level3.net (4.68.37.73) 25.064 ms 18.798 ms 22.253 ms
8 4.69.217.46 (4.69.217.46) 27.868 ms 27.620 ms 27.222 ms
9 4.71.162.30 (4.71.162.30) 31.412 ms 30.613 ms 30.988 ms
10 dr5-po32.ral.hostedsolutions.com (216.27.48.66) 29.535 ms 28.772 ms 27.100 ms

Lead Moderator

Re: Routing issues with routes including 66.109.7.162

Good afternoon.

 

We are seeing multiple reports of problems and are underway with investigation.

 

I don't have any further information I'm afraid but rest assured it will be resolved 

as quickly as possible. 

 

Regards,

Julia

Social Media Customer Service-Lead Moderator

Sharer

Re: Routing issues with routes including 66.109.7.162

As suspected.... at the hand-off to a peer's network:

8 * 66.109.7.162 (66.109.7.162) 539.639 ms 635.064 ms
9 ae-0-11.bar1.Raleigh1.Level3.net (4.69.137.177) 37.035 ms * 30.639 ms

Imagine that... Level3/Century Link.

Had issues with them in the same region back when FFXIV had it's 2.0 reboot. Routing along the eastern seaboard to Montreal for Square-Eidos servers would go AWOL when sent to Level3, but were nice and crisp with TATA or Cogentco.

While TWC hammered out their peering/QoS situation, we used VPN's as a sort of bandaid for it. Not always practical/successful (depends on exactly what is causing the issue), but may be an option to consider in the short term.
Proven Sharer

Re: Routing issues with routes including 66.109.7.162

Whenever you see a table like that it pays to also suspect that trouble is really at the next site upstream, which is 66.109.5.125.  Sure enough, I can tell just by reading the next device name that it's Spectrum's edge router feeding Level3, Spectrum's contracted IXP for the southeast region.  As @RAIST5150 said, it's certainly not the first time that a link between Spectrum and Level3 link has misbehaved.

 

Remember that Spectrum does not do any routing based on geographic IP location or circuit traffic loads or QOS.  Every data packet sent from a particular subscriber address gets routed to the same IXP site every time, unless hardware or media on the specified route has failed and someone manually builds all the bypass routing tables. 

Highlighted
Observer

Re: Routing issues with routes including 66.109.7.162

This has been resolved.  66.109.7.162 no longer appears in traceroutes to Level 3.

Re: Routing issues with routes including 66.109.7.162

We have been seeing this issue accross multiple of our clients on and off over that past several months. Today we were finally able to get traceroutes from multiple client networks at multiple geographic locations at the A and Z ends. So far we have seen the issue from Lexington KY to Wilmington PA, Dayton OH to Wilmington PA, Columbia SC to Columbus OH, Dayton OH to Columbus OH, Winchester KY to Columbus OH. The issue does not appear to affect general Internet traffic, only certain protocols -- we have noticed it due to our L2L VPN tunnels between these geographic locations intermittently being unable to pass IPsec VPN traffic.

 

The attached photos illustrate a traceroute with probe 10/50 for repeated traceroute data.

 

There is definitely an issue at this hop and I am so relieved we are not the only one experiencing this. We have been fighting with Spectrum over this for almost three months now with absolutely zero progress.

 

Where are you at Spectrum !?

 

screenshot3.jpgscreenshot2.jpg