Downstream Routing Issue a) How to really trace b) How to report

[EddieA]

Because a) traceroute and ping don't really cut it based on how many routers/firewalls drop the packets.  And b) I don't need to be told to power cycle my modem and re-boot my PC.

Around 5:30am Friday morning (according to my logs) I lost connectivity to my mail host.  All attempts at connecting via POP3 were timing out.  Later, trying to connect to their website I had the same timeout issue.  After checking and trying a bunch of things I came to the conclusion it was a routing issue.  Turning on my VPN to direct all the traffic to an exit point outside of Spectrum's (RR's) network circumvents the issue.  Unfortunately that doen't automatically point the smoking gun at Spectrum based on the different entry point to the routing which causes a different path to be taken to the destination.

First a couple of things to make the tests as level of a playing field as possible:

[root@Nethserver ~]#  host webhosting.att.com
webhosting.att.com is an alias for sushi.carrierzone.com.
sushi.carrierzone.com has address 66.175.49.6
[root@Nethserver ~]# host websitesmail.att.com
websitesmail.att.com is an alias for websitesmailc45.carrierzone.com.
websitesmailc45.carrierzone.com has address 216.55.149.49
[root@Nethserver ~]#

With all the traffic directed to Spectrum.  Yeah, I know I said that traceroute doesn't really help, but the output is there for completeness:

[root@Nethserver ~]# telnet 66.175.49.6 443
Trying 66.175.49.6...
telnet: connect to address 66.175.49.6: Connection timed out
[root@Nethserver ~]# telnet 216.55.149.49 995
Trying 216.55.149.49...
telnet: connect to address 216.55.149.49: Connection timed out
[root@Nethserver ~]# traceroute 66.175.49.6
traceroute to 66.175.49.6 (66.175.49.6), 30 hops max, 60 byte packets
 1  142.254.237.37 (142.254.237.37)  9.449 ms  9.430 ms  9.389 ms
 2  agg61.igwdcaew01h.socal.rr.com (24.30.172.209)  10.697 ms  10.686 ms  11.795 ms
 3  agg30.lsaicaev01r.socal.rr.com (72.129.17.150)  16.277 ms  16.264 ms  16.301 ms
 4  agg26.lsancarc01r.socal.rr.com (72.129.17.0)  11.728 ms  11.699 ms  11.676 ms
 5  bu-ether16.lsancarc0yw-bcr00.tbone.rr.com (66.109.6.102)  16.051 ms  16.046 ms  16.029 ms
 6  bu-ether45.chctilwc00w-bcr00.tbone.rr.com (107.14.19.36)  22.935 ms  13.845 ms  17.921 ms
 7  xe-9-3-0.edge4.Frankfurt1.level3.net (4.68.111.101)  12.895 ms  13.747 ms  13.672 ms
 8  * * *
Remainder of traceroute all "* * *"
[root@Nethserver ~]# traceroute 216.55.149.49
traceroute to 216.55.149.49 (216.55.149.49), 30 hops max, 60 byte packets
 1  142.254.237.37 (142.254.237.37)  9.386 ms  9.345 ms  9.281 ms
 2  agg61.igwdcaew01h.socal.rr.com (24.30.172.209)  10.285 ms  10.956 ms  10.967 ms
 3  agg30.lsaicaev01r.socal.rr.com (72.129.17.150)  15.114 ms  15.114 ms  15.100 ms
 4  agg26.lsancarc01r.socal.rr.com (72.129.17.0)  11.279 ms  11.225 ms  11.241 ms
 5  bu-ether26.lsancarc0yw-bcr00.tbone.rr.com (66.109.3.230)  17.182 ms  17.154 ms  17.135 ms
 6  bu-ether45.chctilwc00w-bcr00.tbone.rr.com (107.14.19.36)  18.246 ms bu-ether11.tustca4200w-bcr00.tbone.rr.com (66.109.6.5)  16.898 ms  17.254 ms
 7  * * *
Remainder of traceroute all "* * *"

Then after starting the VPN:

[root@Nethserver openvpn]# systemctl start openvpn-client@LosAngeles
[root@Nethserver openvpn]#

I get this:

[root@Nethserver ~]# telnet 66.175.49.6 443
Trying 66.175.49.6...
Connected to 66.175.49.6.
Escape character is '^]'.
^]
telnet> quit
Connection closed.
[root@Nethserver ~]# telnet 216.55.149.49 995
Trying 216.55.149.49...
Connected to 216.55.149.49.
Escape character is '^]'.
^]
telnet> quit
Connection closed.
[root@Nethserver ~]# traceroute 66.175.49.6
traceroute to 66.175.49.6 (66.175.49.6), 30 hops max, 60 byte packets
 1  10.132.0.1 (10.132.0.1)  14.609 ms  14.613 ms  14.598 ms
 2  v434.br01.lax-11.leaseweb.net (23.19.3.81)  31.168 ms  31.161 ms  31.159 ms
 3  69.31.114.133 (69.31.114.133)  14.511 ms  14.510 ms  14.531 ms
 4  et-0-0-71.cr4-lax2.ip4.gtt.net (89.149.140.73)  14.647 ms  14.655 ms  14.655 ms
 5  as174.lax21.ip4.gtt.net (173.205.53.114)  15.211 ms  15.118 ms  15.177 ms
 6  be3271.ccr41.lax01.atlas.cogentco.com (154.54.42.101)  15.176 ms  11.865 ms  12.104 ms
 7  be2932.ccr32.phx01.atlas.cogentco.com (154.54.45.161)  26.266 ms be2931.ccr31.phx01.atlas.cogentco.com (154.54.44.85)  26.265 ms  26.198 ms
 8  be2929.ccr21.elp01.atlas.cogentco.com (154.54.42.66)  34.769 ms be2930.ccr21.elp01.atlas.cogentco.com (154.54.42.78)  34.781 ms  34.783 ms
 9  be2928.ccr42.iah01.atlas.cogentco.com (154.54.30.161)  50.263 ms  50.210 ms  50.768 ms
10  be2069.ccr21.mia01.atlas.cogentco.com (66.28.4.238)  78.169 ms  78.184 ms be2071.ccr22.mia01.atlas.cogentco.com (154.54.0.221)  75.311 ms
11  be3401.ccr21.mia03.atlas.cogentco.com (154.54.47.30)  74.909 ms  77.703 ms  75.401 ms
12  * * *
Remainder of traceroute all "* * *"
[root@Nethserver ~]# traceroute 216.55.149.49
traceroute to 216.55.149.49 (216.55.149.49), 30 hops max, 60 byte packets
 1  10.132.0.1 (10.132.0.1)  15.281 ms  15.282 ms  15.278 ms
 2  v434.br01.lax-11.leaseweb.net (23.19.3.81)  15.284 ms  15.272 ms  15.296 ms
 3  69.31.114.133 (69.31.114.133)  15.294 ms  15.389 ms  15.436 ms
 4  et-0-0-71.cr4-lax2.ip4.gtt.net (89.149.140.73)  15.434 ms  15.432 ms  15.438 ms
 5  as174.lax21.ip4.gtt.net (173.205.53.114)  16.073 ms  16.061 ms  15.958 ms
 6  be3271.ccr41.lax01.atlas.cogentco.com (154.54.42.101)  15.975 ms be3360.ccr42.lax01.atlas.cogentco.com (154.54.25.149)  13.014 ms be3271.ccr41.lax01.atlas.cogentco.com (154.54.42.101)  13.241 ms
 7  be2931.ccr31.phx01.atlas.cogentco.com (154.54.44.85)  25.084 ms be2932.ccr32.phx01.atlas.cogentco.com (154.54.45.161)  24.585 ms be2931.ccr31.phx01.atlas.cogentco.com (154.54.44.85)  24.590 ms
 8  be2929.ccr21.elp01.atlas.cogentco.com (154.54.42.66)  32.943 ms  33.349 ms be2930.ccr21.elp01.atlas.cogentco.com (154.54.42.78)  33.364 ms
 9  be2927.ccr41.iah01.atlas.cogentco.com (154.54.29.221)  49.158 ms  49.191 ms  49.154 ms
10  be2071.ccr22.mia01.atlas.cogentco.com (154.54.0.221)  76.788 ms  76.806 ms  76.805 ms
11  be3400.ccr21.mia03.atlas.cogentco.com (154.54.47.18)  77.283 ms be3401.ccr21.mia03.atlas.cogentco.com (154.54.47.30)  77.297 ms be3400.ccr21.mia03.atlas.cogentco.com (154.54.47.18)  74.935 ms
12  * * *
Remainder of traceroute all "* * *"

So, as asked.  Is there a way to nail down exactly where this routing issue is happening and then how do I report this without talking to someone reading a script.

As an aside, at the same time my mail host was unreachable, so were a bunch (but strangely not all) of the .mil sites I connect to for work.  This issue of the .mil routing appears to have been corrected sometime over the weekend, but not the mail host.

Cheers.

[MsRaye]

ok,  bottom line is you're trying to get ATT mail thru a spectrum connection?

I think you need to contact them since their site has a DNS lookup issue or incorrect entry.

I assume you cleared your DNS cache and tried a non TWC/ DNS like 8.8.8.8

websitesmail.att.com

[EddieA]

This is an externally hosted mail acount that 'should' be accessable from anywhere.

Notice that all the tests were done via IP, not DNS names.

Oh, and by the way:

[root@Nethserver ~]# dig @8.8.8.8 +short webhosting.att.com
sushi.carrierzone.com.
66.175.49.6
[root@Nethserver ~]# dig @8.8.8.8 +short websitesmail.att.com
websitesmailc45.carrierzone.com.
216.55.149.49
[root@Nethserver ~]#
[root@Nethserver ~]# dig @209.18.47.62 +short webhosting.att.com
sushi.carrierzone.com.
66.175.49.6
[root@Nethserver ~]# dig @209.18.47.62 +short websitesmail.att.com
websitesmailc45.carrierzone.com.
216.55.149.49
[root@Nethserver ~]#

Cheers.

[MsRaye]

As far as I know, the DNS is where the routing table  is created... Domain Names or IP addresses.

If there's no dns, nothing gets anywhere. Your tracert's are showing domain names, not straight IP's, if one is bad, "whois" it and contact them to update routing info:

NetRange:       216.55.144.0 - 216.55.159.255
CIDR:           216.55.144.0/20
NetName:        MEGA-8
NetHandle:      NET-216-55-144-0-1
Parent:         NET216 (NET-216-0-0-0-0)
NetType:        Direct Allocation
OriginAS:       AS30447
Organization:   InternetNamesForBusiness.com (INFB)
RegDate:        1999-05-27
Updated:        2012-02-24
Ref:            https://whois.arin.net/rest/net/NET-216-55-144-0-1


OrgName:        InternetNamesForBusiness.com
OrgId:          INFB
Address:        110 East Broward Blvd
Address:        Suite 1650
City:           Fort Lauderdale
StateProv:      FL
PostalCode:     33301
Country:        US
RegDate:        
Updated:        2017-01-28
Ref:            https://whois.arin.net/rest/org/INFB


OrgAbuseHandle: ZI51-ARIN
OrgAbuseName:   InternetNamesForBusiness
OrgAbusePhone:  +1-800-322-9438 
OrgAbuseEmail:  admin@internetnamesforbusiness.com
OrgAbuseRef:    https://whois.arin.net/rest/poc/ZI51-ARIN

[EddieA]

---

@MsRaye wrote:

As far as I know, the DNS is where the routing table  is created... Domain Names or IP addresses.

If there's no dns, nothing gets anywhere. Your tracert's are showing domain names, not straight IP's, if one is bad, "whois" it and contact them to update routing info:

---

All internet routing is via IPs.  The only time DNS is involved, in normal routing, is to convert the destination host name to an IP before the packet is sent out to the router used as the gateway.  All routing from that point onward is only via IP, because that's all that the packet contains (well, it does contain a little more data ) and definitely not any host name information.

If you look at the traceroute, it does show the IP of the router involved.  It's the traceroute program that's trying to be helpful, by using a DNS reverse look-up, to convert that back to it's DNS name to make the output more readable.

Cheers.

[redrock]

1. based on your telnet results the end points are either unreachable due to routing or other network issues or the response is not returning to your host. The data does not indicate which direction has the problems.
2. The destinations could be blocked by a network protection device  along the way if some bad behavior was detected and the network device blocked those addresses. Think denial of service, high rate tcp/udp traffic, etc. These blocks may be of temporary duration, minutes to hours.
3. traceroute is a very limited tool. see notes below for more.
4. suggestion: use the vpn method and check later on status.

mtr, winmtr, traceroute, tracert

• mtr on linux (opensuse)
• sends icmp ping requests to target ip with incrementing ttl values starting at 1
• intermediate nodes decrement ttl and if zero do not forward ping icmp request but respond with “ttl exceeded” message.
• target ip responds with ping reply when presented with ping request.
• traceroute on linux (opensuse)
• sends 3 udp packets to target ip with incrementing ttl values starting at 1
• intermediate nodes decrement ttl and if zero do not forward udp packet but respond with “ttl exceeded” message.
• target ip responds with icmp, type 3, code 3, destination unreachable, port unreachable. (host response if  process port is not active)
• more points on traceroute
• traceroute only does ping, if using icmp, on the target ip address. all intermediate addresses are responding with a status message - TTL exceeded, usually a low priority message and usually handled by software as opposed to the hardware that is forwarding packets, the primary function of router.
• traceroute is only reveals 1/2 the round trip, the outbound ip addresses. to get a sense of inbound path, do a traceroute from an online server, location dependent.
• to get a better sense of any single ip address, do a ping to that address with a 100 count. On linux systems, there are ping parameters that can issue pings as soon as a previous ping response is received, not flooding the target but not waiting either.
• general packet loss interpretation.
• if packet loss does not continue or rise on subsequent hops the hop/router in question is doing its job of forwarding packets.
• return path of packets is not revealed by traceroute or ping plotter. packets from hop in question may be using different return path than target ip.
• routers along path except for target ip normally will respond with "ttl exceeded" messages, as required by tools traceroute and  ping plotter, but the lack of a response does not indicate a problem if target ip does not have packet loss.

[EddieA]

@redrock

Yeah, I fully understand the problems with telnet and, particularly traceroute, with trying to debug a routing issue and also how both programs work internally, hence my comment (a) at the start of the thread.  Yes, I'm going with (4) until I can get this resolved, but that causes a couple of other issues.

With regard to my comment (b).  If there are any Spectrum folks monitoring this, I would like to throw out some kudos  to the people I spoke to earlier today.  Firstly, the first line tech, who after I diverted him away from his script, "The first thing I need to do is reboot your modem", with a "no way in h*ll", understood that he was out of his depth and transferred my to a network technician who, even after validating at his end that the web site was unreachable, waited on the phone while I e-mailed my results to him, to ensure that we were both seeing the exact same routing issue, before escalating it further.  I'll be the first to admit that I was truly surprised that I was able to get, so easily, to a qualified network person who fully understood the issue.

Cheers.

[redrock]

@EddieA,

I am interested in what you and network tech discovered, with details as to why/what caused the end site to be unreachable.

Thanks for the other feedback also.

[karlbeckman]

As a very-long-time subscriber to a former AT&T regional company for my email service, I've recently been subjected to many shuffles and redirects. Incoming emails go through reroutes between AT&T dba SBCGlobal, Yahoo.com, and Verizon hiding up in the ceiling somewhere. 

In my case they moved my existing SBCGlobal mail records prior to Dec 20 to a Yahoo account name, but the new incoming messages still go to SBCGlobal.  They refuse to move my older messages back to the SBCGlobal account. 

Anyhow, based on that you may find it beneficial to use site names for AT&T rather than IP addresses, so the DNS can "correctly" route your session traffic to the IP address of your latest mail server as found in the IXP gateway's DNS records. 

[EddieA]

Why is everyone focusing on DNS.  It's a routing issue.

@redrock

The network tech was goint to escalate to the responsible parties for: 

bu-ether45.chctilwc00w-bcr00.tbone.rr.com (107.14.19.36)

To investigate the rules, although that didn't appear to be the "stall point", but for the odd time it did work correctly, that hop reacted the opposite way around, so appeared to be suspect:

[root@Nethserver ~]# traceroute  216.55.149.49
traceroute to 216.55.149.49 (216.55.149.49), 30 hops max, 60 byte packets
 1  142.254.237.37 (142.254.237.37)  8.694 ms  8.166 ms  8.634 ms
-- snip --
 6  bu-ether45.chctilwc00w-bcr00.tbone.rr.com (107.14.19.36)  18.510 ms  18.394 ms  13.135 ms
 7  xe-9-3-0.edge4.Frankfurt1.level3.net (4.68.111.101)  12.329 ms  13.778 ms  13.289 ms
 

[root@Nethserver ~]# traceroute  66.175.49.6
traceroute to 66.175.49.6 (66.175.49.6), 30 hops max, 60 byte packets
 1  142.254.237.37 (142.254.237.37)  6.772 ms  8.468 ms  8.458 ms
-- snip --
 6  bu-ether45.chctilwc00w-bcr00.tbone.rr.com (107.14.19.36)  13.899 ms bu-ether11.tustca4200w-bcr00.tbone.rr.com (66.109.6.5)  15.990 ms  18.769 ms\

So far I haven't heard anything back from them.

However, starting Tuesday, I started to see the .mil sites I mentioned earlier re-appear as a hard issue, which is a real pain, as I need to access those to work.  Again, it's not all .mil either.  And the point in the path where these diverge from the traceroutes pasted here is after this hop:

lsancarc0yw-bcr00.tbone.rr.com

So I'm at a total loss as to why these seem to be interconnected.

Since then I have tried twice to reach out to the same team as I was originally routed to, but all I get now is the stone-wall treatment from first-level support.  So, I guess I retract my views on the service now.

Cheers.