Seasoned Contributor

A year on and still not resolved

OK, this is (partially) a continuation of a previous thread of mine.

This time around I'm only going to concentrate on the dot com issues and ignore the dot mil, because at that point everyone seems to start talking about black helicopters/conspiracy theories. As the issues started at the same time, I'm assuming that when one is fixed, the other will work as well.

Basically there are 2 websites and one POP3 site I can recreate this issue on every single time and have been using for all my tracing/testing. The websites are my e-mail hosting: https://webhosting.att.com and an e-mail marketing organisation that a couple of mailing lists use as a redirection site in mail campaigns: https://cts.vresp.com (Note: entering just that portion of the web address will only return a "Forbidden" message, but it does prove that the site is reachable). The POP3 site is also my e-mail hosting: websitesmail.att.com:995

Description of the issue. Following a re-boot, where (this is the most important part) a new IP is assigned, I can reach the 3 sites without any issue. This will continue throughout the day, until around 9:10pm PDT. At this point all 3 sites become unreachable for about 2 hours, when they are accessible again for about 3 minutes after which they are unreachable again for 2 hours, followed by allowed access for around 3 minutes again. This 2 hour, 3 minute pattern then repeats for the next day, following which the sites become permanently unreachable. Here is the last of the traces I took of this pattern:

05/05/19-14:25:31 - Script started online
05/05/19-21:12:13 - Online for 24402 seconds
05/05/19-23:03:20 - Offline for 6667 seconds
05/05/19-23:06:15 - Online for 175 seconds
05/06/19-01:03:23 - Offline for 7028 seconds
05/06/19-01:06:28 - Online for 185 seconds
05/06/19-03:03:21 - Offline for 7013 seconds
05/06/19-03:06:31 - Online for 190 seconds
05/06/19-05:03:36 - Offline for 7025 seconds
05/06/19-05:06:26 - Online for 170 seconds
05/06/19-07:03:34 - Offline for 7028 seconds
05/06/19-07:06:30 - Online for 176 seconds
05/06/19-08:05:15 - Offline for 3525 seconds
05/06/19-08:09:23 - Online for 248 seconds
05/06/19-09:03:21 - Offline for 3238 seconds
05/06/19-09:06:21 - Online for 180 seconds
05/06/19-11:03:34 - Offline for 7033 seconds
05/06/19-11:06:29 - Online for 175 seconds
05/06/19-13:03:37 - Offline for 7028 seconds
05/06/19-13:06:27 - Online for 170 seconds
05/06/19-15:03:21 - Offline for 7014 seconds
05/06/19-15:06:21 - Online for 180 seconds
05/06/19-17:03:18 - Offline for 7017 seconds
05/06/19-17:06:08 - Online for 170 seconds
05/06/19-19:03:17 - Offline for 7029 seconds
05/06/19-19:06:18 - Online for 181 seconds
05/19/19-15:24:21 - Script killed after 1109883 seconds

As I stated above, this happens every time I force a re-boot with a new IP being allocated, by spoofing the MAC address and resetting the modem. I have done this now at least half a dozen times, or more, with always exactly the same results. I have similar trace files available from the last 4 reboots, if needed.

All of these sites are fully accessible 24 hours a day if I run a VPN and then try to access.

I have issues with other sites also, the CA DMV and the USPS Tracking site, but they exhibit different issues and I haven't been able (yet) to determine the reason/pattern.

When I first reported this, over a year ago, I tried everything I knew to try and get this in front of a network engineer who might be able to diagnose, but to no avail. I had first level techs insist a re-boot of the modem would fix it, or if not, then a modem replacement would. I had supervisors, who promised faithfully to follow up and get back to me. None ever did. I had a 2nd level support person, who I was referred to via a technician who came out to check the signals/modem, who for the first few times I e-mailed for status usually replied: I saw the ticket closed, so assumed it was resolved. Sorry, I'll re-open another. They eventually stopped responding to all my e-mails.

So the desperation question is how to get this to someone who knows how internet routing works, has access to the various Charter/Spectrum/TW servers involved, and has an interest in fixing customer issues. I'm kinda making an assumption that this has to be happening in the first couple of hops from my modem, otherwise there would be more people with similar issues.

I'd also be interested in anyone in the Los Angeles area can try those sites listed above to see if they have issues or not, especially if the gateway after the modem is: 142.254.237.37.

Cheers.

6 REPLIES 6
Community Manager

Re: A year on and still not resolved

Can you also provide screen shots of any error message you are getting?  

Seasoned Contributor

Re: A year on and still not resolved

No errors, just time-outs:

[root@Nethserver ~]# nc webhosting.att.com 443 < /dev/null
Ncat: Connection timed out.
[root@Nethserver ~]#

Here are some traceroutes immediately after a reboot, with a new IP:

root@Nethserver ~# traceroute -T -p 443 webhosting.att.com
traceroute to webhosting.att.com (66.175.49.6), 30 hops max, 60 byte packets
 1  142.254.237.37 (142.254.237.37)  4.590 ms *  4.568 ms
 2  * * *
 3  * * *
 4  * * *
 5  * * *
 6  * * *
 7  * * *
 8  HOSTOPIA.ear3.Miami2.Level3.net (4.59.242.58)  75.385 ms  75.843 ms  75.848 ms
 9  core2-vl51.50.mia.megarouting.com (64.29.159.13)  77.161 ms  77.152 ms  77.139 ms
10  sushi.carrierzone.com (66.175.49.6)  76.683 ms  77.044 ms  76.996 ms
root@Nethserver ~# traceroute -T -p 443 cts.vresp.com
traceroute to cts.vresp.com (74.116.90.99), 30 hops max, 60 byte packets
 1  142.254.237.37 (142.254.237.37)  5.246 ms  5.214 ms  11.545 ms
 2  agg61.igwdcaew02h.socal.rr.com (24.30.172.213)  13.084 ms  13.084 ms  13.063 ms
 3  agg30.lsaicaev02r.socal.rr.com (72.129.17.152)  17.242 ms  17.252 ms  17.230 ms
 4  agg26.tustcaft01r.socal.rr.com (72.129.17.2)  25.465 ms  25.474 ms  25.463 ms
 5  bu-ether16.tustca4200w-bcr00.tbone.rr.com (66.109.6.64)  25.906 ms  25.917 ms  25.905 ms
 6  * * *
 7  * * *
 8  HOSTOPIA.ear3.Miami2.Level3.net (4.59.242.58)  74.933 ms  72.459 ms  74.571 ms
 9  core2-vl51.50.mia.megarouting.com (64.29.159.13)  78.877 ms  78.655 ms  78.614 ms
10  74.116.90.99.ip.verticalresponse.com (74.116.90.99)  76.947 ms  78.346 ms  90.984 ms
root@Nethserver ~# traceroute -T -p 995 websitesmail.att.com
traceroute to websitesmail.att.com (216.55.149.49), 30 hops max, 60 byte packets
 1  142.254.237.37 (142.254.237.37)  10.095 ms  10.080 ms  10.070 ms
 2  agg61.igwdcaew02h.socal.rr.com (24.30.172.213)  13.136 ms  13.134 ms  13.121 ms
 3  agg30.lsaicaev02r.socal.rr.com (72.129.17.152)  18.969 ms  18.957 ms  18.968 ms
 4  agg26.tustcaft01r.socal.rr.com (72.129.17.2)  18.856 ms  18.868 ms  18.858 ms
 5  bu-ether16.tustca4200w-bcr00.tbone.rr.com (66.109.6.64)  22.492 ms  22.503 ms  22.491 ms
 6  * * *
 7  * * *
 8  HOSTOPIA.ear3.Miami2.Level3.net (4.59.242.58)  77.692 ms  78.101 ms  78.022 ms
 9  core2-vl51.50.mia.megarouting.com (64.29.159.13)  77.085 ms  77.133 ms  77.099 ms
10  websitesmailc45.carrierzone.com (216.55.149.49)  74.422 ms  74.154 ms  74.044 ms
root@Nethserver ~#

And the same after connectivity is lost:

root@Nethserver ~# traceroute -T -p 443 webhosting.att.com
traceroute to webhosting.att.com (66.175.49.6), 30 hops max, 60 byte packets
 1  142.254.237.37 (142.254.237.37)  10.864 ms  10.862 ms  10.853 ms
 2  agg61.igwdcaew02h.socal.rr.com (24.30.172.213)  12.982 ms  12.980 ms  12.972 ms
 3  agg30.lsaicaev02r.socal.rr.com (72.129.17.152)  17.523 ms  17.527 ms  17.518 ms
 4  agg26.tustcaft01r.socal.rr.com (72.129.17.2)  23.315 ms  23.307 ms  23.360 ms
 5  ae-5-0.cr0.chi10.tbone.rr.com (66.109.6.202)  23.211 ms  23.223 ms  23.215 ms
 6  * * *
 7  * * *
 8  * * *
 9  * * *
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *
root@Nethserver ~# traceroute -T -p 443 cts.vresp.com
traceroute to cts.vresp.com (74.116.90.99), 30 hops max, 60 byte packets
 1  142.254.237.37 (142.254.237.37)  10.030 ms  10.388 ms  10.013 ms
 2  agg61.igwdcaew02h.socal.rr.com (24.30.172.213)  11.675 ms  11.687 ms  11.676 ms
 3  agg30.lsaicaev02r.socal.rr.com (72.129.17.152)  13.812 ms  13.826 ms  13.814 ms
 4  agg26.tustcaft01r.socal.rr.com (72.129.17.2)  23.327 ms  23.323 ms  23.314 ms
 5  bu-ether16.tustca4200w-bcr00.tbone.rr.com (66.109.6.64)  21.620 ms  21.633 ms  21.623 ms
 6  * * *
 7  ae-1-51.ear3.Miami2.Level3.net (4.69.207.29)  76.408 ms  76.138 ms  75.697 ms
 8  * * *
 9  * * *
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *
root@Nethserver ~# traceroute -T -p 995 websitesmail.att.com
traceroute to websitesmail.att.com (216.55.149.49), 30 hops max, 60 byte packets
 1  142.254.237.37 (142.254.237.37)  10.005 ms  10.004 ms *
 2  * * *
 3  * * *
 4  * * *
 5  * * *
 6  * * *
 7  * * *
 8  * * *
 9  * * *
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *
root@Nethserver ~#

Cheers.

 

Browser

Re: A year on and still not resolved

The first link opened fine while the second one gave the Forbidden window.  The webmail link opened login window for email and password to be entered. Probably not helpful but just getting back to let you know since you wanted feedback from others.

Seasoned Contributor

Re: A year on and still not resolved

Is your first hop the same as mine on a traceroute: 

142.254.237.37

Cheers.

 

Browser

Re: A year on and still not resolved

I am probably not doing it correctly but tried tracert of https://webhosting.att.com/ but did not work.  I then tried using this website: https://www.site24x7.com/dns-lookup.html and showed 198.41.0.4.  Not sure how else to look at this.

Seasoned Contributor

Issue Not Fixed, Now Support are Ghosting Me

The support folks on this forum were able to get a ticket opened for me at the beginning of June. This was then put on hold, as I was going to be out of the country for 3 weeks, so it really started being looked at towards the end of June. For the next couple of weeks I was getting call backs, on what seemed like a daily basis. Although based on the questions and/or supposed answers it was obvious that the issue had not been looked at by anyone who really understood what was really happening, or by people who had not read through the ticket, as they were repeating the same questions/answers multiple times.

 

Following this, I was told that no further research could be done as I was using my own modem, which meant they couldn't access it remotely. I asked how accessing my modem could help in researching a network routing issue, as it didn't make sense to me, but couldn't get any sensible answer. Anyway, I said if that's what they wanted, go ahead, send someone out and install your modem, which they then did.

 

It was at this point I was "ghosted". Now instead of being called daily, I have tried numerous times, for around 3 weeks, to get a call back for a status. Either no call back happens, or (twice !!) the ticket was routed to the "truck roll" dispatch, who obviously couldn't do anything.

 

On one of those numerous requests for a call-back, I asked for some further information to be added to the ticket, as I discovered that another website was inaccessible, the CA DMV !!!! For this one, I was able to capture some packet traces that show (as I had suspected and reported a number of times) that the issue appears to be in the return path, back to me. Here are the traces I captured:

ca_dmv.jpg

You can see that the 3-way TCP SYN/ACK handshake succeeds, followed by a send of the "Client Hello", which is correctly ACK'd. But following this, the "Server Hello" packet never reaches me, and a few seconds later, the server does a RST to break the communication.

 

I tried to explain this to the "support" person, but as they had no clue about tcpdump, packet traces, TCP negotiation, etc. so I have absolutely no idea what is actually reported in the ticket or where it has been routed.

 

How do I get this escalated to someone who *does* understand these things, as I believe (and so do some of my network savvy friends) that this clearly (??) shows an issue with some of the packets sent back to me being blocked. Unfortunatley I can see no pattern as to why this is only on some websites and also not on all the packet data.

 

And to answer the first question, before it's asked: These traces were taken on the machine hard-wired to the modem, so there is no PC, router, server, etc. interferring with the flow.

 

Cheers.