OK, this is (partially) a continuation of a previous thread of mine.
This time around I'm only going to concentrate on the dot com issues and ignore the dot mil, because at that point everyone seems to start talking about black helicopters/conspiracy theories. As the issues started at the same time, I'm assuming that when one is fixed, the other will work as well.
Basically there are 2 websites and one POP3 site I can recreate this issue on every single time and have been using for all my tracing/testing. The websites are my e-mail hosting: https://webhosting.att.com and an e-mail marketing organisation that a couple of mailing lists use as a redirection site in mail campaigns: https://cts.vresp.com (Note: entering just that portion of the web address will only return a "Forbidden" message, but it does prove that the site is reachable). The POP3 site is also my e-mail hosting: websitesmail.att.com:995
Description of the issue. Following a re-boot, where (this is the most important part) a new IP is assigned, I can reach the 3 sites without any issue. This will continue throughout the day, until around 9:10pm PDT. At this point all 3 sites become unreachable for about 2 hours, when they are accessible again for about 3 minutes after which they are unreachable again for 2 hours, followed by allowed access for around 3 minutes again. This 2 hour, 3 minute pattern then repeats for the next day, following which the sites become permanently unreachable. Here is the last of the traces I took of this pattern:
05/05/19-14:25:31 - Script started online 05/05/19-21:12:13 - Online for 24402 seconds 05/05/19-23:03:20 - Offline for 6667 seconds 05/05/19-23:06:15 - Online for 175 seconds 05/06/19-01:03:23 - Offline for 7028 seconds 05/06/19-01:06:28 - Online for 185 seconds 05/06/19-03:03:21 - Offline for 7013 seconds 05/06/19-03:06:31 - Online for 190 seconds 05/06/19-05:03:36 - Offline for 7025 seconds 05/06/19-05:06:26 - Online for 170 seconds 05/06/19-07:03:34 - Offline for 7028 seconds 05/06/19-07:06:30 - Online for 176 seconds 05/06/19-08:05:15 - Offline for 3525 seconds 05/06/19-08:09:23 - Online for 248 seconds 05/06/19-09:03:21 - Offline for 3238 seconds 05/06/19-09:06:21 - Online for 180 seconds 05/06/19-11:03:34 - Offline for 7033 seconds 05/06/19-11:06:29 - Online for 175 seconds 05/06/19-13:03:37 - Offline for 7028 seconds 05/06/19-13:06:27 - Online for 170 seconds 05/06/19-15:03:21 - Offline for 7014 seconds 05/06/19-15:06:21 - Online for 180 seconds 05/06/19-17:03:18 - Offline for 7017 seconds 05/06/19-17:06:08 - Online for 170 seconds 05/06/19-19:03:17 - Offline for 7029 seconds 05/06/19-19:06:18 - Online for 181 seconds 05/19/19-15:24:21 - Script killed after 1109883 seconds
As I stated above, this happens every time I force a re-boot with a new IP being allocated, by spoofing the MAC address and resetting the modem. I have done this now at least half a dozen times, or more, with always exactly the same results. I have similar trace files available from the last 4 reboots, if needed.
All of these sites are fully accessible 24 hours a day if I run a VPN and then try to access.
I have issues with other sites also, the CA DMV and the USPS Tracking site, but they exhibit different issues and I haven't been able (yet) to determine the reason/pattern.
When I first reported this, over a year ago, I tried everything I knew to try and get this in front of a network engineer who might be able to diagnose, but to no avail. I had first level techs insist a re-boot of the modem would fix it, or if not, then a modem replacement would. I had supervisors, who promised faithfully to follow up and get back to me. None ever did. I had a 2nd level support person, who I was referred to via a technician who came out to check the signals/modem, who for the first few times I e-mailed for status usually replied: I saw the ticket closed, so assumed it was resolved. Sorry, I'll re-open another. They eventually stopped responding to all my e-mails.
So the desperation question is how to get this to someone who knows how internet routing works, has access to the various Charter/Spectrum/TW servers involved, and has an interest in fixing customer issues. I'm kinda making an assumption that this has to be happening in the first couple of hops from my modem, otherwise there would be more people with similar issues.
I'd also be interested in anyone in the Los Angeles area can try those sites listed above to see if they have issues or not, especially if the gateway after the modem is: 188.8.131.52.
No errors, just time-outs:
[root@Nethserver ~]# nc webhosting.att.com 443 < /dev/null Ncat: Connection timed out. [root@Nethserver ~]#
Here are some traceroutes immediately after a reboot, with a new IP:
root@Nethserver ~# traceroute -T -p 443 webhosting.att.com traceroute to webhosting.att.com (184.108.40.206), 30 hops max, 60 byte packets 1 220.127.116.11 (18.104.22.168) 4.590 ms * 4.568 ms 2 * * * 3 * * * 4 * * * 5 * * * 6 * * * 7 * * * 8 HOSTOPIA.ear3.Miami2.Level3.net (22.214.171.124) 75.385 ms 75.843 ms 75.848 ms 9 core2-vl51.50.mia.megarouting.com (126.96.36.199) 77.161 ms 77.152 ms 77.139 ms 10 sushi.carrierzone.com (188.8.131.52) 76.683 ms 77.044 ms 76.996 ms root@Nethserver ~# traceroute -T -p 443 cts.vresp.com traceroute to cts.vresp.com (184.108.40.206), 30 hops max, 60 byte packets 1 220.127.116.11 (18.104.22.168) 5.246 ms 5.214 ms 11.545 ms 2 agg61.igwdcaew02h.socal.rr.com (22.214.171.124) 13.084 ms 13.084 ms 13.063 ms 3 agg30.lsaicaev02r.socal.rr.com (126.96.36.199) 17.242 ms 17.252 ms 17.230 ms 4 agg26.tustcaft01r.socal.rr.com (188.8.131.52) 25.465 ms 25.474 ms 25.463 ms 5 bu-ether16.tustca4200w-bcr00.tbone.rr.com (184.108.40.206) 25.906 ms 25.917 ms 25.905 ms 6 * * * 7 * * * 8 HOSTOPIA.ear3.Miami2.Level3.net (220.127.116.11) 74.933 ms 72.459 ms 74.571 ms 9 core2-vl51.50.mia.megarouting.com (18.104.22.168) 78.877 ms 78.655 ms 78.614 ms 10 22.214.171.124.ip.verticalresponse.com (126.96.36.199) 76.947 ms 78.346 ms 90.984 ms root@Nethserver ~# traceroute -T -p 995 websitesmail.att.com traceroute to websitesmail.att.com (188.8.131.52), 30 hops max, 60 byte packets 1 184.108.40.206 (220.127.116.11) 10.095 ms 10.080 ms 10.070 ms 2 agg61.igwdcaew02h.socal.rr.com (18.104.22.168) 13.136 ms 13.134 ms 13.121 ms 3 agg30.lsaicaev02r.socal.rr.com (22.214.171.124) 18.969 ms 18.957 ms 18.968 ms 4 agg26.tustcaft01r.socal.rr.com (126.96.36.199) 18.856 ms 18.868 ms 18.858 ms 5 bu-ether16.tustca4200w-bcr00.tbone.rr.com (188.8.131.52) 22.492 ms 22.503 ms 22.491 ms 6 * * * 7 * * * 8 HOSTOPIA.ear3.Miami2.Level3.net (184.108.40.206) 77.692 ms 78.101 ms 78.022 ms 9 core2-vl51.50.mia.megarouting.com (220.127.116.11) 77.085 ms 77.133 ms 77.099 ms 10 websitesmailc45.carrierzone.com (18.104.22.168) 74.422 ms 74.154 ms 74.044 ms root@Nethserver ~#
And the same after connectivity is lost:
root@Nethserver ~# traceroute -T -p 443 webhosting.att.com traceroute to webhosting.att.com (22.214.171.124), 30 hops max, 60 byte packets 1 126.96.36.199 (188.8.131.52) 10.864 ms 10.862 ms 10.853 ms 2 agg61.igwdcaew02h.socal.rr.com (184.108.40.206) 12.982 ms 12.980 ms 12.972 ms 3 agg30.lsaicaev02r.socal.rr.com (220.127.116.11) 17.523 ms 17.527 ms 17.518 ms 4 agg26.tustcaft01r.socal.rr.com (18.104.22.168) 23.315 ms 23.307 ms 23.360 ms 5 ae-5-0.cr0.chi10.tbone.rr.com (22.214.171.124) 23.211 ms 23.223 ms 23.215 ms 6 * * * 7 * * * 8 * * * 9 * * * 10 * * * 11 * * * 12 * * * 13 * * * 14 * * * 15 * * * 16 * * * 17 * * * 18 * * * 19 * * * 20 * * * 21 * * * 22 * * * 23 * * * 24 * * * 25 * * * 26 * * * 27 * * * 28 * * * 29 * * * 30 * * * root@Nethserver ~# traceroute -T -p 443 cts.vresp.com traceroute to cts.vresp.com (126.96.36.199), 30 hops max, 60 byte packets 1 188.8.131.52 (184.108.40.206) 10.030 ms 10.388 ms 10.013 ms 2 agg61.igwdcaew02h.socal.rr.com (220.127.116.11) 11.675 ms 11.687 ms 11.676 ms 3 agg30.lsaicaev02r.socal.rr.com (18.104.22.168) 13.812 ms 13.826 ms 13.814 ms 4 agg26.tustcaft01r.socal.rr.com (22.214.171.124) 23.327 ms 23.323 ms 23.314 ms 5 bu-ether16.tustca4200w-bcr00.tbone.rr.com (126.96.36.199) 21.620 ms 21.633 ms 21.623 ms 6 * * * 7 ae-1-51.ear3.Miami2.Level3.net (188.8.131.52) 76.408 ms 76.138 ms 75.697 ms 8 * * * 9 * * * 10 * * * 11 * * * 12 * * * 13 * * * 14 * * * 15 * * * 16 * * * 17 * * * 18 * * * 19 * * * 20 * * * 21 * * * 22 * * * 23 * * * 24 * * * 25 * * * 26 * * * 27 * * * 28 * * * 29 * * * 30 * * * root@Nethserver ~# traceroute -T -p 995 websitesmail.att.com traceroute to websitesmail.att.com (184.108.40.206), 30 hops max, 60 byte packets 1 220.127.116.11 (18.104.22.168) 10.005 ms 10.004 ms * 2 * * * 3 * * * 4 * * * 5 * * * 6 * * * 7 * * * 8 * * * 9 * * * 10 * * * 11 * * * 12 * * * 13 * * * 14 * * * 15 * * * 16 * * * 17 * * * 18 * * * 19 * * * 20 * * * 21 * * * 22 * * * 23 * * * 24 * * * 25 * * * 26 * * * 27 * * * 28 * * * 29 * * * 30 * * * root@Nethserver ~#
The first link opened fine while the second one gave the Forbidden window. The webmail link opened login window for email and password to be entered. Probably not helpful but just getting back to let you know since you wanted feedback from others.
The support folks on this forum were able to get a ticket opened for me at the beginning of June. This was then put on hold, as I was going to be out of the country for 3 weeks, so it really started being looked at towards the end of June. For the next couple of weeks I was getting call backs, on what seemed like a daily basis. Although based on the questions and/or supposed answers it was obvious that the issue had not been looked at by anyone who really understood what was really happening, or by people who had not read through the ticket, as they were repeating the same questions/answers multiple times.
Following this, I was told that no further research could be done as I was using my own modem, which meant they couldn't access it remotely. I asked how accessing my modem could help in researching a network routing issue, as it didn't make sense to me, but couldn't get any sensible answer. Anyway, I said if that's what they wanted, go ahead, send someone out and install your modem, which they then did.
It was at this point I was "ghosted". Now instead of being called daily, I have tried numerous times, for around 3 weeks, to get a call back for a status. Either no call back happens, or (twice !!) the ticket was routed to the "truck roll" dispatch, who obviously couldn't do anything.
On one of those numerous requests for a call-back, I asked for some further information to be added to the ticket, as I discovered that another website was inaccessible, the CA DMV !!!! For this one, I was able to capture some packet traces that show (as I had suspected and reported a number of times) that the issue appears to be in the return path, back to me. Here are the traces I captured:
You can see that the 3-way TCP SYN/ACK handshake succeeds, followed by a send of the "Client Hello", which is correctly ACK'd. But following this, the "Server Hello" packet never reaches me, and a few seconds later, the server does a RST to break the communication.
I tried to explain this to the "support" person, but as they had no clue about tcpdump, packet traces, TCP negotiation, etc. so I have absolutely no idea what is actually reported in the ticket or where it has been routed.
How do I get this escalated to someone who *does* understand these things, as I believe (and so do some of my network savvy friends) that this clearly (??) shows an issue with some of the packets sent back to me being blocked. Unfortunatley I can see no pattern as to why this is only on some websites and also not on all the packet data.
And to answer the first question, before it's asked: These traces were taken on the machine hard-wired to the modem, so there is no PC, router, server, etc. interferring with the flow.