Hello all – I’m going to try and cover some of the things you mention here in this thread.
Sorry, very long post; I recommend you get your favorite beverage.
But first a recent network/disconnect experience I had with a realtime online bike simulator I use for my training: Zwift (https://www.zwift.com/) that I’ll use for comparison and reference. Zwift is a single-shard multi-world setup, just like EVE is a single-shard multi-solarsystem setup. When I was riding in the afternoon on 26 Jan (doing a warmup) I noticed that the rider list disappeared and then the event I was going to didn’t show up. I finished the warmup and restarted to see if the issue would correct itself but then I could not even log in since the Zwift application couldn’t fetch the world list.
Very long story short: All the normal network debugging, flushes, reboots and resets/reinstalls didn’t help. Zwift’s Tech Support couldn’t help. The 1st level Tech Support at my ISP couldn’t see anything wrong. This is where I contacted directly a network admin at my ISP, a former network admin at CCP and also a Zwifter – so he understood all sides of this problem. There were at least three parts to the mystery:
- A fiber cut on the US East Coast in the afternoon (UTC) of 26 Jan, see e.g. https://www.washingtonpost.com/technology/2021/01/26/internet-outage-east-coast/.
- A fiber cut in Northern Iceland at the same time in the afternoon of 26 Jan.
- Congested overland backhauls in the United Kingdom (London) for submarine cables from the US, over which traffic coming from AWS us-east-1 (N. Virginia) and us-east-2 (Ohio) was carried, but Zwift hosts their game in us-east-1.
The network admin at my ISP redirected this traffic from the UK (London) to the Netherlands (Amsterdam) through BGP advertisement changes to mitigate the issue and then we simply waited for these fiber cuts to be repaired.
The moral of the story: Zwift’s Tech Support could not do anything and had no way to get any insight into what was happening. I was able to quickly mitigate the issue for myself since I had a personal contact at my ISP with understanding of all sides of the problem and the necessary tools.
So how does this relate to EVE? Well, network issues are just the nature of the internet. Very often it works to reboot your computer, router, fiber box – and/or waiting for an hour or two. Sometimes you need to contact your ISP to try and find out what’s going on. We always want to hear about these issues (through Technical Support Requests) since sometimes we can help or we can see a larger pattern we can act on.
In my Zwift example, I was testing on 3 different computers and on my iPad (and testing my iPad both on my local network and also while hotspotted through my phone). One of the computers and the iPad while hotspotted could connect but the others could not.
Well, that depends. In my Zwift example then everything else on my network was fine. I speed-checked the network both internally and externally and checked for packet drops, tested video streaming, my kids were playing games, I tested EVE, I tested various websites. All was good, except Zwift. Do you play other realtime games than EVE that are hosted in London in datacenters close to Heathrow?
That ‘whatever system’ is Cloudflare. Tranquility is both behind Cloudflare’s Magic Transit DDoS Mitigation and it also behind Cloudflare’s Spectrum Proxy. Tranquility is in general inaccessible from the internet, connections must come through Cloudflare’s systems. When we moved to Cloudflare we saw issues relating to specific ISPs and we resolved them by coordinating action between those ISPs and Cloudflare. Issues that have cropped up since then, if they can’t be resolved through normal debugging tactics on your local network, need to be escalated to your ISP.
Now, we still absolutely want to hear about your issues, both to help you debug your local network (normal reboot, flush, reset stuff, but we also know about routers that don’t work with EVE) but also in particular in case there is a larger pattern that we can act on with Cloudflare. When you send us a Technical Support Request, please make sure to include information on your geographical location, if you use VPN or not, what DNS resolver you use, who is your ISP, and a traceroute to tranquility.servers.eveonline.com so we can see what Cloudflare colocation center you connect through.
Word.