OUR servers are fine, everything is working perfect!

Seem to have caught this problem aswell…

Yeah for me it’s always the 2. client to log in that gets DCd within less than 5minutes. Not the SAME client, the second one to log in.

Been really annoying for more than a week now…

The interesting part is that I keep getting disconnects for the first 5-10 minutes after starting first client and then once that stops it’s rock solid for hours. That is until I shut down all clients and then start them up again a bit later, then the enjoyment of constant disconnects starts again for 5-10 minutes and then things are stable again.
Probably their “DDoS” protection working as intended or CCP is getting desperate to get those player engagement numbers up so is trying to nudge is in the direction of never logging off clients.

This sounds as exactely the same problems as i have. think its a CCP problem so i simply wait for fixes before throwing away my whole network configutation… maybe they are protecting something “too good” . we will see…

1 Like

Might be something to do with this - aint an IT person so I dont know what “BGP flapping” or whatever is ^^’

It’s really weird that it affects some clients and not all, I would expect any BGP or similar to kill all clients. I really get just a subset disconnecting often and then being stable for hours. Today, 3 out of 4 clients were stable from startup while 1 kept disconnecting for 10-15 min and has now been stable (and probably will for hours).

If you are a Deutsche Telekom AG customer and have been experiencing more disconnects since 1-2 August, then please create a support ticket. If you are playing from Germany and have been experiencing disconnects since the beginning of August even if your ISP is not DTAG, then we want to hear from you. We are trying to understand what is happening in this graph:

1 Like

Submitted. Thanks for looking into this.

Submitted.
Thanks CCP Explorer for looking into this and giving us hope :slight_smile:

First off: Please DO NOT post any of your details as a reply in this thread as they will contain private information. Rather send us a support ticket with those details; but most importantly open a support ticket with Deutsche Telekom AG about network route instability (if DTAG is your ISP).

Earlier this year then customers of Virgin Media UK experienced a very similar issue as is described in this thread and as we are seeing in metrics now for Germany. See this forum thread: 20220225 - Connection Issues (UK).

See also these posts from me for that incident:

At the time when we were debugging this with Virgin Media UK customer, there was this helpful advice from @Alaska: 20220225 - Connection Issues (UK) - #382 by Alaska

Open a blank text file. Copy the following into your file:

:START
@echo Started: %date% %time%
tracert tranquility.servers.eveonline.com
@echo Completed: %date% %time%
timeout 60
GOTO START

Save your file (ie filename.bat) to your desired folder. MUST be .bat

Open a Command prompt window. Drag your saved file into the command prompt window (it will show the location when dragged) and hit enter/return to run it.

When disconnected, switch to that CMD window and allow a traceroute to run. Then stop it, and scroll through the latest traceroutes and see if any of them changed; i.e. did the network traffic suddenly go a different route on its way to Tranquility (which is tranquility.servers.eveonline.com // 172.65.201.188). If there is a route change at the time of the disconnect, then we want to hear about it (in a support ticket).

BTW, this issue can’t be fixed in a patch and can’t be fixed on our side. This needs to be fixed by DTAG and Cloudflare, possibly with Level3’s involvement since we have seen Level3 in Frankfurt in traces that have been sent to us.

I’m not clear how route change would impact only some of the connected accounts. Especially as mine (the ones that are DCing, several are rock stable from start, but not always the same) are DCing in the beginning and then stabilise after 10-15 min to be stable for hours on end (until I restart them).

That is indeed a bit of a mystery.

A sudden route change will result in stateful network devices on the new path to reply with a TCP RST to reset the connection. At that point the EVE Client needs to reestablish the connection to Tranquility through the new Cloudflare location that the network connection is going though. Normally this will result in all running EVE Clients to reset since normally they would all be transmitting and receiving messages.

There is an edge case if the network route flips back and forth very quickly and it so happens that one the clients was not transmitting/receiving then it would stay connected (or there would be packet loss but a re-transmission would be requested and succeed and no RST packets would be sent.

Hence we suspect some other network equipment failure at Deutsche Telekom AG where only some of the connections are terminated (for reasons unknown to us).

We are not seeing this pattern from other countries starting 1-2 August, either they are the same or better (e.g. connectivity in Australia improved significantly at the tail end of July so we just came off that investigation to this one), and so far the only German players that have contacted us are with Deutsche Telekom AG.

At this time i am unable to log into eve, no matter what toon I try to use.
I figured that it has to do with a node along the way to tranquility server.
Now one of two things comes to mind. The node where the event is occurring is either suffering from a hardware failure or it is under attack. Kinda looks like a new play on the DDOS attacks Eve dealt with awhile back. Just pinged the Tranquility server no packet loss. So that is telling me that the port may have a guardian or gate keeper on it. Which is blocking connections… And please do not bother asking, I am not being paid to figure out Eve’s connection issues. I can find another game to play.

This is neither helpful nor fair. CCP can’t control internet traffic and networks of other companies. Plus, with these very complex systems mistakes happen.

They are trying to figure out what is going on and are actively monitoring disconnects which is great proactive move.

1 Like

The flapping could be the case but it happens only in the first 5-10 min of starting up the client. Then its perfectly stable for hours on end. I’ve submitted loges showing initial instability and then stable for almost 2 hours at which point I stopped logging and did my support ticket (clients stayed stable for far longer). Hopefully they help shine some light on this.

I guess the next step would be logging the packet traffic somehow. :grimacing:

1 Like

Was there a dev post about this I missed ?

The internet system we have here is awful ( known as the NBN ) and last month was the worst for disconnects. Most of the EVE players I know try and mitigate this by using a VPN, but I wonder why this is even necessary if the local cloud flare node is correctly routing the game through its own backhaul?

ok tbh the last months i am not motivated a lot to play eve… but i discovered a few weeks ago when collecting at least login rewards i get more and more frequent DCs even with just logging into the charakter selection screen.

I mean… i witnessed the times when you had like every few minutes dcs after the chat server changes back than… or frequently chat disconnects etc… but before even beeing in space is new for me.

Happens one out of 3 trys in average (up to 10/10 trys before it works) atm in like the 1-3 minute after starting the client.
before it was at least rare… now its atm very common… but at least… if i make it past the first few minutes its stable for a few hours… i mean didnt had a DC mid game… just right after the login between char selection and first few minutes in game.

And had no issues on any other service or online game the last months on my end.
But well… eve is special sensitive i know… xD

when i took some sleep i will read here a bit more closely (did join and watch a 24 hour stream before and now i am dead but stumbled over this thread and it sounded like my issues the last weeks so i did save it now for later ;p)

No.

The issues were happening in ISPs’ network systems before reaching the local Cloudflare locations in Australia.

See also my next post with more details about Deutsche Telekom AG as that may be relevant to players in Australia and New Zealand.

Some EVE players who are customers of Deutsche Telekom AG (DTAG) and have been affected recently by random disconnects have reported to us that they have resolved the issue by installing https://cloudflarewarp.com/, which is software from Cloudflare, and using it in full WARP mode. You can read more about this software here https://blog.cloudflare.com/warp-for-desktop/ and here Windows desktop client · Cloudflare WARP client docs. Please note that I’m not requiring, asking or recommending that you install any software on your computer and take no responsibility for this particular software; I’m only relaying the experience of other DTAG customers.

This software is essentially a Cloudflare-specific VPN: “The WARP application uses BoringTun to encrypt all the traffic from your device and send it directly to Cloudflare’s edge, ensuring that no one in between is snooping on what you’re doing. If the site you are visiting is already a Cloudflare customer, the content is immediately sent down to your device.”

Given that Tranquility is serviced by Cloudflare’s network, then if customers of DTAG are able to bypass the disconnect issues by using a Cloudflare VPN-style application, then the root of the disconnect issue is clearly at DTAG.

If you do decide to try this out, please let us know if the disconnects stop.

Further, DTAG customers may want to consider adding their complaint to https://telekomhilft.telekom.de/t5/Telefonie-Internet/BGP-Flaps-Long-lived-TCP-Connections/td-p/5814336 on DTAG’s forums.

I want to comment on that thread:

Ist halt kein Telekom Problem … die nutzen das Peering was der Anbieter vorgibt.

We have seen this happen at ISPs, where they do get the correct BGP information from Cloudflare on the route to Tranquility but the ISPs’ equipment either has out-of-memory issues or a prefix limit and throws the correct routing information away.

Level3 kann halt nen Problem sein, da die neben ner super ausgebauten Autobahn zum Telekom Netz auch ne billige Schotterpiste haben.

We have seen both versions of this issue. Firstly then we actually disable all routing via Level 3 as much as we can on and from our network equipment as many EVE players have had Level 3 related issues in the past. In particular then EVE players in South America and Mexico at certain ISPs were earlier this year unable to even connect to Tranquility before we cut Level 3 off as much as we could by not peering with them in London. Secondly, some ISPs may choose to ignore BGP routing if they have a cheaper exit point off their network.

Telekom ist nen Tier-1 Provider … wenn CCP den größten Kundenstamm in Europa (das Telekom AS ist auch für deren Konzerneinheiten in den anderen Ländern da) erreichen will, müssen se halt bissel Geld in die Hand nehmen.

Such as hosting Tranquility at Cloudflare? We have already done that, but even then we have seen large ISPs then fail at keeping steady connections to Cloudflare.

Im Telekom Netz ist alles fein. CCP bzw. deren Provider muss schauen, was se da in deren Peering Bereich so treiben bzw. bestellen.

I can’t make any claims on what is wrong in this particular case, but experience from similar incidents tells us that the issues such as this one are always in the ISPs’ networks. I’ve also repeatedly had network admins at large ISPs tell me that all is fine in their network but when presented with date over the course of days or even weeks then they ultimately find the cause in their networks. Which is why I always start with finding a possible cause at the ISP before moving to finding a cause at Cloudflare or in our own network equipment. Issues in our own equipment manifests very differently - normally then multiple users spread over the world are having issues. With high probability then localised issues have localised root causes.

Dazu ist es bei “long lived” TCP-Connections (was für ein Buzzword, es sind stinknormale TCP-Verbindungen) vollkommen egal ob sich dabei zwischendurch die Route ändert, davon bekommen die Pakete gar nichts mit.

All stateful devices on the route will notice and send RST in response. No one connects directly to Tranquility but rather to firewalls, DDoS protection and TCP proxies on Cloudflare’s edge. You can read more about the setup here CCP Games | Cloudflare.

1 Like

We are moving all discussion on disconnect issues affecting customers of Deutsche Telekom AG to this new thread: