2023/05/11 - Ongoing - Intermittent chat issues

Nope.

Let’s say the issue was not enough server resources, they can click a few buttons or programmatically provision more server hardware… chuck a load balancer in front of it and away you go.

I also highly doubt it is the chat server for #reasons. If I were to make a prediction it will either be a code change in the chat client within the game, or a configuration issue with the load balancer(s) or the server. Either way, they can make changes on the server.

The cat was here and meowed

LOL Increased Tidi. Tidi has been increasing even for minor character movements for years now, even with the chat and lots of other systems externalized from the sol server cluster. If anything, all these steps have made the sol performance, or at the very least TQ performance, much worse compared to what it used to be before these steps were taken.
Outside tidi environments, the server performance has gone down a lot in the last half year as well. Ever since Photon was made mandatory and a few months prior to that, I kept wondering where CCP’s much lauded server performance improvements are happening. They elude me.

This sucks ngl

I have never liked this interaction of text chat. Those who were around when they made the switch will remember it was a cluster ■■■■. Local and chat channels didn’t work, leading to one friend of mine losing a Hel, which CCP said was his fault off the back of the new chat system “not having issues” - this despite CCP releasing an acknowledgement that the switch over was a ■■■■ show.

The new chat system also removed some useful features for moderators and channel operators, looking at you system messages… and it has pretty predictably led to issues where you can DC from the chat servers but not the game servers which gets obnoxious really fast.

But with that said, if their dev blog announcing the changes were to be believed moving chat out of TQ and off to other hardware provided a performance boost, see the excerpt of the dev blog below:

With the March release, we will replace the text chat backend for EVE Online to use ejabberd, a ‘Rock Solid, Massively Scalable, Infinitely Extensible XMPP Server’. Ejabberd will run on a cluster of machines in the cloud, so we’re not just separating chat from the game server application, we’re also taking it outside of the EVE cluster.

Why?
There are several reasons why we want to do this:

  • We want to break up the EVE game server into more manageable parts and this is one of the steps that gets us closer to a Microservices Architecture.
  • Text chat accounts for a significant portion of CPU use on the game server’s proxy nodes.

source: https://www.eveonline.com/news/view/new-chat-backend-coming-with-the-march-release.

I have long criticised the implementation of the new chat system, and let’s be honest the game’s general stability and performance have been degrading for ages. Prior to the Citadel interior overhaul, I was able to go near Citadel, now docking in one can give my computer a heart attack. - There is a huge amount of work CCP needs to do to make the game more stable and more useable; however, I don’t believe moving the chat services into the TQ data centre will do that.

One last thought before I pop off for the evening. This has been a pain in the arse. It’s been what four days now? That means 13.333% of my billing subscription has been partially wasted due to all the loopholes I just don’t want to deal with (e.g., missing fleet, corp and private channels).

I’ve protested in the form of chastising CCP here, filing a support ticket and bug reports. But what is really pissing me off is how radio silent they’ve become about all of this. I’m really thinking it’s just about the time we all start sending in reimbursement request support tickets if for no other reason than to create a paper trail and stack of tickets the GMs have to deal with.

4 Likes

This is epic! Thanks for the laughs y’all …

Been playing EVE since the beginning and coding even longer. To be fair, I haven’t actually logged in or played EVE in many years now but I just reinstalled the client thinking it might be fun to pew pew in low sec for the 20th anniversary with some T1 frigates or something … anywho, while downloading client I found this thread which is even more entertaining than Jita chat would have been. You know, if that was still a thing :wink:

On a more serious note about technical debt in software. It’s real. If you don’t burn it down and start from scratch with a new architecture every 3-5 years you are quite literally more likely to break anything you change. I can only imagine what a chat system rollback would look like in this catch 22 nightmare. Not to say I am a better coder than folks at CCP but they are definitely supporting some ancient code and any efforts to modernize without a nuke are going to end up something like this:

3 Likes

please add auto subscan feature )

It’s just still incomprehensible that such a company can’t get a grip on such a problem after years…main thing is that skins are being developed for citadels and more and more skins for ships instead of taking on the real problems…after the price increase this almost borders on rip-off… collect money, but slowly but surely drive the game to the wall…

But with that said, if their dev blog announcing the changes were to be believed moving chat out of TQ and off to other hardware provided a performance boost, see the excerpt of the dev blog below:

That’s what they claimed but it never materialized in a noticeable way. In contrast, chat performance has gotten worse and the general server performance is getting worse as well. The only thing that has actually gotten better with chats is the ability to spam harder, faster, more than ever before. And with the removal of features thanks to Photon it has become harder, more difficult and annoying to moderate chats.
I can understand their radio silence. I would also not talk about my work if it’s just so abysmally bad. What irks me, however, is that they keep boasting about how amazing everything is. They are completely delusional in this regard.
The other irksome thing with these things and CCP is that they apparently actively and on purpose screw something up that is going well. In the beginning, chats were really, really bad, then they managed to fix a lot of things and for a while chats were at least stable (still nowhere near the user comfort and stability of the old chat, but at least no disconnects every 5 minutes anymore). This lasted for a few months. And then, for some unannounced, inexplicable reason, things started to get worse again, disconnects started to become frequent again. And now this mess as culmination. It’s amazing in its own right how they manage to be so bad at this whole enterprise.

Enjoy your evening.

2 Likes

Heya CCP folks. While I’ve enjoyed having a giggle at all the incoherent screeching here, and personally don’t need local to undock things and have fun, it does seem like bad form not to drop any updates.

I’m sure there’s folks having their weekend ruined by this outage, so would be nice to share their pain (as an offering to the mob) with even just a little “we’re still working on this” update.

4 Likes

Thinking back to that post CCP put up on the forum where they solemnly promised that if they couldn’t get the new chat working within a month, they’d revert back to the old system.

That was in early 2020.

I mean, I sympathize, I really do. Like Tessa Lee said a couple posts up, if you don’t burn down the whole system every 3-5 years and start over, you end up with legacy-code debt that you can’t really untangle without breaking everything. And what they’re asking the chat system to do is something no off-the-shelf chat architecture is really designed to manage, so there aren’t any good alternatives to building a bespoke system that’s going to be inherently very fragile.

So here we are. Hope it gets fixed soon! Let’s all remember what happened to daily logins after a month or so of Blackout, and hope that doesn’t happen here.

1 Like

Why not just rollback the updates that are causing the chat issue, until you can figure out the programming error thats causing them? Why ruin the 20 years of excellence? :smiley:

2 Likes

Just imagine the poor developer who needs to work all night to solve this problem. Can we buy them a coffee or what? :smiley:

Or imagine calling those third party hosting companies and they are like “Sry It’s weekend, come back on monday.” :rofl:
Or try to get into contact with Amazon about the problems and they have only those automated robot mails google uses as well that don’t even try to hide they care about you, no matter how big of a fish you think you are :rofl:

Some Dev is gonna loose a weekend of sleep and you only loose like a weekend of ratting and mining in null.

1 Like

‘ancient code’ i think is the base game’s core. some kind of legacy here and it was working fine with old chats wich even had voice channels. But CCP decided to do new chats and to make problems to the players and to themselves.

1 Like

Don’t worry about that. Cloud providers like AWS have a very high SLA, and they lose money if they fail to meet it, so assuming this was an AWS problem which I highly doubt, they’d have people around to help out 24/7. That and well, typically when you rent compute power (or other cloud services) you tend to get management access to your part of the cluster.

At present, EVE Voice is used by just 0.4% of our active pilots and it’s currently one of several more things holding us back from developing a 64-bit EVE client after the retirement of Captain’s Quarters last summer.

source: Preparing For The Future – Retirement of EVE Voice | EVE Online

Why are we causing local blackouts? Man up and admit your intentions this isnt accidental
just roll out the black out allready. ffs

You are pitiful, when I lost my career because of you, you blamed me, I hope you will not blame the players for this major failure.
Who compensates me for the days lost without being able to enjoy this wonderful game?

3 Likes

i doubt they will, they are not bold enough to claim they want a black out so they just pull this petty stunt

Legacy code has one simple rule - dont touch if it works. Doesn’t matter how many people are using this functionality. But it works.

1 Like

While I completely agree more, looks at his own horrible yet functional code, the issue at the time was CCP was trying to remove roadblocks that were stopping them from releasing a 64-bit client so we could use more RAM.

1 Like