Devblog: Update On Chat System Issues

XDQhz · January 24, 2019, 1:34pm

In my experience RabbitMQ/AMQP really only suit for stream processing not something need synced.
A key-value store like redis might better suited when two part need decouple while still synced.

XDQhz · January 24, 2019, 2:23pm

CCP_Explorer:

Correct; but it is not the leaving/joining in isolation but also because the game cluster must announce every pilot in local channels (through admin messages that all clients pick up and process but do not display) even if the client of that pilot has not joined that local channel.

A practical example: You are jumping from A to B.

In normal chat the game cluster would call the chat cluster and revoke your access to local A and grant you access to local B. The chat cluster would in response kick you from local A and notify you and all clients in local A. It would then wait for your client to connect to local B and then announce you in local B. For large channels you would not get the full member list and any member list updates might be batched and/or delayed. All very minimal and moderately time critical; very chill.

In EVE chat the game cluster calls the chat cluster, revokes your access to local A, grants you access to local B and then through admin messages immediately announces you having left local A and joined local B. The chat cluster kicks you from local A and notifies your client and other clients in local A. It then waits for your client to connect to local B and then announces you in local B (again; these action/messages may happen in various order since this is a distributed system, but there is always double notification). For large channels you get the full member list when you join and any member list changes are broadcasted immediately. Care must be taken in heavy traffic channels to ensure that you get all individual changes compared to the full member list set you received at the beginning; if the list is snapshotted at time X then you must receive all individual updates at X+ even if you haven’t received the full list yet (you might not have received the list because it takes time to transmit that large list and process it compared to the smaller individual notifications).

Recently we have been looking at the performance of ejabberd, e.g. how it caches data and performance optimizing ejabberd code, and have been deploying such improvements regularly.

Today at downtime (16 Jan) we deployed a rewrite of this snapshotting and transmitting of the initial full member list because the previous implementation was causing issues (including blocking of subsequent actions).

I think doing so really pushed your self into the corners. Local in eve really didn’t need much fancy features. I belive a far simple inhouse broadcast style system will both more reliable and much easy to maintain than ejabberd. You can still use xmpp protocal to send data to the client if you want.

Nevyn_Auscent · January 24, 2019, 4:27pm

You might be right that it is a distant flash back, probably it went with the “old chat”.
I can’t find it in game any more, but it certainly used to exist.

mkint · January 24, 2019, 7:55pm

There was a while I was dropping in to Constellation and Region especially at trade hubs, to see if anybody else was ever in there. It was almost eerie. I believe it was a recent speakers or delayed channel, so you never really knew if you were alone or not when you joined it. It felt like talking into an empty auditorium while wearing a blindfold.
Oh, and the game automatically kicked you out of them, I think at every session change. They were effectively broken, wouldn’t be surprised if they got nuked.

zen-vexor01_LSG · January 26, 2019, 11:31am

don’t mind me im just here for the comments

Sebat_Hadah · January 30, 2019, 12:37am

Chan_aar · February 2, 2019, 11:09am

Well @CCP_Falcon you have had your three weeks, we are past the end of January so whats it going to be?

Peter_Yurgin · February 2, 2019, 11:27am

Yo @CCP_Falcon! Whats the verdict? What’s up? To many times local now writes names on top of other names of people in local and you only see the name on top. Whats the fix?

Dread_Saboteur · February 2, 2019, 11:51am

@CCP_Falcon submit! You guys failed to fix it

Cypherous · February 5, 2019, 12:54am

Well, January has ended, when can we expect to see the promised update regarding chat?

Peter_Yurgin · February 9, 2019, 10:16pm

@CCP_Falcon does it again. Speaks about a deadline and PVP’s it right to death. Nice explosion guys. There will be no change here guys.

Pogmo_Thon · February 10, 2019, 4:00am

The chats just all crashed…again

Chan_aar · February 10, 2019, 11:12am

Damn, just when CCP thought they had it sorted.

Broadhead · February 15, 2019, 8:01pm

Oh, and did I mention that local is still not updating properly?

shaun_27 · February 16, 2019, 5:48pm

I did notice that today. Saw someone in station but local was empty and client was running really smooth for once irc

Broadhead · February 16, 2019, 7:44pm

Oh, but it can’t be, because according to CCP they’ve fixed it. “Logs show nothing?”

Nevyn_Auscent · February 16, 2019, 7:53pm

Literacy seems to be an issue for you, because CCP didn’t say it was perfect and no errors ever would happen.
They said big improvements had been made in the leave/join area.

shaun_27 · February 16, 2019, 8:51pm

You know it most likely doesn’t help their servers for chat are not located in or near TQ game server.

Peter_Yurgin · February 18, 2019, 12:20am

Here is one problem the names write on top of each other now sometimes. But you never know when they do it and to fix it you have to adjust the side of local that has the names.

Jian_Mira · March 13, 2019, 8:58pm

Local chat not working today flying from Jita to Yoma, chat keeps resetting…

What the hell is going on!