ESI - Public Character Data

EVE Technology and Research Center Third Party Developers
1

Hey everyone!

I was advised today that there was a change made on Friday to:

GET esi-characters/characters/{character_id}/ route, AKA the get character information route. The change was that the cache time has been updated from 1 day to 7 days in response to an increase in the number of projects scraping it.

Apologies the messaging for this one wasn’t made sooner, I’m endeavoring to try and get as much messaging about ESI changes as quickly as we’re made aware of them.

e: Via @CCP_Swift

I will add a bit more context as to some of the ESI failures, as it’s, unfortunately, more widespread than this endpoint. he Market data endpoint has been down, and now two relating to character corporation history (though afaik not alliance history), are down or impaired.

The TL;DR for why these endpoints were taken offline is that there is a third-party developer, or a series of third-party devs, using AWS and requesting information from these endpoints so frequently that it puts the rest of the server at risk. These are on the scale of a 6,000% increase in hourly requests. Typically we’d use IP bans to stop the attack and then players would reach out to us, however, that strategy is no longer working in a climate where AWS can assign another IP immediately.

The team is aware of how severe these outages are and are actively pursuing solutions that will restore functionality and give us more tools to block errant programs or bad actors without taking the entire service offline.

e: back from @CCP_Zelus

We have adjusted this back from 7 days to 1, which was what it originally was this afternoon. This may again be increased in the future if the requests are found to impair the server.

A recommendation from the team also suggests using as many of the auth’d endpoints as possible for future development, as those are easier to maintain uptime for. For this case, the affiliation endpoint may prove to be more beneficial in the long run!

8 Likes
2

Is there a way to use cloudflare to filter these ‘attacks’ by these spammers?

I have scripts I use to identify spammers and insert rules if you want copies of them using cloudflare.

2 Likes
3

That is insane. Anything that relies on this is rendered all but useless. Might as well rely on a quarterly newsletter delivered by post on recycled paper.

A week? Unbelievable.

2 Likes
4

Does this mean that WH corps will not be able to get new members onto their mapping tool for one week?
That’s hilarious…

1 Like
5

I really cant express how much this breaks the opsec of wormhole corps and wider corps that use ESI for their out of game Info/Op Sec.

This also currently prevents us from getting new members onboarded for 7 days, do you think that a reasonable time for a player to get set up with a mapper in this day and age ?

2 Likes
7

Use the affiliation endpoint, that cache is only 1 hour.

5 Likes
8

inb4 it gets cached too next month because it’s spammed

1 Like
9

potentially not an issue - as depending on how it is implemented the lookup could be a lot easier and faster than grabbing all the character information each time.

10

Yes that is actively happening to my friend right now lol

11

this is the way

12

Just the most recent in a series of devastating blows targeted at third party developers.

Whilst the affiliations endpoint exists, frankly it sucks and has been subject to breaking changes over the last year. A single doomheim character will break a 500 character batch with no indication. Needing to check each individually.

Nearly all of Alliance Auths 40k users relies on this endpoint for some of our minor apps, (not core affiliation checks). And these will either need to have features removed or the entire app decommissioned.

We just want to stay engaged and playing EVE without being shafted every week

12 Likes
13

https://esistatus.com/

3 Likes
14

HIghly Displeased with this change. Please stop making breaking changes without consulting the CSM and the players that its going to affect.

3 Likes
15

Meanwhile in the CCP boardroom - Devs are in a question-and-answer session with CCP leaders.

CCP Leaders - "What can we do to piss off almost everyone that pays for our game platform this week?

Rando CCP Dev #1 - " Lets change ESI endpoint stuff, without discussing it with anyone from the player base. "

Rando CCP Dev #2 - " Should we tell the CSM or leak this somewhere? "

CCP leaders - " No, screw what they want. We charged them more, and yet they still paid more. " Maybe we can use this as another attempt to gain additional income. "

Rando CCP Dev #1 - " We need more money for the ESI servers login rewards campaign?"

CCP Leaders - " Roll it out next week! "

This is posted in Satire - But really, this really messes stuff up for corporations that have zero IT folks. It also makes things harder for the people ( like me and the dudes above ) that help those zero IT corporations with ESI Stuff.

Can we get a detailed what in the heck is going on over there from someone important please?

#CCPLEASE

2 Likes
16

afaik, any corp using the esi as their part of the recruitment progress is affected

If one register the esi program/website before joining the corp in game, he/she have to wait for 7 days to make the cache flushed. That’s a bad news.

1 Like
17

Not a software expert, but this isn’t an acceptable solution. This is really burning everyone!

Edit: saw this on reddit, valid solution? Ugh. “Just put every endpoint behind auth, make omega a requirement and just permaban the people who F*around instead of breaking the ESI further every single week.”

18

Does CCP have employ any actual engineers? (Rhetorical question since decisions like this tell us all that the answer is “No.”)
There are more elegant solutions to this than increasing the TTL for that endpoint’s data.

19

In the CCP Twitch stream a week or two ago, there was a small segment where they actually talked about ESI. I believe it was Rattati who mentioned many players are unaware of how much of their character data is exposed via ESI, or understand what the impact of that is. He may have even used the phrase, “spied on.”

I have a recollection of him also saying ESI hadn’t been touched in a long time and they didn’t really have a roadmap for it. Then he mentioned, briefly, CCP were looking at ESI through a “why do we have each of these end points? What do we want ESI to do/be for? What end points are needed to support that?” lens, then going from there. I think, from a developer perspective, this is good and worthwhile introspection. I haven’t reviewed the VOD of that stream to validate my recolleciton, so I could be off.

There is clearly a benefit in having some of the character data available to improve the player experience via out-of-game tools. Current character affiliation (corporation and alliance) is a good example. Useful for automatically assigning Discord roles, granting access to shared/corporate mappers, etc. My opinion is CCP should make that available, and on something considerably less than a 7-day refresh cycle, for the reasons people have mentioned above, and because it obviously enhances the player experience. On the other hand, maybe there are a lot of, or at least some, ESI end points that really don’t need to be there.

I have no way of knowing if this ESI change is part of their briefly-mentioned broader re-assessment of ESI or not. If it is, it could/should have been handled differently.

1 Like
20

Why isn’t this endpoint used in the first place?
/corporations/{corporation_id}/members/

1 Like
21

This is really unacceptable. For wormholers, not only can new member not access our maps for up to a week, but way worse, anyone removed from a corp can STILL ACCESS our maps for that same period. This needs to be addressed.

1 Like