Dang DDoSing knucklheads.
After seeing this error when testing the ESI, lead me here. At least there is a good reason.
Any update on progress so far?
1 Like
Maybe just limit the number of requests to the API from one address per day, for example? Well, or turn on at a certain time. It’s kind of boring without the API.
All ESI bans are IP bans. problems only happens when the attacker are evading the IP ban, as they were in this case and have been in other cases as well. With cloud services like AWS IP rotation is pretty easy. So any limitation by IP will probably be ineffective, as attackers will just rotate the IP. That is why a solution is auth, as you can not rotate eve characters as easy as you can rotate IPs. ESI probably just isn’t hardened enough, because, people used to be super thankful for it and treat it with respect, being good API citizens.
Now also serving static CSV exports that cover the price or order volume (not trade!) history of all items in an entire region on a day or week basis via https://static.adam4eve.eu/. Should help reduce the effort to scrape the API.
2 Likes
The best way I’ve found to handle api calls like this would be to put it behind cloudfront which will cache similar requests and return them without touching your API, ec2 instance, lambda, or whatever you have configured.
It will also cache it at different locations so not only will you have insanely fast cached results, but you’ll also have it geographically closest to the IP making the call so it will be that much faster.
Caching on a CDN is a solution, given that once generated on a daily basis, is static for the rest of the day.
Also, auth would help weed out anyone truly hammering that API. At the very least, make it harder for a to-be DDoSer.
It is really unfortunate that this end point is down. My own service which many traders use was 100% reliant on this data particularly and there is nothing like it out there, so my site is down until this is (hopefully) up again soon. Ethan’s data is something different though I’m very grateful for what he does.
Also just let this data be a static download, it was already a pain to call every item for every region you required one by one even if you only had to do it once per day.
1 Like
It can do even nginx.
This endpoint was taken down after a DDoS on 4 November. 80 days later and it’s still down. I’m sorry to say but that was a hell of a successful DDoS.
And from what I can tell, no news since this topic was created, more than a month ago. Any updates on this issue would be appreciated.
Just a few days ago, ccp swift went into further detail what happened and why its not a quick fix
I think it’s safe to say that the wait for the ESI Market History endpoint has now (2023-02-10) reached a critical stage. Over two months without access to this essential api endpoint is having a huge impact on many tools, developers, and players.
The fact that the endpoint had to be taken down due to the constant spikes in CPU use and the subsequent performance issues is concerning, and it’s clear that immediate action needed to be taken. The market is a crucial component for the game, and the failure of this specific endpoint is having a severe impact on the entire community.
The plans to redesign or add authentication to the endpoint are a step in the right direction, but I think it’s imperative that the work being done is expedited as much as possible. The longer this issue persists, the more damage it will cause to the community and the game as a whole.
I implore the teams involved to find a solution to this situation as soon as possible. The community has been patient for too long, and it’s high time that a resolution is found. I hope that everyone involved is working diligently to restore the market and its endpoint to full functionality.
We can’t afford to wait any longer, and I hope that a solution is found soon. Thank you for the update (although to late and to less) and effort being made to resolve this situation.
GO!
3 Likes
is there no rate limiter in place?
They “temporarily” removed the bounty system in October 2020.
CCP just deleting the game piece by piece.
1 Like
when?
any update?
2 Likes
anyone home?
@CCP_Zelus In the short term, has the team considered working with the larger partners and building a static IP whitelist for the API?
It’s a good thought in the event that anything like this happens in the future, we actually considered it as well as a form of mitigation, but the system was designed to be open to all, then closed to specific IPs as required (i.e a blacklist system).
We also don’t want to punish any (new) developers trying to access the endpoints trying to develop apps that are using dynamic IP ranges that need constant updates too.
1 Like