ESI Market History Endpoint

Dang DDoSing knucklheads.

After seeing this error when testing the ESI, lead me here. At least there is a good reason.

Any update on progress so far?

1 Like

Maybe just limit the number of requests to the API from one address per day, for example? Well, or turn on at a certain time. It’s kind of boring without the API.

All ESI bans are IP bans. problems only happens when the attacker are evading the IP ban, as they were in this case and have been in other cases as well. With cloud services like AWS IP rotation is pretty easy. So any limitation by IP will probably be ineffective, as attackers will just rotate the IP. That is why a solution is auth, as you can not rotate eve characters as easy as you can rotate IPs. ESI probably just isn’t hardened enough, because, people used to be super thankful for it and treat it with respect, being good API citizens.

Now also serving static CSV exports that cover the price or order volume (not trade!) history of all items in an entire region on a day or week basis via https://static.adam4eve.eu/. Should help reduce the effort to scrape the API.

2 Likes

The best way I’ve found to handle api calls like this would be to put it behind cloudfront which will cache similar requests and return them without touching your API, ec2 instance, lambda, or whatever you have configured.
It will also cache it at different locations so not only will you have insanely fast cached results, but you’ll also have it geographically closest to the IP making the call so it will be that much faster.

Caching on a CDN is a solution, given that once generated on a daily basis, is static for the rest of the day.

Also, auth would help weed out anyone truly hammering that API. At the very least, make it harder for a to-be DDoSer.

It is really unfortunate that this end point is down. My own service which many traders use was 100% reliant on this data particularly and there is nothing like it out there, so my site is down until this is (hopefully) up again soon. Ethan’s data is something different though I’m very grateful for what he does.

Also just let this data be a static download, it was already a pain to call every item for every region you required one by one even if you only had to do it once per day.

1 Like

This post was flagged by the community and is temporarily hidden.

It can do even nginx.

This endpoint was taken down after a DDoS on 4 November. 80 days later and it’s still down. I’m sorry to say but that was a hell of a successful DDoS.

And from what I can tell, no news since this topic was created, more than a month ago. Any updates on this issue would be appreciated.

Just a few days ago, ccp swift went into further detail what happened and why its not a quick fix

For things like market history that will never change going forward, is it not worth creating bulk files for older data and only having recent data (say, last 7 days) on the ESI. People can then continue to use the ESI and build up their own data over time, but people wanting to scrape all of the data could just grab the bulk files and ingest them rather than scraping the endpoint.