I’ve recently created a new service that I think could be useful for the third-party developer community, and wanted to share it with you and ask for your feedback.
The service is an API for the SDE that provides the data via REST, which I believe could be a more convenient alternative to Steve’s SDE complements in SQL form (which are amazing!) in some instances.
For example, if you’re developing web-based tools that don’t really need/have a database, this API could simplify things a lot.
Thanks for this. I did wind up uploading a (processed) version of the SDE for myself in a DB but had originally coded hooks to read it from a (different) API.
To me the big question is the speed of an update propagating: when CCP updates the SDE how quickly is it reflected in your API?
Right now my struggle is detecting when the SDE changes and propagating it to my DB.
Generating all files and uploading them took me about 5 hours.
It should be feasible to have it update within 6 hours after the SDE changes to have it reflected in the API. This is not implemented yet, though! Updates are triggered manually, so expect 24h delays.
For your use-case I have some other tool planned in which I’d notify you (via webhooks) that the SDE changed, and give you a JSON with only the changes.
This could avoid the very unnecessary (IMHO) “ping API every X minutes and download everything” pull-based messaging the whole EVE 3rd party development currently relies on.
Sweet. This would be the single most & incredibly valuable reason for me to use the API. Instead of me attempting to manually trigger updates in a manner exactly as you described. Checking the page, downloading, unzipping, moving, generating updates, etc.
One additional concern is that I’m not (currently) behind cloudflare, so when I tried to use other SDE providers my traffic gets rejected. Not really sure why but probably because of bot related protections. I punted on figuring that out for now, but I would need to work through any similar issues before becoming an adopter.
Most likely their bot protections indeed. We would need to find a balance between users not getting captchas and me not having to sell my house due to bot attacks.
I’ve set it to “Essentially Off”, but that doesn’t mean it won’t happen, and I might have to re-enable it to manage costs. Will have to see how it goes.
Worse-case scenario API goes down and you’ll have the script to generate all files and you’ll be able to host it yourself.
So, since the checksum file is a really tiny file (32 bytes), you can just make periodical HTTP calls to its location, and parse the “Last-Modified” header from response. Probably most of HTTP-libraries from most of programming languages are able to get a header value from a server response, so it shouldn’t be a big challenge.
Good job but you are missing the endpoint to get the solar system name based on solarSystemNameID which it can be found in the mapDenormalize table/data pack
I think the schema and data format are stable enough for public use:
All data from the SDE released by CCP is available;
Additional data published by Hoboleaks is also available in their own endpoints (though the data won’t be updated daily, like Hoboleaks).
If you don’t trust me to run the service, the tool that generates the data is available via npm and can be executed via:
npx @jitaspace/sde-converter generate
It will download and extract the SDE and generate all the individual files that compose the API to a ./sde-workdir folder (or any other folder via the -w flag).
Source code is still not publicly available (though I’ll happily grant you access — just PM me!).
Please note it will download over 100mb and write ~2.5M files / 10GB of data.
