Keeping market history updated using ESI

[Dalden_V]

Hi there!

I’ve recently returned to the game and I’m busy updating all my tools to use the new ESI API.

I’d like to keep my local copy of the market history for The Forge region updated daily and I’m currently using the GET /markets/{region_id}history/ endpoint to do so.

I call this endpoint for each type_id, and this is working, but I was wondering if there’s a more efficient way of doing this without retrieving redundant data daily?

If I updated the history for all items yesterday then today I don’t need the history going back a year, I just need the one day I’m missing. Is there a way to do that? Or even better, is there a way to get the history for all item_types for just the last day in a single call?

[Blacksmoke16]

No, you would have to just parse the response and filter out the date range you need.

[Dalden_V]

Thanks @Blacksmoke16, that’s what I’m currently doing, just seems wasteful throwing away data I’m not using. CCP doesn’t mind us doing this request every day for all items in the region?

[Piotr_Leusten]

I haven’t actually tried programming this, but might this be what etags are for?
https://developers.eveonline.com/blog/article/esi-etag-best-practices

[Steve_Ronuken]

etags are just to identify a version of the content.

So you send an etag with the request, and if you have the most recent, the server responds with a 304 response, saying “you’ve got it.”

If it doesn’t match, you get the content with a 200 tag.

[Dalden_V]

Yeah, etags won’t help here, because each day there will be a new version that includes the new day’s history as well as a year’s worth of history I already have

[Catherine_Solenne]

They’ll still help for low-volume types that don’t necessarily sell every day

[Ethan02]

See ESI ticket https://github.com/esi/esi-issues/issues/706 regarding this and feel free to upvote to get it to the attention of the devs quicker

[Dalden_V]

Thanks @Ethan02, that’s exactly the issue, I guess it’s a trade-off between the cost of transmitting the data and the cost of filtering the data which could impact caching on the server side. I’ll keep an eye on the GitHub issue and in the meantime just continue filtering the data locally.

[Krysenth]

The etags wouldnt work entirely well with this endpoint as whenever there’s new data (aka “today”), it’s likely dropping off the oldest entry. That cascade’s every page and etags arent useful.

Now, if it grouped the response by month objects instead, and returned those as “pages” etags would work, as if you’ve already got last july through may historical prices already, you’re just interested in the new ones for June. But then you’d just call the latest month anyways.