Problem while downloading all market orders

There is an endpoint /dev/markets/{region_id}/orders/
While i am trying to get all orders for The Forge, its 90% feils to finish, because there are 285 pages for now, it is 285 requests in a cycle, 285 000 of orders in The Forge, so very often in process i reciave error from server.
For example i reciaved an error on page 97. I am trying to w8 for a moment and request this page agein, even if i will try to request this page 10 times, i will reciave error until i w8 for a long time. And when i w8 for a long time, data that i allready reciaved becomes too old. I wanna fetch prices every hour.

Is there an easyer way to reciave from api best buy and best sell jita prices in one request without overloading server with thousands of requests? If i will try to get prices using type_ids, there are 35780 type_ids in game now :frowning:

Easiest way would be to just get the aggregate data from like https://market.fuzzwork.co.uk/api/, which would save you the hassle of having to parse the entire market and aggregate it.

1 Like

238 KB - weight of 1 orders page
60 MB - weight of all orders in The Forge
60x24 = 1440 MB per day if refresh prices once per hour. And it is if player use cache and store orders somewhere and then use this data from other scripts.
CCP need to create best prices endpoint with ability to check bestprice range. For region, for system, for station\structure. Or even with using cache, players generate abnormal abount of traffic.

I think a fair number of people retrieve all the market data for Forge. You will get occasional connection and rate limiting issues (technically error limiting, but…), but if you really want to do it, it can indeed be done. My process takes about 2.5 minutes to retrieve Forge where I retrieve each page one at a time. As long as you’re careful not to cross over the cache timer expiry, it works fine.

I retrieve all market data across all regions every five minutes (current cache timer) and archive all the data for public download. You can read all about it here. I don’t compute the aggregates so this is overkill if that’s all you need. Steve’s data is pretty good. If memory serves, he’s pulling every 20 minutes or so.

Compressed, each of my snaps of Forge is 5.9M and I’m creating 288 a day so all in it’s about 1.7G. I post-process at end of day into a format more convenient for research which makes things much smaller. A typical day of order data for all regions runs about 200M or so.

I am working in google apps script, lots of limitations, one of them, that i cannot manage frequency of requests in one run, but i solve my problem in such way:

  1. I am running order retrieving script every 5 minutes
  2. It scnas cache for the 1st not cached page
  3. It starts to retrieve pages from 1st not cached to 1000
  4. When page downloaded, it splitted in to 3 parts and stored in to cache (because limitations of 100kb per 1 cache record) for 65 minutes
  5. If it recieve an error of last page found it will stop until next run.

So even if in current session i was not able to retrieve ALL orders, i retrieved some part, in next run i will retrieve one more part and so while running every 5 minutes it will cache all orders, and after 65 minutes pass, it will start to refresh orders older 65 minutes.

So each moment i have all Jita sell and buy bestprices with maximum lag of 2 hours. :slight_smile:

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.