Greetings, I am trying to develop an application to analyze makets around a specific location in search of profitable opportunities but for a limited number of blueprints. This requires grabbing market data for items themselves, raw materials, data cores, descryptors, etc. Assuming I have to analyze ~1000 items and have 5 markets around me, it will require 10K+ requests to market/orders and market/history to build up my cache. First question: is that load too much and prone to be flagged as abuse? Is there any better way to accomplish this data load than hammering servers 10K times?
Thanks for your anticipated help.
Just respect the cache timers and you’ll be fine.
In addition to Blacksmoke16’s excellent advice, you also have to respect the error limit.
Unless you’re only getting a few types, it may be faster to get everything for /markets/{region_id}/orders/. Using the type_id parameter quickly become ineffective if you’re getting a lot of types.
1 Like
Thank you both for your replies. Here is my problem with both approaches:
Blacksmoke16: with market/orders, for example, ESI cache expires after 300 seconds (5 mins). This means that, if I query say every hour, all data in may cache is already expired so I have to pull all data again, re-issuing 10K+ reqs. I utilize etag to reduce server effort whenever possible but since orders data is very dynamic, etag is useless most of the times.
Golden Gnu: market/orders without type_id for The Forge can easily return 300+ pages that then must be retrieved one by one. Considering say five regions around me, I may need to generate +1000 requests. This approach effectively reduces number of reqs from ~10k to ~1k compared to other scenario but with ten fold amount of data. Not sure what is worse for servers: #reqs or #gigs.
With market/history data, it is worse since 98% of history is already on my cache but I don’t have any way to ask for incremental data only (say starting at specific date or number of days in the past). Huge waste of bandwidth.
You don’t have to worry about the ESI server, it can handle all the request you throw at it and don’t worry about the amount of data, either. Instead do what works best for you.
How often you update from ESI, is up to you, as long as you do not do it before the cache time expires. What is expensive for ESI is errors, that is why we have the error limit, so it’s very important to handle that and back off if you hit it.
I pull all the regions, once every 30 minutes. It’s fine.
It’s a far more efficient way to do it, pulling everything, because there are enough people doing it that the cache is primed already.
I pull all regions every 5 minutes with some not very fancy code. It’s very doable and the servers don’t complain. You can pull all of Forge serially in about 2 minutes, but many people send parallel requests to pull the pages faster.
Here’s a recent breakdown of the page counts to give you an idea what you’re in for:
| Region ID | Name | Page Count |
|---|---|---|
| 10000001 | Derelik | 18 |
| 10000002 | The Forge | 289 |
| 10000003 | Vale of the Silent | 1 |
| 10000004 | UUA-F4 | 1 |
| 10000005 | Detorid | 1 |
| 10000006 | Wicked Creek | 1 |
| 10000007 | Cache | 1 |
| 10000008 | Scalding Pass | 1 |
| 10000009 | Insmother | 1 |
| 10000010 | Tribute | 1 |
| 10000011 | Great Wildlands | 3 |
| 10000012 | Curse | 14 |
| 10000013 | Malpais | 1 |
| 10000014 | Catch | 1 |
| 10000015 | Venal | 5 |
| 10000016 | Lonetrek | 68 |
| 10000017 | J7HZ-F | 1 |
| 10000018 | The Spire | 1 |
| 10000019 | A821-A | 1 |
| 10000020 | Tash-Murkon | 39 |
| 10000021 | Outer Passage | 1 |
| 10000022 | Stain | 11 |
| 10000023 | Pure Blind | 5 |
| 10000025 | Immensea | 1 |
| 10000027 | Etherium Reach | 1 |
| 10000028 | Molden Heath | 22 |
| 10000029 | Geminate | 2 |
| 10000030 | Heimatar | 69 |
| 10000031 | Impass | 1 |
| 10000032 | Sinq Laison | 106 |
| 10000033 | The Citadel | 51 |
| 10000034 | The Kalevala Expanse | 1 |
| 10000035 | Deklein | 1 |
| 10000036 | Devoid | 13 |
| 10000037 | Everyshore | 24 |
| 10000038 | The Bleak Lands | 11 |
| 10000039 | Esoteria | 2 |
| 10000040 | Oasa | 1 |
| 10000041 | Syndicate | 14 |
| 10000042 | Metropolis | 96 |
| 10000043 | Domain | 149 |
| 10000044 | Solitude | 17 |
| 10000045 | Tenal | 1 |
| 10000046 | Fade | 1 |
| 10000047 | Providence | 1 |
| 10000048 | Placid | 36 |
| 10000049 | Khanid | 9 |
| 10000050 | Querious | 1 |
| 10000051 | Cloud Ring | 1 |
| 10000052 | Kador | 25 |
| 10000053 | Cobalt Edge | 1 |
| 10000054 | Aridia | 17 |
| 10000055 | Branch | 1 |
| 10000056 | Feythabolis | 1 |
| 10000057 | Outer Ring | 4 |
| 10000058 | Fountain | 5 |
| 10000059 | Paragon Soul | 1 |
| 10000060 | Delve | 7 |
| 10000061 | Tenerifis | 1 |
| 10000062 | Omist | 1 |
| 10000063 | Period Basis | 1 |
| 10000064 | Essence | 44 |
| 10000065 | Kor-Azor | 21 |
| 10000066 | Perrigen Falls | 1 |
| 10000067 | Genesis | 31 |
| 10000068 | Verge Vendor | 28 |
| 10000069 | Black Rise | 17 |
I left of the 11* and 12* regions. They all only have one page each.
I keep and publish a post-processed version of all the data. See here if you need historical data for some reason.
1 Like
My main concern about server load is now resolved with your valuable help. Thanks all for tips and techniques too. You guys have a cool community here. Good hunting…
1 Like
This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.