How much is too much?

Greetings, I am trying to develop an application to analyze makets around a specific location in search of profitable opportunities but for a limited number of blueprints. This requires grabbing market data for items themselves, raw materials, data cores, descryptors, etc. Assuming I have to analyze ~1000 items and have 5 markets around me, it will require 10K+ requests to market/orders and market/history to build up my cache. First question: is that load too much and prone to be flagged as abuse? Is there any better way to accomplish this data load than hammering servers 10K times?
Thanks for your anticipated help.

Just respect the cache timers and you’ll be fine.

In addition to Blacksmoke16’s excellent advice, you also have to respect the error limit.
Unless you’re only getting a few types, it may be faster to get everything for /markets/{region_id}/orders/. Using the type_id parameter quickly become ineffective if you’re getting a lot of types.

1 Like

Thank you both for your replies. Here is my problem with both approaches:
Blacksmoke16: with market/orders, for example, ESI cache expires after 300 seconds (5 mins). This means that, if I query say every hour, all data in may cache is already expired so I have to pull all data again, re-issuing 10K+ reqs. I utilize etag to reduce server effort whenever possible but since orders data is very dynamic, etag is useless most of the times.
Golden Gnu: market/orders without type_id for The Forge can easily return 300+ pages that then must be retrieved one by one. Considering say five regions around me, I may need to generate +1000 requests. This approach effectively reduces number of reqs from ~10k to ~1k compared to other scenario but with ten fold amount of data. Not sure what is worse for servers: #reqs or #gigs.
With market/history data, it is worse since 98% of history is already on my cache but I don’t have any way to ask for incremental data only (say starting at specific date or number of days in the past). Huge waste of bandwidth.

You don’t have to worry about the ESI server, it can handle all the request you throw at it and don’t worry about the amount of data, either. Instead do what works best for you.
How often you update from ESI, is up to you, as long as you do not do it before the cache time expires. What is expensive for ESI is errors, that is why we have the error limit, so it’s very important to handle that and back off if you hit it.

I pull all the regions, once every 30 minutes. It’s fine.

It’s a far more efficient way to do it, pulling everything, because there are enough people doing it that the cache is primed already.

I pull all regions every 5 minutes with some not very fancy code. It’s very doable and the servers don’t complain. You can pull all of Forge serially in about 2 minutes, but many people send parallel requests to pull the pages faster.

Here’s a recent breakdown of the page counts to give you an idea what you’re in for:

Region ID Name Page Count
10000001 Derelik 18
10000002 The Forge 289
10000003 Vale of the Silent 1
10000004 UUA-F4 1
10000005 Detorid 1
10000006 Wicked Creek 1
10000007 Cache 1
10000008 Scalding Pass 1
10000009 Insmother 1
10000010 Tribute 1
10000011 Great Wildlands 3
10000012 Curse 14
10000013 Malpais 1
10000014 Catch 1
10000015 Venal 5
10000016 Lonetrek 68
10000017 J7HZ-F 1
10000018 The Spire 1
10000019 A821-A 1
10000020 Tash-Murkon 39
10000021 Outer Passage 1
10000022 Stain 11
10000023 Pure Blind 5
10000025 Immensea 1
10000027 Etherium Reach 1
10000028 Molden Heath 22
10000029 Geminate 2
10000030 Heimatar 69
10000031 Impass 1
10000032 Sinq Laison 106
10000033 The Citadel 51
10000034 The Kalevala Expanse 1
10000035 Deklein 1
10000036 Devoid 13
10000037 Everyshore 24
10000038 The Bleak Lands 11
10000039 Esoteria 2
10000040 Oasa 1
10000041 Syndicate 14
10000042 Metropolis 96
10000043 Domain 149
10000044 Solitude 17
10000045 Tenal 1
10000046 Fade 1
10000047 Providence 1
10000048 Placid 36
10000049 Khanid 9
10000050 Querious 1
10000051 Cloud Ring 1
10000052 Kador 25
10000053 Cobalt Edge 1
10000054 Aridia 17
10000055 Branch 1
10000056 Feythabolis 1
10000057 Outer Ring 4
10000058 Fountain 5
10000059 Paragon Soul 1
10000060 Delve 7
10000061 Tenerifis 1
10000062 Omist 1
10000063 Period Basis 1
10000064 Essence 44
10000065 Kor-Azor 21
10000066 Perrigen Falls 1
10000067 Genesis 31
10000068 Verge Vendor 28
10000069 Black Rise 17

I left of the 11* and 12* regions. They all only have one page each.

I keep and publish a post-processed version of all the data. See here if you need historical data for some reason.

1 Like

My main concern about server load is now resolved with your valuable help. Thanks all for tips and techniques too. You guys have a cool community here. Good hunting…

1 Like

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.