EVE Technology Lab

 
  • Topic is locked indefinitely.
2 Pages12Next page
 

Devsite Blog: New CREST Resource for Bulk Market Orders

First post First post
Author
C C P Alliance
#1 - 2016-05-23 14:36:19 UTC  |  Edited by: CCP Logibro
To help with developers working with market data, we're adding a new Bulk Market Order endpoint to CREST. Find all of the details here.

CCP Logibro // EVE Universe Community Team // Distributor of Nanites // Patron Saint of Logistics

@CCP_Logibro

Requiem Eternal
#2 - 2016-05-23 14:58:54 UTC
Thank you

http://foo-eve.blogspot.com.au/

Goonswarm Federation
#3 - 2016-05-23 15:05:48 UTC
This is excellent news, and a great idea.

Unsolicited advice: a thing that stuck out to me was that the size of the returned response was still very large. Perhaps you might look into alternate transport formats for this data? Something like protocol buffers work very well for serializing data in a very small way. It may or may not be practical; just throwing it out there as a way to reduce the payload's size further.

This post was crafted by the wormhole expert of the Goonswarm Economic Warfare Cabal, the foremost authority on Eve: Online economics and gameplay.

Vote Steve Ronuken for CSM
#4 - 2016-05-23 15:16:58 UTC
Foxfour is a lovely lovely man.

Thanks for doing this Big smile And so quickly after I asked.

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter

C C P Alliance
#5 - 2016-05-23 16:12:17 UTC
Querns wrote:
This is excellent news, and a great idea.

Unsolicited advice: a thing that stuck out to me was that the size of the returned response was still very large. Perhaps you might look into alternate transport formats for this data? Something like protocol buffers work very well for serializing data in a very small way. It may or may not be practical; just throwing it out there as a way to reduce the payload's size further.


I would SOOOOO love to do this!

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.

Goonswarm Federation
#6 - 2016-05-23 17:47:09 UTC
CCP FoxFour wrote:
Querns wrote:
This is excellent news, and a great idea.

Unsolicited advice: a thing that stuck out to me was that the size of the returned response was still very large. Perhaps you might look into alternate transport formats for this data? Something like protocol buffers work very well for serializing data in a very small way. It may or may not be practical; just throwing it out there as a way to reduce the payload's size further.


I would SOOOOO love to do this!


Well here's hoping that the serialization layer of CREST is decoupled enough to make it easy. :)

This post was crafted by the wormhole expert of the Goonswarm Economic Warfare Cabal, the foremost authority on Eve: Online economics and gameplay.

#7 - 2016-05-23 21:36:20 UTC
It would be awesome if it was possible to access bulk market, but with some sort of a sieve list. For example, instead of specifying individual type ids or fetching them all, why not allow for multiple types,


https://crest../market/xyz/orders/multiple?type=12345&type=234&type=345...


With maybe maximum number of type ids set to 100 so it doesn't overflow the 10k entries page response?

That way my application could be reduced to one query per region to sync all data, instead of 60+
Vote Steve Ronuken for CSM
#8 - 2016-05-23 21:57:25 UTC
POS Trader wrote:
It would be awesome if it was possible to access bulk market, but with some sort of a sieve list. For example, instead of specifying individual type ids or fetching them all, why not allow for multiple types,


https://crest../market/xyz/orders/multiple?type=12345&type=234&type=345...


With maybe maximum number of type ids set to 100 so it doesn't overflow the 10k entries page response?

That way my application could be reduced to one query per region to sync all data, instead of 60+



A large chunk of this is to do with caching. Caching 'random' requests like this isn't hugely viable. Because you're likely the only person who's asking for that specific set.

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter

C C P Alliance
#9 - 2016-05-23 22:52:37 UTC
POS Trader wrote:
It would be awesome if it was possible to access bulk market, but with some sort of a sieve list. For example, instead of specifying individual type ids or fetching them all, why not allow for multiple types,


https://crest../market/xyz/orders/multiple?type=12345&type=234&type=345...


With maybe maximum number of type ids set to 100 so it doesn't overflow the 10k entries page response?

That way my application could be reduced to one query per region to sync all data, instead of 60+


With this new resource it should be about 30 requests at the high end for regions like The Forge and next week if I am able to get it up to 30,000 orders per page then it is only 10 requests at the most per region. The bigger issue is just the sheer size of it all. Whelp.

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.

Federation Uprising
#10 - 2016-05-24 14:00:12 UTC
Steve Ronuken wrote:
A large chunk of this is to do with caching. Caching 'random' requests like this isn't hugely viable. Because you're likely the only person who's asking for that specific set.



That's really interesting to know! So it's less strain on the server to deliver the entire data set, because it's static (same data to everyone who asks), instead of a specified group of type ids. That makes sense now.

How often is that static data set updated? Hourly? Every few minutes?
Joint Operation Involving Nobodys
#11 - 2016-05-27 14:45:27 UTC  |  Edited by: Jai Blaze
Steve Ronuken wrote:
A large chunk of this is to do with caching. Caching 'random' requests like this isn't hugely viable. Because you're likely the only person who's asking for that specific set.


but if each typeID has been recently cached, couldn't the request including multiple IDs simply collect info on each item and send in bulk rather than triggering a cache for the specific request?

I send requests to eve-central all the time that include 28 itemIDs (I found that once I get into the 5 digit IDs, any more requests break the URL). this is an example of the first request I send eve-central for my spreadsheet.

http://api.eve-central.com/api/marketstat?typeid=34,35,36,37,38,39,40,377,380,393,394,399,400,434,438,439,440,442,443...etc

Couldn't crest handle it similarly?

Edit: I may not understand caching at all, I do think the data transfer from crest could be reduced a lot by letting us choose a station or something to filter the response. Is it a problem of processing vs bandwidth and they just prefer to heap it on bandwidth?
Vote Steve Ronuken for CSM
#12 - 2016-05-27 17:49:58 UTC
Jai Blaze wrote:
Steve Ronuken wrote:
A large chunk of this is to do with caching. Caching 'random' requests like this isn't hugely viable. Because you're likely the only person who's asking for that specific set.


but if each typeID has been recently cached, couldn't the request including multiple IDs simply collect info on each item and send in bulk rather than triggering a cache for the specific request?

I send requests to eve-central all the time that include 28 itemIDs (I found that once I get into the 5 digit IDs, any more requests break the URL). this is an example of the first request I send eve-central for my spreadsheet.

http://api.eve-central.com/api/marketstat?typeid=34,35,36,37,38,39,40,377,380,393,394,399,400,434,438,439,440,442,443...etc

Couldn't crest handle it similarly?

Edit: I may not understand caching at all, I do think the data transfer from crest could be reduced a lot by letting us choose a station or something to filter the response. Is it a problem of processing vs bandwidth and they just prefer to heap it on bandwidth?



Crest caches at the request response level, not the db query level

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter

Goonswarm Federation
#13 - 2016-05-30 05:41:15 UTC
There should be some filtering available. I've done some testing and making ~750 requests for individual items is almost 3x faster than making whole market requests for 12 regions (those include The Forge and Sinq Laison).

http://evernus.com - the ultimate multiplatform EVE trade tool + nullsec Alliance Market tool

Goonswarm Federation
#14 - 2016-05-30 06:41:58 UTC
The main thing killing the performance is the serial nature of the endpoint - we cannot make concurrent requests for different pages; we need to fetch page by page. With Eve having ~60 regions, it means we can make at most ~60 requests per few seconds it takes to get a reply and parse it.

I propose another endpoint which will return the page count upfront, so we can use our rate limit in full and fetch the pages concurrently.

http://evernus.com - the ultimate multiplatform EVE trade tool + nullsec Alliance Market tool

Amarr Empire
#15 - 2016-05-31 00:15:25 UTC
How are the responses sorted? It doesn't seem to be sorted by time issued, time remaining or type id.
C C P Alliance
#16 - 2016-05-31 20:18:34 UTC
Pete Butcher wrote:
There should be some filtering available. I've done some testing and making ~750 requests for individual items is almost 3x faster than making whole market requests for 12 regions (those include The Forge and Sinq Laison).


Can you do that testing again after todays deployment.

@CCP_FoxFour // Technical Designer // Team Tech Co

Third-party developer? Check out the official developers site for dev blogs, resources, and more.

#17 - 2016-06-01 04:10:16 UTC
Pete Butcher wrote:
The main thing killing the performance is the serial nature of the endpoint - we cannot make concurrent requests for different pages; we need to fetch page by page. With Eve having ~60 regions, it means we can make at most ~60 requests per few seconds it takes to get a reply and parse it.

I propose another endpoint which will return the page count upfront, so we can use our rate limit in full and fetch the pages concurrently.


After I get the first reply I check the page count and queue up each of the remaining pages so I'm typically asynchronously downloading some number of them in no particular order (Python asyncio/aiohttp). Isn't this = "making concurrent requests for different pages"? I don't do Python professionally so I could certainly have missed the big picture...

My OCD is struggling with not being able to be sure that the data set that comes back is coherent, I think the cache can update in the middle of the download? It seems like it would be handy to have an indication of which cache "version" the data came from so you could redo the ones from the previous cache. However, I have a vague memory that there may be two cache servers that aren't synced so there may be no way to guarantee that you can get a coherent set?
Vote Steve Ronuken for CSM
#18 - 2016-06-01 11:35:38 UTC
Nuke Beta wrote:
Pete Butcher wrote:
The main thing killing the performance is the serial nature of the endpoint - we cannot make concurrent requests for different pages; we need to fetch page by page. With Eve having ~60 regions, it means we can make at most ~60 requests per few seconds it takes to get a reply and parse it.

I propose another endpoint which will return the page count upfront, so we can use our rate limit in full and fetch the pages concurrently.


After I get the first reply I check the page count and queue up each of the remaining pages so I'm typically asynchronously downloading some number of them in no particular order (Python asyncio/aiohttp). Isn't this = "making concurrent requests for different pages"? I don't do Python professionally so I could certainly have missed the big picture...

My OCD is struggling with not being able to be sure that the data set that comes back is coherent, I think the cache can update in the middle of the download? It seems like it would be handy to have an indication of which cache "version" the data came from so you could redo the ones from the previous cache. However, I have a vague memory that there may be two cache servers that aren't synced so there may be no way to guarantee that you can get a coherent set?



_Strictly_, you shouldn't be asking for pages you haven't been given a link for.

Woo! CSM XI!

Fuzzwork Enterprises

Twitter: @fuzzysteve on Twitter

Goonswarm Federation
#19 - 2016-06-01 15:26:11 UTC
Nuke Beta wrote:
Pete Butcher wrote:
The main thing killing the performance is the serial nature of the endpoint - we cannot make concurrent requests for different pages; we need to fetch page by page. With Eve having ~60 regions, it means we can make at most ~60 requests per few seconds it takes to get a reply and parse it.

I propose another endpoint which will return the page count upfront, so we can use our rate limit in full and fetch the pages concurrently.


After I get the first reply I check the page count and queue up each of the remaining pages so I'm typically asynchronously downloading some number of them in no particular order (Python asyncio/aiohttp). Isn't this = "making concurrent requests for different pages"? I don't do Python professionally so I could certainly have missed the big picture...

My OCD is struggling with not being able to be sure that the data set that comes back is coherent, I think the cache can update in the middle of the download? It seems like it would be handy to have an indication of which cache "version" the data came from so you could redo the ones from the previous cache. However, I have a vague memory that there may be two cache servers that aren't synced so there may be no way to guarantee that you can get a coherent set?


The problem is - you don't know the url structure to make requests basing only on page count. If you assert one, your app will stop working on any change. That's why an endpoint with links to each page should solve all issues.

http://evernus.com - the ultimate multiplatform EVE trade tool + nullsec Alliance Market tool

Blades of Grass
#20 - 2016-06-01 17:01:29 UTC
I understand i can get these bulk orders from The Forge for example by using https://crest-tq.eveonline.com/market/10000002/orders/all/ but how do I walk to that path from the root instead of hard coding it?

I've tried walking down these paths found in the root, but they don't seem to provide a way to the bulk path:
https://crest-tq.eveonline.com/market/prices/
https://crest-tq.eveonline.com/market/groups/
https://crest-tq.eveonline.com/market/types/
2 Pages12Next page
Forum Jump