There should be a method to incrementally update market data via the API instead of pulling the entire data set every 5 minutes. Provide a complete dump every day, and then release an update every 5 minutes against the last 5 minutes: new, updated and removed orders only.
I can see issues with that – like, if you do your first pull at midday, you’d need to pull the original dump plus every incremental dump.
Still, the idea of only pulling in changes is a solid one. How about a flag on each order that indicates whether it has changed from the previous five minutes? Or a “last changed” timestamp, so I can check to see if the order has changed since my last data pull?
you can’t make such a change ATM . This would need to have a sorted market, by order id, so that if two market data are the same semantically then the outut is also the same.
eg if you have two orders o with volume v and price p : [o1:v1:p1, o2:v2:p2]
This is semantically the same as [o2:v2:p2, o1:v1:p1] but the diff will be non-null. so if later monolith fetches the orders and returns the second, then the path get will be changed even though the data is semantically the same.
In order to ensure a semantic- aware diff, you need to have the list of orders sorted by order id, so to change the whole path.