-
Notifications
You must be signed in to change notification settings - Fork 17
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Storage layer: initial marketstore
tsdb support with async OHLCV history loading.
#308
Conversation
marketstore
tsdb support with async OHLCV history loading.
|
||
tractor.run(main) | ||
# @cli.command() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ahh yeah we can probably drop this.
the way i'd like to deal with "tsdb management" is the new storesh repl (with embedded ipython) which will have a small interactive API for manual db tinkering.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
actually gonna leave it commented in here just in case we decide std cli cmds for this is handy.
@@ -254,61 +254,6 @@ def iterfqsns(self) -> list[str]: | |||
return keys | |||
|
|||
|
|||
def from_df( |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ahh yeah right!
we can drop this (and maybe pandas altogether for now) since i rewrote the ib ohlc frame parser to just cast directly to numpy
.
Ok so my plan forward on this is to try and wrap the final few TODOs and make issues/tasks for all the follow up stuff. |
Bleh/:facepalm:, the ``end_dt`` in scope is not the "earliest" frame's `end_dt` in the async response queue.. Parse the queue's latest epoch and use **that** to compare to the last last pushed datetime index.. Add more detailed logging to help debug any (un)expected datetime index gaps.
It seems once in a while a frame can get missed or dropped (at least with binance?) so in those cases, when the request erlangs is already at max, we just manually request the missing frame and presume things will work out XD Further, discard out of order frames that are "from the future" that somehow end up in the async queue once in a while? Not sure why this happens but it seems thus far just discarding them is nbd.
We return a copy (since since a view doesn't seem to work..) of the (field filtered) shm array contents which is the same index-length as the source data. Further, fence off the resource tracker disable-hack into a helper routine.
Pushed a few more |
Replaces #247 and #305 (merging history from both) and instead adds a more formal "storage layer" for retrieving and storing large ohlcv series from all major backends. The work here is best experienced with the new incremental update patchset from #302 in order to see the new graphics performance improvements at work though this should work moderately well as is.
TL;DR:
marketstore
docker container supervisor actor which allows spawning by passingpikerd --tsdb
-> this closes Supervise the store #143pikerd
) shortly aftermarketstored
(the container super) is startedasync with open_history_client()
support toib
,binance
,kraken
which allows pulling in and storing a large amount of history in the tsdb,marketstore
.datetime
range, new special exception signals can be used (NoData
,DataUnavailable
) to signal that the backend can't deliver data for a given range (or at all).trimeter
in the backends that can handle it (currently onlybinance
).datetime
ranges that are passed to thetrimeter
request scheduler and which dynamically adjusts the request-time index when a gap is detected.piker.data.marketstore.Storage
api/layer which allows async, high level operations. the intention is to eventually have this layer support more tsdb providers likearctic
,techtonicdb
. Currently the only backend ismarketstore
with client-side operations implemented using ouranyio-marketstore
library:fqsn
with appropriate request size limiting with.read_ohlcv()
fqsn
with appropriate limits with.write_ohlcv()
Storage.delete_ts()
What this does (yet) not introduce:
marketstore
storesh
interactive repl for managing the tsdb, there is a minimalipython
embedding at the moment but it is nowhere complete and we need a follow up task-issue to finish this.TODO:
._ahab.py
supervisor:ctx.started()
call?mkts.yaml
config?pikerd
with the tsdb stuff spawned?--tsdb
or--data
or something?pikerd --tsdb
test list:pikerd --tsdb
should raiseDockerNotStarted
and appropriate perms error on nosudo
(for now)marketstore
config and operation:to be done as #313
follow up (to be written in new task-issues and implemented in coming PRs)
moved to #314
marketstore
frombrokerd
feeds and experiment withtechtonicdb
schema (some tinkering was already done in this patchset by @guilledk but is unfinished).1Sec
bucket in order for this to work looking at the code:aggregate()
calls intomodel.FromTrades()
andFromTrades()
only accepts a min step of1Sec
these moved to #312
ipython
) that allows interaction, edit, and general management of the tsdb for both the purposes of research and just plain old data mgmt.