`marketstore` OHCLV storage and client integration #247

guilledk · 2022-01-19T19:12:34Z

Update outdated ingest backend and replace the in-repo async wrapped marketstore client with the new:
https://github.com/pikers/anyio-marketstore

Includes support for a tractor based supervisor actor and thus resolves #143 which was merged in via the sub-PR #284.

ingest TODOs:

re-add destroy cli piker cmd (now implemented as Storage.delete_ts())
tick ingest to marketstore from brokerd feeds:
- experiment with techtonicdb schema and if it can be used with the aggregator plugin (got a feeling we'll need to write at least a 1Sec bucket in order for this to work looking at the code:
  - aggregate() calls into model.FromTrades() and FromTrades() only accepts a min step of 1Sec
  - we might be able to either write a golang plugin/extension to do it from pure tick data or we can make this part of a piker actor?
~~ohlcv ingest and load from multiple backends:~~ this was added as part of the new history in Storage layer: initial marketstore tsdb support with async OHLCV history loading. #308.
- ib seems like the natural place to start since 1s is already offered in their history
  - we'll need separate series for dark vs. regular vlm
  - we'll likely want to write a diffing and syncing system to validate new captured histories from real-time runs (which almost always are slightly different)
- kraken has tick history which has been a todo for a while
benchmark binance tick feed against binancefeeder

`_ahab` TODO:

probably deliver back net-socket info over the ctx.started() call?
- grprc socket
- ws addr
cli support for running pikerd with the tsdb stuff spawned?
- maybe a --tsdb or --data or something?
figure out what to do with the mkts.yaml config?
- we could push a template from code to the user dir?
- or should we just always gen it from python?
step by step interaction test lisit:
pikerd --tsdb should raise DockerNotStarted and appropriate perms error on no sudo (for now)
ctrl-c should kill container instance
- this does not really work when run with sudo since we relinquish root perms in order to create shms..

goodboy · 2022-02-11T16:40:40Z

@guilledk can you rebase this amigo?

goodboy · 2022-02-17T13:15:26Z

Merged in the super stuff from #284 and will begin 1s OHLC ingest from ib shortly 🏄🏼

Also, Start tinkering with `tractor.trionics.ipython_embed()` In effort to get back to a usable REPL around the mkts client this adds usage of the new `tractor` integration api as well as logic for skipping backfilling if existing tsdb arrays are found.

Add some basic `numpy` epoch slice logic to generate append and prepend arrays to write to the db. Mooar cool things, - add a `Storage.delete_ts()` method to wipe a column series from the db easily. - don't attempt to read in any OHLC series by default on client load - add some `pyqtgraph` profiling and drop manual latency measures - if no db series for the fqsn exists write the entire shm array

If `marketstore` is detected try to only load most recent missing data from the data provider (broker) and the rest from the tsdb and push it all to shm for display in the UI. If the provider/broker doesn't have the history client endpoint, just use the old one for now so we can start to incrementally add support. Don't start the ohlc step incrementer task until the backend signals that the feed is live.

It turns out (i guess not so shockingly?) that `marketstore` doesn't always teardown "gracefully" under SIGINT (seems to hang if there are open client connections which are also in the midst of teardown?) so this instead first tries the SIGINT and then fails over to a SIGKILL (destroy loop) which seems to be much more reliable to ensure shutdown without any downside - in terms of a "hard kill". Originally i was thinking the issue was root perms related (which get relegated solely to the `marketstored` daemon actor after spawn) but actually it was indeed the signalling / application layer causing the hold-up/latency on teardown. There's a bunch of lingering (now commented) code which tried to solve this non-problem as well as a bunch logging/prints to help decipher the root of the issue - this will all get cleaned out shortly.

goodboy · 2022-05-07T21:32:59Z

Oh right and I have to go through the todos and update!

Mostly just want get a first set of eyes on the code.

guilledk · 2022-05-08T00:37:34Z

piker/data/_ahab.py

@@ -133,6 +133,136 @@ def unpack_msg(err: Exception) -> str:
                c.kill()


+class Container:


What you think of creating a docker-utils python package with a lot of common abstractions to deal with dockerisms, it would be diferent than pytest-dockerctl cause it wouldnt be only about pytest, I have a bunch of utils id like in a pip instalable package somewhere

Examples: https://github.com/guilledk/py-eosio/blob/71065aad8a792a54b05f63b8ce8b4ac68b00b1e0/py_eosio/sugar.py#L338

wait_for_attr, get_container, docker_open_process, docker_wait_process, docker_move_into

i think this is an amazing idea!

for example here, all the supervisor stuff i had to hack would be great to get into a (somewhat) formalized api layer.

Yeah I'm also not settled on a lot of this api / naming so don't take it too too seriously and feel free to suggest changes.

I'm also fine with landing this code and making it possible for someone else to go through and do a cleanup / reorg if desired.

I mostly just want to get in the ability to store large data sets for users and prime things for proper real-time ingest from tick feeds.

goodboy · 2022-05-08T16:11:12Z

piker/cli/__init__.py

-    """Spawn the piker broker-daemon.
-    """
+@click.option(
+    '--tsdb',


So this is what a user passes to indicate pikerd will spawn the marketstore container in a subactor.

goodboy · 2022-05-08T16:17:48Z

piker/config.py

+
+# taken from ``click`` since apparently they have some
+# super weirdness with sigint and sudo..no clue
+def get_app_dir(app_name, roaming=True, force_posix=False):


Ahh right, yeah this is probably confusing af..

It seems that using SIGIN cancellation was somehow super oddly being swallowed/ignored by click and so pulling out this one routine we needed (which probably isn't a horrible idea to maintain on our own anyway?) was the fast hack.

goodboy · 2022-05-08T16:19:22Z

piker/data/_ahab.py

+            # callers to have root perms?
+            await trio.sleep_forever()
+
+            # await cntr.cancel()


Left over code from when I thought the issues with "graceful cancellation" were to do with root perms stuff. Turns out it wasn't that and it was the actual marketstore app layer not shutting down.

I will pull out this cruft shortly.

Done in #308.

goodboy · 2022-05-08T16:20:06Z

piker/data/_ahab.py

+    Currently the actor calling this task should normally be started
+    with root permissions (until we decide to use something that doesn't
+    require this, like docker's rootless mode or some wrapper project) but
+    te root perms are de-escalated after the docker supervisor sub-actor


goodboy · 2022-05-08T16:21:15Z

piker/data/_source.py

@@ -254,61 +253,6 @@ def iterfqsns(self) -> list[str]:
        return keys


-def from_df(


ahh yeah right!

we can drop this (and maybe pandas altogether for now) since i rewrote the ib ohlc frame parser to just cast directly to numpy.

goodboy · 2022-05-08T16:22:47Z

piker/data/cli.py


-    tractor.run(main)
+# @cli.command()


ahh yeah we can probably drop this.

the way i'd like to deal with "tsdb management" is the new storesh repl (with embedded ipython) which will have a small interactive API for manual db tinkering.

goodboy · 2022-05-08T16:25:07Z

piker/data/feed.py

@@ -188,6 +191,22 @@ async def _setup_persistent_brokerd(
        await trio.sleep_forever()


+async def start_backfill(


Ok so there is a bunch of new code for this in #302 that hasn't yet been factored out. I wasn't sure if that should all come as part of a "backend ohlc history support" PR (aka more of less what #305 is) or be included here?

Technically this PR will land with minimal backend support (i think just ib?) so it's more about how much code ya'll want test review 😂

Yeah this kinda stuff was the main reason for dropping this PR and just aggregating it all in to #308.

goodboy · 2022-05-08T20:02:20Z

Replaced by #308 🏄🏼

guilledk self-assigned this Jan 19, 2022

goodboy requested review from goodboy and iamzoltan January 20, 2022 03:51

guilledk force-pushed the marketstore branch from aae9f34 to 1c22db1 Compare February 11, 2022 18:06

goodboy force-pushed the marketstore branch from 1c22db1 to 5c62ff3 Compare February 17, 2022 13:07

goodboy assigned goodboy and unassigned guilledk Feb 17, 2022

goodboy mentioned this pull request Feb 17, 2022

Ahab's dick #284

Merged

6 tasks

This was referenced Feb 28, 2022

TSDB ingest of historical data sets: a dedicated piker tool to run parallelized history gathering #25

Open

Async hist loading #287

Merged

goodboy force-pushed the marketstore branch 4 times, most recently from b2fe7f2 to 802a3be Compare March 6, 2022 22:05

goodboy force-pushed the marketstore branch from 17bd2f5 to 6d9a940 Compare March 15, 2022 14:46

goodboy mentioned this pull request Mar 19, 2022

Phil MacKraken #214

Merged

goodboy force-pushed the marketstore branch from cc50932 to 8fe5d0d Compare April 8, 2022 15:48

goodboy changed the title ~~New marketstore client integration~~ marketstore OHCLV` storage and client integration Apr 8, 2022

goodboy force-pushed the marketstore branch from 8fe5d0d to 1a7430a Compare April 8, 2022 19:45

goodboy changed the base branch from master to big_data_lines April 8, 2022 19:46

goodboy changed the title ~~marketstore OHCLV` storage and client integration~~ marketstore OHCLV storage and client integration Apr 8, 2022

goodboy force-pushed the big_data_lines branch from a53c123 to 7c93379 Compare April 10, 2022 21:46

goodboy force-pushed the marketstore branch from 1a7430a to ba76f82 Compare April 10, 2022 21:46

goodboy force-pushed the big_data_lines branch 2 times, most recently from 5bea515 to 08b4048 Compare April 11, 2022 20:32

goodboy force-pushed the marketstore branch from ba76f82 to 1aae686 Compare April 11, 2022 20:32

goodboy force-pushed the big_data_lines branch from 28e3b22 to b02a434 Compare April 13, 2022 12:11

goodboy added 16 commits May 7, 2022 16:00

Load any symbol-matching shm array if no marketstored found

115b0f3

Comment each special key combo

c1e02b8

Try downsampling mkts data

0f226f8

Disable re-connect for now in ib script

b85497b

Drop pandas to numpy converter

dfde322

Drop ms-shell, add piker storesh cmd

ab1aaa8

Handle "fatal" level log msgs in docker super

47fe557

Get sync-to-marketstore-tsdb history retrieval workinnn

bb919ef

Limit ohlc queries to 800k datums to avoid purepc size error

6cd3d8e

Handle non-fqsn for derivs and don't put brokername in

4dc15e7

Add anyio-marketstore client as dev dep

5b7aa31

Add docker as tsdb extras dep

27a156c

goodboy force-pushed the marketstore branch from 32c3f63 to 6b64e13 Compare May 7, 2022 20:50

goodboy marked this pull request as ready for review May 7, 2022 20:50

guilledk commented May 8, 2022

View reviewed changes

goodboy reviewed May 8, 2022

View reviewed changes

goodboy mentioned this pull request May 8, 2022

Storage layer: initial marketstore tsdb support with async OHLCV history loading. #308

Merged

16 tasks

goodboy closed this May 8, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

`marketstore` OHCLV storage and client integration #247

`marketstore` OHCLV storage and client integration #247

guilledk commented Jan 19, 2022 •

edited by goodboy

Loading

goodboy commented Feb 11, 2022

goodboy commented Feb 17, 2022

goodboy commented May 7, 2022

guilledk May 8, 2022

guilledk May 8, 2022

goodboy May 8, 2022

goodboy May 8, 2022

goodboy May 8, 2022

goodboy May 8, 2022

goodboy May 8, 2022

goodboy May 8, 2022

goodboy May 8, 2022

goodboy May 8, 2022

goodboy May 8, 2022

goodboy May 8, 2022

goodboy May 8, 2022

goodboy commented May 8, 2022

		@@ -133,6 +133,136 @@ def unpack_msg(err: Exception) -> str:
		c.kill()


		class Container:

		@@ -254,61 +253,6 @@ def iterfqsns(self) -> list[str]:
		return keys


		def from_df(

		@@ -188,6 +191,22 @@ async def _setup_persistent_brokerd(
		await trio.sleep_forever()


		async def start_backfill(

marketstore OHCLV storage and client integration #247

marketstore OHCLV storage and client integration #247

Conversation

guilledk commented Jan 19, 2022 • edited by goodboy Loading

ingest TODOs:

_ahab TODO:

goodboy commented Feb 11, 2022

goodboy commented Feb 17, 2022

goodboy commented May 7, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

goodboy commented May 8, 2022

`marketstore` OHCLV storage and client integration #247

`marketstore` OHCLV storage and client integration #247

guilledk commented Jan 19, 2022 •

edited by goodboy

Loading

`_ahab` TODO: