Document how to bind-mount a mkt.yml #321

goodboy · 2020-05-18T21:53:38Z

Document how to bind-mount in a mkts.yml config for quick testing of
the image included plugins. Fix the python blocks so they highlight
correctly. Mention that you have to build the cmd package before using
the connect command.

codecov-io · 2020-05-18T21:58:11Z

Codecov Report

Merging #321 into master will decrease coverage by 0.03%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master     #321      +/-   ##
==========================================
- Coverage   46.02%   45.99%   -0.04%     
==========================================
  Files          87       87              
  Lines       11200    11200              
==========================================
- Hits         5155     5151       -4     
- Misses       5608     5612       +4     
  Partials      437      437

Impacted Files	Coverage Δ
frontend/stream/stream.go	`67.82% <0.00%> (-3.48%)`	⬇️

tibkiss · 2020-05-30T07:41:00Z

README.md

+```sh
+mkdir ms_etc/
+mv /path/to/mkts.yml ./ms_etc/
+docker run --mount type=bind,source="/full/path/to/ms_etc/",target="/etc" \


I believe providing the normal volume mount would be shorter:
docker run -v /full/path/to/mkts.yml:/etc/mkts.yml

Also, mounting the whole /etc might have undesirable side-effects: host lookup won't work as /etc/resolv.conf will be non-existent.

would be shorter

Indeed. I originally started with that myself but went to the docker docs since I hadn't actually run a container manually in some time. They now suggest this syntax due to explicitness. I'm not very opinionated one way or the other so whatever peeps prefer.

mounting the whole /etc might have undesirable side-effects: host lookup won't work as /etc/resolv.conf will be non-existent.

I'm not sure this is such a worry given that marketstore shouldn't need to query DNS under normal testing operation, but you're right, there's definitely blank hosts, hostname and resolv.conf being generated by the container in the specified host local dir. Maybe it makes sense to instead show how to copy in a custom mkts.yml into the image with something like docker cp ./mkts.yml <containername>/etc/mkts.yml? The main problem with that is I don't see a command to reload the server config?

I haven't had any problems in my usage with the bind but maybe documenting both or at least warning about the networking implications is better? Either way there seems to be no easy way for testing other then without building a new image with the custom mkts.yml moved in as in the Dockerfile. If planning to use the server in a situation where overriding /etc matters for networking purposes maybe a warning in the readme is sufficient?

For me personally having a set of docs going quickly to see how the project works is more important then worrying about more advanced usage details from the outset. When I first started trying to use the project it was pretty unclear go to get going anything practically useful (see #292 for example).

Ahh, just realized the create then cp instructions are already in the readme so disregard the:

Maybe it makes sense to instead show how to copy in a custom mkts.yml into the image with something like docker cp ./mkts.yml /etc/mkts.yml? The main problem with that is I don't see a command to reload the server config?

marketstore uses network for backfiller processes. overwriting /etc for a single file is really not a good idea.

+1 to @tibkiss .
We might want to change the default config file path from /etc/mkts.yml to /etc/mkts/mkts.yml to avoid the above issue?

@dakimura really like that idea. The bind mounting sure is handy for testing (at the least) imo 😸

Would you prefer I remove this documentation for now then until that is discussed and possibly PR-ed separately?

Oh wait, just realizing now that @tibkiss suggestion is to bind the individual file..
I'll just do that with the shorter syntax and drop the etc/ warning.

goodboy · 2020-05-30T20:48:16Z

@tibkiss added a further note to address your concerns as well as a further bind-mount tip to address #292.

Let me know if there's anything else needed 👍

codecov-commenter · 2020-05-30T20:50:12Z

Codecov Report

Merging #321 into master will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master     #321   +/-   ##
=======================================
  Coverage   46.03%   46.03%           
=======================================
  Files          87       87           
  Lines       11203    11203           
=======================================
  Hits         5157     5157           
  Misses       5609     5609           
  Partials      437      437

tibkiss · 2020-05-31T08:02:08Z

README.md

+a local location:
+```sh
+docker run -i -p 5993:5993 alpacamarkets/marketstore:latest \
+  --mount type=bind,source="/<path_to_data>/marketstore",target="/data"


I personally still prefer the shorter -v form, as we don't gain anything being explicit here.
I could also argue, that then you could also write --publish and --interactive as opposed to their short form.

I'll let others (@umitanuki / @dakimura / @leki75 ) to chime in here.

+1 for the short form

prefer the shorter -v form,

yes, I agree.

Sounds good.

goodboy · 2020-06-02T15:03:57Z

@tibkiss @dakimura made all suggested changes. Let me know if there's anything else 👍

goodboy · 2020-06-03T17:54:50Z

@dakimura I've noticed the integration tests aren't so healthy from also trying to use that code manually.

Maybe it's worth opening a new PR to fix those?

dakimura · 2020-06-03T22:51:19Z

@dakimura I've noticed the integration tests aren't so healthy from also trying to use that code manually.

Maybe it's worth opening a new PR to fix those?

ah sorry it's not related to your change... I fixed the issue just now, please pull the latest master to run the test again!

goodboy · 2020-06-04T00:56:52Z

ah sorry it's not related to your change.

Gosh, well I would hope not! 🤣

I fixed the issue just now, please pull the latest master to run the test again!

👍

Document how to bind-mount in a `mkts.yml` config for quick testing of the image included plugins. Fix the python blocks so they highlight correctly. Mention that you have to build the `cmd` package before using the `connect` command.

Mention in the readme how to bind-mount the data dir using `docker` so that persistent storage is possible using the container host. Fixes alpacahq#292

goodboy · 2020-06-07T02:40:52Z

@leki75 & @tibkiss I made the changes requested so I think this is ready to land?

tibkiss · 2020-06-07T05:38:28Z

README.md

+container's host storage. To accomplish this, bind the `data` directory to
+a local location:
+```sh
+docker run -i -p 5993:5993 alpacamarkets/marketstore:latest -v "/path/to/store/data:/data"


this is not valid, we are passing down -v "/path..." to marketstore entrypoint.

-v "/path/to/store/data:/data" should come before the image specification:

docker run -i -p 5993:5993 -v /path/to/store/data:/data alpacamarkets/marketstore:latest

Good catch 😅

tibkiss · 2020-06-07T05:39:37Z

@leki75 & @tibkiss I made the changes requested so I think this is ready to land?

yes! i found one more problem however, would be nice to fix that first.

thank you for your help here!

Also just suggest binding the `mkts.yml` file directly to avoid issues with clobbering `/etc/` on the container.

goodboy · 2020-06-08T16:43:21Z

@tibkiss fixed it! Let me know if there's anything else 🏄

tibkiss · 2020-06-10T11:57:33Z

thank you!

goodboy · 2020-06-10T14:21:06Z

Hmm you know what @tibkiss I'm thinking the bind-mount for /data isn't so clear in terms of the path.

>>> tree /<path_to_data>/marketstore
/<path_to_data>/marketstore
├── category_name
├── WALFile.1590868038674814776.walfile
├── SYMBOL_1
├── SYMBOL_2
├── SYMBOL_3

I'm thinking that should have:

>>> tree /path/to/store/data
/path/to/store/data

To match the -v line from the command above?

What you think?
I can quickly PR this in.
Might not be worth it.

tibkiss reviewed May 30, 2020

View reviewed changes

tibkiss reviewed May 31, 2020

View reviewed changes

dakimura approved these changes Jun 3, 2020

View reviewed changes

goodboy added 2 commits June 3, 2020 20:58

Document how to bind-mount a mkt.yml

2933a8f

Document how to bind-mount in a `mkts.yml` config for quick testing of the image included plugins. Fix the python blocks so they highlight correctly. Mention that you have to build the `cmd` package before using the `connect` command.

Document how to persist data across container runs

b5e3929

Mention in the readme how to bind-mount the data dir using `docker` so that persistent storage is possible using the container host. Fixes alpacahq#292

goodboy force-pushed the doc_bindmount branch from d47bb42 to 594b357 Compare June 4, 2020 00:58

goodboy mentioned this pull request Jun 7, 2020

install issues #323

Open

tibkiss reviewed Jun 7, 2020

View reviewed changes

Use docker's shorter -v style bind mount syntax

281f135

Also just suggest binding the `mkts.yml` file directly to avoid issues with clobbering `/etc/` on the container.

goodboy force-pushed the doc_bindmount branch from 594b357 to 281f135 Compare June 8, 2020 16:43

goodboy mentioned this pull request Jun 9, 2020

How to backup the marketstore database? #330

Closed

tibkiss merged commit fed6997 into alpacahq:master Jun 10, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Document how to bind-mount a mkt.yml #321

Document how to bind-mount a mkt.yml #321

goodboy commented May 18, 2020

codecov-io commented May 18, 2020

tibkiss May 30, 2020

goodboy May 30, 2020 •

edited

Loading

goodboy May 30, 2020

tibkiss May 31, 2020

dakimura Jun 1, 2020 •

edited

Loading

goodboy Jun 2, 2020 •

edited

Loading

goodboy Jun 2, 2020

goodboy commented May 30, 2020

codecov-commenter commented May 30, 2020 •

edited

Loading

tibkiss May 31, 2020

leki75 May 31, 2020

dakimura Jun 1, 2020

goodboy Jun 2, 2020

goodboy commented Jun 2, 2020

goodboy commented Jun 3, 2020

dakimura commented Jun 3, 2020

goodboy commented Jun 4, 2020

goodboy commented Jun 7, 2020

tibkiss Jun 7, 2020

goodboy Jun 8, 2020

tibkiss commented Jun 7, 2020

goodboy commented Jun 8, 2020

tibkiss commented Jun 10, 2020

goodboy commented Jun 10, 2020

Document how to bind-mount a mkt.yml #321

Document how to bind-mount a mkt.yml #321

Conversation

goodboy commented May 18, 2020

codecov-io commented May 18, 2020

Codecov Report

Choose a reason for hiding this comment

goodboy May 30, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dakimura Jun 1, 2020 • edited Loading

Choose a reason for hiding this comment

goodboy Jun 2, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

goodboy commented May 30, 2020

codecov-commenter commented May 30, 2020 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

goodboy commented Jun 2, 2020

goodboy commented Jun 3, 2020

dakimura commented Jun 3, 2020

goodboy commented Jun 4, 2020

goodboy commented Jun 7, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tibkiss commented Jun 7, 2020

goodboy commented Jun 8, 2020

tibkiss commented Jun 10, 2020

goodboy commented Jun 10, 2020

goodboy May 30, 2020 •

edited

Loading

dakimura Jun 1, 2020 •

edited

Loading

goodboy Jun 2, 2020 •

edited

Loading

codecov-commenter commented May 30, 2020 •

edited

Loading