Implemented "plugins" config variable #288

pmconrad · 2017-05-24T17:24:08Z

Combined the various *_node programs into one, where all plugins are available and can be enabled/disabled by setting a config variable. The default is witness, account_history and market_history, like in the normal witness_node.
Note that some combinations of plugins don't make much sense, like delayed_node + witness.
Note also that changing the plugins variable usually means you have to replay the blockchain.

oxarbitrage · 2017-05-24T23:22:59Z

this looks good, ill test it and comment/review.

abitmore · 2017-05-25T10:14:23Z

libraries/app/application.cpp

+      wanted.push_back("account_history");
+      wanted.push_back("market_history");
+   }
+   for (auto it = wanted.cbegin(); it != wanted.cend(); it++)


Better use ++it everywhere, although not a big deal here.

Now using range-based loop as a compromise. :-)

abitmore · 2017-05-25T10:17:10Z

programs/debug_node/README.md

@@ -1,104 +0,0 @@
-
-Introduction
------------


We need to keep this document somewhere. Is it in the docs directory/sub-repository already?

Pushed to bitshares-core.wiki: https://github.com/bitshares/bitshares-core/wiki/README-debug_node

oxarbitrage · 2017-05-29T12:57:03Z

when running the node with this pull request for the first time i am getting error:

root@alfredo:~/bitshares-munich/pull288/bitshares-core#  programs/witness_node/witness_node --data-dir data6/my-blockprod --rpc-endpoint "23.94.69.140:8090" --enabl
e-stale-production -w \""1.6.0"\" \""1.6.1"\" \""1.6.2"\" \""1.6.3"\" \""1.6.4"\" --plugins witness
2895535ms th_a       main.cpp:130                  main                 ] Writing new config file at /root/bitshares-munich/pull288/bitshares-core/data6/my-blockpro
d/config.ini
2895905ms th_a       main.cpp:199                  main                 ] Exiting with error:
13 N5boost16exception_detail10clone_implINS0_19error_info_injectorINS_13property_tree10ini_parser16ini_parser_errorEEEEE: /root/bitshares-munich/pull288/bitshares-c
ore/data6/my-blockprod/config.ini(50): duplicate key name
/root/bitshares-munich/pull288/bitshares-core/data6/my-blockprod/config.ini(50): duplicate key name: 
    {"what":"/root/bitshares-munich/pull288/bitshares-core/data6/my-blockprod/config.ini(50): duplicate key name"}
    th_a  main.cpp:310 load_logging_config_from_ini_file
2896220ms th_a       db_management.cpp:170         close                ] Database close unexpected exception: {"code":10,"name":"assert_exception","message":"Asser
t Exception","stack":[{"context":{"level":"error","file":"index.hpp","line":111,"method":"get","hostname":"","thread_name":"th_a","timestamp":"2017-05-29T12:48:15"}
,"format":"maybe_found != nullptr: Unable to find Object","data":{"id":"2.1.0"}}]}
2896559ms th_a       db_management.cpp:170         close                ] Database close unexpected exception: {"code":10,"name":"assert_exception","message":"Asser
t Exception","stack":[{"context":{"level":"error","file":"index.hpp","line":111,"method":"get","hostname":"","thread_name":"th_a","timestamp":"2017-05-29T12:48:16"}
,"format":"maybe_found != nullptr: Unable to find Object","data":{"id":"2.1.0"}}]}
root@alfredo:~/bitshares-munich/pull288/bitshares-core#

Same error without the --plugins option:

root@alfredo:~/bitshares-munich/pull288/bitshares-core# programs/witness_node/witness_node --data-dir data6/my-blockprod --rpc-endpoint "23.94.69.140:8090" --enable
-stale-production -w \""1.6.0"\" \""1.6.1"\" \""1.6.2"\" \""1.6.3"\" \""1.6.4"\"
3185973ms th_a       main.cpp:130                  main                 ] Writing new config file at /root/bitshares-munich/pull288/bitshares-core/data6/my-blockpro
d/config.ini
3185973ms th_a       main.cpp:199                  main                 ] Exiting with error:
13 N5boost16exception_detail10clone_implINS0_19error_info_injectorINS_13property_tree10ini_parser16ini_parser_errorEEEEE: /root/bitshares-munich/pull288/bitshares-c
ore/data6/my-blockprod/config.ini(50): duplicate key name
/root/bitshares-munich/pull288/bitshares-core/data6/my-blockprod/config.ini(50): duplicate key name: 
    {"what":"/root/bitshares-munich/pull288/bitshares-core/data6/my-blockprod/config.ini(50): duplicate key name"}
    th_a  main.cpp:310 load_logging_config_from_ini_file
3185973ms th_a       db_management.cpp:170         close                ] Database close unexpected exception: {"code":10,"name":"assert_exception","message":"Asser
t Exception","stack":[{"context":{"level":"error","file":"index.hpp","line":111,"method":"get","hostname":"","thread_name":"th_a","timestamp":"2017-05-29T12:53:05"}
,"format":"maybe_found != nullptr: Unable to find Object","data":{"id":"2.1.0"}}]}
3185974ms th_a       db_management.cpp:170         close                ] Database close unexpected exception: {"code":10,"name":"assert_exception","message":"Asser
t Exception","stack":[{"context":{"level":"error","file":"index.hpp","line":111,"method":"get","hostname":"","thread_name":"th_a","timestamp":"2017-05-29T12:53:05"}
,"format":"maybe_found != nullptr: Unable to find Object","data":{"id":"2.1.0"}}]}
root@alfredo:~/bitshares-munich/pull288/bitshares-core#

pmconrad · 2017-05-29T15:51:22Z

This looks like a problem with the default config file in combination with issue #209. (Possibly caused by your multiple "-w" options?)

abitmore · 2017-05-29T17:35:22Z

Multiple -w should be allowed. Something is wrong with your setup. @oxarbitrage

oxarbitrage · 2017-05-29T17:56:42Z

Pretty strange as the current master goes fine following the same steps. Ill look into it further. El may. 29, 2017 2:35 PM, "Abit" <notifications@github.com> escribió:

…

Multiple -w should be allowed. Something is wrong with your setup. @oxarbitrage <https://github.com/oxarbitrage> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#288 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AUrjaUl8jhqZTxKhViylD5GEBacPEFKHks5r-wHdgaJpZM4NlbEz> .

oxarbitrage · 2017-05-29T23:05:45Z

@pmconrad i checked my config file at line 50 where the error is happening and the private-key param is duplicated:

# Tuple of [PublicKey, WIF private key] (may specify multiple times)
private-key = ["BTS6MRyAjQq8ud7hVNYcfnVPJqcVpscN5So8BhtHuGYqET5GDW5CV","5KQwrPbwdL6PhXujxW37FSSQZ1JiwsST4cqQzDeyXtP79zkvFD3"]

# Tuple of [PublicKey, WIF private key] (may specify multiple times)
private-key = ["BTS6MRyAjQq8ud7hVNYcfnVPJqcVpscN5So8BhtHuGYqET5GDW5CV","5KQwrPbwdL6PhXujxW37FSSQZ1JiwsST4cqQzDeyXtP79zkvFD3"]

please note that there was no config file generated from before, this is first time run. i think somehow this pull request is causing this duplicate in the config.

any idea why ?

please note current master node without pull request is working fine.

abitmore · 2017-05-30T01:43:34Z

It may be inserted by witness plugin once, then debug_node plugin or delayed_node plugin once.

oxarbitrage · 2017-05-30T01:56:24Z

make sense @abitmore thanks, however i am calling the node to load only the "witness" plugin or without plugin options at all. this should be i think:

default(to keep it as it was):
load witness
load account history
load market history

with --plugins argument in node start:
load only separated space list of plugins

pmconrad · 2017-05-30T19:41:40Z

The patch collects all available options from all known plugins, independently from the selection of active plugins. The intent is that invoking --help gives you all options from all plugins. I didn't think of collisions there.

The problem only appears when witness_node is run in a clean data dir - in that case it will create a new config file containing all options it knows about. Afterwards it tries to parse the new config file, and fails due to duplicate keys.

Good find, I'll filter out duplicates.

oxarbitrage · 2017-05-31T02:42:47Z

works perfect now @pmconrad

i few tests i made, running node with:

programs/witness_node/witness_node --data-dir data/my-blockprod --rpc-endpoint "localhost:8090" --plugins witness

after sync the memory usage of the witness node in pmap was:

total 2304784K

this is around 2.2 gigs , expected as history is not loaded.

Now starting the node normally with no plugin parameter as:

programs/witness_node/witness_node --data-dir data6/my-blockprod --rpc-endpoint "localhost:8090"

after the sync the memory usage for the process is at:

total 19164108K

around 18 gigs, meaning it haves everything loaded up. didn't made any test with delayed, debug node plugins as i never used them before but with the iterator change, the addition of removed document to wiki and the fix in the duplicated config this pull is looking good to me for a merge.

abitmore · 2017-05-31T18:16:29Z

@oxarbitrage we need to test at least delayed_node. For more info please check this link.

oxarbitrage · 2017-05-31T20:20:22Z

got it, on it.

oxarbitrage · 2017-06-05T19:41:07Z

made some tests with the delayed node, working ok.

first, a trusted node is started and synced listening in port 8090. next a delayed node is started by command:

programs/witness_node/witness_node --plugins delayed_node --trusted-node="127.0.0.1:8090" --rpc-endpoint="127.0.0.1:8091" -d delayed_node -s "0.0.0.0:0" --p2p-endpoint="0.0.0.0:0" --seed-nodes "[]"

it takes a while for the delayed node to get in sync to the trusted node, if everything is ok the delayed node will show this kind of messages while the sync is happening:

...
340736ms th_a       delayed_node_plugin.cpp:106   sync_with_trusted_no ] Pushing block #16617520
...

by opening cli wallets to the trusted and to the delayed we can see the trusted is a bit faster as expected:

in trusted:

new >>> get_dynamic_global_properties 
get_dynamic_global_properties 
{
  "id": "2.1.0",
  "head_block_number": 17165401,
  "head_block_id": "0105ec5940ce3bc66bd988407ff361ec2775e5e8",
  "time": "2017-06-05T19:22:24",
  "current_witness": "1.6.65",
  "next_maintenance_time": "2017-06-05T20:00:00",
  "last_budget_time": "2017-06-05T19:00:00",
  "witness_budget": 75500000,
  "accounts_registered_this_interval": 10,
  "recently_missed_count": 0,
  "current_aslot": 17273164,
  "recent_slots_filled": "340282366920938463463374607431768211455",
  "dynamic_flags": 0,
  "last_irreversible_block_num": 17165383
}
new >>>

in delayed almost at the same time:

new >>> get_dynamic_global_properties 
get_dynamic_global_properties 
{
  "id": "2.1.0",
  "head_block_number": 17165383,
  "head_block_id": "0105ec477c58560877ec3ab2389721d613bafc82",
  "time": "2017-06-05T19:21:30",
  "current_witness": "1.6.74",
  "next_maintenance_time": "2017-06-05T20:00:00",
  "last_budget_time": "2017-06-05T19:00:00",
  "witness_budget": 77300000,
  "accounts_registered_this_interval": 9,
  "recently_missed_count": 0,
  "current_aslot": 17273146,
  "recent_slots_filled": "340282366920938463463374607431768211455",
  "dynamic_flags": 0,
  "last_irreversible_block_num": 17165362
}
new >>>

from the documentation in here:
http://docs.bitshares.org/integration/exchanges/step-by-step.html

we are going to need to change the delayed start command if pull request is merged.
the cli_wallet starting command haves the wrong port, it is connecting to the trusted node instead of the delayed, this should be:

./programs/cli_wallet/cli_wallet --server-rpc-endpoint="ws://127.0.0.1:8091" \
                                 --rpc-http-endpoint="127.0.0.1:8092"

difference is in the server-rpc-endpoint going to port 8091(delayed) instead of 8090(trusted).

also here:
http://docs.bitshares.eu/integration/apps/delayednode.html

we need to update command if merged.
even if not merged the --delay-block-count is not a valid parameter in current delayed node implementation.

abitmore · 2017-06-05T21:39:33Z

@oxarbitrage cli need to connect to trusted node to be able to broadcast transactions, for exchanges, withdrawals.

oxarbitrage · 2017-06-05T21:48:51Z

Got it. The delayed node is only for deposit notifications in the proposed documentation setup for exchanges. Whatever is the case the delayed node seems to be working fine starting it with the main witness executable and the new --plugins argument. I am now giving a try to the debug witness. El jun. 5, 2017 6:39 PM, "Abit" <notifications@github.com> escribió:

…

@oxarbitrage <https://github.com/oxarbitrage> cli need to connect to trusted node to be able to broadcast transactions, for exchanges, withdrawals. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#288 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AUrjaerolkkhZgK-LDaVxqgGAkPahyEkks5sBHWYgaJpZM4NlbEz> .

oxarbitrage · 2017-07-31T23:04:16Z

i know it had been a while but with the next hard fork coming i tested the debug node today with the plugin parameter from this pull. the debug node was the only setup left to test for the complete approval of this pull request.

debug node with new plugins param is started as:

programs/witness_node/witness_node --data-dir data/my-blockprod --rpc-endpoint "127.0.0.1:8090" --max-ops-per-account 100 --partial-operations true --plugins debug_witness

followed the debug node guide at: https://github.com/bitshares/bitshares-core/wiki/README-debug_node . all worked.
if we merge we need to update a bit the documentation there, i can take care of that. there are also changes already mentioned we will need to do at http://docs.bitshares.org/, i can send the pulls there as well.

the only concern may be that the executables for debug_node and delayed_node will not be present anymore, this can be confusing for some new users but if we update the docs should be fine.

i personally think the new parameter is better than having separated stuff.

abitmore · 2017-08-01T08:34:26Z

I'd rather keep this unmerged for now. Although it may work, it may affect most of 3rd parties who rely on delayed nodes. We need a way to disturb them the least, for example, leave the make delayed_node option so they can build a working binary with the old steps (perhaps just a copy of witness_node binary but it should work like a delayed_node), and/or update docs and push-notify the parties.

//Edit: most of the 3rd parties I mentioned above are exchanges, it's important that they play an important role in the ecosystem.

pmconrad · 2017-08-01T08:54:44Z

Would you approve the changes in witness_node if we keep the separate delayed_node?

abitmore · 2017-08-01T09:11:28Z

Would you approve the changes in witness_node if we keep the separate delayed_node?

Yes, I think it's better.

oxarbitrage · 2017-08-05T20:24:19Z

all good here, the delayed node is back as a separated program and can now be launched by the 2 ways:

by the witness_node exe with the plugins arg.
by a separated program at programs/delayed_node/delayed_node

@abitmore let us know if you this is ok to merge. thanks. well done peter.

abitmore

All good except one minor issue.

abitmore · 2017-08-05T22:43:06Z

programs/witness_node/CMakeLists.txt

@@ -11,7 +11,7 @@ endif()

 # We have to link against graphene_debug_witness because deficiency in our API infrastructure doesn't allow plugins to be fully abstracted #246
 target_link_libraries( witness_node
-                       PRIVATE graphene_app graphene_account_history graphene_market_history graphene_witness graphene_chain graphene_debug_witness graphene_egenesis_full fc ${CMAKE_DL_LIBS} ${PLATFORM_SPECIFIC_LIBS} )
+                       PRIVATE graphene_app graphene_delayed_node graphene_account_history graphene_market_history graphene_witness graphene_debug_witness graphene_chain graphene_debug_witness graphene_egenesis_full fc ${CMAKE_DL_LIBS} ${PLATFORM_SPECIFIC_LIBS} )


duplicate graphene_debug_witness.

Fixed, thanks!

From: bitshares/bitshares-core#288

Implemented "plugins" config variable

c4527a8

pmconrad mentioned this pull request May 24, 2017

Witness_node Memory Management Discussion #192

Closed

abitmore reviewed May 25, 2017

View reviewed changes

Use range-based loop instead of iterator

83cc453

Avoid duplicate options in default config file

cbdb015

pmconrad changed the base branch from master to develop July 12, 2017 20:45

oxarbitrage self-requested a review July 31, 2017 22:45

oxarbitrage approved these changes Jul 31, 2017

View reviewed changes

pmconrad mentioned this pull request Aug 3, 2017

Hello plugin #346

Closed

Re-added separate delayed_node

b7f600e

abitmore reviewed Aug 5, 2017

View reviewed changes

Removed duplicate

a4e4257

oxarbitrage merged commit ce15e94 into bitshares:develop Aug 6, 2017

obucina added a commit to peerplays-network/peerplays that referenced this pull request Sep 6, 2019

Implemented "plugins" config variable

604439d

From: bitshares/bitshares-core#288

obucina mentioned this pull request Sep 12, 2019

SON plugin skeleton and plugins config/command line options peerplays-network/peerplays#127

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implemented "plugins" config variable #288

Implemented "plugins" config variable #288

pmconrad commented May 24, 2017

oxarbitrage commented May 24, 2017

abitmore May 25, 2017

pmconrad May 29, 2017

abitmore May 25, 2017

pmconrad May 29, 2017

oxarbitrage commented May 29, 2017

pmconrad commented May 29, 2017

abitmore commented May 29, 2017

oxarbitrage commented May 29, 2017 via email

oxarbitrage commented May 29, 2017

abitmore commented May 30, 2017

oxarbitrage commented May 30, 2017

pmconrad commented May 30, 2017

oxarbitrage commented May 31, 2017

abitmore commented May 31, 2017

oxarbitrage commented May 31, 2017

oxarbitrage commented Jun 5, 2017

abitmore commented Jun 5, 2017

oxarbitrage commented Jun 5, 2017 via email

oxarbitrage commented Jul 31, 2017

abitmore commented Aug 1, 2017 •

edited

Loading

pmconrad commented Aug 1, 2017

abitmore commented Aug 1, 2017

oxarbitrage commented Aug 5, 2017

abitmore left a comment

abitmore Aug 5, 2017

pmconrad Aug 6, 2017

Implemented "plugins" config variable #288

Implemented "plugins" config variable #288

Conversation

pmconrad commented May 24, 2017

oxarbitrage commented May 24, 2017

abitmore May 25, 2017

Choose a reason for hiding this comment

pmconrad May 29, 2017

Choose a reason for hiding this comment

abitmore May 25, 2017

Choose a reason for hiding this comment

pmconrad May 29, 2017

Choose a reason for hiding this comment

oxarbitrage commented May 29, 2017

pmconrad commented May 29, 2017

abitmore commented May 29, 2017

oxarbitrage commented May 29, 2017 via email

oxarbitrage commented May 29, 2017

abitmore commented May 30, 2017

oxarbitrage commented May 30, 2017

pmconrad commented May 30, 2017

oxarbitrage commented May 31, 2017

abitmore commented May 31, 2017

oxarbitrage commented May 31, 2017

oxarbitrage commented Jun 5, 2017

abitmore commented Jun 5, 2017

oxarbitrage commented Jun 5, 2017 via email

oxarbitrage commented Jul 31, 2017

abitmore commented Aug 1, 2017 • edited Loading

pmconrad commented Aug 1, 2017

abitmore commented Aug 1, 2017

oxarbitrage commented Aug 5, 2017

abitmore left a comment

Choose a reason for hiding this comment

abitmore Aug 5, 2017

Choose a reason for hiding this comment

pmconrad Aug 6, 2017

Choose a reason for hiding this comment

abitmore commented Aug 1, 2017 •

edited

Loading