-
Notifications
You must be signed in to change notification settings - Fork 912
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add a bookkeeper
plugin
#5071
Add a bookkeeper
plugin
#5071
Conversation
Ok, I've merged the prerequisites.... |
e510940
to
03e4fd4
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just some preliminary remarks and questions up to commit bd3252a, will continue the review later today.
* It is used to migrate existing databases from a prevoius state, based on | ||
* string indicies */ | ||
static struct migration db_migrations[] = { | ||
{SQL("CREATE TABLE version (version INTEGER);"), NULL}, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we prefix the tables with something (bkp_
?) otherwise we'll have to provision two databases for the postgres case. With a prefix we could just assign the same database and have the plugin and lightningd share it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
well that'd mean that db/exec.c
needs a way to add a prefix to table names. otherwise, seems like a great idea.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Probably a tall ask, but why a separate database? Could we use datastore
instead?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
iiuc datastore
is designed for key-value storage; this application requires more complex relational data storage.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
All relational data storage is based on key-value ultimately (tables have a row ID per row that is the key to the tuple, and indices map tuple values to row ID keys), though I do understand wanting not to have to reimplement SQL when a perfectly good library already exists.
In particular, this seems to be recordkeeping that is "safe" to lose; you lose your records, but none of your funds are at risk if the bookkeeper
-specific database is lost due to inevitable hardware failure. So not using datastore
is fine. But I think you should also point this out in the docs, then, that since it is not stored in lightningd.sqlite3
or in the PostgreSQL database, then any automated backup/replication setup the user uses does not apply to the bookkeeper
data, and if the user recovers from any failure safely, the bookkeeper
data is lost.
Of note is that even if you adapt to PostgreSQL, PostgreSQL users may need to set up a separate database for this bookkeeper
, due to the use of a table name (CREATE TABLE version
) that is the same as the "main" lightningd
database. This complicates an already complicated deployment.
plugins/bkpr/db.c
Outdated
", tag VARCHAR(255)" | ||
", credit BIGINT" | ||
", debit BIGINT" | ||
", output_value BIGINT" | ||
", currency VARCHAR(12)" | ||
", timestamp BIGINT" | ||
", blockheight INTEGER" | ||
", utxo_txid BLOB" | ||
", outnum INTEGER" | ||
", spending_txid BLOB" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Are all these field not nullable?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
spending_txid
is. should I add a default NULL
? i thought fields were null by default.
plugins/bkpr/bookkeeper.c
Outdated
if (e->spending_txid) | ||
err = maybe_update_onchain_fees(cmd, db, e->spending_txid); | ||
else | ||
err = maybe_update_onchain_fees(cmd, db, &e->outpoint.txid); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok, took me a while to find the difference in the two lines. Can we have a local variable with a descriptive name that we then call to a single call instead? Would make following along a bit easier.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
what about
err = maybe_update_onchain_fees(cmd, db, e->spending_txid ? e->spending_txid : &e->outpoint.txid)
plugins/bkpr/recorder.c
Outdated
/* Otherwise, we update the existing record */ | ||
db_col_ignore(stmt, "1"); | ||
if (!amount_msat_sub(¤t_amt, current_amt, debit)) | ||
db_fatal("Overflow when adding onchain fees"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Overflow when adding
-> Underflow while subtracting
db_begin_transaction(db); | ||
log_channel_event(db, acct, chan_ev); | ||
db_commit_transaction(db); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
General: it's often easier and even faster to just wrap the entire event handling in a single large DB transaction rather than trying to do fine-grained control, which may lead to accidentally nested transactions (-> abort), and many transactions being started and committed. Also the one-to-one mapping between event and DB transaction ensure that we don't get partially handled events in case of a crash.
struct channel_event *chan; | ||
struct chain_event *chain; | ||
struct onchain_fee *fee; | ||
u64 lowest = 0; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why not set this to UINT64_MAX? Allows you to treat the "no lowest" case identical to "higher lowest" case.
In addition is seems that this assumes that there'll always be a fee to add at the end, wouldn't it be easier to do a merge-sort like structure where each iteration pops the lowest and prints that before restarting the iteration? This code might get the ordering wrong if we have two consecutive events of the same type followed by a later event, since this will at most print one event for each type before switching to the next type (though there may be a structure in the model I'm not considering here).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this will at most print one event for each type before switching to the next type
We only print one item every loop, there are continue
s at each print action that ensures that there's only one per.
this assumes that there'll always be a fee to add at the end
Since we continue after print, you'll never reach the fee print line if you're out of fees -- another item will have printed and the loop continued.
Generally speaking, the routine here is:
- elect the top item of each list
- from all (still populated) elected items, which has the lowest timestamp?
- print that (with some preference for channel, then chain, then fees in case of a tie)
- return to election phase
I could merge the lowest timestamp identification and 'topmost item' selection.
bca7618
to
10b3547
Compare
Based on comments from @cdecker, I rewrote the way that the 1f364f9#diff-ae7e6b3b74048f72d90eac244f2b7eda49ee6637cf323de3f3cdd2b9af179ed8R197 I wasn't sure about the best way to handle adding a prefix to a table name for the tables, since the shared database code expects certain tables. A better refactorer would probably pull the part of the database migrations out that are expected/core into a top-level thing or something.. |
83513f5
to
b1cf937
Compare
Concept ACK Is the plan that you'll get this working in this repo first and then move the plugin specific stuff (bookkeeper database etc) to the plugins repo? This is essentially laying the groundwork for others to write accounting plugins longer term right? (If there was reason to of course, perhaps this bookkeeper plugin will literally do everything a user could wish for re accounting). |
ef7d476
to
43fdff9
Compare
Looks like valgrind found something... |
52340e3
to
6b94c93
Compare
lgtm! |
After discussion with @niftynei I am bumping this to next release.... |
0c3d53f
to
93f68eb
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Only reviewed half, but minor comments. Mainly, needs rebase and thwacking CI AFAICT...
connectd/test/run-netaddress.c
Outdated
return 0; | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is unnecessary in modern C. And I guess from the commit message, this is simply a rind left over from a previous commit which added common_shutdown() and is now redundant?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
removed! (deleted whole commit, was extraneous)
93f68eb
to
f6c9fbc
Compare
Rebased on master, and reordered and fixed deps so it's possible to bisect, also fixed db_fatal() in plugin to show up in logs. Did some minor fixups which were obvious. The following tests are broken:
The zeroconf ones seem like a conflict introduced in the rebase:
|
Utility method to figure out if an account is "external"
If two events for the same (unlogged) account come in and get run through the "lookup peer" code, we should anticipate that. We do two things here: - one, if it's a duplicate "event" to create the channel open we check for that and just exit early - two, we were using a copy of the account that was fetched/pulled from before the RPC ran, so instead we just re-pull the most up to date account info for the close checks. This fixes the crash I was getting in re-running these things
We weren't passing the command, which meant that it wasn't getting populated correctly. We do that now, it fixes some crashes
It's possible we'll get an "external" event for an account/channel and then try to do close checks on it and it'll bomb out because we didn't go look up the originating account to make sure that that had open info Now, we make sure and do that when we hear about an originating account
We rescan and pick up the channel's close tx, but can't go back and get the open info, since we've already deleted the channel from the database.
nice to have for making tests that assert on description/bolt data
It'll be really nice to be able to read description data about an invoice, if we've got it!
So we print out invoice fees on the same line for those CSVs! This means we have to do a little bit of gymnastics (but not too bad): - we save the fee amount onto the income event now so we can use it later - we ignore every "invoice_fee" event for the koinly/cointracker Note that since we're not skipping income events in the loops we also move the newline character to the start of every `_entry` function so skipped records dont incur empth lines. Changelog-Added: bkpr: print out invoice fees on the same line for `koinly` and `cointracker` csv types
First off, when we pull data out of JSON, unescape it so we don't end up with extraneous escapes in our bookkeeping data. I promise, it's worth it. Then, when we print descriptions out to the csvs, we gotta wrap everything in quotes... but also we have to change all the double-quotes to singles so that adding the quotes doesn't do anything untoward. We also just pass it thru json_escape to get rid of linebreaks etc. Note that in the tests we do a byte comparison instead of converting the CSV dumps to strings because python will escape the strings on conversion...
FXIME: Has a edge case where if you disable the bookkeeper, it'll blowup because you've got an option that isn't present anywhere...
`lightningd` has an option --wallet that lets you supply a database dsn string to connect to a sqlite3/postgres database that's hosted/stored elsewhere. This adds the `--bookkeeper-db` option which does the same, except for the bookkeeping data for a node! Note that the default is to go in the `lightning-dir` in a database called `accounts.sqlite3`
…ce everyone needs it. This was originally done as part of the bookkeeper introduction, but it deserves its own patch (and that one didn't remove the bitcoin/chainparams.o from the individual requirements lines). Suggested-by: @niftynei Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
…gin. We do this for bcli options, too. Good to note if people disable/replace the plugin. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Somehow these two were missed in 79e09b9. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
3a1c228
to
e3c907b
Compare
We need a record of the channel account before you start sending payments through it. Normally we don't start allowing payments to be sent until after the channel has locked in but zeroconf does away with this assumption. Instead we push out a "channel_proposed" event, which should only show up for zeroconfs.
e3c907b
to
e6e2a3b
Compare
bookkeeper
plugin to clightningbookkeeper
plugin
Keep the accounts as an 'append only' log, instead we move the marker for the 'channel_open' forward when a 'channel_open' comes out. We also neatly hide the 'channel_proposed' events in 'inspect' if there's a 'channel_open' for that same event. If you call inspect before the 'channel_open' is confirmed, you'll see the tag as 'channel_proposed', afterwards it shows up as 'channel_open'. However the event log rolls forward -- listaccountevents will show the correct history of the proposal then open confirming (plus any routing that happened before the channel confirmed).
memleak was complaining about dangling refs; they were all allocated off of `cmd` but not technically unreachabe from any in-context memory. Instead we move things over to tmpctx to clean up (in the paths where we're going to move the objects out of reach via intermediate RPC call) ALSO: cleans up unused param in `lookup_invoice_desc` lightningd-1 2022-07-23T19:17:11.192Z **BROKEN** plugin-bookkeeper: MEMLEAK: 0x1511f68 lightningd-1 2022-07-23T19:17:11.194Z **BROKEN** plugin-bookkeeper: label=plugins/bkpr/recorder.c:1048:struct account lightningd-1 2022-07-23T19:17:11.194Z **BROKEN** plugin-bookkeeper: backtrace: lightningd-1 2022-07-23T19:17:11.194Z **BROKEN** plugin-bookkeeper: ccan/ccan/tal/tal.c:442 (tal_alloc_) lightningd-1 2022-07-23T19:17:11.194Z **BROKEN** plugin-bookkeeper: plugins/bkpr/recorder.c:1048 (stmt2account) lightningd-1 2022-07-23T19:17:11.194Z **BROKEN** plugin-bookkeeper: plugins/bkpr/recorder.c:1109 (find_account) lightningd-1 2022-07-23T19:17:11.196Z **BROKEN** plugin-bookkeeper: plugins/bkpr/bookkeeper.c:1606 (parse_and_log_channel_move) lightningd-1 2022-07-23T19:17:11.197Z **BROKEN** plugin-bookkeeper: plugins/bkpr/bookkeeper.c:1713 (json_coin_moved) lightningd-1 2022-07-23T19:17:11.198Z **BROKEN** plugin-bookkeeper: plugins/libplugin.c:1421 (ld_command_handle) lightningd-1 2022-07-23T19:17:11.198Z **BROKEN** plugin-bookkeeper: plugins/libplugin.c:1491 (ld_read_json_one) lightningd-1 2022-07-23T19:17:11.199Z **BROKEN** plugin-bookkeeper: plugins/libplugin.c:1511 (ld_read_json) lightningd-1 2022-07-23T19:17:11.199Z **BROKEN** plugin-bookkeeper: ccan/ccan/io/io.c:59 (next_plan) lightningd-1 2022-07-23T19:17:11.199Z **BROKEN** plugin-bookkeeper: ccan/ccan/io/io.c:407 (do_plan) lightningd-1 2022-07-23T19:17:11.199Z **BROKEN** plugin-bookkeeper: ccan/ccan/io/io.c:417 (io_ready) lightningd-1 2022-07-23T19:17:11.199Z **BROKEN** plugin-bookkeeper: ccan/ccan/io/poll.c:453 (io_loop) lightningd-1 2022-07-23T19:17:11.199Z **BROKEN** plugin-bookkeeper: plugins/libplugin.c:1708 (plugin_main) lightningd-1 2022-07-23T19:17:11.199Z **BROKEN** plugin-bookkeeper: plugins/bkpr/bookkeeper.c:1812 (main) lightningd-1 2022-07-23T19:17:11.200Z **BROKEN** plugin-bookkeeper: ../csu/libc-start.c:308 (__libc_start_main) lightningd-1 2022-07-23T19:17:11.200Z **BROKEN** plugin-bookkeeper: parents: lightningd-1 2022-07-23T19:17:11.200Z **BROKEN** plugin-bookkeeper: plugins/libplugin.c:1378:struct command lightningd-1 2022-07-23T19:17:11.200Z **BROKEN** plugin-bookkeeper: MEMLEAK: 0x15305b8 lightningd-1 2022-07-23T19:17:11.200Z **BROKEN** plugin-bookkeeper: label=plugins/bkpr/bookkeeper.c:1643:enum mvt_tag[] lightningd-1 2022-07-23T19:17:11.200Z **BROKEN** plugin-bookkeeper: backtrace: lightningd-1 2022-07-23T19:17:11.200Z **BROKEN** plugin-bookkeeper: ccan/ccan/tal/tal.c:442 (tal_alloc_) lightningd-1 2022-07-23T19:17:11.201Z **BROKEN** plugin-bookkeeper: ccan/ccan/tal/tal.c:471 (tal_alloc_arr_) lightningd-1 2022-07-23T19:17:11.201Z **BROKEN** plugin-bookkeeper: plugins/bkpr/bookkeeper.c:1643 (parse_tags) lightningd-1 2022-07-23T19:17:11.201Z **BROKEN** plugin-bookkeeper: plugins/bkpr/bookkeeper.c:1686 (json_coin_moved) lightningd-1 2022-07-23T19:17:11.201Z **BROKEN** plugin-bookkeeper: plugins/libplugin.c:1421 (ld_command_handle) lightningd-1 2022-07-23T19:17:11.201Z **BROKEN** plugin-bookkeeper: plugins/libplugin.c:1491 (ld_read_json_one) lightningd-1 2022-07-23T19:17:11.201Z **BROKEN** plugin-bookkeeper: plugins/libplugin.c:1511 (ld_read_json) lightningd-1 2022-07-23T19:17:11.201Z **BROKEN** plugin-bookkeeper: ccan/ccan/io/io.c:59 (next_plan) lightningd-1 2022-07-23T19:17:11.201Z **BROKEN** plugin-bookkeeper: ccan/ccan/io/io.c:407 (do_plan) lightningd-1 2022-07-23T19:17:11.202Z **BROKEN** plugin-bookkeeper: ccan/ccan/io/io.c:417 (io_ready) lightningd-1 2022-07-23T19:17:11.202Z **BROKEN** plugin-bookkeeper: ccan/ccan/io/poll.c:453 (io_loop) lightningd-1 2022-07-23T19:17:11.202Z **BROKEN** plugin-bookkeeper: plugins/libplugin.c:1708 (plugin_main) lightningd-1 2022-07-23T19:17:11.203Z **BROKEN** plugin-bookkeeper: plugins/bkpr/bookkeeper.c:1812 (main) lightningd-1 2022-07-23T19:17:11.204Z **BROKEN** plugin-bookkeeper: ../csu/libc-start.c:308 (__libc_start_main) lightningd-1 2022-07-23T19:17:11.204Z **BROKEN** plugin-bookkeeper: parents: lightningd-1 2022-07-23T19:17:11.204Z **BROKEN** plugin-bookkeeper: plugins/libplugin.c:1378:struct command lightningd-1 2022-07-23T19:17:11.204Z **BROKEN** plugin-bookkeeper: MEMLEAK: 0x1508da8 lightningd-1 2022-07-23T19:17:11.204Z **BROKEN** plugin-bookkeeper: label=plugins/bkpr/bookkeeper.c:1568:struct channel_event lightningd-1 2022-07-23T19:17:11.204Z **BROKEN** plugin-bookkeeper: backtrace: lightningd-1 2022-07-23T19:17:11.204Z **BROKEN** plugin-bookkeeper: ccan/ccan/tal/tal.c:442 (tal_alloc_) lightningd-1 2022-07-23T19:17:11.204Z **BROKEN** plugin-bookkeeper: plugins/bkpr/bookkeeper.c:1568 (parse_and_log_channel_move) lightningd-1 2022-07-23T19:17:11.204Z **BROKEN** plugin-bookkeeper: plugins/bkpr/bookkeeper.c:1713 (json_coin_moved) lightningd-1 2022-07-23T19:17:11.205Z **BROKEN** plugin-bookkeeper: plugins/libplugin.c:1421 (ld_command_handle) lightningd-1 2022-07-23T19:17:11.205Z **BROKEN** plugin-bookkeeper: plugins/libplugin.c:1491 (ld_read_json_one) lightningd-1 2022-07-23T19:17:11.205Z **BROKEN** plugin-bookkeeper: plugins/libplugin.c:1511 (ld_read_json) lightningd-1 2022-07-23T19:17:11.205Z **BROKEN** plugin-bookkeeper: ccan/ccan/io/io.c:59 (next_plan) lightningd-1 2022-07-23T19:17:11.205Z **BROKEN** plugin-bookkeeper: ccan/ccan/io/io.c:407 (do_plan) lightningd-1 2022-07-23T19:17:11.205Z **BROKEN** plugin-bookkeeper: ccan/ccan/io/io.c:417 (io_ready) lightningd-1 2022-07-23T19:17:11.205Z **BROKEN** plugin-bookkeeper: ccan/ccan/io/poll.c:453 (io_loop) lightningd-1 2022-07-23T19:17:11.205Z **BROKEN** plugin-bookkeeper: plugins/libplugin.c:1708 (plugin_main) lightningd-1 2022-07-23T19:17:11.205Z **BROKEN** plugin-bookkeeper: plugins/bkpr/bookkeeper.c:1812 (main) lightningd-1 2022-07-23T19:17:11.206Z **BROKEN** plugin-bookkeeper: ../csu/libc-start.c:308 (__libc_start_main) lightningd-1 2022-07-23T19:17:11.206Z **BROKEN** plugin-bookkeeper: parents: lightningd-1 2022-07-23T19:17:11.206Z **BROKEN** plugin-bookkeeper: plugins/libplugin.c:1378:struct command
e6e2a3b
to
da5d702
Compare
@@ -225,7 +225,8 @@ static struct chain_event **find_txos_for_tx(const tal_t *ctx, | |||
" ORDER BY " | |||
" e.utxo_txid" | |||
", e.outnum" | |||
", e.spending_txid NULLS FIRST")); | |||
", e.spending_txid NULLS FIRST" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
NULLS FIRST, hmm, I've never seen that before! TIL...
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ack da5d702
Ack 4522951 |
Bookkeeper plugin keeps track of all the coin movements that happen on the attached clightning node. It also exposes a number of commands for inspecting these events, as well as some 'on top' accounting to figure out what you've spent on onchain fees.
bkpr-listbalances
: shows all accounts and their balances (a channel is an account)bkpr-listaccountevents
: dump of all raw events (incl onchain fee updates, as they happen). Can be filtered byaccount
bkpr-inspect
: show the onchain footprint of thisaccount
. Nice for seeing unilateral close resolution progress.bkpr-listincome
: list every 'income impacting' event (internal transfers etc are filtered). Defaults to consolidating fees.bkpr-dumpincomecsv
: Dump out the income events to a csv formatted for a given provider. Options include: cointracker, koinly, harmony, and quickbooks.bkpr-channelsapy
: Summary of fees earned on incoming/outgoing payments and fee leases on a channel by channel basis.Note that any coin movements that occurred before the bookkeeper plugin was initialized won't be counted.