-
Notifications
You must be signed in to change notification settings - Fork 895
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Autoclean for giant nodes #7298
Autoclean for giant nodes #7298
Conversation
b7bfa88
to
2a8065e
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Concept-ACK
I left some minors comments
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should remove also the docs
plugins/autoclean.c
Outdated
#define NUM_SUBSYSTEM_TYPES (INVOICES + 1) | ||
}; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
uh I did not know about this! looks amazing!
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This was really fun to read through.
Notably, the counter is now off in test_autoclean
(predates this change).
> assert l3.rpc.autoclean_status()['autoclean']['expiredinvoices']['cleaned'] == 3
FAILED tests/test_plugin.py::test_autoclean - assert 5 == 3
Really nice usage of filter
and the pagination APIs imo!!
@@ -88,20 +91,23 @@ static const struct subsystem_ops subsystem_ops[NUM_SUBSYSTEM_TYPES] = { | |||
{ {"succeededforwards", "failedforwards"}, | |||
"listforwards", | |||
"forwards", | |||
"\"in_channel\":true,\"in_htlc_id\":true,\"resolved_time\":true,\"received_time\":true,\"status\":true", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
oh you weren't joking about hacky!
plugins/libplugin.h
Outdated
@@ -455,6 +455,9 @@ bool plugin_developer_mode(const struct plugin *plugin); | |||
#define plugin_option_dev(name, type, description, set, arg) \ | |||
plugin_option_((name), (type), (description), (set), (arg), true, NULL, NULL, false) | |||
|
|||
#define plugin_option_dev_dynamic(name, type, description, set, arg) \ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit: should be its own commit?
Use modern-style iterators: this can be huge and we will run out of memory! Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
Changelog-Removed: JSON-RPC: `autocleaninvoice` command (deprecated v22.11, EOL v24.02) Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We're about to make this a more complex struct, so introducing a new_clean_info() function unifies the code paths. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This was actually a bug! (Yay for CI!) We weren't clearing the "done" counter and counting things twice... |
2a8065e
to
064b679
Compare
There are really three subsystems: invoices, forwards and sendpays, each of which has two variants we care about (successes and failures). If we split the code that way, we can extract the core differences in each of these cases and share most of the logic. It's a bit awkward to iterate over each "subsystem" in the JSON parameter sense, so we have some iteration code to do that where we need to. The result is going to be much easier to paginate! Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
This is crude, handing a raw JSON string, but it works. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
…n coming). Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
We can filter down to only the list* fields we need. In the case of a node with 1M forwards, this reduces listforwards time from 5 seconds to 4 seconds. It will also reduce memory consumption. Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
listforwards on a large node can easily run out of memory. Sip, don't gulp! Signed-off-by: Rusty Russell <rusty@rustcorp.com.au>
064b679
to
e6cfe91
Compare
Autoclean cannot handle giant nodes:
There are improvements across the board, but the real win comes from only listing 10,000 entries at a time.