Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixes the total num of payments returned by QueryPayments when there are dates restrictions #8883

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

MPins
Copy link
Contributor

@MPins MPins commented Jul 1, 2024

Fixes #8530

Change Description

The function QueryPayments was not considering the dates restriction when counting the total num of payments.

Steps to Test

lncli listpayments --creation_date_start 1650000000 --count_total_payments --creation_date_end 1650000001 | grep total_num_payments

This command should return 0 as there are no payments between the date start and the date end.

Pull Request Checklist

Testing

  • Your PR passes all CI checks.
  • Tests covering the positive and negative (error paths) are included.
  • Bug fixes contain tests triggering the bug to prevent regressions.

Code Style and Documentation

📝 Please see our Contribution Guidelines for further guidance.

Copy link
Contributor

coderabbitai bot commented Jul 1, 2024

Important

Review skipped

Auto reviews are limited to specific labels.

Labels to auto review (1)
  • llm-review

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.


Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media?

Share
Tips

Chat

There are 3 ways to chat with CodeRabbit:

  • Review comments: Directly reply to a review comment made by CodeRabbit. Example:
    • I pushed a fix in commit <commit_id>.
    • Generate unit testing code for this file.
    • Open a follow-up GitHub issue for this discussion.
  • Files and specific lines of code (under the "Files changed" tab): Tag @coderabbitai in a new review comment at the desired location with your query. Examples:
    • @coderabbitai generate unit testing code for this file.
    • @coderabbitai modularize this function.
  • PR comments: Tag @coderabbitai in a new PR comment to ask questions about the PR branch. For the best results, please provide a very specific query, as very limited context is provided in this mode. Examples:
    • @coderabbitai generate interesting stats about this repository and render them as a table.
    • @coderabbitai show all the console.log statements in this repository.
    • @coderabbitai read src/utils.ts and generate unit testing code.
    • @coderabbitai read the files in the src/scheduler package and generate a class diagram using mermaid and a README in the markdown format.
    • @coderabbitai help me debug CodeRabbit configuration file.

Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments.

CodeRabbit Commands (invoked as PR comments)

  • @coderabbitai pause to pause the reviews on a PR.
  • @coderabbitai resume to resume the paused reviews.
  • @coderabbitai review to trigger an incremental review. This is useful when automatic reviews are disabled for the repository.
  • @coderabbitai full review to do a full review from scratch and review all the files again.
  • @coderabbitai summary to regenerate the summary of the PR.
  • @coderabbitai resolve resolve all the CodeRabbit review comments.
  • @coderabbitai configuration to show the current CodeRabbit configuration for the repository.
  • @coderabbitai help to get help.

Additionally, you can add @coderabbitai ignore anywhere in the PR description to prevent this PR from being reviewed.

CodeRabbit Configuration File (.coderabbit.yaml)

  • You can programmatically configure CodeRabbit by adding a .coderabbit.yaml file to the root of your repository.
  • Please see the configuration documentation for more information.
  • If your editor has YAML language server enabled, you can add the path at the top of this file to enable auto-completion and validation: # yaml-language-server: $schema=https://coderabbit.ai/integrations/schema.v2.json

Documentation and Community

  • Visit our Documentation for detailed information on how to use CodeRabbit.
  • Join our Discord Community to get help, request features, and share feedback.
  • Follow us on X/Twitter for updates and announcements.

Copy link
Member

@yyforyongyu yyforyongyu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR. Please check the contribution guidelines regarding Ideal Git Commit Structure.

@MPins MPins requested a review from yyforyongyu July 4, 2024 19:36
Copy link
Member

@yyforyongyu yyforyongyu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved CI run.

@MPins MPins force-pushed the issue-8530 branch 5 times, most recently from 44987f4 to 07bf0a1 Compare July 10, 2024 19:59
@MPins MPins requested a review from yyforyongyu July 10, 2024 20:12
@feelancer21
Copy link
Contributor

feelancer21 commented Jul 11, 2024

@MPins I don't think your fix solves the issue. Using len caps the number of payments at max_payments, which contradicts the definition of the flag count_total_payments. You need to count all payments that are in the database and within the specified date range.

lnd/lnrpc/lightning.proto

Lines 4178 to 4184 in a965535

/*
If set, all payments (complete and incomplete, independent of the
max_payments parameter) will be counted. Note that setting this to true will
increase the run time of the call significantly on systems that have a lot
of payments, as all of them have to be iterated through to be counted.
*/
bool count_total_payments = 5;

@MPins
Copy link
Contributor Author

MPins commented Jul 11, 2024

independent of the
max_payments parameter

At this point, the offset was exhausted and all invoices, that respect the query, were collected to resp.Payments, so uint64(len(resp.Payments)) will return that total num of payments.

@feelancer21
Copy link
Contributor

feelancer21 commented Jul 11, 2024

independent of the
max_payments parameter

At this point, the offset was exhausted and all invoices, that respect the query, were collected to resp.Payments, so uint64(len(resp.Payments)) will return that total num of payments.

I am also beginner. As far I understand the code, the maximum length of res.Payments is restricted to MaxPayments because of the paginator. And hence it is not independent of max_payments.

Edit:
I made a test on testnet lncli listpayments --count_total_payments --max_payments 1. And total_num_payments changed to 1 now. I think it is not as expected.

@MPins
Copy link
Contributor Author

MPins commented Jul 11, 2024

independent of the
max_payments parameter

At this point, the offset was exhausted and all invoices, that respect the query, were collected to resp.Payments, so uint64(len(resp.Payments)) will return that total num of payments.

I am also beginner. As far I understand the code, the maximum length of res.Payments is restricted to MaxPayments because of the paginator. And hence it is not independent of max_payments.

Edit: I made a test on testnet lncli listpayments --count_total_payments --max_payments 1. And total_num_payments changed to 1 now. I think it is not as expected.

You right! Thanks! I'll be working on it.

@MPins
Copy link
Contributor Author

MPins commented Jul 18, 2024

independent of the
max_payments parameter

At this point, the offset was exhausted and all invoices, that respect the query, were collected to resp.Payments, so uint64(len(resp.Payments)) will return that total num of payments.

I am also beginner. As far I understand the code, the maximum length of res.Payments is restricted to MaxPayments because of the paginator. And hence it is not independent of max_payments.
Edit: I made a test on testnet lncli listpayments --count_total_payments --max_payments 1. And total_num_payments changed to 1 now. I think it is not as expected.

You right! Thanks! I'll be working on it.

@yyforyongyu now the correction is considering the point indicate by @feelancer21, when the request comes with --count_total_payments and --max_payments causing pagination. Ready to go, thank you both!

@feelancer21
Copy link
Contributor

Thanks. I see some redundant code now, reading the payment and checking the criteria are mainly the same in accumulatePayments and countFn. You should check if it is possible to extract this part in a seperate functions processing the steps of reading and checking.

@MPins MPins force-pushed the issue-8530 branch 2 times, most recently from a29b155 to d91efbc Compare July 18, 2024 22:30
@MPins
Copy link
Contributor Author

MPins commented Jul 18, 2024

Thanks. I see some redundant code now, reading the payment and checking the criteria are mainly the same in accumulatePayments and countFn. You should check if it is possible to extract this part in a seperate functions processing the steps of reading and checking.

Yes, thats make sense. Done!

Copy link
Member

@yyforyongyu yyforyongyu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Linter failed, and a nit in the method name. Could you add a release note to 0.18.3? Then it should be good to go.

@yyforyongyu yyforyongyu added this to the v0.18.3 milestone Jul 22, 2024
@MPins MPins force-pushed the issue-8530 branch 5 times, most recently from 0469d98 to b5d7255 Compare July 22, 2024 10:05
@MPins
Copy link
Contributor Author

MPins commented Jul 22, 2024

Linter failed, and a nit in the method name. Could you add a release note to 0.18.3? Then it should be good to go.

Thanks!

The linter failed because there was a missing blank line before a 'return'. I can see that in others parts of the code, is there an specific case where the blank line is a must?

The release note could be the following:

  • Fixed a bug when RPC call is made to list payments counting the total number of payments.

Should I update the release note file, or you do?

@yyforyongyu
Copy link
Member

The linter failed because there was a missing blank line before a 'return'. I can see that in others parts of the code, is there an specific case where the blank line is a must?

It's needed for readability. The linter only checks new code based on a configured look-ahead commit hash. That's probably why you'd see some other parts still use this pattern.

Should I update the release note file, or you do?

Yeah it's usually the author who does the update. You'll find the right section and add the note, also update the author list in the end of the release note file.

@MPins
Copy link
Contributor Author

MPins commented Jul 25, 2024

@yyforyongyu just commited the release notes 0.18.3 changes. Thank you!

@MPins MPins force-pushed the issue-8530 branch 3 times, most recently from 66fae02 to 6fd6272 Compare July 29, 2024 14:49
@MPins MPins requested a review from yyforyongyu July 29, 2024 16:10
@MPins
Copy link
Contributor Author

MPins commented Jul 30, 2024

@yyforyongyu I did not understand the lint fail!! Any Tip?

@MPins MPins requested a review from guggero July 31, 2024 01:47
@yyforyongyu
Copy link
Member

hmm this has failed in two of the itests,

--- FAIL: TestLightningNetworkDaemon/tranche02/45-of-159/bitcoind/list_payments (22.88s)
        --- FAIL: TestLightningNetworkDaemon/tranche02/45-of-159/bitcoind/list_payments/payment_exact_start_date (0.01s)

harness_rpc.go:97: 
        	Error Trace:	/home/runner/work/lnd/lnd/lntest/rpc/harness_rpc.go:97
        	            				/home/runner/work/lnd/lnd/lntest/rpc/lnd.go:187
        	            				/home/runner/work/lnd/lnd/itest/lnd_payment_test.go:286
        	Error:      	Received unexpected error:
        	            	rpc error: code = Unknown desc = error rolling back tx: conn busy
        	Messages:   	Alice: failed to call ListPayments

Think it's related?

@MPins
Copy link
Contributor Author

MPins commented Jul 31, 2024

list_payments/payment_exact_start_date

I have run the tests after the change and they are running properly!

Running integration tests with btcd backend.
rm -rf itest/.log itest/.logs-; date
qua 31 jul 2024 18:45:10 -03
EXEC_SUFFIX= scripts/itest_part.sh 0 1 -test.run="TestLightningNetworkDaemon/tranche./.-of-././list_payments" -test.timeout=180m
/home/pins/Projects/lnd/itest/itest.test -test.v -test.run=TestLightningNetworkDaemon/tranche./.-of-././list_payments -test.timeout=180m -logoutput -logdir=.logs-tranche0 -lndexec=/home/pins/Projects/lnd/itest/lnd-itest -btcdexec=/home/pins/Projects/lnd/itest/btcd-itest -splittranches=1 -runtranche=0
=== RUN TestLightningNetworkDaemon
harness_setup.go:25: Setting up HarnessTest...
harness_setup.go:38: Prepare the miner and mine blocks to activate segwit...
harness_setup.go:46: Connecting the miner at 127.0.0.1:10446 with the chain backend...
harness.go:313: Setting up standby nodes Alice and Bob...
harness.go:352: Finished the setup, now running tests...
=== RUN TestLightningNetworkDaemon/tranche00/45-of-156/btcd/list_payments
=== RUN TestLightningNetworkDaemon/tranche00/45-of-156/btcd/list_payments/payment_exact_start_date
=== RUN TestLightningNetworkDaemon/tranche00/45-of-156/btcd/list_payments/payment_earlier_start_date
=== RUN TestLightningNetworkDaemon/tranche00/45-of-156/btcd/list_payments/payment_future_start_date
=== RUN TestLightningNetworkDaemon/tranche00/45-of-156/btcd/list_payments/payment_exact_end_date
=== RUN TestLightningNetworkDaemon/tranche00/45-of-156/btcd/list_payments/payment_future_end_date
=== RUN TestLightningNetworkDaemon/tranche00/45-of-156/btcd/list_payments/payment_earlier_end_date
=== RUN TestLightningNetworkDaemon/tranche00/45-of-156/btcd/list_payments/payment_earlier_start_and_end_date
=== RUN TestLightningNetworkDaemon/tranche00/45-of-156/btcd/list_payments/invoice_exact_start_date
=== RUN TestLightningNetworkDaemon/tranche00/45-of-156/btcd/list_payments/invoice_earlier_start_date
=== RUN TestLightningNetworkDaemon/tranche00/45-of-156/btcd/list_payments/invoice_future_start_date
=== RUN TestLightningNetworkDaemon/tranche00/45-of-156/btcd/list_payments/invoice_exact_end_date
=== RUN TestLightningNetworkDaemon/tranche00/45-of-156/btcd/list_payments/invoice_future_end_date
=== RUN TestLightningNetworkDaemon/tranche00/45-of-156/btcd/list_payments/invoice_earlier_end_date
=== RUN TestLightningNetworkDaemon/tranche00/45-of-156/btcd/list_payments/invoice_earlier_start_and_end_date
=== NAME TestLightningNetworkDaemon/tranche00/45-of-156/btcd/list_payments
harness.go:440: finished test: list_payments, start height=440, end height=447, mined blocks=7
=== NAME TestLightningNetworkDaemon
lnd_test.go:155: =========> tests finished for tranche: 0, tested 156 cases, end height: 440
--- PASS: TestLightningNetworkDaemon (90.83s)
--- PASS: TestLightningNetworkDaemon/tranche00/45-of-156/btcd/list_payments (19.19s)
--- PASS: TestLightningNetworkDaemon/tranche00/45-of-156/btcd/list_payments/payment_exact_start_date (0.00s)
--- PASS: TestLightningNetworkDaemon/tranche00/45-of-156/btcd/list_payments/payment_earlier_start_date (0.00s)
--- PASS: TestLightningNetworkDaemon/tranche00/45-of-156/btcd/list_payments/payment_future_start_date (0.00s)
--- PASS: TestLightningNetworkDaemon/tranche00/45-of-156/btcd/list_payments/payment_exact_end_date (0.00s)
--- PASS: TestLightningNetworkDaemon/tranche00/45-of-156/btcd/list_payments/payment_future_end_date (0.00s)
--- PASS: TestLightningNetworkDaemon/tranche00/45-of-156/btcd/list_payments/payment_earlier_end_date (0.00s)
--- PASS: TestLightningNetworkDaemon/tranche00/45-of-156/btcd/list_payments/payment_earlier_start_and_end_date (0.00s)
--- PASS: TestLightningNetworkDaemon/tranche00/45-of-156/btcd/list_payments/invoice_exact_start_date (0.00s)
--- PASS: TestLightningNetworkDaemon/tranche00/45-of-156/btcd/list_payments/invoice_earlier_start_date (0.00s)
--- PASS: TestLightningNetworkDaemon/tranche00/45-of-156/btcd/list_payments/invoice_future_start_date (0.00s)
--- PASS: TestLightningNetworkDaemon/tranche00/45-of-156/btcd/list_payments/invoice_exact_end_date (0.00s)
--- PASS: TestLightningNetworkDaemon/tranche00/45-of-156/btcd/list_payments/invoice_future_end_date (0.00s)
--- PASS: TestLightningNetworkDaemon/tranche00/45-of-156/btcd/list_payments/invoice_earlier_end_date (0.00s)
--- PASS: TestLightningNetworkDaemon/tranche00/45-of-156/btcd/list_payments/invoice_earlier_start_and_end_date (0.00s)
PASS

Copy link
Member

@yyforyongyu yyforyongyu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To see the failure, you need to run it against the SQL db via make itest icase=list_payments dbbackend=postgres.

Also I'm not sure if #8530 is indeed an issue - what if a user actually wants to know the total num of payments instead of the ranged payments?

If we want to go with the current design, we should also update the count_total_payments here to be count_payments, and explain in details when the total num will be counted, and when the ranged num will be counted,

lnd/lnrpc/lightning.proto

Lines 4184 to 4190 in 04dde98

/*
If set, all payments (complete and incomplete, independent of the
max_payments parameter) will be counted. Note that setting this to true will
increase the run time of the call significantly on systems that have a lot
of payments, as all of them have to be iterated through to be counted.
*/
bool count_total_payments = 5;


countFn := func(sequenceKey, hash []byte) error {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This func won't work in SQL unless we change ForAll to ForEach below.

Copy link
Contributor Author

@MPins MPins Aug 13, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I did the change ... now it is working even to SQL databases ... but as you said there is no otimization when there is a SQL database available because we cannot use the ForAll instead of ForEach as it is done before this fix.

As you said below maybe we shouldn't touch it until we can find an efficient counting method. Maybe this is another PR to come!


countFn := func(sequenceKey, hash []byte) error {
r := bytes.NewReader(hash)
paymentHash, err := deserializePaymentIndex(r)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since counting payments is already slow, this would make it even slower - for nodes with millions of payments this would take a very long time. I think at least we should make this query conditional, that,

  1. if there aren't any date restrictions, use the original totalPayments++ approach.
  2. if there is a date range, use this countFn.

But I still think this would be very slow, cc @bhandras for some help here, maybe we shouldn't touch it until we can find an efficient counting method?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe there is a way to measure the performance impact and only move forward if it is near the same of we have today without the fix.

@MPins
Copy link
Contributor Author

MPins commented Aug 1, 2024

sue - what if a user actually wants to know the total num of payments instead of the ranged payments?

About it is an issue or not, I have the following opinion:
1 - If the user want the total num of payments with no date restrictions, why user would set the date range? It is counter intuitive.
2 - Without this fix there is no way to the user know the total num of payments in a date range.
3 - In the text you highlighted it says "independent of the max_payments parameter" not independent of date range.

@saubyk saubyk modified the milestones: v0.18.3, v0.19.0 Aug 6, 2024
@guggero
Copy link
Collaborator

guggero commented Aug 9, 2024

I'm not sure how to proceed here...

It sounds to me like the expectation for what the --count_total_payments actually does is different between the author of the issue and the reviewers.

Maybe what we need is a new flag, --count_payments_in_range that does the new count as implemented currently but warns the user about the potential performance impact.

Then we can change the description of the --count_total_payments flag to say "counts all payments in the database independent of any filter/range criteria given".

@guggero guggero removed their request for review August 9, 2024 09:31
@yyforyongyu
Copy link
Member

Thanks a lot for the input so far @MPins ! I'd recommend pausing the development of this PR at the moment as we will soon bring native SQL support for payments in 0.19.0 - maybe we can find a more efficient way to perform the queries by then.

@saubyk saubyk changed the title Fixes the total num of payments returned by QueryPaymenst when there are dates restrictions Fixes the total num of payments returned by QueryPayments when there are dates restrictions Aug 13, 2024
MPins added 3 commits August 13, 2024 16:29
QueryPayments function was changed to return the total num of payments
considering the date start and date end restrictions existing on the query.
When there are no payments expected returning, besides
checking if the paymensts is empty, check also if the
total num of payments is zero.
@lightninglabs-deploy
Copy link

@MPins, remember to re-request review from reviewers when ready

@yyforyongyu
Copy link
Member

!lightninglabs-deploy mute

@saubyk saubyk modified the milestones: v0.19.0, v0.20.0 Jan 23, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[bug]: lncli listpayments --count_total_payments should honor --creation_date_start --creation_date_end
6 participants