algod importer: Auto catchpoint label lookup. #65

winder · 2023-04-28T17:58:52Z

Summary

Adds a new option to automatically lookup an appropriate catchpoint label during startup.

Test Plan

New unit tests.

Manually run conduit with -r 27000001 to see that the correct catchpoint label is automatically populated.

codecov · 2023-04-28T18:06:26Z

Codecov Report

Merging #65 (1e2d90f) into master (442791a) will increase coverage by 1.79%.
The diff coverage is 75.41%.

@@            Coverage Diff             @@
##           master      #65      +/-   ##
==========================================
+ Coverage   67.66%   69.45%   +1.79%     
==========================================
  Files          32       33       +1     
  Lines        1976     2118     +142     
==========================================
+ Hits         1337     1471     +134     
+ Misses        570      569       -1     
- Partials       69       78       +9

Impacted Files	Coverage Δ
conduit/data/block_export_data.go	`100.00% <ø> (+92.30%)`	⬆️
conduit/plugins/importers/algod/metrics.go	`100.00% <ø> (ø)`
...lugins/exporters/postgresql/postgresql_exporter.go	`66.66% <51.21%> (-11.54%)`	⬇️
conduit/pipeline/pipeline.go	`66.09% <72.10%> (+0.64%)`	⬆️
conduit/data/config.go	`76.47% <76.47%> (ø)`
conduit/plugins/importers/algod/algod_importer.go	`84.69% <88.17%> (-3.62%)`	⬇️
conduit/pipeline/errors.go	`100.00% <100.00%> (ø)`

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

conduit/plugins/importers/algod/README.md

docs/tutorials/IndexerWriter.md

conduit/plugins/importers/algod/algod_importer.go

tzaffi

Still reading but it's a good juncture to submit some feedback.

tzaffi · 2023-05-03T16:25:08Z

conduit/plugins/importers/algod/README.md

-        # testnet: https://algorand-catchpoints.s3.us-east-2.amazonaws.com/consolidated/testnet_catchpoints.txt
+        # Automatically download an appropriate catchpoint label. If false, you
+        # must specify a catchpoint to use fast catchup.
+        auto: false


Given the language of the algod importer plugin's README, auto is a bit confusing.

It says:

If ... the plugin will automatically run fast catchup on startup if the node is behind the current pipeline round

So to me, "auto" seems redundant, though clearly it isn't.

I see two approaches. Remove "automatically" from the README

Rename auto to something else like latest-available, or latest, or pipeline-latest, or something like that.

Good point. Which approach would you recommend? I updated the readme to clarify what was being done automatically. But maybe the setting name could also be clearer?

To answer your question, I'm leaning towards just modding the README. auto is a sensible name (or the opposite manual as well)

And to answer my own question, the following script seems to work. So yeah, you do need to call sync before starting conduit:

%%bash export DATA_NODE=follower just stop_and_nuke || true just pub-create conduit mainnet just start just algod /v2/ledger/sync/26589217 POST just algod /v2/ledger/sync ../cmd/conduit/conduit -d ./data --next-round-override 26589217

Sorry for the confusion. I've figured out the problem. You need the changes in go-algorand#5349. Without them you run into this error: https://github.com/algorand/go-algorand/pull/5349/files#r1185050820

Hmmm... I did pull down and make the latest from 5349. When I run:

❯ goal --version 12886130966 3.16.180502.dev [HEAD] (commit #28487271+) go-algorand is licensed with AGPLv3.0 source code available at https://github.com/algorand/go-algorand

and git log gives:

commit 284872714e46ae7504e02a7288c9837a000942a3 (HEAD, will/will/follower-catchup-pause) Author: Will Winder <wwinder.unh@gmail.com> Date: Wed May 3 07:19:57 2023 -0400 Remove old test.

So are the errors I'm seeing necessarily from your link?

One possible issue is that I'm merging into another branch. I'll try a clean version of the commit.

Ran again with a clean version of this PR. Taking some notes:

Command:

❯ conduit -d ./data2 -r 10009217 ... after around 30 seconds ... {"__type":"Conduit","_name":"main","level":"info","msg":"Pipeline round: 10009217","time":"2023-05-04T10:25:38-05:00"} ... another minute or so ... {"__type":"processor","_name":"algod","level":"error","msg":"error getting block for round 10009217 (attempt 0)","time":"2023-05-04T10:26:38-05:00"} {"__type":"processor","_name":"algod","level":"error","msg":"error getting block for round 10009217 (attempt 1)","time":"2023-05-04T10:27:38-05:00"}

at the same time getting algod's status (30s and around 3 minutes in)

❯ just status Last committed block: 6483 Time since last block: 0.0s Sync Time: 33.6s Last consensus protocol: https://github.com/algorandfoundation/specs/tree/5615adc36bad610c7f165fa2967f4ecfa75125f0 Next consensus protocol: https://github.com/algorandfoundation/specs/tree/5615adc36bad610c7f165fa2967f4ecfa75125f0 Round for next consensus protocol: 6484 Next consensus protocol supported: true Last Catchpoint: Genesis ID: mainnet-v1.0 Genesis hash: wGHE2Pwdvd7S12BL5FaOP20EGYesN73ktiC1qzkkit8= RUNNING ❯ just status Last committed block: 39148 Time since last block: 0.0s Sync Time: 225.6s Last consensus protocol: https://github.com/algorandfoundation/specs/tree/5615adc36bad610c7f165fa2967f4ecfa75125f0 Next consensus protocol: https://github.com/algorandfoundation/specs/tree/5615adc36bad610c7f165fa2967f4ecfa75125f0 Round for next consensus protocol: 39149 Next consensus protocol supported: true Last Catchpoint: Genesis ID: mainnet-v1.0 Genesis hash: wGHE2Pwdvd7S12BL5FaOP20EGYesN73ktiC1qzkkit8= RUNNING

Ended up failing the same way as before.

tzaffi

not finished reading, but want to address a question

docs/tutorials/IndexerMigration.md

conduit/plugins/importers/algod/README.md

-        # testnet: https://algorand-catchpoints.s3.us-east-2.amazonaws.com/consolidated/testnet_catchpoints.txt
+        # Automatically download an appropriate catchpoint label. If false, you
+        # must specify a catchpoint to use fast catchup.
+        auto: false


tzaffi

Convinced that the issues I'm encountering are un-related to this PR, and that this work solves the problems in question.

winder added 2 commits April 28, 2023 13:46

algod importer: Auto catchpoint label lookup.

677b4f8

update docs

214544e

Replace redundant check.

46f3ba7

winder requested review from Eric-Warehime, shiqizng and a team April 28, 2023 18:47

winder marked this pull request as ready for review April 28, 2023 18:47

Eric-Warehime approved these changes Apr 28, 2023

View reviewed changes

winder added 6 commits April 28, 2023 15:25

Fix catchpoint variable.

674a7a6

Fix unit test.

7b625e4

Don't monitor unless we actually start the catchup

4b5d838

Handle catching up to an earlier round.

9d899ed

Fix unit tests.

f174e71

More tests.

423c9f3

winder requested a review from Eric-Warehime May 3, 2023 12:09

winder added 4 commits May 3, 2023 08:12

Move starting catchup message to better location.

6985d3e

Workaround for catching up to the exact round.

0d58499

Update test.

bf55b66

Fix another test...

d0e1c1c

winder requested review from tzaffi and algochoi May 3, 2023 13:40

shiqizng reviewed May 3, 2023

View reviewed changes

conduit/plugins/importers/algod/README.md Outdated Show resolved Hide resolved

docs/tutorials/IndexerWriter.md Outdated Show resolved Hide resolved

conduit/plugins/importers/algod/algod_importer.go Outdated Show resolved Hide resolved

tzaffi reviewed May 3, 2023

View reviewed changes

winder added 2 commits May 3, 2023 15:16

Mention label and auto precedence.

b644257

Don't overuse the word automatic.

1dea076

tzaffi reviewed May 3, 2023

View reviewed changes

Clarify comment and tutorial.

dcb0cde

shiqizng approved these changes May 3, 2023

View reviewed changes

Catchup by default without the auto parameter.

1e2d90f

tzaffi approved these changes May 4, 2023

View reviewed changes

winder merged commit 86f8b0c into algorand:master May 4, 2023

winder deleted the will/auto-fast-catchup branch May 4, 2023 17:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

algod importer: Auto catchpoint label lookup. #65

algod importer: Auto catchpoint label lookup. #65

winder commented Apr 28, 2023 •

edited

Loading

codecov bot commented Apr 28, 2023 •

edited

Loading

tzaffi left a comment

tzaffi May 3, 2023

winder May 3, 2023

This comment was marked as duplicate.

tzaffi May 3, 2023

tzaffi May 3, 2023

winder May 4, 2023

tzaffi May 4, 2023

tzaffi May 4, 2023

tzaffi May 4, 2023

tzaffi May 4, 2023

tzaffi left a comment

This comment was marked as duplicate.

tzaffi left a comment

algod importer: Auto catchpoint label lookup. #65

algod importer: Auto catchpoint label lookup. #65

Conversation

winder commented Apr 28, 2023 • edited Loading

Summary

Test Plan

codecov bot commented Apr 28, 2023 • edited Loading

Codecov Report

tzaffi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

This comment was marked as duplicate.

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tzaffi left a comment

Choose a reason for hiding this comment

This comment was marked as duplicate.

tzaffi left a comment

Choose a reason for hiding this comment

winder commented Apr 28, 2023 •

edited

Loading

codecov bot commented Apr 28, 2023 •

edited

Loading