feat: validate data column sidecars #7073
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
g11tech
reviewed
Sep 9, 2024
g11tech
reviewed
Sep 9, 2024
g11tech
approved these changes
Sep 17, 2024
matthewkeil
added a commit
that referenced
this pull request
Jan 21, 2025
commit e8bc729 Author: Matthew Keil <me@matthewkeil.com> Date: Tue Dec 3 10:08:30 2024 -0500 refactor: peerdas types (#7243) * refactor: organize peerDAS types * refactor: DataColumnsData * refactor: rename BlockInputBlobs BlockInputColumnData * refactor: split up and rename BlockInputData * refactor: clean up BlobsData * refactor: clean up CachedData types * refactor: change from interface to type and update enum values for grafana * chore: lint * fix: remove extraneous lint fix commit c8075d0 Author: Matthew Keil <me@matthewkeil.com> Date: Mon Nov 25 15:41:41 2024 +0800 feat: log peer disconnect info (#7231) * feat: log disconnect reason * feat: log peerScore update * fix: pretty print peerId * fix: use prettyPrintPeerId commit 8689c76 Author: Matthew Keil <me@matthewkeil.com> Date: Tue Oct 22 06:11:13 2024 -0400 feat: check for no commitments on block or column in sidecar validation (#7184) * feat: check for no commitments on block or column in sidecar validation * test: add sanity check for empty blob commitments in column validation * fix: test bug * fix: but in test passing commitments commit fccf9a2 Author: harkamal <gajinder@g11.in> Date: Tue Oct 8 12:44:11 2024 +0530 add prevdownload tracker commit 513bccc Author: harkamal <gajinder@g11.in> Date: Wed Oct 2 16:46:40 2024 +0530 fix commit 1c08ab3 Author: harkamal <gajinder@g11.in> Date: Wed Oct 2 16:40:40 2024 +0530 some fixing of beacon params commit 7c9a01c Author: harkamal <gajinder@g11.in> Date: Wed Oct 2 14:58:11 2024 +0530 add debug and fix datacolumns migration and improve log commit b04aaef Author: harkamal <gajinder@g11.in> Date: Wed Oct 2 12:52:24 2024 +0530 migrate datacolumns to finalized commit a0e0087 Author: harkamal <gajinder@g11.in> Date: Wed Oct 2 00:36:03 2024 +0530 more log commit 2736b8c Author: harkamal <gajinder@g11.in> Date: Tue Oct 1 19:43:59 2024 +0530 add enhance datacolumn serving logs commit cce193b Author: harkamal <gajinder@g11.in> Date: Tue Oct 1 17:36:14 2024 +0530 improve logging for debugging commit a3de70f Author: harkamal <gajinder@g11.in> Date: Thu Sep 26 14:43:49 2024 +0530 turn persisting network identity to default true commit cec27d6 Author: harkamal <gajinder@g11.in> Date: Sat Sep 21 22:07:58 2024 +0530 handle edge case commit 6a77828 Author: harkamal <gajinder@g11.in> Date: Sat Sep 21 20:33:53 2024 +0530 add debug console log commit 574837a Author: harkamal <gajinder@g11.in> Date: Sat Sep 21 18:39:36 2024 +0530 use sample subnets for data availability commit fee7c08 Author: harkamal <gajinder@g11.in> Date: Tue Sep 17 16:42:45 2024 +0530 validate inclusion proof commit 20ef4c6 Author: Matthew Keil <me@matthewkeil.com> Date: Tue Sep 17 05:33:40 2024 -0400 feat: validate data column sidecars (#7073) * feat: update c-kzg to final DAS version * refactor: use trusted-setup from c-kzg package * feat: implement validateDataColumnsSidecars * feat: check block and column commitments match * test: add unit test for validateDataColumnsSidecars * fix: invalid build and update validity condition of validateDataColumnsSidecars * fix: make error messages better * fix: electra vs peerdas type conflict commit b1940ee Author: Matthew Keil <me@matthewkeil.com> Date: Tue Sep 17 05:26:05 2024 -0400 fix: remove ckzg build script (#7089) * fix: remove unused ckzg build script * fix: remove unused rsync dep from Dockerfile commit bd4f7f9 Author: Matthew Keil <me@matthewkeil.com> Date: Mon Sep 16 06:31:52 2024 -0400 feat: update ckzg to final DAS version (#7050) * feat: update c-kzg to final DAS version * refactor: use trusted-setup from c-kzg package commit a33303f Author: Matthew Keil <me@matthewkeil.com> Date: Mon Sep 16 04:08:29 2024 -0400 feat: refactor and unit test getDataColumnSidecars (#7072) * refactor: getDataColumnSidecars * test: unit test getDataColumnSidecars with mocks from c-kzg library * refactor: use fromHex util * chore: update numbering on mocks * chore: update c-kzg to latest version * chore: fix type export syntax * test: add verification for cells from sidecars * test: add verification to DataColumnSidecars tests * refactor: getDataColumnSidecars for PR comments * feat: narrow type and remove unnecessary conditional * fix: getDataColumnSidecars param type * refactor: rename to computeDataColumnSidecars commit 4ec7aff Author: harkamal <gajinder@g11.in> Date: Fri Sep 13 18:19:45 2024 +0530 edge case optimization commit 3470076 Author: harkamal <gajinder@g11.in> Date: Fri Sep 13 17:31:52 2024 +0530 more debug log commit c4d04ee Author: harkamal <gajinder@g11.in> Date: Fri Sep 13 15:35:32 2024 +0530 fix the column id compute commit cdd9bae Author: harkamal <gajinder@g11.in> Date: Thu Sep 12 17:04:07 2024 +0530 update compute spec tests commit 2b10e4d Author: harkamal <gajinder@g11.in> Date: Thu Sep 12 15:25:00 2024 +0530 datacolumns retrival fix commit 56c8c6e Author: harkamal <gajinder@g11.in> Date: Thu Sep 12 13:57:51 2024 +0530 custodied column fetch debugging log commit af933fb Author: harkamal <gajinder@g11.in> Date: Wed Sep 11 22:11:16 2024 +0530 some fixes commit 74d8122 Author: harkamal <gajinder@g11.in> Date: Wed Sep 11 21:57:41 2024 +0530 add some log for debugging inbound data columns request commit f7571f4 Author: harkamal <gajinder@g11.in> Date: Wed Sep 11 17:07:52 2024 +0530 add some more loggig and availaibility tracking commit d35873e Author: harkamal <gajinder@g11.in> Date: Wed Sep 11 15:02:31 2024 +0530 further wait till cutoff for all data to be available commit 8c21168 Author: harkamal <gajinder@g11.in> Date: Wed Sep 11 00:02:09 2024 +0530 make pull a little less agressive commit de341b5 Author: harkamal <gajinder@g11.in> Date: Tue Sep 10 22:21:37 2024 +0530 add send more log commit bd84892 Author: harkamal <gajinder@g11.in> Date: Tue Sep 10 22:01:43 2024 +0530 more log commit d7721f8 Author: harkamal <gajinder@g11.in> Date: Tue Sep 10 21:19:11 2024 +0530 fix bug commit 5e1de6f Author: harkamal <gajinder@g11.in> Date: Tue Sep 10 20:45:05 2024 +0530 trying some fix commit 2bc1a0d Author: harkamal <gajinder@g11.in> Date: Tue Sep 10 20:11:05 2024 +0530 add cache tracking commit 387da88 Author: harkamal <gajinder@g11.in> Date: Tue Sep 10 19:27:05 2024 +0530 add more log commit aece0ab Author: harkamal <gajinder@g11.in> Date: Tue Sep 10 17:46:52 2024 +0530 fix add missing data availability resolutions commit 006e781 Author: harkamal <gajinder@g11.in> Date: Sat Sep 7 20:35:00 2024 +0530 add debug log commit bf08852 Author: harkamal <gajinder@g11.in> Date: Sat Sep 7 20:08:10 2024 +0530 resolve availability when datacolumns are downloaded and matched commit 2833ac0 Author: harkamal <gajinder@g11.in> Date: Thu Sep 5 20:17:04 2024 +0530 make the csc encoding updates as per latest spec commit 585165e Author: harkamal <gajinder@g11.in> Date: Wed Aug 28 17:35:31 2024 +0530 remove banning unknown block, addmore log commit a33a72f Author: harkamal <gajinder@g11.in> Date: Tue Aug 27 21:59:03 2024 +0530 subnet count 128 commit ae7678e Author: harkamal <gajinder@g11.in> Date: Tue Aug 27 21:29:50 2024 +0530 fx commit 4b6f167 Author: harkamal <gajinder@g11.in> Date: Tue Aug 27 19:37:37 2024 +0530 fix bug commit 180f7d8 Author: harkamal <gajinder@g11.in> Date: Tue Aug 27 18:58:26 2024 +0530 fix log commit 54579b0 Author: harkamal <gajinder@g11.in> Date: Tue Aug 27 17:21:07 2024 +0530 add more info for debugging commit a3533f8 Author: harkamal <gajinder@g11.in> Date: Tue Aug 27 17:13:41 2024 +0530 add supernode flag to configure node custody requirement and make it not required for validator commit e6c613f Author: harkamal <gajinder@g11.in> Date: Tue Aug 27 15:51:20 2024 +0530 rename electra fork to peerdas for rebase and make csc in metadata uint8 commit 81aaeb5 Author: harkamal <gajinder@g11.in> Date: Mon Aug 12 15:43:39 2024 +0530 feat: add and use metadatav3 for peer custody subnet fixes for metadata, working locally change the condition to update metadata csc change commit c7f6341 Author: harkamal <gajinder@g11.in> Date: Fri Aug 9 17:06:52 2024 +0530 fix the types/test rebase fixes commit a0c5d27 Author: harkamal <gajinder@g11.in> Date: Tue Jul 16 18:54:18 2024 +0530 fix: refactor to add and use nodeid computation and clear out nodeid tracking nodeid cleanup for network commit d423004 Author: harkamal <gajinder@g11.in> Date: Mon Jul 15 03:12:52 2024 +0530 feat: add the modifications to work with devnet2 some network options to control peering behavior allow setting node custody capability via --params use eip754 names for the peerdas config commit 47eedae Author: harkamal <gajinder@g11.in> Date: Sun Jul 14 17:56:59 2024 +0530 feat: get various sync mechanisms working with/without sharded data commit 156ef53 Author: matthewkeil <me@matthewkeil.com> Date: Fri Jun 21 17:09:44 2024 +0200 fix: docker build issue for c-kzg wip: REPLACE THIS COMMIT commit yarn lock rebase fixes fix: update c-zkg install workflow feat: add trustedSetupPrecompute cli flag fix: update trusted-setup for testing fix: update c-zkg install workflow to remove sudo fix: add rsync to apk deps commit 499d93c Author: harkamal <gajinder@g11.in> Date: Wed Jan 24 18:40:25 2024 +0530 feat: implement peerDAS on electra add some presets add further params and types add data column to types repo and network move to max request data columns to preset add the datacolumns data in blockinput and fix breaking errors in seen gossip blockinput handle data columns in gossip and the seengossip further propagate forkaware blockdata and resolve build/type issues further handle datacolumns sync by range by root and forkaware data handling fix issues chore: update c-kzg to peerDas version feat: add peerDas ckzg functions to interface fix the lookups handle the publishing flow various sync try fixes fixes compute blob side car various misl debuggings and fixes debug and apply fixes and get range and by root sync to work will full custody enable syncing with lower custody requirement use node peerid rather than a dummy string get and use the nodeid from enr and correctly compute subnets and column indexes filterout and connect to peers only matching out custody requiremnt try adding custody requirement add protection for subnet calc get the sync working with devnet 0 correctly set the enr with custody subnet info rebase fixes small refactor commit 4805a2e Author: harkamal <gajinder@g11.in> Date: Wed Jan 24 17:38:11 2024 +0530 feat: placeholder PR for electra add types stub and epoch config fix types
matthewkeil
added a commit
that referenced
this pull request
Jan 21, 2025
commit e8bc729 Author: Matthew Keil <me@matthewkeil.com> Date: Tue Dec 3 10:08:30 2024 -0500 refactor: peerdas types (#7243) * refactor: organize peerDAS types * refactor: DataColumnsData * refactor: rename BlockInputBlobs BlockInputColumnData * refactor: split up and rename BlockInputData * refactor: clean up BlobsData * refactor: clean up CachedData types * refactor: change from interface to type and update enum values for grafana * chore: lint * fix: remove extraneous lint fix commit c8075d0 Author: Matthew Keil <me@matthewkeil.com> Date: Mon Nov 25 15:41:41 2024 +0800 feat: log peer disconnect info (#7231) * feat: log disconnect reason * feat: log peerScore update * fix: pretty print peerId * fix: use prettyPrintPeerId commit 8689c76 Author: Matthew Keil <me@matthewkeil.com> Date: Tue Oct 22 06:11:13 2024 -0400 feat: check for no commitments on block or column in sidecar validation (#7184) * feat: check for no commitments on block or column in sidecar validation * test: add sanity check for empty blob commitments in column validation * fix: test bug * fix: but in test passing commitments commit fccf9a2 Author: harkamal <gajinder@g11.in> Date: Tue Oct 8 12:44:11 2024 +0530 add prevdownload tracker commit 513bccc Author: harkamal <gajinder@g11.in> Date: Wed Oct 2 16:46:40 2024 +0530 fix commit 1c08ab3 Author: harkamal <gajinder@g11.in> Date: Wed Oct 2 16:40:40 2024 +0530 some fixing of beacon params commit 7c9a01c Author: harkamal <gajinder@g11.in> Date: Wed Oct 2 14:58:11 2024 +0530 add debug and fix datacolumns migration and improve log commit b04aaef Author: harkamal <gajinder@g11.in> Date: Wed Oct 2 12:52:24 2024 +0530 migrate datacolumns to finalized commit a0e0087 Author: harkamal <gajinder@g11.in> Date: Wed Oct 2 00:36:03 2024 +0530 more log commit 2736b8c Author: harkamal <gajinder@g11.in> Date: Tue Oct 1 19:43:59 2024 +0530 add enhance datacolumn serving logs commit cce193b Author: harkamal <gajinder@g11.in> Date: Tue Oct 1 17:36:14 2024 +0530 improve logging for debugging commit a3de70f Author: harkamal <gajinder@g11.in> Date: Thu Sep 26 14:43:49 2024 +0530 turn persisting network identity to default true commit cec27d6 Author: harkamal <gajinder@g11.in> Date: Sat Sep 21 22:07:58 2024 +0530 handle edge case commit 6a77828 Author: harkamal <gajinder@g11.in> Date: Sat Sep 21 20:33:53 2024 +0530 add debug console log commit 574837a Author: harkamal <gajinder@g11.in> Date: Sat Sep 21 18:39:36 2024 +0530 use sample subnets for data availability commit fee7c08 Author: harkamal <gajinder@g11.in> Date: Tue Sep 17 16:42:45 2024 +0530 validate inclusion proof commit 20ef4c6 Author: Matthew Keil <me@matthewkeil.com> Date: Tue Sep 17 05:33:40 2024 -0400 feat: validate data column sidecars (#7073) * feat: update c-kzg to final DAS version * refactor: use trusted-setup from c-kzg package * feat: implement validateDataColumnsSidecars * feat: check block and column commitments match * test: add unit test for validateDataColumnsSidecars * fix: invalid build and update validity condition of validateDataColumnsSidecars * fix: make error messages better * fix: electra vs peerdas type conflict commit b1940ee Author: Matthew Keil <me@matthewkeil.com> Date: Tue Sep 17 05:26:05 2024 -0400 fix: remove ckzg build script (#7089) * fix: remove unused ckzg build script * fix: remove unused rsync dep from Dockerfile commit bd4f7f9 Author: Matthew Keil <me@matthewkeil.com> Date: Mon Sep 16 06:31:52 2024 -0400 feat: update ckzg to final DAS version (#7050) * feat: update c-kzg to final DAS version * refactor: use trusted-setup from c-kzg package commit a33303f Author: Matthew Keil <me@matthewkeil.com> Date: Mon Sep 16 04:08:29 2024 -0400 feat: refactor and unit test getDataColumnSidecars (#7072) * refactor: getDataColumnSidecars * test: unit test getDataColumnSidecars with mocks from c-kzg library * refactor: use fromHex util * chore: update numbering on mocks * chore: update c-kzg to latest version * chore: fix type export syntax * test: add verification for cells from sidecars * test: add verification to DataColumnSidecars tests * refactor: getDataColumnSidecars for PR comments * feat: narrow type and remove unnecessary conditional * fix: getDataColumnSidecars param type * refactor: rename to computeDataColumnSidecars commit 4ec7aff Author: harkamal <gajinder@g11.in> Date: Fri Sep 13 18:19:45 2024 +0530 edge case optimization commit 3470076 Author: harkamal <gajinder@g11.in> Date: Fri Sep 13 17:31:52 2024 +0530 more debug log commit c4d04ee Author: harkamal <gajinder@g11.in> Date: Fri Sep 13 15:35:32 2024 +0530 fix the column id compute commit cdd9bae Author: harkamal <gajinder@g11.in> Date: Thu Sep 12 17:04:07 2024 +0530 update compute spec tests commit 2b10e4d Author: harkamal <gajinder@g11.in> Date: Thu Sep 12 15:25:00 2024 +0530 datacolumns retrival fix commit 56c8c6e Author: harkamal <gajinder@g11.in> Date: Thu Sep 12 13:57:51 2024 +0530 custodied column fetch debugging log commit af933fb Author: harkamal <gajinder@g11.in> Date: Wed Sep 11 22:11:16 2024 +0530 some fixes commit 74d8122 Author: harkamal <gajinder@g11.in> Date: Wed Sep 11 21:57:41 2024 +0530 add some log for debugging inbound data columns request commit f7571f4 Author: harkamal <gajinder@g11.in> Date: Wed Sep 11 17:07:52 2024 +0530 add some more loggig and availaibility tracking commit d35873e Author: harkamal <gajinder@g11.in> Date: Wed Sep 11 15:02:31 2024 +0530 further wait till cutoff for all data to be available commit 8c21168 Author: harkamal <gajinder@g11.in> Date: Wed Sep 11 00:02:09 2024 +0530 make pull a little less agressive commit de341b5 Author: harkamal <gajinder@g11.in> Date: Tue Sep 10 22:21:37 2024 +0530 add send more log commit bd84892 Author: harkamal <gajinder@g11.in> Date: Tue Sep 10 22:01:43 2024 +0530 more log commit d7721f8 Author: harkamal <gajinder@g11.in> Date: Tue Sep 10 21:19:11 2024 +0530 fix bug commit 5e1de6f Author: harkamal <gajinder@g11.in> Date: Tue Sep 10 20:45:05 2024 +0530 trying some fix commit 2bc1a0d Author: harkamal <gajinder@g11.in> Date: Tue Sep 10 20:11:05 2024 +0530 add cache tracking commit 387da88 Author: harkamal <gajinder@g11.in> Date: Tue Sep 10 19:27:05 2024 +0530 add more log commit aece0ab Author: harkamal <gajinder@g11.in> Date: Tue Sep 10 17:46:52 2024 +0530 fix add missing data availability resolutions commit 006e781 Author: harkamal <gajinder@g11.in> Date: Sat Sep 7 20:35:00 2024 +0530 add debug log commit bf08852 Author: harkamal <gajinder@g11.in> Date: Sat Sep 7 20:08:10 2024 +0530 resolve availability when datacolumns are downloaded and matched commit 2833ac0 Author: harkamal <gajinder@g11.in> Date: Thu Sep 5 20:17:04 2024 +0530 make the csc encoding updates as per latest spec commit 585165e Author: harkamal <gajinder@g11.in> Date: Wed Aug 28 17:35:31 2024 +0530 remove banning unknown block, addmore log commit a33a72f Author: harkamal <gajinder@g11.in> Date: Tue Aug 27 21:59:03 2024 +0530 subnet count 128 commit ae7678e Author: harkamal <gajinder@g11.in> Date: Tue Aug 27 21:29:50 2024 +0530 fx commit 4b6f167 Author: harkamal <gajinder@g11.in> Date: Tue Aug 27 19:37:37 2024 +0530 fix bug commit 180f7d8 Author: harkamal <gajinder@g11.in> Date: Tue Aug 27 18:58:26 2024 +0530 fix log commit 54579b0 Author: harkamal <gajinder@g11.in> Date: Tue Aug 27 17:21:07 2024 +0530 add more info for debugging commit a3533f8 Author: harkamal <gajinder@g11.in> Date: Tue Aug 27 17:13:41 2024 +0530 add supernode flag to configure node custody requirement and make it not required for validator commit e6c613f Author: harkamal <gajinder@g11.in> Date: Tue Aug 27 15:51:20 2024 +0530 rename electra fork to peerdas for rebase and make csc in metadata uint8 commit 81aaeb5 Author: harkamal <gajinder@g11.in> Date: Mon Aug 12 15:43:39 2024 +0530 feat: add and use metadatav3 for peer custody subnet fixes for metadata, working locally change the condition to update metadata csc change commit c7f6341 Author: harkamal <gajinder@g11.in> Date: Fri Aug 9 17:06:52 2024 +0530 fix the types/test rebase fixes commit a0c5d27 Author: harkamal <gajinder@g11.in> Date: Tue Jul 16 18:54:18 2024 +0530 fix: refactor to add and use nodeid computation and clear out nodeid tracking nodeid cleanup for network commit d423004 Author: harkamal <gajinder@g11.in> Date: Mon Jul 15 03:12:52 2024 +0530 feat: add the modifications to work with devnet2 some network options to control peering behavior allow setting node custody capability via --params use eip754 names for the peerdas config commit 47eedae Author: harkamal <gajinder@g11.in> Date: Sun Jul 14 17:56:59 2024 +0530 feat: get various sync mechanisms working with/without sharded data commit 156ef53 Author: matthewkeil <me@matthewkeil.com> Date: Fri Jun 21 17:09:44 2024 +0200 fix: docker build issue for c-kzg wip: REPLACE THIS COMMIT commit yarn lock rebase fixes fix: update c-zkg install workflow feat: add trustedSetupPrecompute cli flag fix: update trusted-setup for testing fix: update c-zkg install workflow to remove sudo fix: add rsync to apk deps commit 499d93c Author: harkamal <gajinder@g11.in> Date: Wed Jan 24 18:40:25 2024 +0530 feat: implement peerDAS on electra add some presets add further params and types add data column to types repo and network move to max request data columns to preset add the datacolumns data in blockinput and fix breaking errors in seen gossip blockinput handle data columns in gossip and the seengossip further propagate forkaware blockdata and resolve build/type issues further handle datacolumns sync by range by root and forkaware data handling fix issues chore: update c-kzg to peerDas version feat: add peerDas ckzg functions to interface fix the lookups handle the publishing flow various sync try fixes fixes compute blob side car various misl debuggings and fixes debug and apply fixes and get range and by root sync to work will full custody enable syncing with lower custody requirement use node peerid rather than a dummy string get and use the nodeid from enr and correctly compute subnets and column indexes filterout and connect to peers only matching out custody requiremnt try adding custody requirement add protection for subnet calc get the sync working with devnet 0 correctly set the enr with custody subnet info rebase fixes small refactor commit 4805a2e Author: harkamal <gajinder@g11.in> Date: Wed Jan 24 17:38:11 2024 +0530 feat: placeholder PR for electra add types stub and epoch config fix types
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
Implements
validateDataColumnsSidecarsin place of stub function