Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

v0.8 spec sync #77

Merged
merged 18 commits into from
Jul 11, 2019
Merged

v0.8 spec sync #77

merged 18 commits into from
Jul 11, 2019

Conversation

hwwhww
Copy link
Contributor

@hwwhww hwwhww commented Jul 4, 2019

What was wrong?

Fix #76

How was it fixed?

  1. Using merkle_minimal/zerohashes style Merkleization since with max_length, we do need zerohashes approach for saving computations of a lot of padding.
  2. Add max_length in List and add mix_in_length for List.hash_tree_root().
  3. Add Bitvector and Bitlist:
    NOTE: Use spec patch Define Bitlist/Bitvector serialization using bytes, not bigints consensus-specs#1267 for Bitvector and Bitlist serialization. Should wait for Define Bitlist/Bitvector serialization using bytes, not bigints consensus-specs#1267 getting merged.
  4. Passed the fuzzing (test_decorder.py) in pyspec (enable BeaconState, BeaconBlock in pyspec side).

Cute Animal Picture

persian-cat-2461526_640

@hwwhww hwwhww force-pushed the list_max_length branch from f531f6b to 75de33e Compare July 4, 2019 07:38
@hwwhww hwwhww force-pushed the list_max_length branch 4 times, most recently from 25037b9 to a9bd4fe Compare July 5, 2019 08:03
@hwwhww hwwhww force-pushed the list_max_length branch from a9bd4fe to 47373b5 Compare July 5, 2019 08:20
@hwwhww hwwhww marked this pull request as ready for review July 5, 2019 08:23
@hwwhww hwwhww requested a review from jannikluhn July 8, 2019 12:55
Copy link
Contributor

@jannikluhn jannikluhn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Preliminary review, only managed to go through bitlist so far, will continue soon.

ssz/sedes/bitlist.py Show resolved Hide resolved
ssz/sedes/bitlist.py Outdated Show resolved Hide resolved

class Bitlist(BaseCompositeSedes[BytesOrByteArray, bytes]):
def __init__(self, bit_count: int) -> None:
if bit_count < 0:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
if bit_count < 0:
if bit_count < 1:

as the new spec version now explicitly forbids zero-length vectors..

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks like I was wrong, empty lists (but not vectors) should be fine.

ssz/sedes/bitlist.py Outdated Show resolved Hide resolved
ssz/sedes/bitlist.py Outdated Show resolved Hide resolved
ssz/sedes/bitlist.py Outdated Show resolved Hide resolved
ssz/sedes/bitlist.py Outdated Show resolved Hide resolved
ssz/sedes/bitlist.py Outdated Show resolved Hide resolved
if len_value == 0:
return b'\x01'

as_bytearray = [0] * (len_value // 8 + 1)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is there a reason you aren't actually using a bytearray? I think it will give you some added type/value safety since it won't allow assignment of values outside of the 0-256 range.

as_bytearray = bytearray(len_value // 8 + 1)

Also, won't len_value // 8 + 1 produce arrays with an extra element in the case that len_value is a clean multiple of 8?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, I should have used bytearray, thanks!

Also, won't len_value // 8 + 1 produce arrays with an extra element in the case that len_value is a clean multiple of 8?

It is. https://github.com/ethereum/eth2.0-specs/blob/dev/specs/simple-serialize.md#bitlistn:
"Note that from the offset coding, the length (in bytes) of the bitlist is known. An additional leading 1 bit is added so that the length in bits will also be known."

ssz/utils.py Outdated Show resolved Hide resolved
ssz/utils.py Outdated Show resolved Hide resolved
ssz/utils.py Outdated Show resolved Hide resolved

class Bitlist(BaseCompositeSedes[BytesOrByteArray, bytes]):
def __init__(self, bit_count: int) -> None:
if bit_count < 0:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks like I was wrong, empty lists (but not vectors) should be fine.

ssz/sedes/bitvector.py Show resolved Hide resolved
ssz/sedes/bitlist.py Outdated Show resolved Hide resolved
ssz/sedes/bitvector.py Outdated Show resolved Hide resolved
ssz/sedes/bitvector.py Outdated Show resolved Hide resolved
tests/tree_hash/test_composite_tree_hash.py Show resolved Hide resolved
tests/tree_hash/test_composite_tree_hash.py Outdated Show resolved Hide resolved
tests/tree_hash/test_composite_tree_hash.py Outdated Show resolved Hide resolved
tests/tree_hash/test_composite_tree_hash.py Show resolved Hide resolved
tests/tree_hash/test_composite_tree_hash.py Outdated Show resolved Hide resolved
@hwwhww hwwhww force-pushed the list_max_length branch from bb301ed to 81456de Compare July 11, 2019 06:31
@hwwhww
Copy link
Contributor Author

hwwhww commented Jul 11, 2019

@jannikluhn @pipermerriam
Thanks for the review, the PR feedback is applied or replied.

@jannikluhn while the minimum length check is pending on the spec side discussion, we can go back to check it later after this PR.

Copy link
Contributor

@jannikluhn jannikluhn left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Probably worth revisiting some parts in the feature for cleanups, but since those wouldn't affect the external API and it would be helpful to get a feeling of the performance requirements first, I think we can merge this now.

@@ -142,20 +153,18 @@ def _deserialize_stream(self, stream: IO[bytes]) -> Iterable[TDeserialized]:
# Tree hashing
#
def hash_tree_root(self, value: Iterable[TSerializable]) -> bytes:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was just thinking about input validation, to prevent people from accidentally hashing inputs that are too big. Don't think it's super necessary though, we can always add this later.

for layer in range(chunk_depth, max_depth):
tmp_list[layer + 1] = hash_eth2(tmp_list[layer] + zerohashes[layer])

return tmp_list[max_depth]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, you're right. In the interest of time I'd be ok with keeping it as it is for now. We might have to revisit the implementation anyway when we're going to implement tree caching.

@hwwhww hwwhww merged commit a200d0b into ethereum:master Jul 11, 2019
pacrob added a commit to pacrob/py-ssz that referenced this pull request Apr 14, 2023
* convert bash scripts to py
pacrob added a commit to pacrob/py-ssz that referenced this pull request Dec 1, 2023
* convert bash scripts to py
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Sync with spec v0.8
3 participants