VID dispersal, commitment should take an iterator over u8 instead of &[u8]

[ggutoski]

jellyfish/primitives/src/vid/mod.rs

Lines 56 to 59 in f6866c5

	fn commit_only(&self, payload: &[u8]) -> VidResult<Self::Commit>;
	/// Compute shares to send to the storage nodes
	fn disperse(&self, payload: &[u8]) -> VidResult<VidDisperse<Self>>;

This forces the caller to marshal its data as a contiguous u8 slice, which is a waste of time and memory.

[ggutoski]

Problem

bytes_to_field_elements and bytes_from_field_elements also need to change to take an iterator:

jellyfish/utilities/src/conversion.rs

Lines 129 to 131 in f6866c5

	pub fn bytes_to_field_elements<B, F>(bytes: B) -> Vec<F>
	where
	B: Borrow<[u8]>,

Unfortunately, current impls of these functions need to know the size up front. We could do it if we require the iterator to be ExactSizeIterator, but the expected use case is an iterator formed by chaining a bunch of other iterators. (Example: Vec<Vec<u8>>). Sadly, chained iterators can never be ExactSizeIterator, even when composed of iterators that are: rust-lang/rust#66531

Thus, any solution to this issue requires a change to bytes_[to|from]_field_elements that does not require ExactSizeIterator. That's annoying because we need to find a new way to keep the invertibility guarantee without ExactSizeIterator. 😕

[ggutoski]

duplicate of #300 . But the above comment is new.

[ggutoski]

Candidate solution

As described in #300, bytes_[to|from]_field_elements should be an "iterator adaptor".

Currently we prefix the output with a field element that encodes the byte length of the input. Don't do this because it requires knowing the byte length at the beginning. Instead, append an additional field element at the end of the iterator that encodes the number of bytes to discard from the final field element.

Example: Field elements take 32 bytes, but the input byte length is 14 mod 32, so we append an additional field element that encodes the number 14, indicating that only the first 14 bytes of the final field element are part of the input and so the remaining 18 bytes should be discarded.

This solution requires peekable (the ability to peek 1 item ahead). Fortunately, peekable is part of the Iterator trait, so all iterators have it (including chained iterators).