Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce data access recording capabilities #162

Merged
merged 2 commits into from
Jun 24, 2023
Merged

Conversation

frisitano
Copy link
Contributor

@frisitano frisitano commented Jun 16, 2023

Describe your changes

This PR introduces the required changes to support recording of data access. This includes the introduction of the following components and changes:

  • KvMap: A trait that defines the interface for a key-value map.
  • RecordingMap: A [RecordingMap] that records read requests to the underlying key-value map. The data recorder is used to generate a proof for read requests.
  • MerkleMapT: A supertrait that defines the required traits for a type to be used as a data map backend for the [GenericMerkleStore]
  • MerkleStore -> GenericMerkleStore: Made MerkleStore generic over the key value map backend.

src/data.rs Outdated Show resolved Hide resolved
src/data.rs Outdated Show resolved Hide resolved
src/data.rs Outdated
}

// TODO: Consider if we are happy with the fact that duplicates are counted twice in both
// `len(..)` and `iter(..)` methods.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we are unhappy with double counting then we can introduce a counter for duplicates which we deduct when len is called. To make the iter(..) method only return unique values we could use itertools or alternatively copy the Unique implementation we require .

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we do want to handle this properly (especially for iteration) as it may affect how things work downstream. Depending on how much work it is, copying the implementation would be better (though, if it proves rather complex, adding itertools as a dependency may be OK.

Another potential alternative is to change how this data structure keeps track of updates. Specifically, it could look something like this:

pub struct DataRecorder<K, V> {
    data: BTreeMap<K, V>,
    updates: BTreeSet<K>,
    trace: RefCell<BTreeMap<K, V>>,
}

And something like get() method could then work like so:

pub fn get(&self, key: &K) -> Option<&V> {
    match self.state.get(key) {
        None => None,
        Some(value) => {
            if !self.updates.contains(key) {
                self.trace.borrow_mut().insert(key.clone(), value);                
            }
            Some(value)
        }
    }
}

This does do more work - so, not sure if it is worth it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does state represent in this instance?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oh - sorry, state should have been data.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think your proposal will work. It will always return the original value even if it has been updated. I implemented an alternate scheme, this involves the introduction of updated_keys which keeps track of the keys from the original set which have been updated.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've implemented this approach in 80a7f01 as it simplified the implementation a bit. The tests seem to be passing (and I've added some more checks) - but take a look and see if I missed anything.

src/data.rs Outdated Show resolved Hide resolved
src/data.rs Outdated Show resolved Hide resolved
src/data.rs Outdated Show resolved Hide resolved
src/data.rs Outdated Show resolved Hide resolved
@frisitano frisitano requested a review from bobbinth June 23, 2023 13:38
@frisitano frisitano changed the title Frisitano tx executor Introduce data access recording capabilities Jun 23, 2023
@bobbinth
Copy link
Contributor

I pushed a commit which simplifies things a bit. This includes:

  • Refactoring of RecordingMap.
  • Getting rid of some extra traits in favor of having a single KvMap trait which inherits from other traits such as Extend, FromIterator etc.
  • Reorganizing crate structure a bit (i.e., moving KvMap under collections).

@bobbinth bobbinth merged commit f52ac29 into next Jun 24, 2023
@bobbinth bobbinth deleted the frisitano-tx-executor branch June 24, 2023 06:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants