Skip to content
This repository has been archived by the owner on Nov 15, 2023. It is now read-only.

Implementer's Guide: Integrate DMP into PersistentValidationData + DMQ watermarks #1629

Closed
wants to merge 4 commits into from

Conversation

pepyakin
Copy link
Contributor

@pepyakin pepyakin commented Aug 24, 2020

Based on top of #1588
Closes #1643

Summary

  • Main theme of this PR is integration of DMP into the PersistentValidationData ("ground truth") through a DMQ MQC head.
  • Introduce DMQ watermarks

This PR is accompanied by some drive-by wording fixes here and there.

Description

I decided to take the approach of putting only the MQC head into PersistentValidationData to allow the authorization of the messages that a collator provides while building a parablock.

Background The alternative is to put the DMQ contents, all messages, into `PersistentValidationData`. We need to put all messages in because `PersistentValidationData` is used to form the inputs and is collected before the execution of the PVF. Putting only data to authorize the sequence of messages into `PersistentValidationData` allows the collator to include the messages that were actually consumed into the PoV.

All in all, this should positively affect the amount of data made available.

While working on it I realized that it might be useful to supply the block number in which a particular downward message was put in the DMQ. That also makes it similar to HRMP/XCMP MQC which may be good from the implementation standpoint.

It seemed to me that it would be harder for the PVF to return processed_downward_messages and trying to keep track of the on-the-relay-chain state of DMQ rather than dealing with watermarks. So, i finally introduced DMQ watermarks. They cost a bit more, since we need to keep track the digest (the list of all blocks where the DMP messages were sent to correctly check the DMQ watermark)

Those DMQ watermarks, however, are still different from HRMP watermarks. I am still not confident about merging them.

What to look for

Here are some clues how to approach this PR

  • The TransientValidationData has all data to check if the resulting candidate passes the acceptance criteria (by collator or primary checker)
  • The acceptance criteria establishes sensible invariants
    • That the enactment function can rely on
    • Won't change in between backing and the enactment
  • Unused data is properly cleaned and doesn't pile up
  • Among other things from areas of safety, consistency, etc

@pepyakin pepyakin added A3-in_progress Pull request is in progress. No review needed at this stage. B0-silent Changes should not be mentioned in any release notes C1-low PR touches the given topic and has a low impact on builders. labels Aug 24, 2020
@pepyakin pepyakin added this to the Grosvenor milestone Aug 24, 2020
@pepyakin pepyakin force-pushed the ser-dmq-persistent branch from 0c2c250 to a2510c1 Compare August 24, 2020 17:56
Copy link
Contributor

@drahnr drahnr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks mostly good as far as I can tell, it would be nice to avoid using acronyms in headers,

@pepyakin pepyakin force-pushed the ser-dmq-persistent branch from 7bf4578 to 398cf4a Compare August 31, 2020 12:25
@pepyakin
Copy link
Contributor Author

On the second thought, the DMQ watermarks are not such a great idea. The problem with them is that they are coarse grained and require reading all messages within a block. Earlier, we decided that we don't want to restrict the relay-chain on how many messages to send to a parachain. With that, we can potentially end up in a situation the relay-chain sent a too many messages to even fit into a parablock.

We could potentially leverage a compound watermark, similar to what we are going to use for XCMP where the watermark consists of the block number and para id, which allows to read only a part of messages in a block.

In the DMP we have only the single sender - the relay-chain and only meaningful tie-breaker for the messages within the block is the order in which they were sent, e.g. (block number 5, message number 3 of 5).

That, however, seems to be an over-engineered version of the original processed_downward_messages approach. I think we would better to leave it.

Therefore, I am closing this PR

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
A3-in_progress Pull request is in progress. No review needed at this stage. B0-silent Changes should not be mentioned in any release notes C1-low PR touches the given topic and has a low impact on builders.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Guide: Pass DMP messages into the PVF
2 participants