Store the domain genesis storage on the consensus chain #1824

NingLin-P · 2023-08-15T21:05:04Z

In domain v2, the domain's genesis state root is driven and stored on the consensus chain when the domain is instantiated, and we need to ensure when the operator node is started it will construct the same genesis state root.

Currently, this is done by storing the domain runtime code and optionally the domain GenesisRuntimeConfig on the consensus chain, and the operator node will fetch these data from consensus chain (use GenesisRuntimeConfig::default in case it is not stored) and them construct the genesis state by <GenesisRuntimeConfig as BuildStorage>::build_storage.

Unfortunately, both GenesisRuntimeConfig::default and <GenesisRuntimeConfig as BuildStorage>::build_storage are called on the client side and the return value is determined by the native runtime that compiled with the consensus chain, thus even if we store the same WASM runtime code onchain the operator may still get a different genesis state root if the operator is running with a newer release of subspace-node, see #1798 for more detail.

This PR fixes this issue by directly storing the domain genesis storage onchain when domain is instantiated instead of building the genesis storage when the operator is started. This won't bring much cost compared with the main branch because:

~~We remove the runtime code from the genesis storage to avoid duplicating with RuntimeRegistry~~
~~We just changed to store the genesis storage instead of GenesisRuntimeConfig onchain~~

This PR fixes the issue by storing the domain genesis storage in the runtime registry, so when registering a runtime instead of providing just the runtime code we need to provide the full genesis storage (which includes the runtime code), and every domain instance will reuse the same genesis storage with a slight adjustment of the SelfDomainId storage, without the need of rebuilding the genesis storage again.

This PR also comes with a domain subcommand build-genesis-storage that we can use to get the raw genesis storage for registering runtime.

This PR also gets rid of the ugly workaround of constructing chain spec with the GenesisRuntimeConfig which is introduced in #1668

Code contributor checklist:

I have read, understood and followed contributing guide

vedhavyas

I'm still not sure the exact reason we need to hold entire genesis state 🤔
If the build_storage relies on std, then other substrate chains might not sync at all but that is not the case.
Can you add more detail as to what is actually causing the different state root? If there is an actual requirement for std, then the fix should be sent upstream instead of trying to patch it here IMO

NingLin-P · 2023-09-06T18:38:34Z

Please take a look when you have time @nazar-pc

nazar-pc · 2023-09-08T02:50:31Z

Will take a look tomorrow, this seems to be pretty involved

vedhavyas

I'm still not sure why we need the host function 🤔. We already have the raw chainspec stored, when the domain instance is created, we can make necessary domain specific changes and generate the state root which is what the host function is doing now. I was hoping the host function is completely removed in this change

crates/sp-domains/src/lib.rs

vedhavyas · 2023-09-08T13:45:47Z

crates/pallet-domains/src/runtime_registry.rs

@@ -134,19 +94,26 @@ pub(crate) fn register_runtime_at_genesis<T: Config>(
 /// Schedules a runtime upgrade after `DomainRuntimeUpgradeDelay` from current block number.
 pub(crate) fn do_schedule_runtime_upgrade<T: Config>(
    runtime_id: RuntimeId,
-    code: Vec<u8>,
+    raw_genesis_storage: Vec<u8>,


we already have the raw_genesis when we register a runtime. Why do we need this again ?

Because the genesis state will be changed as runtime version changes as observed previously, so every different runtime needs a corresponding genesis storage.

This doesn't seem right. If we upgrade existing domain, we shouldn't wipe the state and replace it with one from genesis. Domain should do the actual upgrade instead with provided updated code.

Domain should do the actual upgrade instead with provided updated code.

Yeah, you are right!

Still not sure why we need to pass the entire geneis storage here. We just new runtime and the rest should stay as is. What am I missing here ?

Because the genesis state will be changed as runtime version changes as observed previously, so every different runtime needs a corresponding genesis storage.

^

@nazar-pc 's comment is wrong, plz take a look at #1824 (comment)

Passing genesis storage means you can instantitate domain with correct storage right away, otherwise you'll need to do multiple runtime upgrades right after instantiation, which makes little sense.

NingLin-P · 2023-09-08T15:42:41Z

we can make necessary domain specific changes and generate the state root which is what the host function is doing now

The reason we still need host function is these operator is not supported in runtime currently:

Decode/Encode the genesis state
- For adjusting the SelfDomainId storage and getting the runtime code to extract the runtime version/hash
Compute the state root out of the genesis state

vedhavyas · 2023-09-08T16:27:23Z

The reason we still need host function is these operator is not supported in runtime currently:
Decode/Encode the genesis state
For adjusting the SelfDomainId storage and getting the runtime code to extract the runtime version/hash
Compute the state root out of the genesis state

serde-json-core supports WASM, so we can already decode those key values and replace the necessary keys. As for the state root, we just need a trie builder to construct the state root. trie-db as well supports WASM. These should completely eliminate the Host functions on the domains.

Update:
Thinking further, we just need to decode only once when the runtime added.
On the encoder, why not use scale instead ?

nazar-pc

I will not pretend to understand what is going on, but I have questions and concerns. The goal was to remove host calls, but we seem to still have them, which is not good.

crates/sp-domains/src/storage.rs

domains/pallets/domain-id/src/lib.rs

test/subspace-test-client/src/domain_chain_spec.rs

crates/sp-domains/src/lib.rs

crates/pallet-domains/src/lib.rs

nazar-pc · 2023-09-08T19:15:49Z

crates/pallet-domains/src/runtime_registry.rs

@@ -134,19 +94,26 @@ pub(crate) fn register_runtime_at_genesis<T: Config>(
 /// Schedules a runtime upgrade after `DomainRuntimeUpgradeDelay` from current block number.
 pub(crate) fn do_schedule_runtime_upgrade<T: Config>(
    runtime_id: RuntimeId,
-    code: Vec<u8>,
+    raw_genesis_storage: Vec<u8>,


This doesn't seem right. If we upgrade existing domain, we shouldn't wipe the state and replace it with one from genesis. Domain should do the actual upgrade instead with provided updated code.

This storage key will be used in upcoming commit to reset the SelfDomainId storage item to the correct value in the domain genesis storage Signed-off-by: linning <linningde25@gmail.com>

…Info supported RawGenesis is mostly port from upstream with slight changes to support our usecase Signed-off-by: linning <linningde25@gmail.com>

…ime code In this way the domain instance can reuse the genesis storage, which including both the wasm runtime code and genesis config of other pallet, and avoid to build the genesis storage again Signed-off-by: linning <linningde25@gmail.com>

…e root for domain instance This commit remove the usage of host function to derive genesis receipt and also get rid of the ugly workaround of constructing chain spec with the RuntimeGenesisConfig and global variable Signed-off-by: linning <linningde25@gmail.com>

Signed-off-by: linning <linningde25@gmail.com>

Previously we need to delay this to block one due to the host function is unavailable but now the host function is removed. Also, because the block hash of the genesis block is unavailable thus the genesis receipt's consensus_block_hash is set to the default value Signed-off-by: linning <linningde25@gmail.com>

Since we use runtime_object.raw_genesis to derive the genesis state for domain instance and which already contains the sudo account, raw_genesis_config is needless now Signed-off-by: linning <linningde25@gmail.com>

…-storage Since now we don't need the raw_genesis_config when instantiate domain and we need the raw_genesis_storage when register runtime Signed-off-by: linning <linningde25@gmail.com>

This is used to simulate the situation where the domain runtime is upgraded after domain instantiation, thus the domain tests will ensure the domain genesis state will always be the same even it is generated in different place with different native runtime. This commit also change the domain_id of the RuntimeGenesisConfig to an arbitrary value to ensure the domain_id will be reset to the correct value in the genesis storage Signed-off-by: linning <linningde25@gmail.com>

NingLin-P · 2023-09-14T01:36:33Z

PR is rebased, and now all the domain-related host functions are removed, PTAL.

vedhavyas

Make sense. Some nits and questions.

vedhavyas · 2023-09-14T14:29:04Z

crates/pallet-domains/src/lib.rs

-                    )
-                    .expect("Genesis domain instantiation must always succeed");
+                let domain_id =
+                    do_instantiate_domain::<T>(domain_config, domain_owner.clone(), One::one(), None)


we do not need the raw_genesis_config to instantiate domain anymore. We should also remove the sudo related things from the domain_registry as well

It is removed in a later commit but I'm not sure what you means by remove the sudo related things from the domain_registry as well

vedhavyas · 2023-09-14T14:37:20Z

crates/pallet-domains/src/lib.rs

    /// A temporary storage to hold any previous epoch details for a given domain
    /// if the epoch transitioned in this block so that all the submitted bundles
    /// within this block are verified.
-    /// The storage is cleared on block finalization.
+    /// TODO: The storage is cleared on block finalization that means this storage is already cleared when
+    /// verifying the `submit_bundle` extrinsic and not used at all


this is never meant to be used outside the runtime. This storage purely exists to ensure POE is verified if there was a Epoch transition in this block

Yeah, but the storage is inserted during block execution and cleared on block finalization which means it only exists during block execution, and we need to verify POE whenever a node receives a submit_bundles extrinsic.

vedhavyas · 2023-09-14T14:39:26Z

crates/pallet-domains/src/lib.rs

-                    .get(operator_id)
-                    .ok_or(BundleError::BadOperator)?;
-                Ok((*operator_stake, pending_election_params.total_domain_stake))
+        if let Some(pending_election_params) = LastEpochStakingDistribution::<T>::get(domain_id) {


This refactor seems unrelated 🤔

It is related to the change of registering the genesis domain at block #0 instead of block #1.

Because when in block #1, the LastEpochStakingDistribution is cleared when verifying the bundle that is to be included at #2 so we always use DomainStakingSummary.

But when in block #0 namely the genesis block there is no block finalization thus the LastEpochStakingDistribution is used when verifying the bundle that is to be included at #1, and it will fail with BadOperator due to the operator id not present in LastEpochStakingDistribution::operators

vedhavyas · 2023-09-14T14:40:51Z

crates/sp-domains/src/lib.rs

        ExecutionReceipt {
            domain_block_number: Zero::zero(),
            domain_block_hash: Default::default(),
            domain_block_extrinsic_root: Default::default(),
            parent_domain_block_receipt_hash: Default::default(),
-            consensus_block_hash: consensus_genesis_hash,
+            consensus_block_hash: Default::default(),


not sure why? Ideally, this should be the block hash in which the domain was instantiated. Is that not so ?

Sorry, there should be a TODO here, but the change is due to the genesis domain being registered at the genesis block now, and we can't know the hash of the genesis block at that time.

crates/pallet-domains/src/lib.rs

vedhavyas · 2023-09-14T14:44:41Z

test/subspace-test-client/src/domain_chain_spec.rs

@@ -89,12 +88,12 @@ pub fn testnet_evm_genesis() -> RuntimeGenesisConfig {
                .collect(),
            ..Default::default()
        },
-        ethereum: Default::default(),
-        base_fee: Default::default(),
        self_domain_id: evm_domain_test_runtime::SelfDomainIdConfig {


why not just default instead ?

the domain_id is set to Some(DomainId::new(123)) for testing purpose

nazar-pc

I don't think I follow everything through, but I certainly like this more than previous iteration

nazar-pc · 2023-09-14T17:35:50Z

crates/sp-domains/src/storage.rs

+/// NOTE: this function is port from `sp_state_machine::in_memory_backend::new_in_mem` which is
+/// only export for `std` but we need to use it in `no_std`


What about submitting PR upstream that exposes it in other environments? I'd pull it earlier today with Substrate upgrade already.

I want to get some feedback for this direction first, if it is the right direction I will surely contribute it upstream.

nazar-pc · 2023-09-14T17:36:31Z

crates/sp-domains/src/storage.rs

+    TrieBackendBuilder::new(db, empty_trie_root::<LayoutV0<H>>()).build()
+}
+
+// NOTE: this is port from `sp_core::storage::StorageKey` with `TypeInfo` supported


Again, looks like something worth contributing upstream.

nazar-pc · 2023-09-14T17:54:33Z

crates/pallet-domains/src/runtime_registry.rs

+    let mut raw_genesis: RawGenesis = Decode::decode(&mut raw_genesis_storage.as_slice())
+        .map_err(|_| Error::FailedToDecodeRawGenesis)?;
+
+    // Remove the runtime code from `raw_genesis` before storing it


Why removing runtime code from genesis state? It would make sense to me to leave it in there so you don't need to put it back in later.

Because runtime code is updatable, it is weird to extract the code, update it, and store the updated code back to the genesis state.

Because runtime code is updatable, it is weird to extract the code, update it, and store the updated code back to the genesis state.

This should never be necessary, runtime code in genesis state doesn't need to be updatable. It needs to be canonical instead and come as a bundle, just like it is before you remove runtime code out of it. For updated runtime genesis state will be different too. State and code should come in pairs when it comes to instantiation.

You don't update genesis state without updating runtime code and you don't update runtime code without updating genesis state.

Okay, I may be misled by your reply #1824 (comment), that is what I thought at first! the genesis state and the runtime code should come in pairs. This should be true for both the first registration and the later runtime upgrade. The new domain instance should use both the upgraded runtime code and genesis state to initialize its genesis block, this is the only place the genesis state is involved. The existing domain instance should just upgrade its runtime code don't care about the updated genesis state.

The last part of above comment is correct. They come together, but during upgrades we only upgrade code and let it change the state accordingly.

It still means we need to update state one step at a time when we deploy new domain. Maybe this is acceptable, but feels inefficient.

I'm okay with storing the runtime code along with the genesis state, but what we need to discuss is whether the genesis state should be updated as the domain runtime upgraded in the consensus chain. My opinion is yes, the domain genesis state and runtime code should come in pairs and be updated at the same time.

What you're suggesting seems to be that domain code is updated "in place", which is confusing for other domains that use the old code
I say, we store all the new runtime seperately as they are submitted but leave the first one in place. This also avoids the manually refilling them when we need the full genesis

Suppose the domain is instantiated, it commits to a runtime object with the specified RuntimeId, and uses both the runtime object's runtime code and genesis state to initialize the domain chain.

Later, the runtime object is updated (both the code and genesis state), and the existing domain instance will upgrade its runtime code unconditionally and automatically as the upgrade is derived from the consensus block deterministically. However, the existing domain doesn't care about the updated genesis state because it only needs this when building its genesis block.

Then there is a new domain instance instantiated and committed to the same runtime object with the same RuntimeId, now the new domain instance should use the newest version of both the runtime code and genesis state to initialize its genesis block. This is the reason why I think we should update the runtime object's runtime code and genesis state at the same time.

And I can't see why we need to store all the version of the runtime code in the consensus chain state to support instantiating new domain instances with an older version of the runtime code.

My opinion is yes, the domain genesis state and runtime code should come in pairs and be updated at the same time.

That is my expectation and preference as well.

And I can't see why we need to store all the version of the runtime code in the consensus chain state to support instantiating new domain instances with an older version of the runtime code.

Then whatever the latest is will have to support runtime upgrades from whatever the oldest registered domain is (even if it is not running right now) because otherwise domain will not resume. I feel like this is an issue and storing all used versions is very desirable.

Then whatever the latest is will have to support runtime upgrades from whatever the oldest registered domain is (even if it is not running right now) because otherwise domain will not resume

No, as said above the domain runtime upgrade is not controlled by the domain owner but by the runtime object, whenever the runtime object is upgraded the domain instance will upgrade its runtime unconditionally and automatically.

If the domain instance node is not running for a while, then it should catch up to the domain chain tip by deriving the domain block from the consensus block, it can't miss any consensus block since it is instantiated unless we support syncing the domain chain from other nodes, but still all the consensus block should be handled by some (but not all) domain nodes in case any potential bundle/runtime upgrade is missed.

The PR is updated with runtime code and genesis state stored together and updated together, PTAL.

crates/pallet-domains/src/lib.rs

…ain_runtime call Signed-off-by: linning <linningde25@gmail.com>

Signed-off-by: linning <linningde25@gmail.com>

NingLin-P · 2023-10-02T09:56:31Z

All comments are addressed/replied, please take a look again, thanks!

vedhavyas

General Direction make sense but left some coments

crates/pallet-domains/src/lib.rs

vedhavyas · 2023-10-02T11:16:00Z

crates/pallet-domains/src/lib.rs

@@ -1379,7 +1361,7 @@ impl<T: Config> Pallet<T> {

    pub fn domain_runtime_code(domain_id: DomainId) -> Option<Vec<u8>> {
        RuntimeRegistry::<T>::get(Self::runtime_id(domain_id)?)
-            .map(|runtime_object| runtime_object.code)
+            .and_then(|mut runtime_object| runtime_object.raw_genesis.take_runtime_code())


This doe not consider runtime upgrades and instead return the first runtime

We don't maintain historical domain runtime code, they are updated in place and the returned value is determined by the consensus_hash used to call the runtime API.

No, that should not be case. We should never touch the raw chainspec and treat it just like that for domain raw spec as well. Updating in place means, raw chainspec is always updating whenever there is a runtime upgrade.

This is for the purposes of future instantiation such that we don't have to go through potentially many runtime upgrades and keep the whole upgrade path support in the runtime forever (though maybe we do need that for other reasons).

Then we will essential break the exisiting domains for the operators who want to sync from beginning, IIUC. If we do want to support runtime upgrades such that new domains do not need to go through the upgrade path. Then each domain should ideally carry their current rutime hash and fetch the right runtime based on that hash. This way, exisiting operator domains can sync from the beginning, while new domains and their operators can start from the latest upgrade as their genesis runtime.

Update: Okay I see where my confusion is coming from. Fetching this from chain_spec and the fact that we are updating this in place leads to this confusion. We already update the chainspec with domain_id, why not have a Runtime storage against runtime hash and then return that instead ?

Sorry, I don't get your point. raw_genesis is not fetched from the chain spec but built from chain spec, different runtime code can lead to different raw_genesis even if the rest of the chain spec is the same.

We already update the chainspec with domain_id, why not have a Runtime storage against runtime hash and then return that instead ?

Do you mean storing the runtime code separately? the runtime code and raw_genesis are come together and updated together, domain_runtime_code is the only place that runtime code is used alone so I don't see much benefit to doing so, also see #1824 (comment)

crates/pallet-domains/src/runtime_registry.rs

vedhavyas · 2023-10-03T08:52:14Z

crates/pallet-domains/src/runtime_registry.rs

@@ -134,19 +94,26 @@ pub(crate) fn register_runtime_at_genesis<T: Config>(
 /// Schedules a runtime upgrade after `DomainRuntimeUpgradeDelay` from current block number.
 pub(crate) fn do_schedule_runtime_upgrade<T: Config>(
    runtime_id: RuntimeId,
-    code: Vec<u8>,
+    raw_genesis_storage: Vec<u8>,


Still not sure why we need to pass the entire geneis storage here. We just new runtime and the rest should stay as is. What am I missing here ?

crates/sp-domains/src/lib.rs

vedhavyas · 2023-10-03T08:59:14Z

crates/subspace-node/src/domain/evm_chain_spec.rs

        balances: BalancesConfig {
+            // TODO: remove `endowed_accounts` once XDM is ready


I dont understand this TODO. Why do we want to remove endowed accounts ?

Because the genesis storage is registered with the runtime code and is reused by every domain instance, so it basically means we mint SSC whenever a domain is instantiated.

sure, for the production runtime these will be empty anyway. For the tests, I would prefer having these endowed accounts rather than using the XDM to moving funds from the Consensus. This is a bad ux when testing or developing

Okay, will update the comment

vedhavyas

Okay. Lets move on.
Thank you for being patient

NingLin-P requested review from vedhavyas, nazar-pc and rg3l3dr as code owners August 15, 2023 21:05

NingLin-P force-pushed the genesis-storage branch from a6591cf to 9797d2a Compare August 16, 2023 15:33

vedhavyas reviewed Aug 17, 2023

View reviewed changes

NingLin-P mentioned this pull request Aug 17, 2023

The way of generating the genesis state root for the domain instance is not deterministic #1798

Closed

NingLin-P marked this pull request as draft August 17, 2023 20:09

NingLin-P force-pushed the genesis-storage branch from 9797d2a to 7cb396a Compare September 4, 2023 19:17

NingLin-P marked this pull request as ready for review September 4, 2023 19:25

NingLin-P requested a review from vedhavyas September 5, 2023 19:49

vedhavyas reviewed Sep 8, 2023

View reviewed changes

nazar-pc reviewed Sep 8, 2023

View reviewed changes

NingLin-P added 9 commits September 14, 2023 07:35

Refactor the pallet-domain-id and export the storage key of SelfDomainId

a1cc353

This storage key will be used in upcoming commit to reset the SelfDomainId storage item to the correct value in the domain genesis storage Signed-off-by: linning <linningde25@gmail.com>

Introduce RawGenesis which is the genesis storage with codec and Type…

02163c7

…Info supported RawGenesis is mostly port from upstream with slight changes to support our usecase Signed-off-by: linning <linningde25@gmail.com>

Remove the domain related host function

3154370

Signed-off-by: linning <linningde25@gmail.com>

Remove the raw_genesis_config from domain registry

a524100

Since we use runtime_object.raw_genesis to derive the genesis state for domain instance and which already contains the sudo account, raw_genesis_config is needless now Signed-off-by: linning <linningde25@gmail.com>

Replace the domain subcommand build-genesis-config with build-genesis…

fb9a367

…-storage Since now we don't need the raw_genesis_config when instantiate domain and we need the raw_genesis_storage when register runtime Signed-off-by: linning <linningde25@gmail.com>

NingLin-P force-pushed the genesis-storage branch from 7cb396a to eb8036d Compare September 14, 2023 01:34

vedhavyas self-requested a review September 14, 2023 14:45

vedhavyas reviewed Sep 14, 2023

View reviewed changes

nazar-pc reviewed Sep 14, 2023

View reviewed changes

NingLin-P mentioned this pull request Sep 14, 2023

Invalid total rewards fraud proof #1957

Merged

1 task

NingLin-P added 5 commits September 21, 2023 07:51

Update both the runtime code and the genesis state in the upgrade_dom…

d8fe8bf

…ain_runtime call Signed-off-by: linning <linningde25@gmail.com>

Apply review suggestions

3cc6914

Signed-off-by: linning <linningde25@gmail.com>

Merge branch 'main' of github.com:subspace/subspace into genesis-storage

2768da8

Use String for runtime_name/domain_name

245f219

Signed-off-by: linning <linningde25@gmail.com>

Merge branch 'main' of github.com:subspace/subspace into genesis-storage

59f9e64

nazar-pc requested review from vedhavyas and nazar-pc October 2, 2023 09:57

vedhavyas reviewed Oct 3, 2023

View reviewed changes

vedhavyas approved these changes Oct 3, 2023

View reviewed changes

NingLin-P added this pull request to the merge queue Oct 3, 2023

Merged via the queue into main with commit 94e2fb0 Oct 3, 2023

NingLin-P deleted the genesis-storage branch October 3, 2023 19:18

		/// NOTE: this function is port from `sp_state_machine::in_memory_backend::new_in_mem` which is
		/// only export for `std` but we need to use it in `no_std`

		balances: BalancesConfig {
		// TODO: remove `endowed_accounts` once XDM is ready

Store the domain genesis storage on the consensus chain #1824

Store the domain genesis storage on the consensus chain #1824

Conversation

NingLin-P commented Aug 15, 2023 • edited Loading

Code contributor checklist:

vedhavyas left a comment

Choose a reason for hiding this comment

NingLin-P commented Sep 6, 2023

nazar-pc commented Sep 8, 2023

vedhavyas left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

NingLin-P commented Sep 8, 2023

vedhavyas commented Sep 8, 2023 • edited Loading

nazar-pc left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

NingLin-P commented Sep 14, 2023

vedhavyas left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nazar-pc left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

NingLin-P Sep 14, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

NingLin-P Sep 14, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

NingLin-P Sep 15, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

NingLin-P commented Oct 2, 2023

vedhavyas left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nazar-pc Oct 3, 2023 • edited Loading

Choose a reason for hiding this comment

vedhavyas Oct 3, 2023 • edited Loading

Choose a reason for hiding this comment

NingLin-P Oct 3, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vedhavyas left a comment

Choose a reason for hiding this comment

NingLin-P commented Aug 15, 2023 •

edited

Loading

vedhavyas commented Sep 8, 2023 •

edited

Loading

NingLin-P Sep 14, 2023 •

edited

Loading

NingLin-P Sep 14, 2023 •

edited

Loading

NingLin-P Sep 15, 2023 •

edited

Loading

nazar-pc Oct 3, 2023 •

edited

Loading

vedhavyas Oct 3, 2023 •

edited

Loading

NingLin-P Oct 3, 2023 •

edited

Loading