check_dependent_project: Patches are not respected for all crates in Cumulus

[joao-paulo-parity]

https://matrix.to/#/!gJeGMHCcDoIwsHIJri:matrix.parity.io/$zfa4QuUiNc4ugd5RZ-DIJD5jg0elvWVSdCaKUkjREH0?via=matrix.parity.io

So I've merged my substrate PR using the bot: paritytech/substrate#11232
But the bot failed to automatically merge the Polkadot companion: paritytech/polkadot#5337
And the cumulus job in my Polkadot PR also seems to not be using the changes from my PR: https://gitlab.parity.io/parity/mirrors/polkadot/-/jobs/1577990
So now I can't merge it manually.

So is the cumulus job in Polkadot also broken?

[joao-paulo-parity]

And the cumulus job in my Polkadot PR also seems to not be using the changes from my PR: https://gitlab.parity.io/parity/mirrors/polkadot/-/jobs/1577990

According to https://gitlab.parity.io/parity/mirrors/polkadot/-/jobs/1577990#L519 the Polkadot PR should've been patched into the cloned Cumulus repository, but as shown in https://gitlab.parity.io/parity/mirrors/polkadot/-/jobs/1577990#L2216 it appears that not all references were changed with diener.

I'll try to reproduce the patching procedure locally and see where the problem lies.

[joao-paulo-parity]

Reproduction steps

git clone https://github.com/paritytech/polkadot
cd polkadot
git fetch up pull/5337/head:5337
git checkout 5337

cd ..
git clone https://github.com/paritytech/cumulus
cd cumulus
git reset --hard b1a5d70256f7344a23bedaa72924b6559b187845
diener patch --target "https://github.com/paritytech/polkadot" --crates-to-patch .. --path Cargo.toml)
git diff

Outdated findings (under revision)

This is the output of git diff

diff --git a/Cargo.toml b/Cargo.toml
index f9a0bcc8..54e04354 100644
--- a/Cargo.toml
+++ b/Cargo.toml
@@ -49,3 +49,102 @@ opt-level = 3
 inherits = "release"
 lto = true
 codegen-units = 1
+[patch."https://github.com/paritytech/polkadot"]
+polkadot-cli ={path = "/home/user/w/polkadot/cli" }
+polkadot-client ={path = "/home/user/w/polkadot/node/client" }
+kusama-runtime ={path = "/home/user/w/polkadot/runtime/kusama" }
+kusama-runtime-constants ={path = "/home/user/w/polkadot/runtime/kusama/constants" }
+polkadot-primitives ={path = "/home/user/w/polkadot/primitives" }
+polkadot-core-primitives ={path = "/home/user/w/polkadot/core-primitives" }
+polkadot-parachain ={path = "/home/user/w/polkadot/parachain" }
+polkadot-runtime-common ={path = "/home/user/w/polkadot/runtime/common" }
+polkadot-runtime-parachains ={path = "/home/user/w/polkadot/runtime/parachains" }
+polkadot-runtime-metrics ={path = "/home/user/w/polkadot/runtime/metrics" }
+xcm ={path = "/home/user/w/polkadot/xcm" }
+xcm-procedural ={path = "/home/user/w/polkadot/xcm/procedural" }
+xcm-executor ={path = "/home/user/w/polkadot/xcm/xcm-executor" }
+polkadot-primitives-test-helpers ={path = "/home/user/w/polkadot/primitives/test-helpers" }
+slot-range-helper ={path = "/home/user/w/polkadot/runtime/common/slot_range_helper" }
+pallet-xcm ={path = "/home/user/w/polkadot/xcm/pallet-xcm" }
+xcm-builder ={path = "/home/user/w/polkadot/xcm/xcm-builder" }
+pallet-xcm-benchmarks ={path = "/home/user/w/polkadot/xcm/pallet-xcm-benchmarks" }
+polkadot-node-core-parachains-inherent ={path = "/home/user/w/polkadot/node/core/parachains-inherent" }
+tracing-gum ={path = "/home/user/w/polkadot/node/gum" }
+tracing-gum-proc-macro ={path = "/home/user/w/polkadot/node/gum/proc-macro" }
+polkadot-node-jaeger ={path = "/home/user/w/polkadot/node/jaeger" }
+polkadot-node-primitives ={path = "/home/user/w/polkadot/node/primitives" }
+polkadot-erasure-coding ={path = "/home/user/w/polkadot/erasure-coding" }
+polkadot-node-subsystem ={path = "/home/user/w/polkadot/node/subsystem" }
+polkadot-node-subsystem-types ={path = "/home/user/w/polkadot/node/subsystem-types" }
+polkadot-node-network-protocol ={path = "/home/user/w/polkadot/node/network/protocol" }
+polkadot-overseer-gen ={path = "/home/user/w/polkadot/node/overseer/overseer-gen" }
+metered-channel ={path = "/home/user/w/polkadot/node/metered-channel" }
+polkadot-overseer-gen-proc-macro ={path = "/home/user/w/polkadot/node/overseer/overseer-gen/proc-macro" }
+polkadot-statement-table ={path = "/home/user/w/polkadot/statement-table" }
+polkadot-overseer ={path = "/home/user/w/polkadot/node/overseer" }
+polkadot-node-metrics ={path = "/home/user/w/polkadot/node/metrics" }
+polkadot-test-service ={path = "/home/user/w/polkadot/node/test/service" }
+polkadot-rpc ={path = "/home/user/w/polkadot/rpc" }
+polkadot-service ={path = "/home/user/w/polkadot/node/service" }
+polkadot-approval-distribution ={path = "/home/user/w/polkadot/node/network/approval-distribution" }
+polkadot-node-subsystem-util ={path = "/home/user/w/polkadot/node/subsystem-util" }
+polkadot-node-subsystem-test-helpers ={path = "/home/user/w/polkadot/node/subsystem-test-helpers" }
+polkadot-availability-bitfield-distribution ={path = "/home/user/w/polkadot/node/network/bitfield-distribution" }
+polkadot-availability-distribution ={path = "/home/user/w/polkadot/node/network/availability-distribution" }
+polkadot-availability-recovery ={path = "/home/user/w/polkadot/node/network/availability-recovery" }
+polkadot-collator-protocol ={path = "/home/user/w/polkadot/node/network/collator-protocol" }
+polkadot-dispute-distribution ={path = "/home/user/w/polkadot/node/network/dispute-distribution" }
+polkadot-gossip-support ={path = "/home/user/w/polkadot/node/network/gossip-support" }
+polkadot-network-bridge ={path = "/home/user/w/polkadot/node/network/bridge" }
+polkadot-node-collation-generation ={path = "/home/user/w/polkadot/node/collation-generation" }
+polkadot-node-core-approval-voting ={path = "/home/user/w/polkadot/node/core/approval-voting" }
+polkadot-node-core-av-store ={path = "/home/user/w/polkadot/node/core/av-store" }
+polkadot-node-core-backing ={path = "/home/user/w/polkadot/node/core/backing" }
+polkadot-node-core-bitfield-signing ={path = "/home/user/w/polkadot/node/core/bitfield-signing" }
+polkadot-node-core-candidate-validation ={path = "/home/user/w/polkadot/node/core/candidate-validation" }
+polkadot-node-core-pvf ={path = "/home/user/w/polkadot/node/core/pvf" }
+test-parachain-adder ={path = "/home/user/w/polkadot/parachain/test-parachains/adder" }
+test-parachain-halt ={path = "/home/user/w/polkadot/parachain/test-parachains/halt" }
+polkadot-node-core-chain-api ={path = "/home/user/w/polkadot/node/core/chain-api" }
+polkadot-node-core-chain-selection ={path = "/home/user/w/polkadot/node/core/chain-selection" }
+polkadot-node-core-dispute-coordinator ={path = "/home/user/w/polkadot/node/core/dispute-coordinator" }
+polkadot-node-core-provisioner ={path = "/home/user/w/polkadot/node/core/provisioner" }
+polkadot-node-core-pvf-checker ={path = "/home/user/w/polkadot/node/core/pvf-checker" }
+polkadot-node-core-runtime-api ={path = "/home/user/w/polkadot/node/core/runtime-api" }
+polkadot-runtime ={path = "/home/user/w/polkadot/runtime/polkadot" }
+polkadot-runtime-constants ={path = "/home/user/w/polkadot/runtime/polkadot/constants" }
+polkadot-statement-distribution ={path = "/home/user/w/polkadot/node/network/statement-distribution" }
+rococo-runtime ={path = "/home/user/w/polkadot/runtime/rococo" }
+bp-messages ={path = "/home/user/w/polkadot/bridges/primitives/messages" }
+bp-runtime ={path = "/home/user/w/polkadot/bridges/primitives/runtime" }
+bp-rococo ={path = "/home/user/w/polkadot/bridges/primitives/chain-rococo" }
+bp-polkadot-core ={path = "/home/user/w/polkadot/bridges/primitives/polkadot-core" }
+bp-wococo ={path = "/home/user/w/polkadot/bridges/primitives/chain-wococo" }
+bridge-runtime-common ={path = "/home/user/w/polkadot/bridges/bin/runtime-common" }
+bp-message-dispatch ={path = "/home/user/w/polkadot/bridges/primitives/message-dispatch" }
+pallet-bridge-dispatch ={path = "/home/user/w/polkadot/bridges/modules/dispatch" }
+pallet-bridge-grandpa ={path = "/home/user/w/polkadot/bridges/modules/grandpa" }
+bp-header-chain ={path = "/home/user/w/polkadot/bridges/primitives/header-chain" }
+bp-test-utils ={path = "/home/user/w/polkadot/bridges/primitives/test-utils" }
+pallet-bridge-messages ={path = "/home/user/w/polkadot/bridges/modules/messages" }
+rococo-runtime-constants ={path = "/home/user/w/polkadot/runtime/rococo/constants" }
+westend-runtime ={path = "/home/user/w/polkadot/runtime/westend" }
+westend-runtime-constants ={path = "/home/user/w/polkadot/runtime/westend/constants" }
+polkadot-test-client ={path = "/home/user/w/polkadot/node/test/client" }
+polkadot-test-runtime ={path = "/home/user/w/polkadot/runtime/test-runtime" }
+test-runtime-constants ={path = "/home/user/w/polkadot/runtime/test-runtime/constants" }
+polkadot-performance-test ={path = "/home/user/w/polkadot/node/test/performance-test" }
+xcm-executor-integration-tests ={path = "/home/user/w/polkadot/xcm/xcm-executor/integration-tests" }
+xcm-simulator ={path = "/home/user/w/polkadot/xcm/xcm-simulator" }
+xcm-simulator-example ={path = "/home/user/w/polkadot/xcm/xcm-simulator/example" }
+xcm-simulator-fuzzer ={path = "/home/user/w/polkadot/xcm/xcm-simulator/fuzzer" }
+polkadot-test-malus ={path = "/home/user/w/polkadot/node/malus" }
+zombienet-backchannel ={path = "/home/user/w/polkadot/node/zombienet-backchannel" }
+test-parachains ={path = "/home/user/w/polkadot/parachain/test-parachains" }
+test-parachain-adder-collator ={path = "/home/user/w/polkadot/parachain/test-parachains/adder/collator" }
+test-parachain-undying ={path = "/home/user/w/polkadot/parachain/test-parachains/undying" }
+test-parachain-undying-collator ={path = "/home/user/w/polkadot/parachain/test-parachains/undying/collator" }
+staking-miner ={path = "/home/user/w/polkadot/utils/staking-miner" }
+remote-ext-tests-bags-list ={path = "/home/user/w/polkadot/utils/remote-ext-tests/bags-list" }
+polkadot-voter-bags ={path = "/home/user/w/polkadot/utils/generate-bags" }
+polkadot ={path = "/home/user/w/polkadot" }

In the diff above there's no mention of polkadot-node-core-pvf (from https://gitlab.parity.io/parity/mirrors/polkadot/-/jobs/1577990#L2216) being patched to /home/user/w/polkadot.

The current lead is on the patching procedure being incomplete or diener being at fault.

[joao-paulo-parity]

Might be related to paritytech/diener#15

[joao-paulo-parity]

paritytech/diener#16 is a potential solution

[joao-paulo-parity]

paritytech/diener#16 overcame the problem mentioned in #45 (comment) of not all packages being properly detected and patched.

Now a new problem appeared in https://gitlab.parity.io/parity/mirrors/polkadot/-/jobs/1579158#L2287

= note: perhaps two different versions of crate polkadot_parachain are being used?

https://gitlab.parity.io/parity/mirrors/polkadot/-/jobs/1579158#L2211 shows

Checking polkadot-parachain v0.9.22 (/builds/parity/mirrors/polkadot/parachain)

While https://gitlab.parity.io/parity/mirrors/polkadot/-/jobs/1579158#L2213 shows

Checking polkadot-parachain v0.9.22 (https://github.com/paritytech/polkadot?branch=master#64f09487)

Indeed two versions of polkadot-parachain are being (mis)used. I think this is related to a bug paritytech/diener#16 which should be fixed by paritytech/diener@d1fe2f6.

A new job is scheduled to run in https://gitlab.parity.io/parity/mirrors/polkadot/-/jobs/1579376 to verify that this problem doesn't come up again.

[joao-paulo-parity]

Now the aforementioned problems (#45 (comment) and #45 (comment)) seem to be gone according to https://gitlab.parity.io/parity/mirrors/polkadot/-/jobs/1579787#L2032.

#46.

[joao-paulo-parity]

paritytech/substrate#11506 can be merged after this is resolved

[mordamax]

merged

[joao-paulo-parity]

It should not be closed because the bug still exists. See #46 and paritytech/diener#15.

We can close this either

1. After we get confirmation that diener doesn't find all crates in a repository diener#15 no longer affects the projects
2. After Fix patching #46 is merged

[joao-paulo-parity]

closing as per paritytech/diener#15 (comment)

[joao-paulo-parity]

Leaving this opened until paritytech/cumulus#1344 is merged