build(yarn): migrate from v1.2.17 to v3.6.0 #1142
Assignees
Labels
dependencies
Pull requests that update a dependency file
Developer_Experience
good-first-issue
Good for newcomers
good-first-issue-400-expert
Hacktoberfest
Hacktoberfest participants are welcome to take a stab at issues marked with this label.
help wanted
Extra attention is needed
Comments
petermetz
added
good-first-issue
|
Issues have been reported with older versions of yarn so this just became a little more urgent than before. |
|
Renamed the issue to match the fact that there is now Yarn v3 released instead of v2 being the latest and greatest. |
petermetz
added a commit
to petermetz/cacti
that referenced
this issue
Nov 16, 2021
We hope to solve issues with this that were plaguing contributors due to bugs in the much older Yarn 1.x versions that we had to use because Yarn V2 had it's own set of problems when it came to linking the .bin folders for child packages in a monorepo. Fixes hyperledger-cacti#1142 Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
petermetz
added a commit
to petermetz/cacti
that referenced
this issue
Nov 16, 2021
We hope to solve issues with this that were plaguing contributors due to bugs in the much older Yarn 1.x versions that we had to use because Yarn V2 had it's own set of problems when it came to linking the .bin folders for child packages in a monorepo. Fixes hyperledger-cacti#1142 Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
petermetz
added a commit
to petermetz/cacti
that referenced
this issue
Nov 16, 2021
We hope to solve issues with this that were plaguing contributors due to bugs in the much older Yarn 1.x versions that we had to use because Yarn V2 had it's own set of problems when it came to linking the .bin folders for child packages in a monorepo. Fixes hyperledger-cacti#1142 Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
petermetz
added a commit
to petermetz/cacti
that referenced
this issue
Nov 16, 2021
We hope to solve issues with this that were plaguing contributors due to bugs in the much older Yarn 1.x versions that we had to use because Yarn V2 had it's own set of problems when it came to linking the .bin folders for child packages in a monorepo. Fixes hyperledger-cacti#1142 Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
petermetz
added a commit
to petermetz/cacti
that referenced
this issue
Nov 16, 2021
We hope to solve issues with this that were plaguing contributors due to bugs in the much older Yarn 1.x versions that we had to use because Yarn V2 had it's own set of problems when it came to linking the .bin folders for child packages in a monorepo. Fixes hyperledger-cacti#1142 Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
petermetz
added a commit
to petermetz/cacti
that referenced
this issue
Dec 8, 2021
We hope to solve issues with this that were plaguing contributors due to bugs in the much older Yarn 1.x versions that we had to use because Yarn V2 had it's own set of problems when it came to linking the .bin folders for child packages in a monorepo. Fixes hyperledger-cacti#1142 Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
|
The problem I'm having right now is that although the build works, the dependency linking phase takes about an hour instead of "just" a few minutes (which was already too long IMO...) Look at the last line of the logs for the timing data: $ yarn
➤ YN0000: ┌ Project validation
➤ YN0057: │ @hyperledger/cactus-cmd-socket-server: 'nohoist' is deprecated, please use 'installConfig.hoistingLimits' instead
➤ YN0000: └ Completed
➤ YN0000: ┌ Resolution step
➤ YN0002: │ @angular-builders/custom-webpack@npm:12.1.0 doesn't provide @angular/compiler-cli (pe9f85), requested by @angular-devkit/build-angular
➤ YN0002: │ @angular-builders/custom-webpack@npm:12.1.0 doesn't provide @types/node (p25b75), requested by ts-node
➤ YN0002: │ @angular-builders/custom-webpack@npm:12.1.0 doesn't provide typescript (p19cdd), requested by @angular-devkit/build-angular
➤ YN0002: │ @angular-builders/custom-webpack@npm:12.1.0 doesn't provide typescript (p50e10), requested by ts-node
➤ YN0002: │ @endemolshinegroup/cosmiconfig-typescript-loader@npm:3.0.2 [8cf92] doesn't provide typescript (p5535d), requested by ts-node
➤ YN0002: │ @hyperledger/cactus-example-carbon-accounting-frontend@workspace:examples/cactus-example-carbon-accounting-frontend doesn't provide @angular-devkit/architect (pcaed6), requested by @ionic/angular-toolkit
➤ YN0002: │ @hyperledger/cactus-example-carbon-accounting-frontend@workspace:examples/cactus-example-carbon-accounting-frontend doesn't provide @angular-devkit/schematics (p8d85a), requested by @ionic/angular-toolkit
➤ YN0060: │ @hyperledger/cactus-example-carbon-accounting-frontend@workspace:examples/cactus-example-carbon-accounting-frontend provides rxjs (pc148d) with version 7.3.0, which doesn't satisfy what @ionic-native/core requests
➤ YN0060: │ @hyperledger/cactus-example-carbon-accounting-frontend@workspace:examples/cactus-example-carbon-accounting-frontend provides rxjs (pfe0e5) with version 7.3.0, which doesn't satisfy what @ionic-native/splash-screen requests
➤ YN0060: │ @hyperledger/cactus-example-carbon-accounting-frontend@workspace:examples/cactus-example-carbon-accounting-frontend provides rxjs (p0ff63) with version 7.3.0, which doesn't satisfy what @ionic-native/status-bar requests
➤ YN0002: │ @hyperledger/cactus-example-supply-chain-frontend@workspace:examples/cactus-example-supply-chain-frontend doesn't provide @angular-devkit/architect (pa007a), requested by @ionic/angular-toolkit
➤ YN0002: │ @hyperledger/cactus-example-supply-chain-frontend@workspace:examples/cactus-example-supply-chain-frontend doesn't provide @angular-devkit/schematics (p12956), requested by @ionic/angular-toolkit
➤ YN0060: │ @hyperledger/cactus-example-supply-chain-frontend@workspace:examples/cactus-example-supply-chain-frontend provides rxjs (p8ee71) with version 7.3.0, which doesn't satisfy what @ionic-native/core requests
➤ YN0060: │ @hyperledger/cactus-example-supply-chain-frontend@workspace:examples/cactus-example-supply-chain-frontend provides rxjs (p29437) with version 7.3.0, which doesn't satisfy what @ionic-native/splash-screen requests
➤ YN0060: │ @hyperledger/cactus-example-supply-chain-frontend@workspace:examples/cactus-example-supply-chain-frontend provides rxjs (p64485) with version 7.3.0, which doesn't satisfy what @ionic-native/status-bar requests
➤ YN0002: │ @hyperledger/cactus-test-cmd-api-server@workspace:packages/cactus-test-cmd-api-server doesn't provide @types/express (p681aa), requested by express-jwt-authz
➤ YN0002: │ @hyperledger/cactus-test-cmd-api-server@workspace:packages/cactus-test-cmd-api-server doesn't provide express (paf000), requested by express-jwt-authz
➤ YN0002: │ @ionic/angular-toolkit@npm:4.0.0 [00bbd] doesn't provide webpack (p7a42e), requested by copy-webpack-plugin
➤ YN0002: │ dns-over-http-resolver@npm:1.2.3 doesn't provide node-fetch (p89343), requested by native-fetch
➤ YN0002: │ ws-identity-client@npm:1.0.2 doesn't provide request (p3ccc8), requested by request-promise-native
➤ YN0000: │ Some peer dependencies are incorrectly met; run yarn explain peer-requirements <hash> for details, where <hash> is the six-letter p-prefixed code
➤ YN0000: └ Completed in 1s 682ms
➤ YN0000: ┌ Fetch step
➤ YN0019: │ npm-packlist-npm-1.4.8-ddca195225-85f764bd0f.zip appears to be unused - removing
➤ YN0019: │ protoc-gen-ts-npm-0.4.0-1333fe4b6f-3a69eaae62.zip appears to be unused - removing
➤ YN0019: │ sjcl-npm-1.0.7-2fbb0e93e4-4eec2df27c.zip appears to be unused - removing
➤ YN0019: │ tar-stream-npm-2.1.4-5f7040c9c5-ba02688aa2.zip appears to be unused - removing
➤ YN0019: │ util-npm-0.10.4-7c577db41a-913f9a90d0.zip appears to be unused - removing
➤ YN0000: └ Completed in 3s 952ms
➤ YN0000: ┌ Link step
➤ YN0062: │ fsevents@patch:fsevents@npm%3A2.3.2#~builtin<compat/fsevents>::version=2.3.2&hash=1cc4b2 The platform linux is incompatible with this module, link skipped.
➤ YN0062: │ fsevents@patch:fsevents@npm%3A2.1.3#~builtin<compat/fsevents>::version=2.1.3&hash=1cc4b2 The platform linux is incompatible with this module, link skipped.
➤ YN0062: │ esbuild-android-arm64@npm:0.13.8 The platform linux is incompatible with this module, link skipped.
➤ YN0062: │ esbuild-darwin-64@npm:0.13.8 The platform linux is incompatible with this module, link skipped.
➤ YN0062: │ esbuild-darwin-arm64@npm:0.13.8 The platform linux is incompatible with this module, link skipped.
➤ YN0062: │ esbuild-freebsd-64@npm:0.13.8 The platform linux is incompatible with this module, link skipped.
➤ YN0062: │ esbuild-freebsd-arm64@npm:0.13.8 The platform linux is incompatible with this module, link skipped.
➤ YN0063: │ esbuild-linux-32@npm:0.13.8 The CPU architecture x64 is incompatible with this module, link skipped.
➤ YN0063: │ esbuild-linux-arm64@npm:0.13.8 The CPU architecture x64 is incompatible with this module, link skipped.
➤ YN0063: │ esbuild-linux-arm@npm:0.13.8 The CPU architecture x64 is incompatible with this module, link skipped.
➤ YN0063: │ esbuild-linux-mips64le@npm:0.13.8 The CPU architecture x64 is incompatible with this module, link skipped.
➤ YN0063: │ esbuild-linux-ppc64le@npm:0.13.8 The CPU architecture x64 is incompatible with this module, link skipped.
➤ YN0062: │ esbuild-netbsd-64@npm:0.13.8 The platform linux is incompatible with this module, link skipped.
➤ YN0062: │ esbuild-openbsd-64@npm:0.13.8 The platform linux is incompatible with this module, link skipped.
➤ YN0062: │ esbuild-sunos-64@npm:0.13.8 The platform linux is incompatible with this module, link skipped.
➤ YN0062: │ esbuild-windows-32@npm:0.13.8 The platform linux is incompatible with this module, link skipped.
➤ YN0062: │ esbuild-windows-64@npm:0.13.8 The platform linux is incompatible with this module, link skipped.
➤ YN0062: │ esbuild-windows-arm64@npm:0.13.8 The platform linux is incompatible with this module, link skipped.
➤ YN0062: │ fsevents@patch:fsevents@npm%3A1.2.13#~builtin<compat/fsevents>::version=1.2.13&hash=1cc4b2 The platform linux is incompatible with this module, link skipped.
➤ YN0007: │ electron@npm:13.3.0 must be built because it never has been before or the last one failed
➤ YN0007: │ grpc-tools@npm:1.11.2 must be built because it never has been before or the last one failed
➤ YN0007: │ secp256k1@npm:4.0.2 must be built because it never has been before or the last one failed
➤ YN0007: │ protobufjs@npm:6.11.2 must be built because it never has been before or the last one failed
➤ YN0007: │ web3-bzz@npm:1.5.2 must be built because it never has been before or the last one failed
➤ YN0007: │ web3-shh@npm:1.5.2 must be built because it never has been before or the last one failed
➤ YN0007: │ pkcs11js@npm:1.2.6 must be built because it never has been before or the last one failed
➤ YN0007: │ cpu-features@npm:0.0.2 must be built because it never has been before or the last one failed
➤ YN0007: │ bufferutil@npm:4.0.5 must be built because it never has been before or the last one failed
➤ YN0007: │ utf-8-validate@npm:5.0.7 must be built because it never has been before or the last one failed
➤ YN0007: │ keccak@npm:3.0.2 must be built because it never has been before or the last one failed
➤ YN0007: │ @nestjs/core@npm:8.2.0 [4e575] must be built because it never has been before or the last one failed
➤ YN0007: │ core-js-pure@npm:3.19.1 must be built because it never has been before or the last one failed
➤ YN0007: │ @angular/cli@npm:12.2.0 must be built because it never has been before or the last one failed
➤ YN0007: │ keccak@npm:1.4.0 must be built because it never has been before or the last one failed
➤ YN0007: │ secp256k1@npm:3.8.0 must be built because it never has been before or the last one failed
➤ YN0007: │ web3-bzz@npm:1.3.5 must be built because it never has been before or the last one failed
➤ YN0007: │ web3-shh@npm:1.3.5 must be built because it never has been before or the last one failed
➤ YN0007: │ core-js@npm:2.6.12 must be built because it never has been before or the last one failed
➤ YN0007: │ esbuild@npm:0.12.17 must be built because it never has been before or the last one failed
➤ YN0007: │ core-js@npm:3.16.0 must be built because it never has been before or the last one failed
➤ YN0007: │ esbuild@npm:0.13.8 must be built because it never has been before or the last one failed
➤ YN0007: │ nice-napi@npm:1.0.2 must be built because it never has been before or the last one failed
➤ YN0007: │ iso-constants@npm:0.1.2 must be built because it never has been before or the last one failed
➤ YN0007: │ web3-bzz@npm:1.6.1 must be built because it never has been before or the last one failed
➤ YN0007: │ web3-shh@npm:1.6.1 must be built because it never has been before or the last one failed
➤ YN0007: │ aws-sdk@npm:2.965.0 must be built because it never has been before or the last one failed
➤ YN0007: │ keytar@npm:7.7.0 must be built because it never has been before or the last one failed
➤ YN0007: │ deasync@npm:0.1.24 must be built because it never has been before or the last one failed
➤ YN0007: │ grpc@npm:1.24.11 must be built because it never has been before or the last one failed
➤ YN0007: │ cbor@npm:6.0.1 must be built because it never has been before or the last one failed
➤ YN0007: │ web3@npm:1.2.7 must be built because it never has been before or the last one failed
➤ YN0007: │ @web3-js/websocket@npm:1.0.30 must be built because it never has been before or the last one failed
➤ YN0007: │ @web3-js/scrypt-shim@npm:0.1.0 must be built because it never has been before or the last one failed
➤ YN0007: │ nodemon@npm:2.0.15 must be built because it never has been before or the last one failed
➤ YN0009: │ cpu-features@npm:0.0.2 couldn't be built successfully (exit code 1, logs can be found here: /tmp/xfs-3c1a4bc9/build.log)
➤ YN0009: │ cpu-features@npm:0.0.2 couldn't be built successfully (exit code 1, logs can be found here: /tmp/xfs-bc9e3266/build.log)
➤ YN0009: │ cpu-features@npm:0.0.2 couldn't be built successfully (exit code 1, logs can be found here: /tmp/xfs-faa463e9/build.log)
➤ YN0009: │ cpu-features@npm:0.0.2 couldn't be built successfully (exit code 1, logs can be found here: /tmp/xfs-b2610671/build.log)
➤ YN0007: │ @openapitools/openapi-generator-cli@npm:2.4.14 must be built because it never has been before or the last one failed
➤ YN0007: │ web3@npm:1.5.2 must be built because it never has been before or the last one failed
➤ YN0007: │ ssh2@npm:1.5.0 must be built because it never has been before or the last one failed
➤ YN0007: │ web3@npm:1.3.5 must be built because it never has been before or the last one failed
➤ YN0007: │ web3@npm:1.6.1 must be built because it never has been before or the last one failed
➤ YN0000: └ Completed in 48m 3s
➤ YN0000: Done with warnings in 48m 9s |
|
There are some problems with the CI as well it seems: logs_11870.zip |
|
@petermetz please assign this to me. thanks! |
@zondervancalvez Assigning now, thank you! |
@zondervancalvez Please make sure to check the related work on this (I have a branch that has partial resolution for this) |
petermetz
added a commit
to petermetz/cacti
that referenced
this issue
Dec 2, 2022
We hope to solve issues with this that were plaguing contributors due to bugs in the much older Yarn 1.x versions that we had to use because Yarn V2 had it's own set of problems when it came to linking the .bin folders for child packages in a monorepo. Fixes hyperledger-cacti#1142 Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
|
This PR/issue depends on:
|
zondervancalvez
added a commit
to zondervancalvez/cactus
that referenced
this issue
Jul 7, 2023
Fixes hyperledger-cacti#1142 Signed-off-by: zondervancalvez <zondervan.v.calvez@accenture.com>
zondervancalvez
added a commit
to zondervancalvez/cactus
that referenced
this issue
Jul 7, 2023
Fixes hyperledger-cacti#1142 Signed-off-by: zondervancalvez <zondervan.v.calvez@accenture.com>
petermetz
added a commit
to petermetz/cacti
that referenced
this issue
Jul 8, 2023
We hope to solve issues with this that were plaguing contributors due to bugs in the much older Yarn 1.x versions that we had to use because Yarn V3 had it's own set of problems when it came to linking the .bin folders for child packages in a monorepo. 1. We've moved from hoisting by default to NOT hoisting by default for all packages which results in slightly slower installs for the dependencies when setting up the project build but it does have the beneficial effect of much easier fixes for compiler errors because each package can be provided with their own version of whatever dependencies they need. 2. The "pre" and "post" npm scripts are no longer supported by Yarn which was an annoying breaking change so we had to restructure some of the scripts which had pre and post scripts of their own to get equal behavior. 3. Upgraded to Yarn v3.6.0 4. packages/cactus-api-client/src/main/typescript/socketio-api-client.ts has a line where the Typescript compiler has been disabled and an issue opened for the same: https://github.com/hyperledger/cacti/issues/2523 5. Added several missing dependencies to the individual package.json files of the packages. These were resolving fine earlier because of hoisting but the problem of undeclared dependencies went undetected because of this convenience feature of the NodeJS module resolution algorithm. Now, instead of having to catch these problems in production, we can fix them up-front at development time which is great because there were dozens of them! 6. Migrated from `import` to `import type` in a few places in the code where it was applicable and useful: when you only import a dependency via `import type` then it's safe to move it to "devDependencies" because at runtime the `import type` syntax is completely omitted by the compiler. 7. Refactored the importing of "internal-ip" in a couple of test cases because the newer version does not export `v4` and `v6` but instead `internalIpV4`. 7. Renamed 2 environment variables that are used by ci.yaml to control the steps taken by ci.sh because they had a `YARN_` prefix in their names that Yarn v3 appears to be automatically parsing and assuming to be configuration for itself - which then leads to crashes because it does not recognize the custom variable name that is not meant for it. The variables renamed are: `TOOLS_VALIDATE_BUNDLE_NAMES_DISABLED` and `CUSTOM_CHECKS_DISABLED` 8. The configure script now runs the init-registries sub-script via npm instead of yarn because yarn is unable to run scripts prior to having ran a yarn install on a fresh clone (which was making the configure script bail out on the CI) 9. The `--mode=skip-build` parameter is specified for yarn in the configure script so that during installation it doesn't spend an extra 15 minutes compiling dependencies with native code in them. Yarn v1 did not do this by default and therefore it is assumed that it should work fine for v3 as well, but this is by no means a guarantee that it will work in all cases. 10. Updated the artillery binary path to be relatively qualified instead of just the binary file's name in the test case that does the API server benchmarking in packages/cactus-cmd-api-server/src/test/typescript/benchmark/artillery-api-benchmark.test.ts so now the binary's path is specified explicitly as ./packages/cactus-cmd-api-server/node_modules/artillery/bin/artillery which is the new path on account of us turning off t he hoisting of the dependencies entirely for the project (e.g. the binary is no longer in the rood node_modules directory) Fixes hyperledger-cacti#1142 Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
petermetz
added a commit
to petermetz/cacti
that referenced
this issue
Jul 8, 2023
We hope to solve issues with this that were plaguing contributors due to bugs in the much older Yarn 1.x versions that we had to use because Yarn V3 had it's own set of problems when it came to linking the .bin folders for child packages in a monorepo. 1. We've moved from hoisting by default to NOT hoisting by default for all packages which results in slightly slower installs for the dependencies when setting up the project build but it does have the beneficial effect of much easier fixes for compiler errors because each package can be provided with their own version of whatever dependencies they need. 2. The "pre" and "post" npm scripts are no longer supported by Yarn which was an annoying breaking change so we had to restructure some of the scripts which had pre and post scripts of their own to get equal behavior. 3. Upgraded to Yarn v3.6.0 4. packages/cactus-api-client/src/main/typescript/socketio-api-client.ts has a line where the Typescript compiler has been disabled and an issue opened for the same: https://github.com/hyperledger/cacti/issues/2523 5. Added several missing dependencies to the individual package.json files of the packages. These were resolving fine earlier because of hoisting but the problem of undeclared dependencies went undetected because of this convenience feature of the NodeJS module resolution algorithm. Now, instead of having to catch these problems in production, we can fix them up-front at development time which is great because there were dozens of them! 6. Migrated from `import` to `import type` in a few places in the code where it was applicable and useful: when you only import a dependency via `import type` then it's safe to move it to "devDependencies" because at runtime the `import type` syntax is completely omitted by the compiler. 7. Refactored the importing of "internal-ip" in a couple of test cases because the newer version does not export `v4` and `v6` but instead `internalIpV4`. 8. Renamed 2 environment variables that are used by ci.yaml to control the steps taken by ci.sh because they had a `YARN_` prefix in their names that Yarn v3 appears to be automatically parsing and assuming to be configuration for itself - which then leads to crashes because it does not recognize the custom variable name that is not meant for it. The variables renamed are: `TOOLS_VALIDATE_BUNDLE_NAMES_DISABLED` and `CUSTOM_CHECKS_DISABLED` 9. The configure script now runs the init-registries sub-script via npm instead of yarn because yarn is unable to run scripts prior to having ran a yarn install on a fresh clone (which was making the configure script bail out on the CI) 10. The `--mode=skip-build` parameter is specified for yarn in the configure script so that during installation it doesn't spend an extra 15 minutes compiling dependencies with native code in them. Yarn v1 did not do this by default and therefore it is assumed that it should work fine for v3 as well, but this is by no means a guarantee that it will work in all cases. 11. Updated the artillery binary path to be relatively qualified instead of just the binary file's name in the test case that does the API server benchmarking in packages/cactus-cmd-api-server/src/test/typescript/benchmark/artillery-api-benchmark.test.ts so now the binary's path is specified explicitly as ./packages/cactus-cmd-api-server/node_modules/artillery/bin/artillery which is the new path on account of us turning off t he hoisting of the dependencies entirely for the project (e.g. the binary is no longer in the rood node_modules directory) Fixes hyperledger-cacti#1142 Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
zondervancalvez
added a commit
to zondervancalvez/cactus
that referenced
this issue
Jul 10, 2023
Fixes hyperledger-cacti#1142 Signed-off-by: zondervancalvez <zondervan.v.calvez@accenture.com>
zondervancalvez
added a commit
to zondervancalvez/cactus
that referenced
this issue
Jul 10, 2023
Fixes hyperledger-cacti#1142 Signed-off-by: zondervancalvez <zondervan.v.calvez@accenture.com>
zondervancalvez
added a commit
to zondervancalvez/cactus
that referenced
this issue
Jul 10, 2023
Fixes hyperledger-cacti#1142 Signed-off-by: zondervancalvez <zondervan.v.calvez@accenture.com>
zondervancalvez
added a commit
to zondervancalvez/cactus
that referenced
this issue
Jul 10, 2023
Fixes hyperledger-cacti#1142 Signed-off-by: zondervancalvez <zondervan.v.calvez@accenture.com>
zondervancalvez
added a commit
to zondervancalvez/cactus
that referenced
this issue
Jul 11, 2023
Fixes hyperledger-cacti#1142 Signed-off-by: zondervancalvez <zondervan.v.calvez@accenture.com>
zondervancalvez
added a commit
to zondervancalvez/cactus
that referenced
this issue
Jul 11, 2023
Fixes hyperledger-cacti#1142 Signed-off-by: zondervancalvez <zondervan.v.calvez@accenture.com>
petermetz
added a commit
to petermetz/cacti
that referenced
this issue
Jul 12, 2023
We hope to solve issues with this that were plaguing contributors due to bugs in the much older Yarn 1.x versions that we had to use because Yarn V3 had it's own set of problems when it came to linking the .bin folders for child packages in a monorepo. 1. We've moved from hoisting by default to NOT hoisting by default for all packages which results in slightly slower installs for the dependencies when setting up the project build but it does have the beneficial effect of much easier fixes for compiler errors because each package can be provided with their own version of whatever dependencies they need. 2. The "pre" and "post" npm scripts are no longer supported by Yarn which was an annoying breaking change so we had to restructure some of the scripts which had pre and post scripts of their own to get equal behavior. 3. Upgraded to Yarn v3.6.0 4. packages/cactus-api-client/src/main/typescript/socketio-api-client.ts has a line where the Typescript compiler has been disabled and an issue opened for the same: https://github.com/hyperledger/cacti/issues/2523 5. Added several missing dependencies to the individual package.json files of the packages. These were resolving fine earlier because of hoisting but the problem of undeclared dependencies went undetected because of this convenience feature of the NodeJS module resolution algorithm. Now, instead of having to catch these problems in production, we can fix them up-front at development time which is great because there were dozens of them! 6. Migrated from `import` to `import type` in a few places in the code where it was applicable and useful: when you only import a dependency via `import type` then it's safe to move it to "devDependencies" because at runtime the `import type` syntax is completely omitted by the compiler. 7. Refactored the importing of "internal-ip" in a couple of test cases because the newer version does not export `v4` and `v6` but instead `internalIpV4`. 8. Renamed 2 environment variables that are used by ci.yaml to control the steps taken by ci.sh because they had a `YARN_` prefix in their names that Yarn v3 appears to be automatically parsing and assuming to be configuration for itself - which then leads to crashes because it does not recognize the custom variable name that is not meant for it. The variables renamed are: `TOOLS_VALIDATE_BUNDLE_NAMES_DISABLED` and `CUSTOM_CHECKS_DISABLED` 9. The configure script now runs the init-registries sub-script via npm instead of yarn because yarn is unable to run scripts prior to having ran a yarn install on a fresh clone (which was making the configure script bail out on the CI) 10. The `--mode=skip-build` parameter is specified for yarn in the configure script so that during installation it doesn't spend an extra 15 minutes compiling dependencies with native code in them. Yarn v1 did not do this by default and therefore it is assumed that it should work fine for v3 as well, but this is by no means a guarantee that it will work in all cases. 11. Updated the artillery binary path to be relatively qualified instead of just the binary file's name in the test case that does the API server benchmarking in packages/cactus-cmd-api-server/src/test/typescript/benchmark/artillery-api-benchmark.test.ts so now the binary's path is specified explicitly as ./packages/cactus-cmd-api-server/node_modules/artillery/bin/artillery which is the new path on account of us turning off t he hoisting of the dependencies entirely for the project (e.g. the binary is no longer in the rood node_modules directory) Fixes hyperledger-cacti#1142 Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
petermetz
added a commit
to petermetz/cacti
that referenced
this issue
Jul 12, 2023
We hope to solve issues with this that were plaguing contributors due to bugs in the much older Yarn 1.x versions that we had to use because Yarn V3 had it's own set of problems when it came to linking the .bin folders for child packages in a monorepo. 1. We've moved from hoisting by default to NOT hoisting by default for all packages which results in slightly slower installs for the dependencies when setting up the project build but it does have the beneficial effect of much easier fixes for compiler errors because each package can be provided with their own version of whatever dependencies they need. 2. The "pre" and "post" npm scripts are no longer supported by Yarn which was an annoying breaking change so we had to restructure some of the scripts which had pre and post scripts of their own to get equal behavior. 3. Upgraded to Yarn v3.6.0 4. packages/cactus-api-client/src/main/typescript/socketio-api-client.ts has a line where the Typescript compiler has been disabled and an issue opened for the same: https://github.com/hyperledger/cacti/issues/2523 5. Added several missing dependencies to the individual package.json files of the packages. These were resolving fine earlier because of hoisting but the problem of undeclared dependencies went undetected because of this convenience feature of the NodeJS module resolution algorithm. Now, instead of having to catch these problems in production, we can fix them up-front at development time which is great because there were dozens of them! 6. Migrated from `import` to `import type` in a few places in the code where it was applicable and useful: when you only import a dependency via `import type` then it's safe to move it to "devDependencies" because at runtime the `import type` syntax is completely omitted by the compiler. 7. Refactored the importing of "internal-ip" in a couple of test cases because the newer version does not export `v4` and `v6` but instead `internalIpV4`. 8. Renamed 2 environment variables that are used by ci.yaml to control the steps taken by ci.sh because they had a `YARN_` prefix in their names that Yarn v3 appears to be automatically parsing and assuming to be configuration for itself - which then leads to crashes because it does not recognize the custom variable name that is not meant for it. The variables renamed are: `TOOLS_VALIDATE_BUNDLE_NAMES_DISABLED` and `CUSTOM_CHECKS_DISABLED` 9. The configure script now runs the init-registries sub-script via npm instead of yarn because yarn is unable to run scripts prior to having ran a yarn install on a fresh clone (which was making the configure script bail out on the CI) 10. The `--mode=skip-build` parameter is specified for yarn in the configure script so that during installation it doesn't spend an extra 15 minutes compiling dependencies with native code in them. Yarn v1 did not do this by default and therefore it is assumed that it should work fine for v3 as well, but this is by no means a guarantee that it will work in all cases. 11. Updated the artillery binary path to be relatively qualified instead of just the binary file's name in the test case that does the API server benchmarking in packages/cactus-cmd-api-server/src/test/typescript/benchmark/artillery-api-benchmark.test.ts so now the binary's path is specified explicitly as ./packages/cactus-cmd-api-server/node_modules/artillery/bin/artillery which is the new path on account of us turning off t he hoisting of the dependencies entirely for the project (e.g. the binary is no longer in the rood node_modules directory) 12. Explicitly disabled the build scripts of a long list of dependencies in the root package.json file because they were greatly and needlessly slowing down the build (we know that we don't need their build scripts because it was working well with yarn v1 without them) Fixes hyperledger-cacti#1142 Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
petermetz
added a commit
to petermetz/cacti
that referenced
this issue
Jul 15, 2023
We hope to solve issues with this that were plaguing contributors due to bugs in the much older Yarn 1.x versions that we had to use because Yarn V3 had it's own set of problems when it came to linking the .bin folders for child packages in a monorepo. 1. We've moved from hoisting by default to NOT hoisting by default for all packages which results in slightly slower installs for the dependencies when setting up the project build but it does have the beneficial effect of much easier fixes for compiler errors because each package can be provided with their own version of whatever dependencies they need. 2. The "pre" and "post" npm scripts are no longer supported by Yarn which was an annoying breaking change so we had to restructure some of the scripts which had pre and post scripts of their own to get equal behavior. 3. Upgraded to Yarn v3.6.0 4. packages/cactus-api-client/src/main/typescript/socketio-api-client.ts has a line where the Typescript compiler has been disabled and an issue opened for the same: https://github.com/hyperledger/cacti/issues/2523 5. Added several missing dependencies to the individual package.json files of the packages. These were resolving fine earlier because of hoisting but the problem of undeclared dependencies went undetected because of this convenience feature of the NodeJS module resolution algorithm. Now, instead of having to catch these problems in production, we can fix them up-front at development time which is great because there were dozens of them! 6. Migrated from `import` to `import type` in a few places in the code where it was applicable and useful: when you only import a dependency via `import type` then it's safe to move it to "devDependencies" because at runtime the `import type` syntax is completely omitted by the compiler. 7. Refactored the importing of "internal-ip" in a couple of test cases because the newer version does not export `v4` and `v6` but instead `internalIpV4`. 8. Renamed 2 environment variables that are used by ci.yaml to control the steps taken by ci.sh because they had a `YARN_` prefix in their names that Yarn v3 appears to be automatically parsing and assuming to be configuration for itself - which then leads to crashes because it does not recognize the custom variable name that is not meant for it. The variables renamed are: `TOOLS_VALIDATE_BUNDLE_NAMES_DISABLED` and `CUSTOM_CHECKS_DISABLED` 9. The configure script now runs the init-registries sub-script via npm instead of yarn because yarn is unable to run scripts prior to having ran a yarn install on a fresh clone (which was making the configure script bail out on the CI) 10. The `--mode=skip-build` parameter is specified for yarn in the configure script so that during installation it doesn't spend an extra 15 minutes compiling dependencies with native code in them. Yarn v1 did not do this by default and therefore it is assumed that it should work fine for v3 as well, but this is by no means a guarantee that it will work in all cases. 11. Updated the artillery binary path to be relatively qualified instead of just the binary file's name in the test case that does the API server benchmarking in packages/cactus-cmd-api-server/src/test/typescript/benchmark/artillery-api-benchmark.test.ts so now the binary's path is specified explicitly as ./packages/cactus-cmd-api-server/node_modules/artillery/bin/artillery which is the new path on account of us turning off t he hoisting of the dependencies entirely for the project (e.g. the binary is no longer in the rood node_modules directory) 12. Explicitly disabled the build scripts of a long list of dependencies in the root package.json file because they were greatly and needlessly slowing down the build (we know that we don't need their build scripts because it was working well with yarn v1 without them) Fixes hyperledger-cacti#1142 Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
petermetz
added a commit
to petermetz/cacti
that referenced
this issue
Jul 15, 2023
We hope to solve issues with this that were plaguing contributors due to bugs in the much older Yarn 1.x versions that we had to use because Yarn V3 had it's own set of problems when it came to linking the .bin folders for child packages in a monorepo. 1. We've moved from hoisting by default to NOT hoisting by default for all packages which results in slightly slower installs for the dependencies when setting up the project build but it does have the beneficial effect of much easier fixes for compiler errors because each package can be provided with their own version of whatever dependencies they need. 2. The "pre" and "post" npm scripts are no longer supported by Yarn which was an annoying breaking change so we had to restructure some of the scripts which had pre and post scripts of their own to get equal behavior. 3. Upgraded to Yarn v3.6.0 4. packages/cactus-api-client/src/main/typescript/socketio-api-client.ts has a line where the Typescript compiler has been disabled and an issue opened for the same: https://github.com/hyperledger/cacti/issues/2523 5. Added several missing dependencies to the individual package.json files of the packages. These were resolving fine earlier because of hoisting but the problem of undeclared dependencies went undetected because of this convenience feature of the NodeJS module resolution algorithm. Now, instead of having to catch these problems in production, we can fix them up-front at development time which is great because there were dozens of them! 6. Migrated from `import` to `import type` in a few places in the code where it was applicable and useful: when you only import a dependency via `import type` then it's safe to move it to "devDependencies" because at runtime the `import type` syntax is completely omitted by the compiler. 7. Refactored the importing of "internal-ip" in a couple of test cases because the newer version does not export `v4` and `v6` but instead `internalIpV4` which meant that the >=7.0.0 versions needed downgrading. 8. Renamed 2 environment variables that are used by ci.yaml to control the steps taken by ci.sh because they had a `YARN_` prefix in their names that Yarn v3 appears to be automatically parsing and assuming to be configuration for itself - which then leads to crashes because it does not recognize the custom variable name that is not meant for it. The variables renamed are: `TOOLS_VALIDATE_BUNDLE_NAMES_DISABLED` and `CUSTOM_CHECKS_DISABLED` 9. The configure script now runs the init-registries sub-script via npm instead of yarn because yarn is unable to run scripts prior to having ran a yarn install on a fresh clone (which was making the configure script bail out on the CI) 10. Configured yarn NOT to rebuild most dependencies that have native code in them. The exception from under this is the gRPC and related dependencies whose native code extensions are being used by the test cases and so we do need them for real. 11. Updated the artillery binary path to be relatively qualified instead of just the binary file's name in the test case that does the API server benchmarking in packages/cactus-cmd-api-server/src/test/typescript/benchmark/artillery-api-benchmark.test.ts so now the binary's path is specified explicitly as ./packages/cactus-cmd-api-server/node_modules/artillery/bin/artillery which is the new path on account of us turning off t he hoisting of the dependencies entirely for the project (e.g. the binary is no longer in the rood node_modules directory) 12. Explicitly disabled the build scripts of a long list of dependencies in the root package.json file because they were greatly and needlessly slowing down the build (we know that we don't need their build scripts because it was working well with yarn v1 without them) 13. Updated the mapping for the foundry/forge dependencies because now that we don't have hoisting enabled by default for any of the packages the binaries end up in the package's own node_modules instead of the root's. 14. Refactored the ci.sh script to not do a lot of the resource intensive diagnostics and the disk space optimization is also disabled by default. These things used to be very important when we were running a monolithic CI job for everything but not so much now that we have a separate job for every single package. This brings down the ci.sh execution time a few minutes at least which adds up to a lot because it is executed 40+ times for each CI run. 15. In general the important change here (apart from the yarn v3 upgrade) is the disablement of the hoisting of dependencies because this makes the build much more reliable and harder to mess up with missing dependencies. Fixes hyperledger-cacti#1142 Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
petermetz
added a commit
to petermetz/cacti
that referenced
this issue
Jul 15, 2023
We hope to solve issues with this that were plaguing contributors due to bugs in the much older Yarn 1.x versions that we had to use because Yarn V3 had it's own set of problems when it came to linking the .bin folders for child packages in a monorepo. 1. We've moved from hoisting by default to NOT hoisting by default for all packages which results in slightly slower installs for the dependencies when setting up the project build but it does have the beneficial effect of much easier fixes for compiler errors because each package can be provided with their own version of whatever dependencies they need. 2. The "pre" and "post" npm scripts are no longer supported by Yarn which was an annoying breaking change so we had to restructure some of the scripts which had pre and post scripts of their own to get equal behavior. 3. Upgraded to Yarn v3.6.0 4. packages/cactus-api-client/src/main/typescript/socketio-api-client.ts has a line where the Typescript compiler has been disabled and an issue opened for the same: https://github.com/hyperledger/cacti/issues/2523 5. Added several missing dependencies to the individual package.json files of the packages. These were resolving fine earlier because of hoisting but the problem of undeclared dependencies went undetected because of this convenience feature of the NodeJS module resolution algorithm. Now, instead of having to catch these problems in production, we can fix them up-front at development time which is great because there were dozens of them! 6. Migrated from `import` to `import type` in a few places in the code where it was applicable and useful: when you only import a dependency via `import type` then it's safe to move it to "devDependencies" because at runtime the `import type` syntax is completely omitted by the compiler. 7. Refactored the importing of "internal-ip" in a couple of test cases because the newer version does not export `v4` and `v6` but instead `internalIpV4` which meant that the >=7.0.0 versions needed downgrading. 8. Renamed 2 environment variables that are used by ci.yaml to control the steps taken by ci.sh because they had a `YARN_` prefix in their names that Yarn v3 appears to be automatically parsing and assuming to be configuration for itself - which then leads to crashes because it does not recognize the custom variable name that is not meant for it. The variables renamed are: `TOOLS_VALIDATE_BUNDLE_NAMES_DISABLED` and `CUSTOM_CHECKS_DISABLED` 9. The configure script now runs the init-registries sub-script via npm instead of yarn because yarn is unable to run scripts prior to having ran a yarn install on a fresh clone (which was making the configure script bail out on the CI) 10. Configured yarn NOT to rebuild most dependencies that have native code in them. The exception from under this is the gRPC and related dependencies whose native code extensions are being used by the test cases and so we do need them for real. 11. Updated the artillery binary path to be relatively qualified instead of just the binary file's name in the test case that does the API server benchmarking in packages/cactus-cmd-api-server/src/test/typescript/benchmark/artillery-api-benchmark.test.ts so now the binary's path is specified explicitly as ./packages/cactus-cmd-api-server/node_modules/artillery/bin/artillery which is the new path on account of us turning off t he hoisting of the dependencies entirely for the project (e.g. the binary is no longer in the rood node_modules directory) 12. Explicitly disabled the build scripts of a long list of dependencies in the root package.json file because they were greatly and needlessly slowing down the build (we know that we don't need their build scripts because it was working well with yarn v1 without them) 13. Updated the mapping for the foundry/forge dependencies because now that we don't have hoisting enabled by default for any of the packages the binaries end up in the package's own node_modules instead of the root's. 14. Refactored the ci.sh script to not do a lot of the resource intensive diagnostics and the disk space optimization is also disabled by default. These things used to be very important when we were running a monolithic CI job for everything but not so much now that we have a separate job for every single package. This brings down the ci.sh execution time a few minutes at least which adds up to a lot because it is executed 40+ times for each CI run. 15. In general the important change here (apart from the yarn v3 upgrade) is the disablement of the hoisting of dependencies because this makes the build much more reliable and harder to mess up with missing dependencies. 16. Had to upgrade protoc-gen-ts to v0.8.6 which then needed typescript v4.9.x or newer which then forced me to fix a few new compiler errors (which seemed like legit bugs so it's a net positive but it's yet another big diff causing change) 17. Had to fix a few webpack import issues related to backend dependencies being used on the front-end - this only affects the example code/GUI Fixes hyperledger-cacti#1142 Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
petermetz
added a commit
to petermetz/cacti
that referenced
this issue
Jul 16, 2023
We hope to solve issues with this that were plaguing contributors due to bugs in the much older Yarn 1.x versions that we had to use because Yarn V3 had it's own set of problems when it came to linking the .bin folders for child packages in a monorepo. 1. We've moved from hoisting by default to NOT hoisting by default for all packages which results in slightly slower installs for the dependencies when setting up the project build but it does have the beneficial effect of much easier fixes for compiler errors because each package can be provided with their own version of whatever dependencies they need. 2. The "pre" and "post" npm scripts are no longer supported by Yarn which was an annoying breaking change so we had to restructure some of the scripts which had pre and post scripts of their own to get equal behavior. 3. Upgraded to Yarn v3.6.0 4. packages/cactus-api-client/src/main/typescript/socketio-api-client.ts has a line where the Typescript compiler has been disabled and an issue opened for the same: https://github.com/hyperledger/cacti/issues/2523 5. Added several missing dependencies to the individual package.json files of the packages. These were resolving fine earlier because of hoisting but the problem of undeclared dependencies went undetected because of this convenience feature of the NodeJS module resolution algorithm. Now, instead of having to catch these problems in production, we can fix them up-front at development time which is great because there were dozens of them! 6. Migrated from `import` to `import type` in a few places in the code where it was applicable and useful: when you only import a dependency via `import type` then it's safe to move it to "devDependencies" because at runtime the `import type` syntax is completely omitted by the compiler. 7. Refactored the importing of "internal-ip" in a couple of test cases because the newer version does not export `v4` and `v6` but instead `internalIpV4` which meant that the >=7.0.0 versions needed downgrading. 8. Renamed 2 environment variables that are used by ci.yaml to control the steps taken by ci.sh because they had a `YARN_` prefix in their names that Yarn v3 appears to be automatically parsing and assuming to be configuration for itself - which then leads to crashes because it does not recognize the custom variable name that is not meant for it. The variables renamed are: `TOOLS_VALIDATE_BUNDLE_NAMES_DISABLED` and `CUSTOM_CHECKS_DISABLED` 9. The configure script now runs the init-registries sub-script via npm instead of yarn because yarn is unable to run scripts prior to having ran a yarn install on a fresh clone (which was making the configure script bail out on the CI) 10. Configured yarn NOT to rebuild most dependencies that have native code in them. The exception from under this is the gRPC and related dependencies whose native code extensions are being used by the test cases and so we do need them for real. 11. Updated the artillery binary path to be relatively qualified instead of just the binary file's name in the test case that does the API server benchmarking in packages/cactus-cmd-api-server/src/test/typescript/benchmark/artillery-api-benchmark.test.ts so now the binary's path is specified explicitly as ./packages/cactus-cmd-api-server/node_modules/artillery/bin/artillery which is the new path on account of us turning off t he hoisting of the dependencies entirely for the project (e.g. the binary is no longer in the rood node_modules directory) 12. Explicitly disabled the build scripts of a long list of dependencies in the root package.json file because they were greatly and needlessly slowing down the build (we know that we don't need their build scripts because it was working well with yarn v1 without them) 13. Updated the mapping for the foundry/forge dependencies because now that we don't have hoisting enabled by default for any of the packages the binaries end up in the package's own node_modules instead of the root's. 14. Refactored the ci.sh script to not do a lot of the resource intensive diagnostics and the disk space optimization is also disabled by default. These things used to be very important when we were running a monolithic CI job for everything but not so much now that we have a separate job for every single package. This brings down the ci.sh execution time a few minutes at least which adds up to a lot because it is executed 40+ times for each CI run. 15. In general the important change here (apart from the yarn v3 upgrade) is the disablement of the hoisting of dependencies because this makes the build much more reliable and harder to mess up with missing dependencies. 16. Had to upgrade protoc-gen-ts to v0.8.6 which then needed typescript v4.9.x or newer which then forced me to fix a few new compiler errors (which seemed like legit bugs so it's a net positive but it's yet another big diff causing change) 17. Had to fix a few webpack import issues related to backend dependencies being used on the front-end - this only affects the example code/GUI 18. Upgraded the eslint typescript plugin and parser dependencies because the linter otherwise couldn't handle the code with the new version of the typescript compiler (v4.9.x). Also had to set the ban on explicit anys to a warn level down from an error because we have hundreds of them in the codebase. There's one linter error left that I won't fix here because there is already another PR for it. Fixes hyperledger-cacti#1142 Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
petermetz
changed the title
petermetz
added a commit
to petermetz/cacti
that referenced
this issue
Jul 20, 2023
We hope to solve issues with this that were plaguing contributors due to bugs in the much older Yarn 1.x versions that we had to use because Yarn V3 had it's own set of problems when it came to linking the .bin folders for child packages in a monorepo. 1. We've moved from hoisting by default to NOT hoisting by default for all packages which results in slightly slower installs for the dependencies when setting up the project build but it does have the beneficial effect of much easier fixes for compiler errors because each package can be provided with their own version of whatever dependencies they need. 2. The "pre" and "post" npm scripts are no longer supported by Yarn which was an annoying breaking change so we had to restructure some of the scripts which had pre and post scripts of their own to get equal behavior. 3. Upgraded to Yarn v3.6.0 4. packages/cactus-api-client/src/main/typescript/socketio-api-client.ts has a line where the Typescript compiler has been disabled and an issue opened for the same: https://github.com/hyperledger/cacti/issues/2523 5. Added several missing dependencies to the individual package.json files of the packages. These were resolving fine earlier because of hoisting but the problem of undeclared dependencies went undetected because of this convenience feature of the NodeJS module resolution algorithm. Now, instead of having to catch these problems in production, we can fix them up-front at development time which is great because there were dozens of them! 6. Migrated from `import` to `import type` in a few places in the code where it was applicable and useful: when you only import a dependency via `import type` then it's safe to move it to "devDependencies" because at runtime the `import type` syntax is completely omitted by the compiler. 7. Refactored the importing of "internal-ip" in a couple of test cases because the newer version does not export `v4` and `v6` but instead `internalIpV4` which meant that the >=7.0.0 versions needed downgrading. 8. Renamed 2 environment variables that are used by ci.yaml to control the steps taken by ci.sh because they had a `YARN_` prefix in their names that Yarn v3 appears to be automatically parsing and assuming to be configuration for itself - which then leads to crashes because it does not recognize the custom variable name that is not meant for it. The variables renamed are: `TOOLS_VALIDATE_BUNDLE_NAMES_DISABLED` and `CUSTOM_CHECKS_DISABLED` 9. The configure script now runs the init-registries sub-script via npm instead of yarn because yarn is unable to run scripts prior to having ran a yarn install on a fresh clone (which was making the configure script bail out on the CI) 10. Configured yarn NOT to rebuild most dependencies that have native code in them. The exception from under this is the gRPC and related dependencies whose native code extensions are being used by the test cases and so we do need them for real. 11. Updated the artillery binary path to be relatively qualified instead of just the binary file's name in the test case that does the API server benchmarking in packages/cactus-cmd-api-server/src/test/typescript/benchmark/artillery-api-benchmark.test.ts so now the binary's path is specified explicitly as ./packages/cactus-cmd-api-server/node_modules/artillery/bin/artillery which is the new path on account of us turning off t he hoisting of the dependencies entirely for the project (e.g. the binary is no longer in the rood node_modules directory) 12. Explicitly disabled the build scripts of a long list of dependencies in the root package.json file because they were greatly and needlessly slowing down the build (we know that we don't need their build scripts because it was working well with yarn v1 without them) 13. Updated the mapping for the foundry/forge dependencies because now that we don't have hoisting enabled by default for any of the packages the binaries end up in the package's own node_modules instead of the root's. 14. Refactored the ci.sh script to not do a lot of the resource intensive diagnostics and the disk space optimization is also disabled by default. These things used to be very important when we were running a monolithic CI job for everything but not so much now that we have a separate job for every single package. This brings down the ci.sh execution time a few minutes at least which adds up to a lot because it is executed 40+ times for each CI run. 15. In general the important change here (apart from the yarn v3 upgrade) is the disablement of the hoisting of dependencies because this makes the build much more reliable and harder to mess up with missing dependencies. 16. Had to upgrade protoc-gen-ts to v0.8.6 which then needed typescript v4.9.x or newer which then forced me to fix a few new compiler errors (which seemed like legit bugs so it's a net positive but it's yet another big diff causing change) 17. Had to fix a few webpack import issues related to backend dependencies being used on the front-end - this only affects the example code/GUI 18. Upgraded the eslint typescript plugin and parser dependencies because the linter otherwise couldn't handle the code with the new version of the typescript compiler (v4.9.x). Also had to set the ban on explicit anys to a warn level down from an error because we have hundreds of them in the codebase. There's one linter error left that I won't fix here because there is already another PR for it. Fixes hyperledger-cacti#1142 Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
petermetz
added a commit
to petermetz/cacti
that referenced
this issue
Jul 20, 2023
We hope to solve issues with this that were plaguing contributors due to bugs in the much older Yarn 1.x versions that we had to use because Yarn V3 had it's own set of problems when it came to linking the .bin folders for child packages in a monorepo. 1. We've moved from hoisting by default to NOT hoisting by default for all packages which results in slightly slower installs for the dependencies when setting up the project build but it does have the beneficial effect of much easier fixes for compiler errors because each package can be provided with their own version of whatever dependencies they need. 2. The "pre" and "post" npm scripts are no longer supported by Yarn which was an annoying breaking change so we had to restructure some of the scripts which had pre and post scripts of their own to get equal behavior. 3. Upgraded to Yarn v3.6.0 4. packages/cactus-api-client/src/main/typescript/socketio-api-client.ts has a line where the Typescript compiler has been disabled and an issue opened for the same: https://github.com/hyperledger/cacti/issues/2523 5. Added several missing dependencies to the individual package.json files of the packages. These were resolving fine earlier because of hoisting but the problem of undeclared dependencies went undetected because of this convenience feature of the NodeJS module resolution algorithm. Now, instead of having to catch these problems in production, we can fix them up-front at development time which is great because there were dozens of them! 6. Migrated from `import` to `import type` in a few places in the code where it was applicable and useful: when you only import a dependency via `import type` then it's safe to move it to "devDependencies" because at runtime the `import type` syntax is completely omitted by the compiler. 7. Refactored the importing of "internal-ip" in a couple of test cases because the newer version does not export `v4` and `v6` but instead `internalIpV4` which meant that the >=7.0.0 versions needed downgrading. 8. Renamed 2 environment variables that are used by ci.yaml to control the steps taken by ci.sh because they had a `YARN_` prefix in their names that Yarn v3 appears to be automatically parsing and assuming to be configuration for itself - which then leads to crashes because it does not recognize the custom variable name that is not meant for it. The variables renamed are: `TOOLS_VALIDATE_BUNDLE_NAMES_DISABLED` and `CUSTOM_CHECKS_DISABLED` 9. The configure script now runs the init-registries sub-script via npm instead of yarn because yarn is unable to run scripts prior to having ran a yarn install on a fresh clone (which was making the configure script bail out on the CI) 10. Configured yarn NOT to rebuild most dependencies that have native code in them. The exception from under this is the gRPC and related dependencies whose native code extensions are being used by the test cases and so we do need them for real. 11. Updated the artillery binary path to be relatively qualified instead of just the binary file's name in the test case that does the API server benchmarking in packages/cactus-cmd-api-server/src/test/typescript/benchmark/artillery-api-benchmark.test.ts so now the binary's path is specified explicitly as ./packages/cactus-cmd-api-server/node_modules/artillery/bin/artillery which is the new path on account of us turning off t he hoisting of the dependencies entirely for the project (e.g. the binary is no longer in the rood node_modules directory) 12. Explicitly disabled the build scripts of a long list of dependencies in the root package.json file because they were greatly and needlessly slowing down the build (we know that we don't need their build scripts because it was working well with yarn v1 without them) 13. Updated the mapping for the foundry/forge dependencies because now that we don't have hoisting enabled by default for any of the packages the binaries end up in the package's own node_modules instead of the root's. 14. Refactored the ci.sh script to not do a lot of the resource intensive diagnostics and the disk space optimization is also disabled by default. These things used to be very important when we were running a monolithic CI job for everything but not so much now that we have a separate job for every single package. This brings down the ci.sh execution time a few minutes at least which adds up to a lot because it is executed 40+ times for each CI run. 15. In general the important change here (apart from the yarn v3 upgrade) is the disablement of the hoisting of dependencies because this makes the build much more reliable and harder to mess up with missing dependencies. 16. Had to upgrade protoc-gen-ts to v0.8.6 which then needed typescript v4.9.x or newer which then forced me to fix a few new compiler errors (which seemed like legit bugs so it's a net positive but it's yet another big diff causing change) 17. Had to fix a few webpack import issues related to backend dependencies being used on the front-end - this only affects the example code/GUI 18. Upgraded the eslint typescript plugin and parser dependencies because the linter otherwise couldn't handle the code with the new version of the typescript compiler (v4.9.x). Also had to set the ban on explicit anys to a warn level down from an error because we have hundreds of them in the codebase. There's one linter error left that I won't fix here because there is already another PR for it. Fixes hyperledger-cacti#1142 Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
petermetz
added a commit
to petermetz/cacti
that referenced
this issue
Jul 20, 2023
We hope to solve issues with this that were plaguing contributors due to bugs in the much older Yarn 1.x versions that we had to use because Yarn V3 had it's own set of problems when it came to linking the .bin folders for child packages in a monorepo. 1. We've moved from hoisting by default to NOT hoisting by default for all packages which results in slightly slower installs for the dependencies when setting up the project build but it does have the beneficial effect of much easier fixes for compiler errors because each package can be provided with their own version of whatever dependencies they need. 2. The "pre" and "post" npm scripts are no longer supported by Yarn which was an annoying breaking change so we had to restructure some of the scripts which had pre and post scripts of their own to get equal behavior. 3. Upgraded to Yarn v3.6.0 4. packages/cactus-api-client/src/main/typescript/socketio-api-client.ts has a line where the Typescript compiler has been disabled and an issue opened for the same: https://github.com/hyperledger/cacti/issues/2523 5. Added several missing dependencies to the individual package.json files of the packages. These were resolving fine earlier because of hoisting but the problem of undeclared dependencies went undetected because of this convenience feature of the NodeJS module resolution algorithm. Now, instead of having to catch these problems in production, we can fix them up-front at development time which is great because there were dozens of them! 6. Migrated from `import` to `import type` in a few places in the code where it was applicable and useful: when you only import a dependency via `import type` then it's safe to move it to "devDependencies" because at runtime the `import type` syntax is completely omitted by the compiler. 7. Refactored the importing of "internal-ip" in a couple of test cases because the newer version does not export `v4` and `v6` but instead `internalIpV4` which meant that the >=7.0.0 versions needed downgrading. 8. Renamed 2 environment variables that are used by ci.yaml to control the steps taken by ci.sh because they had a `YARN_` prefix in their names that Yarn v3 appears to be automatically parsing and assuming to be configuration for itself - which then leads to crashes because it does not recognize the custom variable name that is not meant for it. The variables renamed are: `TOOLS_VALIDATE_BUNDLE_NAMES_DISABLED` and `CUSTOM_CHECKS_DISABLED` 9. The configure script now runs the init-registries sub-script via npm instead of yarn because yarn is unable to run scripts prior to having ran a yarn install on a fresh clone (which was making the configure script bail out on the CI) 10. Configured yarn NOT to rebuild most dependencies that have native code in them. The exception from under this is the gRPC and related dependencies whose native code extensions are being used by the test cases and so we do need them for real. 11. Updated the artillery binary path to be relatively qualified instead of just the binary file's name in the test case that does the API server benchmarking in packages/cactus-cmd-api-server/src/test/typescript/benchmark/artillery-api-benchmark.test.ts so now the binary's path is specified explicitly as ./packages/cactus-cmd-api-server/node_modules/artillery/bin/artillery which is the new path on account of us turning off t he hoisting of the dependencies entirely for the project (e.g. the binary is no longer in the rood node_modules directory) 12. Explicitly disabled the build scripts of a long list of dependencies in the root package.json file because they were greatly and needlessly slowing down the build (we know that we don't need their build scripts because it was working well with yarn v1 without them) 13. Updated the mapping for the foundry/forge dependencies because now that we don't have hoisting enabled by default for any of the packages the binaries end up in the package's own node_modules instead of the root's. 14. Refactored the ci.sh script to not do a lot of the resource intensive diagnostics and the disk space optimization is also disabled by default. These things used to be very important when we were running a monolithic CI job for everything but not so much now that we have a separate job for every single package. This brings down the ci.sh execution time a few minutes at least which adds up to a lot because it is executed 40+ times for each CI run. 15. In general the important change here (apart from the yarn v3 upgrade) is the disablement of the hoisting of dependencies because this makes the build much more reliable and harder to mess up with missing dependencies. 16. Had to upgrade protoc-gen-ts to v0.8.6 which then needed typescript v4.9.x or newer which then forced me to fix a few new compiler errors (which seemed like legit bugs so it's a net positive but it's yet another big diff causing change) 17. Had to fix a few webpack import issues related to backend dependencies being used on the front-end - this only affects the example code/GUI 18. Upgraded the eslint typescript plugin and parser dependencies because the linter otherwise couldn't handle the code with the new version of the typescript compiler (v4.9.x). Also had to set the ban on explicit anys to a warn level down from an error because we have hundreds of them in the codebase. There's one linter error left that I won't fix here because there is already another PR for it. Fixes hyperledger-cacti#1142 Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
petermetz
added a commit
to petermetz/cacti
that referenced
this issue
Jul 20, 2023
We hope to solve issues with this that were plaguing contributors due to bugs in the much older Yarn 1.x versions that we had to use because Yarn V3 had it's own set of problems when it came to linking the .bin folders for child packages in a monorepo. 1. We've moved from hoisting by default to NOT hoisting by default for all packages which results in slightly slower installs for the dependencies when setting up the project build but it does have the beneficial effect of much easier fixes for compiler errors because each package can be provided with their own version of whatever dependencies they need. 2. The "pre" and "post" npm scripts are no longer supported by Yarn which was an annoying breaking change so we had to restructure some of the scripts which had pre and post scripts of their own to get equal behavior. 3. Upgraded to Yarn v3.6.0 4. packages/cactus-api-client/src/main/typescript/socketio-api-client.ts has a line where the Typescript compiler has been disabled and an issue opened for the same: https://github.com/hyperledger/cacti/issues/2523 5. Added several missing dependencies to the individual package.json files of the packages. These were resolving fine earlier because of hoisting but the problem of undeclared dependencies went undetected because of this convenience feature of the NodeJS module resolution algorithm. Now, instead of having to catch these problems in production, we can fix them up-front at development time which is great because there were dozens of them! 6. Migrated from `import` to `import type` in a few places in the code where it was applicable and useful: when you only import a dependency via `import type` then it's safe to move it to "devDependencies" because at runtime the `import type` syntax is completely omitted by the compiler. 7. Refactored the importing of "internal-ip" in a couple of test cases because the newer version does not export `v4` and `v6` but instead `internalIpV4` which meant that the >=7.0.0 versions needed downgrading. 8. Renamed 2 environment variables that are used by ci.yaml to control the steps taken by ci.sh because they had a `YARN_` prefix in their names that Yarn v3 appears to be automatically parsing and assuming to be configuration for itself - which then leads to crashes because it does not recognize the custom variable name that is not meant for it. The variables renamed are: `TOOLS_VALIDATE_BUNDLE_NAMES_DISABLED` and `CUSTOM_CHECKS_DISABLED` 9. The configure script now runs the init-registries sub-script via npm instead of yarn because yarn is unable to run scripts prior to having ran a yarn install on a fresh clone (which was making the configure script bail out on the CI) 10. Configured yarn NOT to rebuild most dependencies that have native code in them. The exception from under this is the gRPC and related dependencies whose native code extensions are being used by the test cases and so we do need them for real. 11. Updated the artillery binary path to be relatively qualified instead of just the binary file's name in the test case that does the API server benchmarking in packages/cactus-cmd-api-server/src/test/typescript/benchmark/artillery-api-benchmark.test.ts so now the binary's path is specified explicitly as ./packages/cactus-cmd-api-server/node_modules/artillery/bin/artillery which is the new path on account of us turning off t he hoisting of the dependencies entirely for the project (e.g. the binary is no longer in the rood node_modules directory) 12. Explicitly disabled the build scripts of a long list of dependencies in the root package.json file because they were greatly and needlessly slowing down the build (we know that we don't need their build scripts because it was working well with yarn v1 without them) 13. Updated the mapping for the foundry/forge dependencies because now that we don't have hoisting enabled by default for any of the packages the binaries end up in the package's own node_modules instead of the root's. 14. Refactored the ci.sh script to not do a lot of the resource intensive diagnostics and the disk space optimization is also disabled by default. These things used to be very important when we were running a monolithic CI job for everything but not so much now that we have a separate job for every single package. This brings down the ci.sh execution time a few minutes at least which adds up to a lot because it is executed 40+ times for each CI run. 15. In general the important change here (apart from the yarn v3 upgrade) is the disablement of the hoisting of dependencies because this makes the build much more reliable and harder to mess up with missing dependencies. 16. Had to upgrade protoc-gen-ts to v0.8.6 which then needed typescript v4.9.x or newer which then forced me to fix a few new compiler errors (which seemed like legit bugs so it's a net positive but it's yet another big diff causing change) 17. Had to fix a few webpack import issues related to backend dependencies being used on the front-end - this only affects the example code/GUI 18. Upgraded the eslint typescript plugin and parser dependencies because the linter otherwise couldn't handle the code with the new version of the typescript compiler (v4.9.x). Also had to set the ban on explicit anys to a warn level down from an error because we have hundreds of them in the codebase. There's one linter error left that I won't fix here because there is already another PR for it. Fixes hyperledger-cacti#1142 Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
petermetz
added a commit
to petermetz/cacti
that referenced
this issue
Jul 20, 2023
We hope to solve issues with this that were plaguing contributors due to bugs in the much older Yarn 1.x versions that we had to use because Yarn V3 had it's own set of problems when it came to linking the .bin folders for child packages in a monorepo. 1. We've moved from hoisting by default to NOT hoisting by default for all packages which results in slightly slower installs for the dependencies when setting up the project build but it does have the beneficial effect of much easier fixes for compiler errors because each package can be provided with their own version of whatever dependencies they need. 2. The "pre" and "post" npm scripts are no longer supported by Yarn which was an annoying breaking change so we had to restructure some of the scripts which had pre and post scripts of their own to get equal behavior. 3. Upgraded to Yarn v3.6.0 4. packages/cactus-api-client/src/main/typescript/socketio-api-client.ts has a line where the Typescript compiler has been disabled and an issue opened for the same: https://github.com/hyperledger/cacti/issues/2523 5. Added several missing dependencies to the individual package.json files of the packages. These were resolving fine earlier because of hoisting but the problem of undeclared dependencies went undetected because of this convenience feature of the NodeJS module resolution algorithm. Now, instead of having to catch these problems in production, we can fix them up-front at development time which is great because there were dozens of them! 6. Migrated from `import` to `import type` in a few places in the code where it was applicable and useful: when you only import a dependency via `import type` then it's safe to move it to "devDependencies" because at runtime the `import type` syntax is completely omitted by the compiler. 7. Refactored the importing of "internal-ip" in a couple of test cases because the newer version does not export `v4` and `v6` but instead `internalIpV4` which meant that the >=7.0.0 versions needed downgrading. 8. Renamed 2 environment variables that are used by ci.yaml to control the steps taken by ci.sh because they had a `YARN_` prefix in their names that Yarn v3 appears to be automatically parsing and assuming to be configuration for itself - which then leads to crashes because it does not recognize the custom variable name that is not meant for it. The variables renamed are: `TOOLS_VALIDATE_BUNDLE_NAMES_DISABLED` and `CUSTOM_CHECKS_DISABLED` 9. The configure script now runs the init-registries sub-script via npm instead of yarn because yarn is unable to run scripts prior to having ran a yarn install on a fresh clone (which was making the configure script bail out on the CI) 10. Configured yarn NOT to rebuild most dependencies that have native code in them. The exception from under this is the gRPC and related dependencies whose native code extensions are being used by the test cases and so we do need them for real. 11. Updated the artillery binary path to be relatively qualified instead of just the binary file's name in the test case that does the API server benchmarking in packages/cactus-cmd-api-server/src/test/typescript/benchmark/artillery-api-benchmark.test.ts so now the binary's path is specified explicitly as ./packages/cactus-cmd-api-server/node_modules/artillery/bin/artillery which is the new path on account of us turning off t he hoisting of the dependencies entirely for the project (e.g. the binary is no longer in the rood node_modules directory) 12. Explicitly disabled the build scripts of a long list of dependencies in the root package.json file because they were greatly and needlessly slowing down the build (we know that we don't need their build scripts because it was working well with yarn v1 without them) 13. Updated the mapping for the foundry/forge dependencies because now that we don't have hoisting enabled by default for any of the packages the binaries end up in the package's own node_modules instead of the root's. 14. Refactored the ci.sh script to not do a lot of the resource intensive diagnostics and the disk space optimization is also disabled by default. These things used to be very important when we were running a monolithic CI job for everything but not so much now that we have a separate job for every single package. This brings down the ci.sh execution time a few minutes at least which adds up to a lot because it is executed 40+ times for each CI run. 15. In general the important change here (apart from the yarn v3 upgrade) is the disablement of the hoisting of dependencies because this makes the build much more reliable and harder to mess up with missing dependencies. 16. Had to upgrade protoc-gen-ts to v0.8.6 which then needed typescript v4.9.x or newer which then forced me to fix a few new compiler errors (which seemed like legit bugs so it's a net positive but it's yet another big diff causing change) 17. Had to fix a few webpack import issues related to backend dependencies being used on the front-end - this only affects the example code/GUI 18. Upgraded the eslint typescript plugin and parser dependencies because the linter otherwise couldn't handle the code with the new version of the typescript compiler (v4.9.x). Also had to set the ban on explicit anys to a warn level down from an error because we have hundreds of them in the codebase. There's one linter error left that I won't fix here because there is already another PR for it. Fixes hyperledger-cacti#1142 Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
petermetz
added a commit
to petermetz/cacti
that referenced
this issue
Jul 20, 2023
We hope to solve issues with this that were plaguing contributors due to bugs in the much older Yarn 1.x versions that we had to use because Yarn V3 had it's own set of problems when it came to linking the .bin folders for child packages in a monorepo. 1. We've moved from hoisting by default to NOT hoisting by default for all packages which results in slightly slower installs for the dependencies when setting up the project build but it does have the beneficial effect of much easier fixes for compiler errors because each package can be provided with their own version of whatever dependencies they need. 2. The "pre" and "post" npm scripts are no longer supported by Yarn which was an annoying breaking change so we had to restructure some of the scripts which had pre and post scripts of their own to get equal behavior. 3. Upgraded to Yarn v3.6.0 4. packages/cactus-api-client/src/main/typescript/socketio-api-client.ts has a line where the Typescript compiler has been disabled and an issue opened for the same: https://github.com/hyperledger/cacti/issues/2523 5. Added several missing dependencies to the individual package.json files of the packages. These were resolving fine earlier because of hoisting but the problem of undeclared dependencies went undetected because of this convenience feature of the NodeJS module resolution algorithm. Now, instead of having to catch these problems in production, we can fix them up-front at development time which is great because there were dozens of them! 6. Migrated from `import` to `import type` in a few places in the code where it was applicable and useful: when you only import a dependency via `import type` then it's safe to move it to "devDependencies" because at runtime the `import type` syntax is completely omitted by the compiler. 7. Refactored the importing of "internal-ip" in a couple of test cases because the newer version does not export `v4` and `v6` but instead `internalIpV4` which meant that the >=7.0.0 versions needed downgrading. 8. Renamed 2 environment variables that are used by ci.yaml to control the steps taken by ci.sh because they had a `YARN_` prefix in their names that Yarn v3 appears to be automatically parsing and assuming to be configuration for itself - which then leads to crashes because it does not recognize the custom variable name that is not meant for it. The variables renamed are: `TOOLS_VALIDATE_BUNDLE_NAMES_DISABLED` and `CUSTOM_CHECKS_DISABLED` 9. The configure script now runs the init-registries sub-script via npm instead of yarn because yarn is unable to run scripts prior to having ran a yarn install on a fresh clone (which was making the configure script bail out on the CI) 10. Configured yarn NOT to rebuild most dependencies that have native code in them. The exception from under this is the gRPC and related dependencies whose native code extensions are being used by the test cases and so we do need them for real. 11. Updated the artillery binary path to be relatively qualified instead of just the binary file's name in the test case that does the API server benchmarking in packages/cactus-cmd-api-server/src/test/typescript/benchmark/artillery-api-benchmark.test.ts so now the binary's path is specified explicitly as ./packages/cactus-cmd-api-server/node_modules/artillery/bin/artillery which is the new path on account of us turning off t he hoisting of the dependencies entirely for the project (e.g. the binary is no longer in the rood node_modules directory) 12. Explicitly disabled the build scripts of a long list of dependencies in the root package.json file because they were greatly and needlessly slowing down the build (we know that we don't need their build scripts because it was working well with yarn v1 without them) 13. Updated the mapping for the foundry/forge dependencies because now that we don't have hoisting enabled by default for any of the packages the binaries end up in the package's own node_modules instead of the root's. 14. Refactored the ci.sh script to not do a lot of the resource intensive diagnostics and the disk space optimization is also disabled by default. These things used to be very important when we were running a monolithic CI job for everything but not so much now that we have a separate job for every single package. This brings down the ci.sh execution time a few minutes at least which adds up to a lot because it is executed 40+ times for each CI run. 15. In general the important change here (apart from the yarn v3 upgrade) is the disablement of the hoisting of dependencies because this makes the build much more reliable and harder to mess up with missing dependencies. 16. Had to upgrade protoc-gen-ts to v0.8.6 which then needed typescript v4.9.x or newer which then forced me to fix a few new compiler errors (which seemed like legit bugs so it's a net positive but it's yet another big diff causing change) 17. Had to fix a few webpack import issues related to backend dependencies being used on the front-end - this only affects the example code/GUI 18. Upgraded the eslint typescript plugin and parser dependencies because the linter otherwise couldn't handle the code with the new version of the typescript compiler (v4.9.x). Also had to set the ban on explicit anys to a warn level down from an error because we have hundreds of them in the codebase. There's one linter error left that I won't fix here because there is already another PR for it. Fixes hyperledger-cacti#1142 Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
petermetz
added a commit
to petermetz/cacti
that referenced
this issue
Jul 20, 2023
We hope to solve issues with this that were plaguing contributors due to bugs in the much older Yarn 1.x versions that we had to use because Yarn V3 had it's own set of problems when it came to linking the .bin folders for child packages in a monorepo. 1. We've moved from hoisting by default to NOT hoisting by default for all packages which results in slightly slower installs for the dependencies when setting up the project build but it does have the beneficial effect of much easier fixes for compiler errors because each package can be provided with their own version of whatever dependencies they need. 2. The "pre" and "post" npm scripts are no longer supported by Yarn which was an annoying breaking change so we had to restructure some of the scripts which had pre and post scripts of their own to get equal behavior. 3. Upgraded to Yarn v3.6.0 4. packages/cactus-api-client/src/main/typescript/socketio-api-client.ts has a line where the Typescript compiler has been disabled and an issue opened for the same: https://github.com/hyperledger/cacti/issues/2523 5. Added several missing dependencies to the individual package.json files of the packages. These were resolving fine earlier because of hoisting but the problem of undeclared dependencies went undetected because of this convenience feature of the NodeJS module resolution algorithm. Now, instead of having to catch these problems in production, we can fix them up-front at development time which is great because there were dozens of them! 6. Migrated from `import` to `import type` in a few places in the code where it was applicable and useful: when you only import a dependency via `import type` then it's safe to move it to "devDependencies" because at runtime the `import type` syntax is completely omitted by the compiler. 7. Refactored the importing of "internal-ip" in a couple of test cases because the newer version does not export `v4` and `v6` but instead `internalIpV4` which meant that the >=7.0.0 versions needed downgrading. 8. Renamed 2 environment variables that are used by ci.yaml to control the steps taken by ci.sh because they had a `YARN_` prefix in their names that Yarn v3 appears to be automatically parsing and assuming to be configuration for itself - which then leads to crashes because it does not recognize the custom variable name that is not meant for it. The variables renamed are: `TOOLS_VALIDATE_BUNDLE_NAMES_DISABLED` and `CUSTOM_CHECKS_DISABLED` 9. The configure script now runs the init-registries sub-script via npm instead of yarn because yarn is unable to run scripts prior to having ran a yarn install on a fresh clone (which was making the configure script bail out on the CI) 10. Configured yarn NOT to rebuild most dependencies that have native code in them. The exception from under this is the gRPC and related dependencies whose native code extensions are being used by the test cases and so we do need them for real. 11. Updated the artillery binary path to be relatively qualified instead of just the binary file's name in the test case that does the API server benchmarking in packages/cactus-cmd-api-server/src/test/typescript/benchmark/artillery-api-benchmark.test.ts so now the binary's path is specified explicitly as ./packages/cactus-cmd-api-server/node_modules/artillery/bin/artillery which is the new path on account of us turning off t he hoisting of the dependencies entirely for the project (e.g. the binary is no longer in the rood node_modules directory) 12. Explicitly disabled the build scripts of a long list of dependencies in the root package.json file because they were greatly and needlessly slowing down the build (we know that we don't need their build scripts because it was working well with yarn v1 without them) 13. Updated the mapping for the foundry/forge dependencies because now that we don't have hoisting enabled by default for any of the packages the binaries end up in the package's own node_modules instead of the root's. 14. Refactored the ci.sh script to not do a lot of the resource intensive diagnostics and the disk space optimization is also disabled by default. These things used to be very important when we were running a monolithic CI job for everything but not so much now that we have a separate job for every single package. This brings down the ci.sh execution time a few minutes at least which adds up to a lot because it is executed 40+ times for each CI run. 15. In general the important change here (apart from the yarn v3 upgrade) is the disablement of the hoisting of dependencies because this makes the build much more reliable and harder to mess up with missing dependencies. 16. Had to upgrade protoc-gen-ts to v0.8.6 which then needed typescript v4.9.x or newer which then forced me to fix a few new compiler errors (which seemed like legit bugs so it's a net positive but it's yet another big diff causing change) 17. Had to fix a few webpack import issues related to backend dependencies being used on the front-end - this only affects the example code/GUI 18. Upgraded the eslint typescript plugin and parser dependencies because the linter otherwise couldn't handle the code with the new version of the typescript compiler (v4.9.x). Also had to set the ban on explicit anys to a warn level down from an error because we have hundreds of them in the codebase. There's one linter error left that I won't fix here because there is already another PR for it. Fixes hyperledger-cacti#1142 Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
petermetz
added a commit
to petermetz/cacti
that referenced
this issue
Jul 20, 2023
We hope to solve issues with this that were plaguing contributors due to bugs in the much older Yarn 1.x versions that we had to use because Yarn V3 had it's own set of problems when it came to linking the .bin folders for child packages in a monorepo. 1. We've moved from hoisting by default to NOT hoisting by default for all packages which results in slightly slower installs for the dependencies when setting up the project build but it does have the beneficial effect of much easier fixes for compiler errors because each package can be provided with their own version of whatever dependencies they need. 2. The "pre" and "post" npm scripts are no longer supported by Yarn which was an annoying breaking change so we had to restructure some of the scripts which had pre and post scripts of their own to get equal behavior. 3. Upgraded to Yarn v3.6.0 4. packages/cactus-api-client/src/main/typescript/socketio-api-client.ts has a line where the Typescript compiler has been disabled and an issue opened for the same: https://github.com/hyperledger/cacti/issues/2523 5. Added several missing dependencies to the individual package.json files of the packages. These were resolving fine earlier because of hoisting but the problem of undeclared dependencies went undetected because of this convenience feature of the NodeJS module resolution algorithm. Now, instead of having to catch these problems in production, we can fix them up-front at development time which is great because there were dozens of them! 6. Migrated from `import` to `import type` in a few places in the code where it was applicable and useful: when you only import a dependency via `import type` then it's safe to move it to "devDependencies" because at runtime the `import type` syntax is completely omitted by the compiler. 7. Refactored the importing of "internal-ip" in a couple of test cases because the newer version does not export `v4` and `v6` but instead `internalIpV4` which meant that the >=7.0.0 versions needed downgrading. 8. Renamed 2 environment variables that are used by ci.yaml to control the steps taken by ci.sh because they had a `YARN_` prefix in their names that Yarn v3 appears to be automatically parsing and assuming to be configuration for itself - which then leads to crashes because it does not recognize the custom variable name that is not meant for it. The variables renamed are: `TOOLS_VALIDATE_BUNDLE_NAMES_DISABLED` and `CUSTOM_CHECKS_DISABLED` 9. The configure script now runs the init-registries sub-script via npm instead of yarn because yarn is unable to run scripts prior to having ran a yarn install on a fresh clone (which was making the configure script bail out on the CI) 10. Configured yarn NOT to rebuild most dependencies that have native code in them. The exception from under this is the gRPC and related dependencies whose native code extensions are being used by the test cases and so we do need them for real. 11. Updated the artillery binary path to be relatively qualified instead of just the binary file's name in the test case that does the API server benchmarking in packages/cactus-cmd-api-server/src/test/typescript/benchmark/artillery-api-benchmark.test.ts so now the binary's path is specified explicitly as ./packages/cactus-cmd-api-server/node_modules/artillery/bin/artillery which is the new path on account of us turning off t he hoisting of the dependencies entirely for the project (e.g. the binary is no longer in the rood node_modules directory) 12. Explicitly disabled the build scripts of a long list of dependencies in the root package.json file because they were greatly and needlessly slowing down the build (we know that we don't need their build scripts because it was working well with yarn v1 without them) 13. Updated the mapping for the foundry/forge dependencies because now that we don't have hoisting enabled by default for any of the packages the binaries end up in the package's own node_modules instead of the root's. 14. Refactored the ci.sh script to not do a lot of the resource intensive diagnostics and the disk space optimization is also disabled by default. These things used to be very important when we were running a monolithic CI job for everything but not so much now that we have a separate job for every single package. This brings down the ci.sh execution time a few minutes at least which adds up to a lot because it is executed 40+ times for each CI run. 15. In general the important change here (apart from the yarn v3 upgrade) is the disablement of the hoisting of dependencies because this makes the build much more reliable and harder to mess up with missing dependencies. 16. Had to upgrade protoc-gen-ts to v0.8.6 which then needed typescript v4.9.x or newer which then forced me to fix a few new compiler errors (which seemed like legit bugs so it's a net positive but it's yet another big diff causing change) 17. Had to fix a few webpack import issues related to backend dependencies being used on the front-end - this only affects the example code/GUI 18. Upgraded the eslint typescript plugin and parser dependencies because the linter otherwise couldn't handle the code with the new version of the typescript compiler (v4.9.x). Also had to set the ban on explicit anys to a warn level down from an error because we have hundreds of them in the codebase. There's one linter error left that I won't fix here because there is already another PR for it. Fixes hyperledger-cacti#1142 Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
This was referenced Jul 24, 2023
sandeepnRES
pushed a commit
to sandeepnRES/cacti
that referenced
this issue
Dec 21, 2023
We hope to solve issues with this that were plaguing contributors due to bugs in the much older Yarn 1.x versions that we had to use because Yarn V3 had it's own set of problems when it came to linking the .bin folders for child packages in a monorepo. 1. We've moved from hoisting by default to NOT hoisting by default for all packages which results in slightly slower installs for the dependencies when setting up the project build but it does have the beneficial effect of much easier fixes for compiler errors because each package can be provided with their own version of whatever dependencies they need. 2. The "pre" and "post" npm scripts are no longer supported by Yarn which was an annoying breaking change so we had to restructure some of the scripts which had pre and post scripts of their own to get equal behavior. 3. Upgraded to Yarn v3.6.0 4. packages/cactus-api-client/src/main/typescript/socketio-api-client.ts has a line where the Typescript compiler has been disabled and an issue opened for the same: https://github.com/hyperledger/cacti/issues/2523 5. Added several missing dependencies to the individual package.json files of the packages. These were resolving fine earlier because of hoisting but the problem of undeclared dependencies went undetected because of this convenience feature of the NodeJS module resolution algorithm. Now, instead of having to catch these problems in production, we can fix them up-front at development time which is great because there were dozens of them! 6. Migrated from `import` to `import type` in a few places in the code where it was applicable and useful: when you only import a dependency via `import type` then it's safe to move it to "devDependencies" because at runtime the `import type` syntax is completely omitted by the compiler. 7. Refactored the importing of "internal-ip" in a couple of test cases because the newer version does not export `v4` and `v6` but instead `internalIpV4` which meant that the >=7.0.0 versions needed downgrading. 8. Renamed 2 environment variables that are used by ci.yaml to control the steps taken by ci.sh because they had a `YARN_` prefix in their names that Yarn v3 appears to be automatically parsing and assuming to be configuration for itself - which then leads to crashes because it does not recognize the custom variable name that is not meant for it. The variables renamed are: `TOOLS_VALIDATE_BUNDLE_NAMES_DISABLED` and `CUSTOM_CHECKS_DISABLED` 9. The configure script now runs the init-registries sub-script via npm instead of yarn because yarn is unable to run scripts prior to having ran a yarn install on a fresh clone (which was making the configure script bail out on the CI) 10. Configured yarn NOT to rebuild most dependencies that have native code in them. The exception from under this is the gRPC and related dependencies whose native code extensions are being used by the test cases and so we do need them for real. 11. Updated the artillery binary path to be relatively qualified instead of just the binary file's name in the test case that does the API server benchmarking in packages/cactus-cmd-api-server/src/test/typescript/benchmark/artillery-api-benchmark.test.ts so now the binary's path is specified explicitly as ./packages/cactus-cmd-api-server/node_modules/artillery/bin/artillery which is the new path on account of us turning off t he hoisting of the dependencies entirely for the project (e.g. the binary is no longer in the rood node_modules directory) 12. Explicitly disabled the build scripts of a long list of dependencies in the root package.json file because they were greatly and needlessly slowing down the build (we know that we don't need their build scripts because it was working well with yarn v1 without them) 13. Updated the mapping for the foundry/forge dependencies because now that we don't have hoisting enabled by default for any of the packages the binaries end up in the package's own node_modules instead of the root's. 14. Refactored the ci.sh script to not do a lot of the resource intensive diagnostics and the disk space optimization is also disabled by default. These things used to be very important when we were running a monolithic CI job for everything but not so much now that we have a separate job for every single package. This brings down the ci.sh execution time a few minutes at least which adds up to a lot because it is executed 40+ times for each CI run. 15. In general the important change here (apart from the yarn v3 upgrade) is the disablement of the hoisting of dependencies because this makes the build much more reliable and harder to mess up with missing dependencies. 16. Had to upgrade protoc-gen-ts to v0.8.6 which then needed typescript v4.9.x or newer which then forced me to fix a few new compiler errors (which seemed like legit bugs so it's a net positive but it's yet another big diff causing change) 17. Had to fix a few webpack import issues related to backend dependencies being used on the front-end - this only affects the example code/GUI 18. Upgraded the eslint typescript plugin and parser dependencies because the linter otherwise couldn't handle the code with the new version of the typescript compiler (v4.9.x). Also had to set the ban on explicit anys to a warn level down from an error because we have hundreds of them in the codebase. There's one linter error left that I won't fix here because there is already another PR for it. Fixes hyperledger-cacti#1142 Signed-off-by: Peter Somogyvari <peter.somogyvari@accenture.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Description
Depends on #1141
This is an expert level good first issue, meaning that you don't need to know much about Cactus specifically, but you do need to be an expert in configuring build systems/package managers/etc.
Yarn v3 has some performance improvements (among other things) compared to Yarn v1 so we should migrate over as soon as possible (when it hits GA).
Acceptance Criteria
yarn jest some/path/to/test.case.test.tsandyarn tap --ts --timeout=9999 some/path/to/other/test-case.test.ts)yarn configureworks as expected - does not crash due to missing or unlinked binaries from the./node_modules/.bin/folderThe text was updated successfully, but these errors were encountered: