Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(puppeteer): Cleanup node_modules #1497

Merged
merged 3 commits into from
Jul 24, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions puppeteer/agama-integration-tests
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,12 @@ MOCHA_OPTIONS=(--bail --slow 10000)

if [ -e "$MYDIR/../.git/" ]; then
npm install --omit=optional

# do the same node_modules cleanup as in the RPM package to have the very same
# environment and have consistent results
"$MYDIR/node-prune.sh"
"$MYDIR/node-puppeteer-prune.sh"

npx mocha "${MOCHA_OPTIONS[@]}" "$@"
else
# set the default load path
Expand Down
159 changes: 159 additions & 0 deletions puppeteer/node-prune.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,159 @@
#! /bin/bash

# This script clean up the node_modules directory which is usually huge and
# contains a lot of not needed files. It was inspired by the node-prune tool
# (https://github.com/tj/node-prune).
#
# Usage:
#
# node-prune.sh [path]
#
# The optional [path] argument is a path to the node_modules directory, if it is
# not specified it uses node_modules in the current directory.
#
# This is a generic tool, you might run it against any node_modules directory,
# not only in Agama Puppeteer tests.

MODULES_PATH="${1:-./node_modules}"

# The list of names/patterns comes from
# https://github.com/tj/node-prune/blob/master/internal/prune/prune.go

# files to delete
FILES=(
Jenkinsfile
Makefile
Gulpfile.js
Gruntfile.js
gulpfile.js
.DS_Store
.tern-project
.gitattributes
.editorconfig
.eslintrc
eslint
.eslintrc.js
.eslintrc.json
.eslintrc.yml
.eslintignore
.stylelintrc
stylelint.config.js
.stylelintrc.json
.stylelintrc.yaml
.stylelintrc.yml
.stylelintrc.js
.htmllintrc
htmllint.js
.lint
.npmrc
.npmignore
.jshintrc
.flowconfig
.documentup.json
.yarn-metadata.json
.travis.yml
appveyor.yml
.gitlab-ci.yml
circle.yml
.coveralls.yml
CHANGES
changelog
# keep the package licenses, it's unclear if we can legally delete them...
# LICENSE.txt
# LICENSE
# LICENSE-MIT
# LICENSE.BSD
# license
# LICENCE.txt
# LICENCE
# LICENCE-MIT
# LICENCE.BSD
# licence
AUTHORS
CONTRIBUTORS
.yarn-integrity
.yarnclean
_config.yml
.babelrc
.yo-rc.json
jest.config.js
karma.conf.js
wallaby.js
wallaby.conf.js
.prettierrc
.prettierrc.yml
.prettierrc.toml
.prettierrc.js
.prettierrc.json
prettier.config.js
.appveyor.yml
tsconfig.json
tslint.json
)

# directories to delete
DIRECTORIES=(
test
tests
powered-test
docs
doc
.idea
.vscode
website
images
assets
example
examples
coverage
.nyc_output
.circleci
.github
)

# delete files with specific extensions
EXTENSIONS=(
markdown
md
mkd
ts
jst
coffee
tgz
swp
)

# delete additional files with specific extensions (not deleted by the original
# node-prune tool)
EXTRA_EXTENSIONS=(
# The map files take almost half of the node_modules content! An they would be
# useful only for reporting bugs in Puppeteer itself or in some dependent
# library.
map
)

echo -n "Before cleanup: "
du -h -s "$MODULES_PATH" | cut -f1

# delete files
for F in "${FILES[@]}"; do
find "$MODULES_PATH" -type f -name "$F" -delete
done

# delete directories recursively
for D in "${DIRECTORIES[@]}"; do
find "$MODULES_PATH" -type d -name "$D" -prune -exec rm -rf \{\} \;
done

# delete files with specific extenstions
for E in "${EXTENSIONS[@]}"; do
find "$MODULES_PATH" -type f -name "*.$E" -delete
done

# delete additional files with extensions
for EE in "${EXTRA_EXTENSIONS[@]}"; do
find "$MODULES_PATH" -type f -name "*.$EE" -delete
done

echo -n "After cleanup: "
du -h -s "$MODULES_PATH" | cut -f1
9 changes: 9 additions & 0 deletions puppeteer/node-puppeteer-prune.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
#! /bin/sh

# This is a helper script which deletes some not needed files from NPM packages.
# This script is specific for Puppeteer installations.

MODULES_PATH="${1:-./node_modules}"

# delete Puppeteer CommonJS modules, we use the ES modules (in lib/esm)
rm -rf "$MODULES_PATH/puppeteer-core/lib/cjs"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I kind of expect that it will run internally that node-prune.sh..so just one command will be enough to cleanup pupeteer modules.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both commands are started in the agama-integration-test scripts which installs the NPM packages and runs the tests. Normally you should not need to run them separately.

6 changes: 6 additions & 0 deletions puppeteer/package/agama-integration-tests.changes
Original file line number Diff line number Diff line change
@@ -1,3 +1,9 @@
-------------------------------------------------------------------
Wed Jul 24 15:03:25 UTC 2024 - Ladislav Slezák <lslezak@suse.com>

- Reduce the node_modules size, delete not needed files
(gh#openSUSE/agama#1497)

-------------------------------------------------------------------
Fri Jul 19 10:18:37 UTC 2024 - Ladislav Slezák <lslezak@suse.com>

Expand Down
6 changes: 6 additions & 0 deletions puppeteer/package/agama-integration-tests.spec
Original file line number Diff line number Diff line change
Expand Up @@ -47,6 +47,12 @@ outside.
rm -f package-lock.json
local-npm-registry %{_sourcedir} install --omit=optional --with=dev --legacy-peer-deps || ( find ~/.npm/_logs -name '*-debug.log' -print0 | xargs -0 cat; false)

# node_modules cleanup
%{_builddir}/agama/node-prune.sh
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should we run it also for agama web directory?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agama-web uses webpack which already bundles only the needed files into the target JS file. It would not help there.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And we do obfuscation and minification as well...


# extra cleanup for the Puppeteer NPM packages
%{_builddir}/agama/node-puppeteer-prune.sh

%install
install -D -d -m 0755 %{buildroot}%{_datadir}/agama/integration-tests
cp -aR node_modules %{buildroot}%{_datadir}/agama/integration-tests
Expand Down