feat(deps): update puppeteer & typescript dependencies #862

yanivfranco · 2024-06-11T21:01:35Z

This PR is big. might be eligible for a new release.
But I think it is long overdue :)

before:

after:

In this PR:

update puppeteer to 22.5.0. major changes:
- xpath selector implementation. Affected scrapers: behatsdaa, BeyahadBishvilhaScraper, LeumiScraper - should be tested
- no page.waitForTimeout - replaced with sleep function
- Load event changed to PuppeteerLifecycleEvent
updated typescript to 4.7.4
This update had few collateral updates:
- jest to 29.7
- ts-jest to 29.1.4
- @types/jest
- babel-jest
- eslint to 8.57 & all eslint plugins & configs
updated uuid to 9.0.1
Removed pupeteer config script, providing chromium revision manually from puppeteer release notes.
For the newer puppeteer version, there is no valid way to get the relevant used chromium revision from puppeteer. (see Programmatically access compatible Chromium revision number puppeteer/puppeteer#8203 (comment)). Regardless, in my opinion, this whole script is redundant and over engineered for this case. The chromium revision is static for the specific used puppeteer version, and can be retrieved easily from the release notes. This will need to be done each time puppeteer version is updated (which seems it didn't happen for 4 years so I think it won't make much of a difference)

Tested scrapers : discount, isracard & cal.

baruchiro · 2024-06-14T14:52:24Z

I will review it deeply later, but I see a lot of changes because you ran lint --fix?

Why did you do it?

yanivfranco · 2024-06-14T19:54:11Z

I will review it deeply later, but I see a lot of changes because you ran lint --fix?

Why did you do it?

updating eslint & all eslint configs caused few linting errors across multiple files. Added the "--fix" options to fix those minor errors and pass the lint stage.

…israeli-bank-scrapers-fork into update-dependencies

esakal · 2024-06-21T19:33:32Z

Hi @yanivfranco, upgrading technology stack is usually painful and I see you had quite a few breaking changes you tackled so thank you for your contribution.

few things:

Supporting the israeli-bank-scraper-core

I see that you removed file utils/update-puppeteer-config.js but this file is needed to "burn" the desired chromium version to be used inside the library when bundling the core version. Are you aware of that? is it not needed anymore?

Node version

it works only with node v18. node v16 causes errors due to unsupported engines. I suggest to:

change the value in .nvmrc file to v18
in package.json remove engines.npm and change engines.node to >= 18.19.0

package-lock.json file sync issues

running npm ci fails due to sync issues with package.json. You should:

delete node_modules folder
run npm i
commit the changes of package-lock.json

vulnerabilities

I see 6 high severity vulnerabilities but it is much better then what we had so....

Thanks again, it is an important PR

regarding the lint, I"m ok with that, anyhow we have lint checks that prevent us from merging PRs with lint issues.

@baruchiro please review it as well.

baruchiro

Thank you, I always give up in the middle of a dependency update process 🙈

I didn't finished my review:

baruchiro · 2024-06-23T03:29:50Z

src/helpers/elements-interactions.ts

@@ -64,7 +64,7 @@ async function pageEvalAll<R>(page: Page | Frame, selector: string,
    result = await page.$$eval(selector, callback, ...args);
  } catch (e) {
    // TODO temporary workaround to puppeteer@1.5.0 which breaks $$eval bevahvior until they will release a new version.


Maybe we can open a new task to remove this part.

sure, this part is probably irrelevant now. should open an issue?

Yes, and write a note that we should start working on it only after merging this PR.

src/helpers/dates.ts

src/helpers/elements-interactions.ts

src/helpers/transactions.ts

src/scrapers/base-beinleumi-group.ts

src/scrapers/base-isracard-amex.ts

…scrapers-fork into update-dependencies

yanivfranco · 2024-06-24T22:57:16Z

@esakal Thanks, I am happy to contribute

Supporting the israeli-bank-scraper-core - From what I understand, this file is only used to get the chromium revision from the puppeteer/-core package. This is not relevant anymore because there is no programmatic way to do it with newer puppeteer. Added the chromium revision as a static string (which should take care of the core package as well... right?) - see index.ts. I made sure there are no more places referencing the deleted file, did I miss something?
Fixed node version
Fixed package-lock issue
You are right, 6 high vulnerabilities, still better than before :)

@baruchiro pushed new iteration

baruchiro · 2024-06-25T05:53:13Z

Hi everyone, we are in the middle of reviewing this, but please test it locally with your accounts to find any problem as soon as possible.

Bank	Owners
Mizrahi	@baruchiro
Cal	@baruchiro ✅ @galbarm @daniel-hauser @yanivfranco ✅
Isracard	@baruchiro ✅ @galbarm @daniel-hauser @yanivfranco ✅
Max	@baruchiro ✅ @galbarm @daniel-hauser @dimdimi4
Discount	@galbarm @yanivfranco ✅
Hapoalim	@galbarm @daniel-hauser
Amex	@daniel-hauser
Behatsdaa	@daniel-hauser
Yahav	@gczobel ✅
Mercantile	@EzzatQ @kfirarad
Leumi	@dimdimi4
Otsar Hahayal	@matanelgabsi

src/scrapers/yahav.ts

src/helpers/fetch.ts

src/scrapers/beyahad-bishvilha.ts

src/scrapers/factory.ts

src/scrapers/leumi.ts

src/scrapers/visa-cal.ts

baruchiro · 2024-06-25T16:38:23Z

src/index.ts

 export function getPuppeteerConfig() {
-  return { ...puppeteerConfig };
+  return { chromiumRevision: '1250580' }; // https://github.com/puppeteer/puppeteer/releases/tag/puppeteer-core-v22.5.0
+
 }


@esakal I reviewed everything, I left the chromium version / scrapers-core to you.

package.json

baruchiro · 2024-06-25T16:41:38Z

package.json

    "build:types": "tsc --emitDeclarationOnly",
-    "build:puppeteer-config": "ncp src/puppeteer-config.json lib/puppeteer-config.json",
    "build:js": "babel src --out-dir lib --extensions \".ts\" --source-maps inline --verbose",
-    "prebuild": "node utils/update-puppeteer-config.js",
    "postbuild": "rimraf lib/tests",


baruchiro · 2024-06-25T16:45:09Z

I finished my review, waiting for @esakal to review the israeli-bank-scrapers-core thing

gczobel · 2024-06-26T08:54:00Z

Hi,
I can confirm that yahav scraper works in the fork yanivfranco:update-dependencies
PASS src/scrapers/yahav.test.ts (9.385 s)

As a side effect, I see a lot of warnings running the test:
ts-jest[ts-jest-transformer] (WARN) Define `ts-jest` config under `globals` is deprecated. Please do transform: { <transform_regex>: ['ts-jest', { /* ts-jest config goes here in Jest */ }], },

yanivfranco · 2024-06-26T21:10:59Z

@gczobel Thanks, fixed

baruchiro · 2024-06-27T01:00:16Z

Well done!!

Looks good to me, I will talk with @esakal to check the -core thing

daniel-hauser · 2024-06-27T03:01:33Z

i tested the following scrapers and all tests pass:

hapoalim
amex
behatsdaa

When running npm audit i see that we still have some issues. this PR seems like a good place to fix it as we test all scrapers anyway.

israeli-bank-scrapers> npm audit
# npm audit report

braces  <3.0.3
Severity: high
Uncontrolled resource consumption in braces - https://github.com/advisories/GHSA-grv7-fg5c-xmjg
fix available via `npm audit fix --force`
Will install jscodeshift@0.16.1, which is a breaking change
node_modules/jscodeshift/node_modules/braces
  micromatch  0.2.0 - 3.1.10
  Depends on vulnerable versions of braces
  node_modules/jscodeshift/node_modules/micromatch
    jscodeshift  0.3.20 - 0.13.1
    Depends on vulnerable versions of micromatch
    node_modules/jscodeshift

ws  8.0.0 - 8.17.0
Severity: high
ws affected by a DoS when handling a request with many HTTP headers - https://github.com/advisories/GHSA-3h5v-q93c-6h6q
fix available via `npm audit fix --force`
Will install puppeteer@22.12.1, which is outside the stated dependency range
node_modules/ws
  puppeteer-core  11.0.0 - 22.11.1
  Depends on vulnerable versions of ws
  node_modules/puppeteer-core
    puppeteer  18.2.0 - 22.11.1
    Depends on vulnerable versions of puppeteer-core
    node_modules/puppeteer

6 high severity vulnerabilities

To address all issues (including breaking changes), run:
  npm audit fix --force

After running npm audit fix --force, all tests passed and the diff of package.json (package-lock.json as well but long) is:

diff --git a/package.json b/package.json
index 2e88e70..89fdd0a 100644
--- a/package.json
+++ b/package.json
@@ -63,7 +63,7 @@
     "fs-extra": "^10.0.0",
     "husky": "^8.0.3",
     "jest": "^29.7.0",
-    "jscodeshift": "^0.11.0",
+    "jscodeshift": "^0.16.1",
     "minimist": "^1.2.5",
     "ncp": "^2.0.0",
     "rimraf": "^3.0.0",
@@ -80,7 +80,7 @@
     "moment": "^2.22.2",
     "moment-timezone": "^0.5.37",
     "node-fetch": "^2.2.0",
-    "puppeteer": "22.5.0",
+    "puppeteer": "^22.12.1",
     "uuid": "^9.0.1"
   },
   "files": [

esakal · 2024-06-28T13:59:38Z

Hi,
Just a quick update. There is an issue with building the core version. try to run npm run prepare:core and you will get compile errors. it happens since the jscodeshift fails to run thus all imports remain to puppeteer instead of being replaced to puppeteer-core.

It happens since babel which uses browserlist throw

Error [BrowserslistError]: Unknown browser query `var updateDb = require('update-browserslist-db')`. Maybe you are using old Browserslist or made typo in query.

I didn't yet figure out how to satisfy browserlist, if you have any idea please share.

yanivfranco · 2024-06-28T17:52:53Z

@daniel-hauser Thanks for the response. I updated jscodeshift to the latest version, however there is an issue with the latest puppeteer version though. The binary still does not exists in the storage.googleapi repository, which breaks download-chromium lib which is recommend in the docs. So i'll leave it for now, can open a task to do it once the revision is available to download.

@esakal I will try to take a look. Can you maybe elaborate why jscodeshift is running in a child process and not as part of node current context?

yanivfranco · 2024-06-28T18:15:34Z

@esakal I got the prepare:core script working again. Changed jscodeshift to run in the same process instead of a child process. let me know if there are any problems.

yanivfranco · 2024-07-08T21:17:34Z

@esakal @baruchiro Any update?

baruchiro · 2024-07-09T00:15:56Z

@esakal @baruchiro Any update?

I think the only thing left is the -core version.

I tried to test it with the npm link and use it in Caspion, but for that, I need to upgrade the NodeJS version in my project and so on...

I hope I will test it in a new fork of Caspion soon.

baruchiro

I tested the israeli-bank-scrapers-core version by running the prepare:core command locally and reference the source as a package in another project.

It works.

Approved!!!

esakal

@yanivfranco thank you! this will have a great impact on the project. Nice job with the code mod library

baruchiro · 2024-07-11T04:08:44Z

@esakal @eshaham I think it must be a BREAKING CHANGES version.

First, it doesn't support NodeJS v16 anymore, as I tried to test it in Caspion.

And all the massive upgrades are significant.

github-actions · 2024-07-11T05:05:19Z

🎉 This PR is included in version 5.0.0 🎉

The release is available on GitHub release

Your semantic-release bot 📦🚀

Yaniv Franco added 4 commits June 10, 2024 01:22

update dependencies

3dbd35d

fix chromium revision

baa1281

rollback puppeteer and revision

8572a86

remove puppeteer config

d36695f

yanivfranco mentioned this pull request Jun 11, 2024

Dependencies versions & security vulnerabilities #855

Open

Merge branch 'master' into update-dependencies

472d95e

yanivfranco added 2 commits June 15, 2024 01:44

fix getPuppeteerConfig result type

fc78842

Merge branch 'update-dependencies' of https://github.com/yanivfranco/…

e736dec

…israeli-bank-scrapers-fork into update-dependencies

baruchiro requested changes Jun 24, 2024

View reviewed changes

yanivfranco added 4 commits June 25, 2024 01:42

fix pr comments

e7793a5

Merge branch 'master' of https://github.com/yanivfranco/israeli-bank-…

0b4a2a4

…scrapers-fork into update-dependencies

package lock merge

49ba3fe

eslint type import fixes

07351cb

baruchiro requested changes Jun 25, 2024

View reviewed changes

fix pr comments

ab531f2

fix ts-jest warning

7e648e7

yanivfranco changed the title ~~Update puppeteer & typescript dependencies~~ feat: Update puppeteer & typescript dependencies Jun 26, 2024

yanivfranco changed the title ~~feat: Update puppeteer & typescript dependencies~~ feat(deps): Update puppeteer & typescript dependencies Jun 26, 2024

fix jscodeshift issue

1aad15b

yanivfranco changed the title ~~feat(deps): Update puppeteer & typescript dependencies~~ feat(deps): update puppeteer & typescript dependencies Jul 1, 2024

Merge remote-tracking branch 'origin/master' into pr/yanivfranco/862

cda66f1

baruchiro approved these changes Jul 9, 2024

View reviewed changes

baruchiro mentioned this pull request Jul 10, 2024

chromium is opening but not doing anything #873

Closed

esakal approved these changes Jul 10, 2024

View reviewed changes

esakal merged commit de0e614 into eshaham:master Jul 11, 2024
6 checks passed

github-actions bot added the released label Jul 11, 2024

baruchiro mentioned this pull request Jul 11, 2024

ci: upgrade workflows to use node 18 #875

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(deps): update puppeteer & typescript dependencies #862

feat(deps): update puppeteer & typescript dependencies #862

yanivfranco commented Jun 11, 2024 •

edited by baruchiro

Loading

baruchiro commented Jun 14, 2024

yanivfranco commented Jun 14, 2024

esakal commented Jun 21, 2024

baruchiro left a comment

baruchiro Jun 23, 2024

yanivfranco Jun 24, 2024

baruchiro Jun 25, 2024

yanivfranco commented Jun 24, 2024 •

edited

Loading

baruchiro commented Jun 25, 2024 •

edited

Loading

baruchiro Jun 25, 2024

baruchiro Jun 25, 2024

baruchiro commented Jun 25, 2024

gczobel commented Jun 26, 2024

yanivfranco commented Jun 26, 2024

baruchiro commented Jun 27, 2024

daniel-hauser commented Jun 27, 2024

esakal commented Jun 28, 2024 •

edited

Loading

yanivfranco commented Jun 28, 2024 •

edited

Loading

yanivfranco commented Jun 28, 2024 •

edited

Loading

yanivfranco commented Jul 8, 2024

baruchiro commented Jul 9, 2024

baruchiro left a comment

esakal left a comment

baruchiro commented Jul 11, 2024

github-actions bot commented Jul 11, 2024

feat(deps): update puppeteer & typescript dependencies #862

feat(deps): update puppeteer & typescript dependencies #862

Conversation

yanivfranco commented Jun 11, 2024 • edited by baruchiro Loading

baruchiro commented Jun 14, 2024

yanivfranco commented Jun 14, 2024

esakal commented Jun 21, 2024

Supporting the israeli-bank-scraper-core

Node version

package-lock.json file sync issues

vulnerabilities

baruchiro left a comment

Choose a reason for hiding this comment

baruchiro Jun 23, 2024

Choose a reason for hiding this comment

yanivfranco Jun 24, 2024

Choose a reason for hiding this comment

baruchiro Jun 25, 2024

Choose a reason for hiding this comment

yanivfranco commented Jun 24, 2024 • edited Loading

baruchiro commented Jun 25, 2024 • edited Loading

baruchiro Jun 25, 2024

Choose a reason for hiding this comment

baruchiro Jun 25, 2024

Choose a reason for hiding this comment

baruchiro commented Jun 25, 2024

gczobel commented Jun 26, 2024

yanivfranco commented Jun 26, 2024

baruchiro commented Jun 27, 2024

daniel-hauser commented Jun 27, 2024

esakal commented Jun 28, 2024 • edited Loading

yanivfranco commented Jun 28, 2024 • edited Loading

yanivfranco commented Jun 28, 2024 • edited Loading

yanivfranco commented Jul 8, 2024

baruchiro commented Jul 9, 2024

baruchiro left a comment

Choose a reason for hiding this comment

esakal left a comment

Choose a reason for hiding this comment

baruchiro commented Jul 11, 2024

github-actions bot commented Jul 11, 2024

yanivfranco commented Jun 11, 2024 •

edited by baruchiro

Loading

yanivfranco commented Jun 24, 2024 •

edited

Loading

baruchiro commented Jun 25, 2024 •

edited

Loading

esakal commented Jun 28, 2024 •

edited

Loading

yanivfranco commented Jun 28, 2024 •

edited

Loading

yanivfranco commented Jun 28, 2024 •

edited

Loading