--audit-level not working, regression of #22?

[u2mejc]

First, thank you for your hard work, it's very appreciated! 🤗

Issue: We're setting up a CI env to automatically run check-audit, and we're setting the --audit-level (or alternatively the env npm_config_audit_level), but if we skip any vulnerabilities in resolve-audit, the exit code still returns a non zero for vulnerabilities below the audit-level. It seems the NPM exit code is being persisted, even though there is an existing audit-resolve.json.

It's mentioned this is supported in #22 so I'm not sure if I'm doing it wrong, but I've tried multiple times.

Expected Result: check-audit should only fail on vulnerabilities equal to or greater than the desired audit level, that do not have a current decisions in the audit-resolve.json file.

Repoduce:

% npm list -g npm-audit-resolver
/home/me/.nvm/versions/node/v20.11.0/lib
└── npm-audit-resolver@3.0.0-RC.0

% resolve-audit
>>>> npm audit --json

 There's 4 fixable vulnerabilities that running 'npm audit fix' could address.

 f) Run: npm/yarn audit fix
 s) Skip this
 q) Quit
What would you like to do? s
Selected: Skip

------------------------------------------------------
[ high ] axios  https://github.com/advisories/GHSA-8hc4-vh64-cxmj
  Server-Side Request Forgery in axios

vulnerable versions >=1.3.2 <=1.7.3 found in:
 - axios

  npm audit fix handles this

 r) remind me in 24h
 i) ignore paths
 del) Remove all listed dependency paths
 s) Skip this
 q) Quit
What would you like to do? r

------------------------------------------------------
[ low ] elliptic  https://github.com/advisories/GHSA-f7q4-pwc6-w24p
  Elliptic's EDDSA missing signature length check

vulnerable versions >=4.0.0 <=6.5.6 found in:
 - elliptic

  npm audit fix handles this

 r) remind me in 24h
 i) ignore paths
 del) Remove all listed dependency paths
 s) Skip this
 q) Quit
What would you like to do? s
Selected: Skip

------------------------------------------------------
[ low ] elliptic  https://github.com/advisories/GHSA-977x-g7h5-7qgw
  Elliptic's ECDSA missing check for whether leading bit of r and s is zero

vulnerable versions >=2.0.0 <=6.5.6 found in:
 - elliptic

  npm audit fix handles this

 r) remind me in 24h
 i) ignore paths
 del) Remove all listed dependency paths
 s) Skip this
 q) Quit
What would you like to do? s
Selected: Skip

------------------------------------------------------
[ low ] elliptic  https://github.com/advisories/GHSA-49q7-c7j4-3p7m
  Elliptic allows BER-encoded signatures

vulnerable versions >=5.2.1 <=6.5.6 found in:
 - elliptic

  npm audit fix handles this

 r) remind me in 24h
 i) ignore paths
 del) Remove all listed dependency paths
 s) Skip this
 q) Quit
What would you like to do? s
Selected: Skip

% check-audit --audit-level high
>>>> npm audit --json --audit-level high
Total of 6 actions to process
--------------------------------------------------
[low] elliptic: Elliptic's EDDSA missing signature length check (1098593)
  elliptic
--------------------------------------------------
[low] elliptic: Elliptic's ECDSA missing check for whether leading bit of r and s is zero (1098594)
  elliptic
--------------------------------------------------
[low] elliptic: Elliptic allows BER-encoded signatures (1098595)
  elliptic
--------------------------------------------------
 😱   Unresolved issues found!
--------------------------------------------------
% echo $?
1

[naugtur]

I think this might be that the support of this flag was always naive - it's being passed to npm audit and so it doesn't report stuff on that level.
I'll need to dive deeper to figure out if that idea is right.

Ping me again if I'm not reporting progress ;)

[u2mejc]

Will do, thank you for the rapid reply! 🚀

[tgauges]

It may be because the report output of npm audit is not changed by the option --audit-level, it only changes the exit code:

https://docs.npmjs.com/cli/commands/npm-audit#description

By default, the audit command will exit with a non-zero code if any vulnerability is found. It may be useful in CI environments to include the --audit-level parameter to specify the minimum vulnerability level that will cause the command to fail. This option does not filter the report output, it simply changes the command's failure threshold.

[u2mejc]

Ping me again if I'm not reporting progress ;)

@naugtur per your request, ping this issue! 😄