Queued scan results in passed test (no waiting until it starts)

[wawaqa]

If a scan is queued, sec-tester doesn't wait until it starts, gets done or finds a vulnerability, but simply lets the test pass.

Steps to reproduce:

1. Get the API key for the organization with engine count=1.
2. Make >1 test files with jest (e.g. 3)
3. Run tests with default jest options (several test files are run in parallel in jest)
4. Observe the results.

Actual: tests with scans of queued status passed, although the scan didn't even start.
Expected: sec-tester-js waits until the scan is done

Looks like the problem is here: https://github.com/NeuraLegion/sec-tester-js/blob/master/packages/scan/src/Scan.ts
private readonly ACTIVE_STATUSES doesn't contain queued, so we don't poll such scans

[derevnjuk]

@wawaqa queued scan can be considered as active. There is no reason to proceed with it at all since it might take days to start such a scan.

[wawaqa]

@derevnjuk - right, we need to think how to handle this situation. Actually, any of our scans can take days. :)
IMHO the correct behavior here would be failing by timeout or smth like that - this way user would know that there were no asserted results from the scan.

[derevnjuk]

There are just two options that are worth to be discussed:

1. Fail the test by timeout.
2. Fail the test immediately to show a message that the user should upgrade his plan to have the ability to run parallel tests.

@ArtLinkov @bararchy @wawaqa fyi

[derevnjuk]

We agreed that in order to avoid “making assumptions” about the user's preferred CI behavior we should not use the exception by default. Instead, we will show log messages that will inform them of the situation and hold until there is a free engine, as follows:

When the scan is queued, log: The maximum amount of concurrent scans has been reached for the organization, the execution will resume once a free engine will be available. If you want to increase the execution concurrency, please upgrade your subscription or contact your system administrator

When the engine is available, log: Connected to the engine, resuming execution