[Security Solution][Endpoint] Cypress test improvements to capture Agent diagnostics file when test fails #202965

paul-tavares · 2024-12-04T15:30:55Z

Summary

the Cypress parallel runner was updated to set tooling logging level first from Env. variables before falling back to the value defined in the Cypress configuration file
- The env. value to set, if wanting to enable a specific logging level, is TOOLING_LOG_LEVEL. The values supported are the same as those used with ToolingLog (here): silent, error, warning, success, info, debug, verbose
- This change makes it easier to run Cypress tests locally with (for example) a logging level of verbose for our tooling without having to modify the Cypress configuration file. Example: export TOOLING_LOG_LEVEL=verbose && yarn cypress:dw:open
Added two new methods to our scripting VM service clients (for Vagrant and Multipass):
- download: allow you to pull files out of the VM and save them locally
- upload: uploads a local file to the VM. (upload already existed as transfer - which has now been marked as deprecated).
Added new service function on our Fleet scripting module to enable us to set the logging level on a Fleet Agent
- Cypress tests were adjusted to automatically set the agent logging to debug when running in CI
A new Cypress task that allows for an Agent Diagnostic file (which includes the Endpoint Log) to be retrieved from the host VM and stored with the CI job (under the artifacts tab)
- A few tests were updated to include this step for failed test

Screen capture showing Agent Diagnostics zip file captured for failing test:

…the runtime env. first before attempting to use the setting from the cypress config

…gin level

…the agent

paul-tavares · 2024-12-04T15:31:04Z

/ci

paul-tavares · 2024-12-04T21:31:43Z

/ci

…mprovements

elasticmachine · 2024-12-05T14:30:09Z

Pinging @elastic/security-defend-workflows (Team:Defend Workflows)

maximpn

@paul-tavares Thanks for improving tests logging 🙏 I reviewed changed to x-pack/plugins/security_solution/scripts/run_cypress/parallel.ts and left comments.

maximpn · 2024-12-10T15:18:36Z

x-pack/plugins/security_solution/scripts/run_cypress/parallel.ts

@@ -54,8 +54,6 @@ export const cli = () => {
        )
        .boolean('inspect');

-      const USE_CHROME_BETA = process.env.USE_CHROME_BETA?.match(/(1|true)/i);


Isn't an ability to test in Chrome beta version required anymore?

oh.... no no. this is mistake. I did not meant to remove this. Will add it back. Thanks for catching it.

maximpn · 2024-12-10T15:23:52Z

x-pack/plugins/security_solution/scripts/run_cypress/parallel.ts

-        createToolingLogger.defaultLogLevel = cypressConfigFile.env.TOOLING_LOG_LEVEL;
+      // Adjust tooling log level based on the `TOOLING_LOG_LEVEL` property, which can be
+      // defined in the cypress config file or set in the `env`
+      {


It might be useful to have the same functionality for Serverless i.e. the same changes are required in x-pack/plugins/security_solution/scripts/run_cypress/parallel_serverless.ts.

Have you considered creation a separate file/function like createLogger() reading env variable under the hood?

Ohhhh. Did not realize there was a serverless parallel runner. I will break this out into a utility and will call it from both parallel runners. Thanks for the feedback

maximpn · 2024-12-10T15:26:33Z

x-pack/plugins/security_solution/scripts/run_cypress/parallel.ts

+
+        if (logLevel) {
+          _cliLogger.info(`Setting tooling log level to [${logLevel}]`);
+          createToolingLogger.defaultLogLevel = logLevel;


Is it expected behavior to affect all the instances created below in the execution flow?

As a safer option it might better to create a ToolingLog instance and pass it to prefixedOutputLogger.

yeah... its expected. The goal is to ensure that all intances of ToolingLog (created by createToolingLogger()) uses by default the log level set via the env. variable or the cypress config file.

…also use it in the `parallel_serverless` runner

…mprovements

maximpn · 2024-12-10T19:06:58Z

x-pack/plugins/security_solution/scripts/run_cypress/utils.ts

+ * It will first check the NodeJs `process.env` to see if an Environment Variable was set
+ * and then, if provided, it will use the value defined in the Cypress Config. file.
+ */
+export const setDefaultToolingLoggingLevel = (cypressConfigFile?: any) => {


There is no point using any type here. cypressConfigFile in upstream parallel and parallel_serverless files already has type any. We could provide a more specific type for cypressConfigFile here like

interface CypressConfigFile { env?: Record<string, string>; }

I used any here because that matches the type I am seeing for the cypressConfigFile in the parallel modules:

And I set this type here to be any more specific I'll likely have to cast the value when I attempt to pass it to this function. I'll instead change the signature of the function to not require the entire config file and instead just pass in the value for TOOLING_LOG_LEVEL instead.

elasticmachine · 2024-12-10T21:05:15Z

💚 Build Succeeded

Buildkite Build
Commit: 0b2fd24

Metrics [docs]

Unknown metric groups

ESLint disabled line counts

id	before	after	diff
`securitySolution`	560	561	+1

Total ESLint disabled count

id	before	after	diff
`securitySolution`	644	645	+1

History

💚 Build #258745 succeeded 9f95a40
💔 Build #258500 failed 6d7244b
💔 Build #258461 failed e221246
💛 Build #257471 was flaky fb1d2d2
💔 Build #257134 failed 69e024c
💔 Build #257010 failed be1b754

cc @paul-tavares

maximpn

@paul-tavares Thanks for addressing my comments 🙏

Changes to x-pack/plugins/security_solution/scripts/run_cypress/parallel.ts and x-pack/plugins/security_solution/scripts/run_cypress/parallel_serverless.ts LGTM

maximpn · 2024-12-11T11:08:27Z

x-pack/plugins/security_solution/scripts/run_cypress/utils.ts

+    process.env.TOOLING_LOG_LEVEL ||
+    process.env.CYPRESS_TOOLING_LOG_LEVEL ||
+    defaultFallbackLoggingLevel ||
+    '';


nit: It looks like empty string isn't necessary here since if condition below is gonna check for all "falsable" values.

kibanamachine · 2024-12-12T15:21:44Z

Starting backport for target branches: 8.x

https://github.com/elastic/kibana/actions/runs/12299431737

kibanamachine · 2024-12-12T15:30:11Z

💔 All backports failed

Status	Branch	Result
❌	8.x	Backport failed because of merge conflicts

Manual backport

To create the backport manually run:

node scripts/backport --pr 202965

Questions ?

Please refer to the Backport tool documentation

kibanamachine · 2024-12-13T15:49:10Z

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add a backport:* label or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 202965 locally

kibanamachine · 2024-12-16T16:48:45Z

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create automatically backports add a backport:* label or prevent reminders by adding the backport:skip label.
You can also create backports manually by running node scripts/backport --pr 202965 locally

paul-tavares · 2024-12-16T20:55:08Z

💚 All backports created successfully

Status	Branch	Result
✅	8.x

Note: Successful backport PRs will be merged automatically after passing CI.

Questions ?

Please refer to the Backport tool documentation

…ent diagnostics file when test fails (elastic#202965) ## Summary - the Cypress `parallel` runner was updated to set tooling logging level first from Env. variables before falling back to the value defined in the Cypress configuration file - The env. value to set, if wanting to enable a specific logging level, is `TOOLING_LOG_LEVEL`. The values supported are the same as those used with `ToolingLog` ([here](https://github.com/elastic/kibana/blob/b6287708f687d4e3288851052c0c6ae4ade8ce60/packages/kbn-tooling-log/src/log_levels.ts#L10)): `silent`, `error`, `warning`, `success`, `info`, `debug`, `verbose` - This change makes it easier to run Cypress tests locally with (for example) a logging level of `verbose` for our tooling without having to modify the Cypress configuration file. Example: `export TOOLING_LOG_LEVEL=verbose && yarn cypress:dw:open` - Added two new methods to our scripting VM service clients (for Vagrant and Multipass): - `download`: allow you to pull files out of the VM and save them locally - `upload`: uploads a local file to the VM. (upload already existed as `transfer` - which has now been marked as deprecated). - Added new service function on our Fleet scripting module to enable us to set the logging level on a Fleet Agent - Cypress tests were adjusted to automatically set the agent logging to debug when running in CI - A new Cypress task that allows for an Agent Diagnostic file (which includes the Endpoint Log) to be retrieved from the host VM and stored with the CI job (under the artifacts tab) - A few tests were updated to include this step for failed test (cherry picked from commit 2ab8a5c) # Conflicts: # x-pack/plugins/security_solution/scripts/endpoint/common/fleet_services.ts

…ure Agent diagnostics file when test fails (#202965) (#204485) # Backport This will backport the following commits from `main` to `8.x`: - [[Security Solution][Endpoint] Cypress test improvements to capture Agent diagnostics file when test fails (#202965)](#202965)  ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport)

…ent diagnostics file when test fails (elastic#202965) ## Summary - the Cypress `parallel` runner was updated to set tooling logging level first from Env. variables before falling back to the value defined in the Cypress configuration file - The env. value to set, if wanting to enable a specific logging level, is `TOOLING_LOG_LEVEL`. The values supported are the same as those used with `ToolingLog` ([here](https://github.com/elastic/kibana/blob/b6287708f687d4e3288851052c0c6ae4ade8ce60/packages/kbn-tooling-log/src/log_levels.ts#L10)): `silent`, `error`, `warning`, `success`, `info`, `debug`, `verbose` - This change makes it easier to run Cypress tests locally with (for example) a logging level of `verbose` for our tooling without having to modify the Cypress configuration file. Example: `export TOOLING_LOG_LEVEL=verbose && yarn cypress:dw:open` - Added two new methods to our scripting VM service clients (for Vagrant and Multipass): - `download`: allow you to pull files out of the VM and save them locally - `upload`: uploads a local file to the VM. (upload already existed as `transfer` - which has now been marked as deprecated). - Added new service function on our Fleet scripting module to enable us to set the logging level on a Fleet Agent - Cypress tests were adjusted to automatically set the agent logging to debug when running in CI - A new Cypress task that allows for an Agent Diagnostic file (which includes the Endpoint Log) to be retrieved from the host VM and stored with the CI job (under the artifacts tab) - A few tests were updated to include this step for failed test

paul-tavares added 7 commits December 3, 2024 16:00

Enhance Cypress "parallel" runner to pickup tooling debug level from …

367f798

…the runtime env. first before attempting to use the setting from the cypress config

Added additional methods to Fleet scripting services to set Agent log…

ed0c28d

…gin level

Added download() and upload() to VM services

7e0013f

Change Cypress create host VM method to also enable debug logging on …

0904273

…the agent

Add code to capture version of fleet-server standalone (serverless)

43a3ebd

Added captureHostVmAgentDiagnostics cy task

050584a

Added captureHostVmAgentDiagnostics to file operations cy test file

be1b754

paul-tavares added release_note:skip Skip the PR/issue when compiling release notes Team:Defend Workflows “EDR Workflows” sub-team of Security Solution backport:prev-minor Backport to (9.0) the previous minor version (i.e. one version back from main) v8.18.0 labels Dec 4, 2024

paul-tavares self-assigned this Dec 4, 2024

paul-tavares added 2 commits December 4, 2024 11:07

Allow captureHostVmAgentDiagnostics() to get a fileNamePrefix on input

944c302

Add fleet server version to log message on success

69e024c

paul-tavares added 2 commits December 5, 2024 09:29

Remove dev test code

43e7b2e

Merge remote-tracking branch 'upstream/main' into task/cypress-test-i…

fb1d2d2

…mprovements

paul-tavares marked this pull request as ready for review December 5, 2024 14:30

paul-tavares requested review from a team, MadameSheema, patrykkopycinski, maximpn and banderror as code owners December 5, 2024 14:30

paul-tavares requested a review from ashokaditya December 5, 2024 14:30

paul-tavares added 3 commits December 9, 2024 14:01

Merge branch 'main' into task/cypress-test-improvements

e221246

Merge branch 'main' into task/cypress-test-improvements

6d7244b

Merge branch 'main' into task/cypress-test-improvements

9f95a40

maximpn reviewed Dec 10, 2024

View reviewed changes

paul-tavares added 2 commits December 10, 2024 11:16

Refactor the setting of Tooling debug in cypress parallel runner and …

19b608f

…also use it in the `parallel_serverless` runner

Merge remote-tracking branch 'upstream/main' into task/cypress-test-i…

06ead4a

…mprovements

paul-tavares requested a review from maximpn December 10, 2024 16:17

maximpn reviewed Dec 10, 2024

View reviewed changes

Adjust function signature

0b2fd24

paul-tavares requested a review from maximpn December 10, 2024 19:32

maximpn approved these changes Dec 11, 2024

View reviewed changes

ashokaditya approved these changes Dec 12, 2024

View reviewed changes

paul-tavares merged commit 2ab8a5c into elastic:main Dec 12, 2024
8 checks passed

paul-tavares deleted the task/cypress-test-improvements branch December 12, 2024 15:21

kibanamachine added the v9.0.0 label Dec 12, 2024

kibanamachine added the backport missing Added to PRs automatically when the are determined to be missing a backport. label Dec 13, 2024

paul-tavares mentioned this pull request Dec 16, 2024

[8.x] [Security Solution][Endpoint] Cypress test improvements to capture Agent diagnostics file when test fails (#202965) #204485

Merged

kibanamachine removed the backport missing Added to PRs automatically when the are determined to be missing a backport. label Dec 17, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Security Solution][Endpoint] Cypress test improvements to capture Agent diagnostics file when test fails #202965

[Security Solution][Endpoint] Cypress test improvements to capture Agent diagnostics file when test fails #202965

paul-tavares commented Dec 4, 2024 •

edited by kibanamachine

Loading

paul-tavares commented Dec 4, 2024

paul-tavares commented Dec 4, 2024

elasticmachine commented Dec 5, 2024

maximpn left a comment

maximpn Dec 10, 2024

paul-tavares Dec 10, 2024

maximpn Dec 10, 2024

paul-tavares Dec 10, 2024

maximpn Dec 10, 2024

paul-tavares Dec 10, 2024

maximpn Dec 10, 2024 •

edited

Loading

paul-tavares Dec 10, 2024

elasticmachine commented Dec 10, 2024

ESLint disabled line counts

Total ESLint disabled count

maximpn left a comment

maximpn Dec 11, 2024

kibanamachine commented Dec 12, 2024

kibanamachine commented Dec 12, 2024

kibanamachine commented Dec 13, 2024

kibanamachine commented Dec 16, 2024

paul-tavares commented Dec 16, 2024

[Security Solution][Endpoint] Cypress test improvements to capture Agent diagnostics file when test fails #202965

[Security Solution][Endpoint] Cypress test improvements to capture Agent diagnostics file when test fails #202965

Conversation

paul-tavares commented Dec 4, 2024 • edited by kibanamachine Loading

Summary

paul-tavares commented Dec 4, 2024

paul-tavares commented Dec 4, 2024

elasticmachine commented Dec 5, 2024

maximpn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

maximpn Dec 10, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

elasticmachine commented Dec 10, 2024

💚 Build Succeeded

Metrics [docs]

ESLint disabled line counts

Total ESLint disabled count

History

maximpn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kibanamachine commented Dec 12, 2024

kibanamachine commented Dec 12, 2024

💔 All backports failed

Manual backport

Questions ?

kibanamachine commented Dec 13, 2024

kibanamachine commented Dec 16, 2024

paul-tavares commented Dec 16, 2024

💚 All backports created successfully

Questions ?

paul-tavares commented Dec 4, 2024 •

edited by kibanamachine

Loading

maximpn Dec 10, 2024 •

edited

Loading