Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add e2e tests #172

Closed
wants to merge 15 commits into from
Closed

feat: add e2e tests #172

wants to merge 15 commits into from

Conversation

alegrey91
Copy link

@alegrey91 alegrey91 commented Jan 11, 2024

User description

Overview

This PR provides the implementation of an e2e test suite using ginkgo.

Additional Information

This PR provides test to verify generation of SBOMSPDXv2p3Filtered resource in the cluster.

How to Test

make e2e/v1.27.3 (you can chose whatever version of Kubernetes you prefer)

Checklist before requesting a review

  • My code follows the style guidelines of this project
  • I have commented on my code, particularly in hard-to-understand areas
  • I have performed a self-review of my code
  • If it is a core feature, I have added thorough tests.
  • New and existing unit tests pass locally with my changes

Type

Tests


Description

This PR introduces end-to-end (e2e) tests for the node-agent component. The main changes include:

  • Addition of utility functions for creating, deleting, and checking the status of Kubernetes Pods and Custom Resources, as well as waiting for them to be ready.
  • Implementation of an e2e test for generating a filtered Software Bill of Materials (SBOM). The test creates a Pod and a Custom Resource, and checks if a filtered SBOM is generated within 2 minutes.
  • Setup of the test suite for the e2e tests, including functions for setting up and tearing down the test environment.
  • Update of the go.mod file to include the dependencies required for the e2e tests, such as ginkgo for BDD testing and gomega for matchers and assertions.
  • Update of the Makefile to include commands for running the e2e tests.
  • Addition of a README.md file providing documentation for the e2e tests.

Changes walkthrough

Relevant files                                                                                                                                 
Tests
e2e_utils.go                                                                                               
    e2e/e2e_utils.go

    This file contains utility functions for the e2e tests. It
    includes functions for creating, deleting, and checking the
    status of Kubernetes Pods and Custom Resources. It also
    includes functions for waiting for a Pod or Custom Resource
    to be ready.

+152/-0
generate_filtered_sbom_test.go                                                           
    e2e/generate_filtered_sbom_test.go

    This file contains the actual e2e test for generating a
    filtered SBOM (Software Bill of Materials). It tests the
    creation of a Pod and a Custom Resource, and checks if a
    filtered SBOM is generated within 2 minutes.

+104/-0
e2e_suite_test.go                                                                                     
    e2e/e2e_suite_test.go

    This file sets up the test suite for the e2e tests. It
    includes functions for setting up and tearing down the test
    environment, which involves creating a Kubernetes cluster
    and ensuring all necessary Pods are running.

+62/-0
Dependencies
go.mod                                                                                                           
    go.mod

    This file has been updated to include the dependencies
    required for the e2e tests, such as ginkgo for BDD testing
    and gomega for matchers and assertions.

+7/-1
Configuration changes
Makefile                                                                                                       
    Makefile

    The Makefile has been updated to include commands for
    running the e2e tests. This includes creating a KinD
    cluster, installing dependencies, loading the Docker image
    into the cluster, running the tests, and cleaning up the
    environment.

+62/-2
Documentation
README.md                                                                                                     
    e2e/README.md

    This file provides documentation for the e2e tests. It
    includes information on the prerequisites for running the
    tests, and instructions on how to run the tests and clean up
    the environment.

+24/-0

✨ Usage guide:

Overview:
The describe tool scans the PR code changes, and generates a description for the PR - title, type, summary, walkthrough and labels. The tool can be triggered automatically every time a new PR is opened, or can be invoked manually by commenting on a PR.

When commenting, to edit configurations related to the describe tool (pr_description section), use the following template:

/describe --pr_description.some_config1=... --pr_description.some_config2=...

With a configuration file, use the following template:

[pr_description]
some_config1=...
some_config2=...
Enabling\disabling automation
  • When you first install the app, the default mode for the describe tool is:
pr_commands = ["/describe --pr_description.add_original_user_description=true" 
                         "--pr_description.keep_original_user_title=true", ...]

meaning the describe tool will run automatically on every PR, will keep the original title, and will add the original user description above the generated description.

  • Markers are an alternative way to control the generated description, to give maximal control to the user. If you set:
pr_commands = ["/describe --pr_description.use_description_markers=true", ...]

the tool will replace every marker of the form pr_agent:marker_name in the PR description with the relevant content, where marker_name is one of the following:

  • type: the PR type.
  • summary: the PR summary.
  • walkthrough: the PR walkthrough.

Note that when markers are enabled, if the original PR description does not contain any markers, the tool will not alter the description at all.

Custom labels

The default labels of the describe tool are quite generic: [Bug fix, Tests, Enhancement, Documentation, Other].

If you specify custom labels in the repo's labels page or via configuration file, you can get tailored labels for your use cases.
Examples for custom labels:

  • Main topic:performance - pr_agent:The main topic of this PR is performance
  • New endpoint - pr_agent:A new endpoint was added in this PR
  • SQL query - pr_agent:A new SQL query was added in this PR
  • Dockerfile changes - pr_agent:The PR contains changes in the Dockerfile
  • ...

The list above is eclectic, and aims to give an idea of different possibilities. Define custom labels that are relevant for your repo and use cases.
Note that Labels are not mutually exclusive, so you can add multiple label categories.
Make sure to provide proper title, and a detailed and well-phrased description for each label, so the tool will know when to suggest it.

Utilizing extra instructions

The describe tool can be configured with extra instructions, to guide the model to a feedback tailored to the needs of your project.

Be specific, clear, and concise in the instructions. With extra instructions, you are the prompter. Notice that the general structure of the description is fixed, and cannot be changed. Extra instructions can change the content or style of each sub-section of the PR description.

Examples for extra instructions:

[pr_description] 
extra_instructions="""
- The PR title should be in the format: '<PR type>: <title>'
- The title should be short and concise (up to 10 words)
- ...
"""

Use triple quotes to write multi-line instructions. Use bullet points to make the instructions more readable.

More PR-Agent commands

To invoke the PR-Agent, add a comment using one of the following commands:

  • /review: Request a review of your Pull Request.
  • /describe: Update the PR title and description based on the contents of the PR.
  • /improve [--extended]: Suggest code improvements. Extended mode provides a higher quality feedback.
  • /ask <QUESTION>: Ask a question about the PR.
  • /update_changelog: Update the changelog based on the PR's contents.
  • /add_docs 💎: Generate docstring for new components introduced in the PR.
  • /generate_labels 💎: Generate labels for the PR based on the PR's contents.
  • /analyze 💎: Automatically analyzes the PR, and presents changes walkthrough for each component.

See the tools guide for more details.
To list the possible configuration parameters, add a /config comment.

See the describe usage page for a comprehensive guide on using this tool.

@alegrey91 alegrey91 requested a review from matthyx January 11, 2024 14:22
@alegrey91 alegrey91 marked this pull request as ready for review January 11, 2024 14:34
Copy link

PR Description updated to latest commit (dec4418)

matthyx
matthyx previously approved these changes Jan 15, 2024
@matthyx
Copy link
Contributor

matthyx commented Jan 15, 2024

@dwertent pls approve, I cannot alone since I contributed

Copy link

Summary:

  • License scan: failure
  • Credentials scan: success
  • Vulnerabilities scan: success
  • Unit test: success
  • Go linting: failure

@matthyx matthyx force-pushed the test/add-e2e-tests branch from 9437788 to 46b4628 Compare January 19, 2024 20:44
Copy link

Summary:

  • License scan: failure
  • Credentials scan: success
  • Vulnerabilities scan: failure
  • Unit test: success
  • Go linting: failure

matthyx
matthyx previously approved these changes Jan 22, 2024
@matthyx matthyx force-pushed the test/add-e2e-tests branch from 46b4628 to 5ca4b63 Compare January 24, 2024 11:13
Copy link

Summary:

  • License scan: failure
  • Credentials scan: success
  • Vulnerabilities scan: failure
  • Unit test: success
  • Go linting: failure

@matthyx
Copy link
Contributor

matthyx commented Jan 24, 2024

@dwertent we should merge this

@alegrey91
Copy link
Author

Any news about the PR? @matthyx

@matthyx matthyx force-pushed the test/add-e2e-tests branch 2 times, most recently from 9c1b759 to 17e1c01 Compare February 16, 2024 16:23
Copy link

Summary:

  • License scan: failure
  • Credentials scan: failure
  • Vulnerabilities scan: failure
  • Unit test: success
  • Go linting: failure

@matthyx matthyx force-pushed the test/add-e2e-tests branch from 17e1c01 to 1894025 Compare March 9, 2024 20:37
@matthyx
Copy link
Contributor

matthyx commented Mar 9, 2024

@dwertent DNS test fails because of test.com - the fix is in my other PR, I'll rebase this one afterwards

Signed-off-by: Alessio Greggi <ale_grey_91@hotmail.it>
Signed-off-by: Alessio Greggi <ale_grey_91@hotmail.it>
Signed-off-by: Alessio Greggi <ale_grey_91@hotmail.it>
Signed-off-by: Alessio Greggi <ale_grey_91@hotmail.it>
Signed-off-by: Alessio Greggi <ale_grey_91@hotmail.it>
Signed-off-by: Alessio Greggi <ale_grey_91@hotmail.it>
Signed-off-by: Alessio Greggi <ale_grey_91@hotmail.it>
alegrey91 and others added 8 commits March 11, 2024 07:41
Signed-off-by: Alessio Greggi <ale_grey_91@hotmail.it>
Signed-off-by: Alessio Greggi <ale_grey_91@hotmail.it>
Signed-off-by: Alessio Greggi <ale_grey_91@hotmail.it>
Signed-off-by: Alessio Greggi <ale_grey_91@hotmail.it>
Signed-off-by: Alessio Greggi <ale_grey_91@hotmail.it>
Signed-off-by: Alessio Greggi <ale_grey_91@hotmail.it>
Signed-off-by: Matthias Bertschy <matthias.bertschy@gmail.com>
Signed-off-by: Matthias Bertschy <matthias.bertschy@gmail.com>
@matthyx matthyx force-pushed the test/add-e2e-tests branch from 1894025 to 272057b Compare March 11, 2024 06:41
Copy link

Summary:

  • License scan: failure
  • Credentials scan: failure
  • Vulnerabilities scan: failure
  • Unit test: success
  • Go linting: failure

Copy link

@dwertent dwertent left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see the tests are using SBOMSPDXv2p3Filtered, but the node-agent no longer creates SBOMs of that kind, we work now with syftsbom instead

@matthyx matthyx closed this Aug 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

3 participants