Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Validate that directory does not contain - #572

Merged
merged 2 commits into from
Jul 3, 2020
Merged

Validate that directory does not contain - #572

merged 2 commits into from
Jul 3, 2020

Conversation

ruflin
Copy link
Member

@ruflin ruflin commented Jun 30, 2020 •
edited
Loading

As we try to ensure no directories contain -, we can also check for this. This adds a check.

The additional NewPackage method with assets was removed and merged into NewPackage for now. This means NewPackage is a bit more heavy but we have all the validation in place. It also means, the memory consumption to keep all packages is a bit higher. For the number of packages we have today, this should be ok.

Instead I plan a future refactoring that serves the /search endpoint only from the info of the BasePackage. So not all package information with the assets has to be kept in memory.

This removes all the - assets in the packages.

@ruflin ruflin self-assigned this Jun 30, 2020
@ruflin ruflin mentioned this pull request Jun 30, 2020
Closed
29 tasks
@elasticmachine
Copy link

elasticmachine commented Jun 30, 2020 •
edited
Loading

💔 Tests Failed

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Branch indexing]

  • Start Time: 2020-07-03T08:01:18.150+0000

  • Duration: 8 min 14 sec

Test stats 🧪

Test Results
Failed 18
Passed 145
Skipped 0
Total 163

Test errors

Expand to view the tests failures

  • Name: Test / TestPackageIndex – package-registry

    • Age: 12
    • Duration: 0.01
    • Error Details: Failed

  • Name: Test / TestPackageIndex//package/example/1.0.0/ – package-registry

    • Age: 12
    • Duration: 0.01
    • Error Details: Failed

  • Name: Test / TestAllPackageIndex – package-registry

    • Age: 12
    • Duration: 0.06
    • Error Details: Failed

  • Name: Test / TestAllPackageIndex/base/0.2.0 – package-registry

    • Age: 12
    • Duration: 0
    • Error Details: Failed

  • Name: Test / TestAllPackageIndex/default_pipeline/0.0.2 – package-registry

    • Age: 12
    • Duration: 0
    • Error Details: Failed

  • Name: Test / TestAllPackageIndex/example/0.0.2 – package-registry

    • Age: 12
    • Duration: 0
    • Error Details: Failed

  • Name: Test / TestAllPackageIndex/example/1.0.0 – package-registry

    • Age: 12
    • Duration: 0.01
    • Error Details: Failed

  • Name: Test / TestAllPackageIndex/multiple_false/0.0.1 – package-registry

    • Age: 12
    • Duration: 0
    • Error Details: Failed

  • Name: Test / TestAllPackageIndex/yamlpipeline/1.0.0 – package-registry

    • Age: 12
    • Duration: 0.02
    • Error Details: Failed

  • Name: TestIntegration / TestPackageIndex – package-registry

    • Age: 12
    • Duration: 0.01
    • Error Details: Failed

  • Name: TestIntegration / TestPackageIndex//package/example/1.0.0/ – package-registry

    • Age: 12
    • Duration: 0.01
    • Error Details: Failed

  • Name: TestIntegration / TestAllPackageIndex – package-registry

    • Age: 12
    • Duration: 0.04
    • Error Details: Failed

  • Name: TestIntegration / TestAllPackageIndex/base/0.2.0 – package-registry

    • Age: 12
    • Duration: 0
    • Error Details: Failed

  • Name: TestIntegration / TestAllPackageIndex/default_pipeline/0.0.2 – package-registry

    • Age: 12
    • Duration: 0
    • Error Details: Failed

  • Name: TestIntegration / TestAllPackageIndex/example/0.0.2 – package-registry

    • Age: 12
    • Duration: 0
    • Error Details: Failed

  • Name: TestIntegration / TestAllPackageIndex/example/1.0.0 – package-registry

    • Age: 12
    • Duration: 0.01
    • Error Details: Failed

  • Name: TestIntegration / TestAllPackageIndex/multiple_false/0.0.1 – package-registry

    • Age: 12
    • Duration: 0
    • Error Details: Failed

  • Name: TestIntegration / TestAllPackageIndex/yamlpipeline/1.0.0 – package-registry

    • Age: 12
    • Duration: 0.01
    • Error Details: Failed

Log output

Expand to view the last 100 lines of log output 

[2020-07-03T08:09:15.206Z] Step 2/19 : FROM golang:${GO_VERSION} AS builder
[2020-07-03T08:09:15.206Z]  ---> 2421885b04da
[2020-07-03T08:09:15.206Z] Step 3/19 : ENV GO111MODULE=on
[2020-07-03T08:09:15.206Z]  ---> Using cache
[2020-07-03T08:09:15.206Z]  ---> 1f0cf3d53d83
[2020-07-03T08:09:15.206Z] Step 4/19 : COPY ./ /package-registry
[2020-07-03T08:09:15.806Z]  ---> 45d30ed9d1ac
[2020-07-03T08:09:15.806Z] Step 5/19 : WORKDIR /package-registry
[2020-07-03T08:09:16.085Z]  ---> Running in fa528ce13911
[2020-07-03T08:09:16.086Z] Removing intermediate container fa528ce13911
[2020-07-03T08:09:16.086Z]  ---> 596865516e36
[2020-07-03T08:09:16.086Z] Step 6/19 : RUN go build .
[2020-07-03T08:09:16.351Z]  ---> Running in 2a58a63df626
[2020-07-03T08:09:16.925Z] go: downloading gopkg.in/yaml.v2 v2.2.8
[2020-07-03T08:09:16.925Z] go: downloading github.com/Masterminds/semver/v3 v3.1.0
[2020-07-03T08:09:16.925Z] go: downloading github.com/gorilla/mux v1.7.4
[2020-07-03T08:09:16.925Z] go: downloading github.com/radovskyb/watcher v1.0.7
[2020-07-03T08:09:16.925Z] go: downloading github.com/aymerick/raymond v2.0.2+incompatible
[2020-07-03T08:09:16.925Z] go: downloading github.com/elastic/go-ucfg v0.8.4-0.20200415140258-1232bd4774a6
[2020-07-03T08:09:16.925Z] go: downloading github.com/pkg/errors v0.9.1
[2020-07-03T08:09:16.926Z] go: downloading github.com/joeshaw/multierror v0.0.0-20140124173710-69b34d4ec901
[2020-07-03T08:09:21.197Z] Removing intermediate container 2a58a63df626
[2020-07-03T08:09:21.197Z]  ---> e8470e3dbfd1
[2020-07-03T08:09:21.197Z] Step 7/19 : FROM centos:7
[2020-07-03T08:09:21.197Z]  ---> b5b4d78bc90c
[2020-07-03T08:09:21.197Z] Step 8/19 : RUN yum install -y zip rsync && yum clean all
[2020-07-03T08:09:21.197Z]  ---> Using cache
[2020-07-03T08:09:21.197Z]  ---> e42f56f81e7c
[2020-07-03T08:09:21.197Z] Step 9/19 : COPY --from=builder /package-registry/package-registry /package-registry/package-registry
[2020-07-03T08:09:21.197Z]  ---> Using cache
[2020-07-03T08:09:21.197Z]  ---> cf7d0d9af5db
[2020-07-03T08:09:21.197Z] Step 10/19 : WORKDIR /package-registry
[2020-07-03T08:09:21.197Z]  ---> Using cache
[2020-07-03T08:09:21.197Z]  ---> 3d988467c62e
[2020-07-03T08:09:21.197Z] Step 11/19 : COPY config.docker.yml /package-registry/config.yml
[2020-07-03T08:09:21.197Z]  ---> 19229b23bad7
[2020-07-03T08:09:21.197Z] Step 12/19 : EXPOSE 8080
[2020-07-03T08:09:21.197Z]  ---> Running in c1eabea83fcd
[2020-07-03T08:09:21.197Z] Removing intermediate container c1eabea83fcd
[2020-07-03T08:09:21.197Z]  ---> 5cf17b1c6f41
[2020-07-03T08:09:21.197Z] Step 13/19 : ENTRYPOINT ["./package-registry"]
[2020-07-03T08:09:21.197Z]  ---> Running in 265940b90cd2
[2020-07-03T08:09:21.463Z] Removing intermediate container 265940b90cd2
[2020-07-03T08:09:21.463Z]  ---> 472b08bbde1c
[2020-07-03T08:09:21.463Z] Step 14/19 : CMD ["--address=0.0.0.0:8080"]
[2020-07-03T08:09:21.463Z]  ---> Running in 46f3de4b3f73
[2020-07-03T08:09:21.725Z] Removing intermediate container 46f3de4b3f73
[2020-07-03T08:09:21.726Z]  ---> 3cc9a2520781
[2020-07-03T08:09:21.726Z] Step 15/19 : HEALTHCHECK --interval=1s --retries=30 CMD curl --silent --fail localhost:8080/health || exit 1
[2020-07-03T08:09:21.726Z]  ---> Running in a0cbebc5fca6
[2020-07-03T08:09:21.992Z] Removing intermediate container a0cbebc5fca6
[2020-07-03T08:09:21.992Z]  ---> b45c60f67acf
[2020-07-03T08:09:21.992Z] Step 16/19 : LABEL BRANCH_NAME=PR-572
[2020-07-03T08:09:21.992Z]  ---> Running in a22f3195757a
[2020-07-03T08:09:22.253Z] Removing intermediate container a22f3195757a
[2020-07-03T08:09:22.253Z]  ---> afd1f84ae529
[2020-07-03T08:09:22.253Z] Step 17/19 : LABEL GIT_SHA=5bf6104468ff4f08439655ba20eea1c1cc2f923c
[2020-07-03T08:09:22.253Z]  ---> Running in 518254c41cfa
[2020-07-03T08:09:22.253Z] Removing intermediate container 518254c41cfa
[2020-07-03T08:09:22.253Z]  ---> 51b94e8c0b1f
[2020-07-03T08:09:22.253Z] Step 18/19 : LABEL GO_VERSION=1.13.0
[2020-07-03T08:09:22.518Z]  ---> Running in d44cca9a7e3a
[2020-07-03T08:09:22.518Z] Removing intermediate container d44cca9a7e3a
[2020-07-03T08:09:22.518Z]  ---> 69db0dbb3437
[2020-07-03T08:09:22.518Z] Step 19/19 : LABEL TIMESTAMP=2020-07-03_08:09
[2020-07-03T08:09:22.518Z]  ---> Running in 1b43da3fa6fd
[2020-07-03T08:09:22.791Z] Removing intermediate container 1b43da3fa6fd
[2020-07-03T08:09:22.791Z]  ---> 8abd0621b1f3
[2020-07-03T08:09:22.791Z] Successfully built 8abd0621b1f3
[2020-07-03T08:09:22.791Z] Successfully tagged docker.elastic.co/observability-ci/package-registry:5bf6104468ff4f08439655ba20eea1c1cc2f923c
[2020-07-03T08:09:23.158Z] + docker push docker.elastic.co/observability-ci/package-registry:5bf6104468ff4f08439655ba20eea1c1cc2f923c
[2020-07-03T08:09:23.158Z] The push refers to repository [docker.elastic.co/observability-ci/package-registry]
[2020-07-03T08:09:23.158Z] 538443ead664: Preparing
[2020-07-03T08:09:23.158Z] 5389b0f5ba5e: Preparing
[2020-07-03T08:09:23.158Z] 62df25568c70: Preparing
[2020-07-03T08:09:23.158Z] edf3aa290fb3: Preparing
[2020-07-03T08:09:23.424Z] edf3aa290fb3: Layer already exists
[2020-07-03T08:09:25.371Z] 538443ead664: Pushed
[2020-07-03T08:09:26.347Z] 5389b0f5ba5e: Pushed
[2020-07-03T08:09:26.935Z] 62df25568c70: Pushed
[2020-07-03T08:09:29.499Z] 5bf6104468ff4f08439655ba20eea1c1cc2f923c: digest: sha256:55adfe3508be8ee3c233a567d851c378fc200b2dc268498e7ec61b713e714c7e size: 1158
[2020-07-03T08:09:29.821Z] + docker tag docker.elastic.co/observability-ci/package-registry:5bf6104468ff4f08439655ba20eea1c1cc2f923c docker.elastic.co/observability-ci/package-registry:PR-572
[2020-07-03T08:09:30.137Z] + docker push docker.elastic.co/observability-ci/package-registry:PR-572
[2020-07-03T08:09:30.137Z] The push refers to repository [docker.elastic.co/observability-ci/package-registry]
[2020-07-03T08:09:30.137Z] 538443ead664: Preparing
[2020-07-03T08:09:30.137Z] 5389b0f5ba5e: Preparing
[2020-07-03T08:09:30.137Z] 62df25568c70: Preparing
[2020-07-03T08:09:30.137Z] edf3aa290fb3: Preparing
[2020-07-03T08:09:30.398Z] edf3aa290fb3: Layer already exists
[2020-07-03T08:09:30.399Z] 5389b0f5ba5e: Layer already exists
[2020-07-03T08:09:30.399Z] 538443ead664: Layer already exists
[2020-07-03T08:09:30.399Z] 62df25568c70: Layer already exists
[2020-07-03T08:09:30.974Z] PR-572: digest: sha256:55adfe3508be8ee3c233a567d851c378fc200b2dc268498e7ec61b713e714c7e size: 1158
[2020-07-03T08:09:31.467Z] Running on Jenkins in /var/lib/jenkins/workspace/Beats_package-registry_PR-572
[2020-07-03T08:09:31.504Z] [INFO] getVaultSecret: Getting secrets
[2020-07-03T08:09:31.606Z] Masking supported pattern matches of $VAULT_ADDR or $VAULT_ROLE_ID or $VAULT_SECRET_ID
[2020-07-03T08:09:32.241Z] + chmod 755 generate-build-data.sh
[2020-07-03T08:09:32.241Z] + ./generate-build-data.sh https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/package-registry/PR-572/ https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/package-registry/PR-572/runs/12 UNSTABLE 493825
[2020-07-03T08:09:32.241Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/package-registry/PR-572/runs/12/steps/?limit=10000 -o steps-info.json
[2020-07-03T08:09:32.492Z] INFO: curl https://beats-ci.elastic.co/blue/rest/organizations/jenkins/pipelines/Beats/package-registry/PR-572/runs/12/tests/?status=FAILED -o tests-errors.json

@ruflin
Validate that directory do not contain -
0d856cd 
As we try to ensure no directories contain -, we can also check for this. This adds a check.

The additional NewPackage method with assets was removed and merged into NewPackage for now. This means NewPackage is a bit more heavy but we have all the validation in place. It also means, the memory consumption to keep all packages is a bit higher. For the number of packages we have today, this should be ok.

Instead I plan a future refactoring that serves the /search endpoint only from the info of the BasePackage. So not all package information with the assets has to be kept in memory.

This removes all the - assets in the packages.
@ruflin ruflin marked this pull request as ready for review July 3, 2020 08:27
@ruflin ruflin requested a review from mtojek July 3, 2020 08:27
mtojek
mtojek approved these changes Jul 3, 2020
View reviewed changes
Copy link
Contributor

@mtojek mtojek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: does not contain (commit message)

@ruflin ruflin merged commit a4caba7 into elastic:master Jul 3, 2020
@ruflin ruflin changed the title Validate that directory do not contain - Validate that directory does not contain - Jul 3, 2020
@ruflin
Copy link
Member Author

ruflin commented Jul 3, 2020

Fixed commit message and merged ;-)

@ruflin ruflin deleted the validate-directory-names branch July 3, 2020 08:52
This was referenced Jul 14, 2020
Closed
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mtojek mtojek mtojek approved these changes

Assignees

@ruflin ruflin

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@ruflin @elasticmachine @mtojek