Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ci] Favor direct mage invocation on CI #19960

Merged
merged 3 commits into from
Jul 17, 2020
Merged

[ci] Favor direct mage invocation on CI #19960

merged 3 commits into from
Jul 17, 2020

Conversation

andrewkroh
Copy link
Member

@andrewkroh andrewkroh commented Jul 15, 2020 •
edited
Loading

What does this PR do?

Favor direct mage invocation on CI.

This changes Jenkins and Travis to directly invoke mage where
possible instead of going through make. Some of the remaining
make don't yet have a mage equivalant (mainly crosscompile).

For Packetbeat this add Jenkins stages to test on darwin and Windows.

There were a few fixes I had to make related to these changes:

  • Add some mage target dependencies to ensure fields and dashboards
    are ready when tests use them.
  • Swap the order of the Go imports for dev-tools/mage/targets/integtest
    and unittest so that unit tests run before integ tests when running
    the 'mage test' target.
  • chown the shared Python venv that is in the root of the repo after
    Dockerized integ tests exit to ensure there are no permissions issues
    caused by root owned files.
  • I found a few Python string encoding issues that caused tests failures.
    I thought we fixed these during the Python 3 conversion, but something
    here exposed a few that we didn't address.

One thing of note that I did not correct. Journalbeat has system tests
but they are not executed. The existing/old Makefile has SYSTEM_TESTS=false
so they are not executed. So when I switched it to mage I left it as
'mage goUnitTest' to avoid the Python when failed when I tried it.

Why is it important?

We want to remove the older Makefile logic so we need to migrate towards mage.

Checklist

  • My code follows the style guidelines of this project
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have made corresponding change to the default configuration files
  • I have added tests that prove my fix is effective or that my feature works
  • I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

@andrewkroh andrewkroh added the ci label Jul 15, 2020
@botelastic botelastic bot added the needs_team Indicates that the issue/PR needs a Team:* label label Jul 15, 2020
@andresrc andresrc added the Team:Automation Label for the Observability productivity team label Jul 16, 2020
@botelastic botelastic bot removed the needs_team Indicates that the issue/PR needs a Team:* label label Jul 16, 2020
@andrewkroh andrewkroh force-pushed the feature/jenkinsfile-favor-mage branch from 0ff075e to 339030a Compare July 16, 2020 19:26
@andrewkroh
Favor direct mage invocation on CI
cce8284 
This changes Jenkins and Travis to directly invoke mage where
possible instead of going through make. Some of the remaining
make don't yet have a mage equivalant (mainly crosscompile).

For Packetbeat this add Jenkins stages to test on darwin and Windows.

There were a few fixes I had to make related to these changes:

- Add some mage target dependencies to ensure fields and dashboards
  are ready when tests use them.
- Swap the order of the Go imports for dev-tools/mage/targets/integtest
  and unittest so that unit tests run before integ tests when running
  the 'mage test' target.
- chown the shared Python venv that is in the root of the repo after
  Dockerized integ tests exit to ensure there are no permissions issues
  caused by root owned files.
- I found a few Python string encoding issues that caused tests failures.
  I thought we fixed these during the Python 3 conversion, but something
  here exposed a few that we didn't address.

One thing of note that I did not correct. Journalbeat has system tests
but they are not executed. The existing/old Makefile has SYSTEM_TESTS=false
so they are not executed. So when I switched it to mage I left it as
'mage goUnitTest' to avoid the Python when failed when I tried it.
@andrewkroh andrewkroh force-pushed the feature/jenkinsfile-favor-mage branch from ada07f4 to cce8284 Compare July 17, 2020 00:22
@andrewkroh andrewkroh changed the title [ci] Use mageTarget instead of makeTarget [ci] Favor direct mage invocation on CI Jul 17, 2020
@andrewkroh andrewkroh marked this pull request as ready for review July 17, 2020 02:22
@andrewkroh andrewkroh requested review from a team as code owners July 17, 2020 02:22
@andrewkroh
Copy link
Member Author

run tests

cachedout
cachedout reviewed Jul 17, 2020
View reviewed changes
Jenkinsfile Outdated Show resolved Hide resolved
@andrewkroh
Copy link
Member Author

The failures in the build are caused by Kafka download failures while building the Docker image.

@andrewkroh andrewkroh merged commit 4a1f800 into elastic:master Jul 17, 2020
@andrewkroh andrewkroh mentioned this pull request Jul 17, 2020
Merged
6 tasks
@andrewkroh andrewkroh mentioned this pull request Jul 17, 2020
Closed
v1v added a commit to v1v/beats that referenced this pull request Jul 20, 2020
@v1v
Merge remote-tracking branch 'upstream/master' into feature/fix-warni…
cf807c2 
…ngs-archive

* upstream/master: (274 commits)
  Test export commands in all Beats (elastic#20016)
  [Ingest Manager] Allow using drop path for agent build (elastic#20019)
  [DOCS] Remove include for deleted file in monitoring docs (elastic#20038)
  Removing xpack.monitoring.* settings (elastic#18608)
  [Heartbeat] Add service_name option for APM integration (elastic#19932)
  [Elastic Agent] Fix merging of fleet.yml. Add --staging to enroll cmd. (elastic#20026)
  Stricter stalebot configuration for PRs (elastic#20004)
  [ci] Favor direct mage invocation on CI (elastic#19960)
  Add cloudwatch input into Filebeat configure inputs documentation (elastic#19973)
  [Filebeat] remove delimiter \n from log line in s3 input (elastic#19972)
  [Metricbeat] Update MySQL dashboard (elastic#19913)
  Packetbeat process monitor: Ignore missing /proc/net/tcp6 (elastic#19945)
  [CI] fix MODULE variable cornercases (elastic#19985)
  Ignore timestamp in fortinet/clientendpoint and netscout/sightline (elastic#19998)
  add 7.9 to .backportrc.json (elastic#19952)
  Update internal links to external (elastic#19947)
  Remove Dynamic Script Compilations warning in Cisco module (elastic#19840)
  [Elastic Agent] Fix RPM and DEB packaging for Elastic Agent (elastic#19959)
  [Ingest Manager] Do not compare err with custom type (elastic#19980)
  Fix nanocore sum for non default intervals on Kubernetes Overview Dashboard (elastic#19675)
  ...
v1v added a commit to v1v/beats that referenced this pull request Jul 20, 2020
@v1v
Merge remote-tracking branch 'upstream/master' into feature/ci-read-o…
a6c5555 
…nly-mod

* upstream/master: (75 commits)
  Test export commands in all Beats (elastic#20016)
  [Ingest Manager] Allow using drop path for agent build (elastic#20019)
  [DOCS] Remove include for deleted file in monitoring docs (elastic#20038)
  Removing xpack.monitoring.* settings (elastic#18608)
  [Heartbeat] Add service_name option for APM integration (elastic#19932)
  [Elastic Agent] Fix merging of fleet.yml. Add --staging to enroll cmd. (elastic#20026)
  Stricter stalebot configuration for PRs (elastic#20004)
  [ci] Favor direct mage invocation on CI (elastic#19960)
  Add cloudwatch input into Filebeat configure inputs documentation (elastic#19973)
  [Filebeat] remove delimiter \n from log line in s3 input (elastic#19972)
  [Metricbeat] Update MySQL dashboard (elastic#19913)
  Packetbeat process monitor: Ignore missing /proc/net/tcp6 (elastic#19945)
  [CI] fix MODULE variable cornercases (elastic#19985)
  Ignore timestamp in fortinet/clientendpoint and netscout/sightline (elastic#19998)
  add 7.9 to .backportrc.json (elastic#19952)
  Update internal links to external (elastic#19947)
  Remove Dynamic Script Compilations warning in Cisco module (elastic#19840)
  [Elastic Agent] Fix RPM and DEB packaging for Elastic Agent (elastic#19959)
  [Ingest Manager] Do not compare err with custom type (elastic#19980)
  Fix nanocore sum for non default intervals on Kubernetes Overview Dashboard (elastic#19675)
  ...
@v1v v1v mentioned this pull request Jul 22, 2020
Merged
v1v pushed a commit to v1v/beats that referenced this pull request Jul 22, 2020
@andrewkroh @v1v
[ci] Favor direct mage invocation on CI (elastic#19960)
244f01f 
This changes Jenkins and Travis to directly invoke mage where
possible instead of going through make. Some of the remaining
make don't yet have a mage equivalant (mainly crosscompile).

For Packetbeat this add Jenkins stages to test on darwin and Windows.

There were a few fixes I had to make related to these changes:

- Add some mage target dependencies to ensure fields and dashboards
  are ready when tests use them.
- Swap the order of the Go imports for dev-tools/mage/targets/integtest
  and unittest so that unit tests run before integ tests when running
  the 'mage test' target.
- chown the shared Python venv that is in the root of the repo after
  Dockerized integ tests exit to ensure there are no permissions issues
  caused by root owned files.
- I found a few Python string encoding issues that caused tests failures.
  I thought we fixed these during the Python 3 conversion, but something
  here exposed a few that we didn't address.

One thing of note that I did not correct. Journalbeat has system tests
but they are not executed. The existing/old Makefile has SYSTEM_TESTS=false
so they are not executed. So when I switched it to mage I left it as
'mage goUnitTest' to avoid the Python when failed when I tried it.
andrewkroh added a commit to andrewkroh/beats that referenced this pull request Jul 28, 2020
@andrewkroh
[ci] Favor direct mage invocation on CI (elastic#19960)
9876b78 
This changes Jenkins and Travis to directly invoke mage where
possible instead of going through make. Some of the remaining
make don't yet have a mage equivalant (mainly crosscompile).

For Packetbeat this add Jenkins stages to test on darwin and Windows.

There were a few fixes I had to make related to these changes:

- Add some mage target dependencies to ensure fields and dashboards
  are ready when tests use them.
- Swap the order of the Go imports for dev-tools/mage/targets/integtest
  and unittest so that unit tests run before integ tests when running
  the 'mage test' target.
- chown the shared Python venv that is in the root of the repo after
  Dockerized integ tests exit to ensure there are no permissions issues
  caused by root owned files.
- I found a few Python string encoding issues that caused tests failures.
  I thought we fixed these during the Python 3 conversion, but something
  here exposed a few that we didn't address.

One thing of note that I did not correct. Journalbeat has system tests
but they are not executed. The existing/old Makefile has SYSTEM_TESTS=false
so they are not executed. So when I switched it to mage I left it as
'mage goUnitTest' to avoid the Python when failed when I tried it.

(cherry picked from commit 4a1f800)
andrewkroh added a commit that referenced this pull request Jul 29, 2020
@andrewkroh
Cherry-pick #19960 to 7.x: [ci] Favor direct mage invocation on CI (#…
4c6dba2 
…20030)

* [ci] Favor direct mage invocation on CI (#19960)

This changes Jenkins and Travis to directly invoke mage where
possible instead of going through make. Some of the remaining
make don't yet have a mage equivalant (mainly crosscompile).

For Packetbeat this add Jenkins stages to test on darwin and Windows.

There were a few fixes I had to make related to these changes:

- Add some mage target dependencies to ensure fields and dashboards
  are ready when tests use them.
- Swap the order of the Go imports for dev-tools/mage/targets/integtest
  and unittest so that unit tests run before integ tests when running
  the 'mage test' target.
- chown the shared Python venv that is in the root of the repo after
  Dockerized integ tests exit to ensure there are no permissions issues
  caused by root owned files.
- I found a few Python string encoding issues that caused tests failures.
  I thought we fixed these during the Python 3 conversion, but something
  here exposed a few that we didn't address.

One thing of note that I did not correct. Journalbeat has system tests
but they are not executed. The existing/old Makefile has SYSTEM_TESTS=false
so they are not executed. So when I switched it to mage I left it as
'mage goUnitTest' to avoid the Python when failed when I tried it.

(cherry picked from commit 4a1f800)
v1v added a commit that referenced this pull request Jul 30, 2020
@v1v
[ci] Favor direct mage invocation on CI (#19960) (#20153)
af1ff5e
melchiormoulin pushed a commit to melchiormoulin/beats that referenced this pull request Oct 14, 2020
@andrewkroh @melchiormoulin
[ci] Favor direct mage invocation on CI (elastic#19960)
c16edba 
This changes Jenkins and Travis to directly invoke mage where
possible instead of going through make. Some of the remaining
make don't yet have a mage equivalant (mainly crosscompile).

For Packetbeat this add Jenkins stages to test on darwin and Windows.

There were a few fixes I had to make related to these changes:

- Add some mage target dependencies to ensure fields and dashboards
  are ready when tests use them.
- Swap the order of the Go imports for dev-tools/mage/targets/integtest
  and unittest so that unit tests run before integ tests when running
  the 'mage test' target.
- chown the shared Python venv that is in the root of the repo after
  Dockerized integ tests exit to ensure there are no permissions issues
  caused by root owned files.
- I found a few Python string encoding issues that caused tests failures.
  I thought we fixed these during the Python 3 conversion, but something
  here exposed a few that we didn't address.

One thing of note that I did not correct. Journalbeat has system tests
but they are not executed. The existing/old Makefile has SYSTEM_TESTS=false
so they are not executed. So when I switched it to mage I left it as
'mage goUnitTest' to avoid the Python when failed when I tried it.
leweafan pushed a commit to leweafan/beats that referenced this pull request Apr 28, 2023
@v1v
[ci] Favor direct mage invocation on CI (elastic#19960) (elastic#20153)
1eb8608
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@cachedout cachedout cachedout approved these changes

Assignees
No one assigned
Labels
ci review Team:Automation Label for the Observability productivity team v7.10.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@andrewkroh @cachedout @andresrc