Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate obs-infraobs-integrations to package-spec v3 #2 #8171

Merged

Conversation

shmsr
Copy link
Member

@shmsr shmsr commented Oct 12, 2023

Migrate a batch of packages from v1/v2 to v3 package-spec. Used the following scripts and some manual changes to generate the changes in this PR:

yq_hack.zsh

#!/bin/bash

set -e
set -u
set -o pipefail
set -x

pushd $1

find data_stream -type f -path "*fields*" -name "*.yml" -exec yq e -i "del .[].release" {} \;
find data_stream -type f -path "*fields*" -name "ecs.yml" -exec yq e -i "unique_by(.name)" {} \;
find data_stream -type f -path "*fields*" -name "*.yml" -exec yq e -i "del .[].fields.[].required" {} \;

yq e -i 'del .release' manifest.yml
yq e -i "with(select(.license != null); .conditions.elastic.subscription = .license) | del .license" manifest.yml

popd

git add -u $1/
git commit -m "[$1]: migration with yq"

ecs-update.zsh

#!/bin/bash

set -u
set -o pipefail
set -x

echo "-- executing ecs-update for $1 --"

ecs-update -format-version=3.0.0 -fix-dotted-yaml-keys -add-owner-type -owner elastic/obs-infraobs-integrations -v -skip-format=true $1

if (( $? )) 
then
    echo "-- write validation for $1 --"
    pushd $1

    go run ../scripting/write_validation.go

    popd

    # Try again?
    git checkout .
    ecs-update -format-version=3.0.0 -fix-dotted-yaml-keys -add-owner-type -owner elastic/obs-infraobs-integrations -v -skip-format=true $1
fi

And another Go program to write the validation.yml by parsing the elastic-package lint errors.

Then I hooked this scripts to run with:

$ pkgs=($(cat .github/CODEOWNERS | grep '@elastic/obs-infraobs-integrations' | cut -d ' ' -f 1 | cut -d '/' -f 3 | sort | uniq))

$ for ((idx=16; idx<=30; idx++))
do
scripting/yq_hack.zsh $pkgs[idx]
done
$ for ((idx=16; idx<=30; idx++))
do
scripting/ecs_update.zsh $pkgs[idx] || break
done

Another variation of the above script:

$ for ((idx=16; idx<=30; idx++))
do
case $pkgs[idx] in
  "nagios_xi" | "ibmmq" | "haproxy") echo "-- skip $pkgs[idx] --";;
  "nginx" | "memcached" | "kafka_log" | "jolokia_input" | "mongodb" | "mysql" | "microsoft_sqlserver" | "kafka" | "influxdb" | "iis" | "golang" | "hadoop") echo "-- done $pkgs[idx] --";;
  *) scripting/ecs_update.zsh $pkgs[idx] || break;;
esac
done

As I am creating a batch of 15 packages per migration, I am using a for loop. In case the migration is not possible for some packages, I put it in the first case to skip them.

Proposed commit message

Migrate packages owned by obs-infraobs-integrations but few couldn't be migrated. Marking them as follows:

  • golang
  • hadoop
  • haproxy
  • ibmmq
  • iis
  • influxdb
  • jolokia_input
  • kafka
  • kafka_log
  • memcached
  • microsoft_sqlserver
  • mongodb
  • mysql
  • nagios_xi
  • nginx

Please see the comment #8171 (comment) in this PR itself to understand why some packages couldn't be migrated and also learn about the challenges faced when migrating.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

  • [ ]

How to test this PR locally

Related issues

Relates to #8170

Screenshots

The format_version in the package manifest changed from 2.0.0 to 3.0.0. Removed
dotted YAML keys from package manifest. Added 'owner.type: elastic' to package
manifest.

[git-generate]
go run github.com/andrewkroh/go-examples/ecs-update@latest -v -format-version=3.0.0 -skip-format -fix-dotted-yaml-keys -add-owner-type packages/golang
The format_version in the package manifest changed from 1.0.0 to 3.0.0. Removed
dotted YAML keys from package manifest. Added 'owner.type: elastic' to package
manifest.

[git-generate]
go run github.com/andrewkroh/go-examples/ecs-update@latest -v -format-version=3.0.0 -skip-format -fix-dotted-yaml-keys -add-owner-type packages/hadoop
The format_version in the package manifest changed from 2.0.0 to 3.0.0. Removed
dotted YAML keys from package manifest. Added 'owner.type: elastic' to package
manifest.

[git-generate]
go run github.com/andrewkroh/go-examples/ecs-update@latest -v -format-version=3.0.0 -skip-format -fix-dotted-yaml-keys -add-owner-type packages/jolokia_input
The format_version in the package manifest changed from 1.0.0 to 3.0.0. Removed
dotted YAML keys from package manifest. Added 'owner.type: elastic' to package
manifest.

[git-generate]
go run github.com/andrewkroh/go-examples/ecs-update@latest -v -format-version=3.0.0 -skip-format -fix-dotted-yaml-keys -add-owner-type packages/kafka_log
The format_version in the package manifest changed from 1.0.0 to 3.0.0. Added
'owner.type: elastic' to package manifest.

[git-generate]
go run github.com/andrewkroh/go-examples/ecs-update@latest -v -format-version=3.0.0 -skip-format -fix-dotted-yaml-keys -add-owner-type packages/memcached
@elasticmachine
Copy link

elasticmachine commented Oct 12, 2023

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2023-10-19T07:50:38.693+0000

  • Duration: 24 min 30 sec

Test stats 🧪

Test Results
Failed 0
Passed 223
Skipped 0
Total 223

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

@elasticmachine
Copy link

elasticmachine commented Oct 12, 2023

🌐 Coverage report

Name Metrics % (covered/total) Diff
Packages 100.0% (17/17) 💚
Files 91.304% (21/23) 👎 -8.696
Classes 91.304% (21/23) 👎 -8.696
Methods 88.066% (214/243) 👍 12.308
Lines 94.99% (2977/3134) 👎 -5.01
Conditionals 100.0% (0/0) 💚

The format_version in the package manifest changed from 1.0.0 to 3.0.0. Removed
dotted YAML keys from package manifest. Added 'owner.type: elastic' to package
manifest.

[git-generate]
go run github.com/andrewkroh/go-examples/ecs-update@latest -v -format-version=3.0.0 -skip-format -fix-dotted-yaml-keys -add-owner-type packages/iis
The format_version in the package manifest changed from 1.0.0 to 3.0.0. Removed
dotted YAML keys from package manifest. Added 'owner.type: elastic' to package
manifest.

[git-generate]
go run github.com/andrewkroh/go-examples/ecs-update@latest -v -format-version=3.0.0 -skip-format -fix-dotted-yaml-keys -add-owner-type packages/influxdb
The format_version in the package manifest changed from 1.0.0 to 3.0.0. Removed
dotted YAML keys from package manifest. Added 'owner.type: elastic' to package
manifest.

[git-generate]
go run github.com/andrewkroh/go-examples/ecs-update@latest -v -format-version=3.0.0 -skip-format -fix-dotted-yaml-keys -add-owner-type packages/kafka
The format_version in the package manifest changed from 1.0.0 to 3.0.0. Removed
dotted YAML keys from package manifest. Added 'owner.type: elastic' to package
manifest.

[git-generate]
go run github.com/andrewkroh/go-examples/ecs-update@latest -v -format-version=3.0.0 -skip-format -fix-dotted-yaml-keys -add-owner-type packages/microsoft_sqlserver
The format_version in the package manifest changed from 1.0.0 to 3.0.0. Removed
dotted YAML keys from package manifest. Added 'owner.type: elastic' to package
manifest.

[git-generate]
go run github.com/andrewkroh/go-examples/ecs-update@latest -v -format-version=3.0.0 -skip-format -fix-dotted-yaml-keys -add-owner-type packages/mongodb
The format_version in the package manifest changed from 1.0.0 to 3.0.0. Removed
dotted YAML keys from package manifest. Added 'owner.type: elastic' to package
manifest.

[git-generate]
go run github.com/andrewkroh/go-examples/ecs-update@latest -v -format-version=3.0.0 -skip-format -fix-dotted-yaml-keys -add-owner-type packages/mysql
The format_version in the package manifest changed from 1.0.0 to 3.0.0. Removed
dotted YAML keys from package manifest. Added 'owner.type: elastic' to package
manifest.

[git-generate]
go run github.com/andrewkroh/go-examples/ecs-update@latest -v -format-version=3.0.0 -skip-format -fix-dotted-yaml-keys -add-owner-type packages/nginx
@shmsr
Copy link
Member Author

shmsr commented Oct 12, 2023

  • haproxy

    • Legacy viz: file "/Users/subhamsarkar/go/src/github.com/elastic/integrations/build/packages/haproxy-1.8.4.zip/kibana/visualization/haproxy-794b6cd0-471d-11e8-bc13-1397384faad3.json" is invalid: found legacy visualization "Active servers in backend [Metrics HAProxy]" (metric, TSVB)
  • ibmmq

    • Legacy viz: file "/Users/subhamsarkar/go/src/github.com/elastic/integrations/build/packages/ibmmq-1.2.3.zip/kibana/dashboard/ibmmq-ba1d8830-7c7b-11e9-9645-e37efaf5baff.json" is invalid: "[Logs IBM MQ] Overview of error log" contains legacy visualization: "Top 5 Errors [Logs IBM MQ]" (table, Aggs-based)
  • nagios_xi

    • file "/Users/subhamsarkar/go/src/github.com/elastic/integrations/build/packages/nagios_xi-1.1.1.zip/data_stream/service/fields/fields.yml" is invalid: field 0.fields.0.fields.20: Additional property decription is not allowed
      • This is not expected. Needs to be checked. Skipping for now.

Copy link
Contributor

@tommyers-elastic tommyers-elastic left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

awesome - no blockers from me. couple of comments still to resolve though. thanks !

@tommyers-elastic tommyers-elastic merged commit 6445a75 into elastic:main Oct 19, 2023
4 checks passed
@elasticmachine
Copy link

Package golang - 1.1.0 containing this change is available at https://epr.elastic.co/search?package=golang

@elasticmachine
Copy link

Package hadoop - 1.3.0 containing this change is available at https://epr.elastic.co/search?package=hadoop

@elasticmachine
Copy link

Package iis - 1.16.0 containing this change is available at https://epr.elastic.co/search?package=iis

@elasticmachine
Copy link

Package jolokia - 0.3.0 containing this change is available at https://epr.elastic.co/search?package=jolokia

@elasticmachine
Copy link

Package kafka - 1.10.0 containing this change is available at https://epr.elastic.co/search?package=kafka

@elasticmachine
Copy link

Package kafka_log - 1.4.0 containing this change is available at https://epr.elastic.co/search?package=kafka_log

@elasticmachine
Copy link

Package memcached - 1.3.0 containing this change is available at https://epr.elastic.co/search?package=memcached

@elasticmachine
Copy link

Package microsoft_sqlserver - 2.3.0 containing this change is available at https://epr.elastic.co/search?package=microsoft_sqlserver

@elasticmachine
Copy link

Package mysql - 1.16.0 containing this change is available at https://epr.elastic.co/search?package=mysql

@elasticmachine
Copy link

Package nginx - 1.16.0 containing this change is available at https://epr.elastic.co/search?package=nginx

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants