Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Ingest] Add support for yaml field types #60440

Merged
merged 3 commits into from
Mar 18, 2020
Merged

[Ingest] Add support for yaml field types #60440

merged 3 commits into from
Mar 18, 2020

Conversation

jen-huang
Copy link
Contributor

Summary

This PR adds support for yaml field types defined in package definition (elastic/package-registry#267):

  • Changes stream config model to save type and value, instead of just value
  • Adds code editor in UI for configuring yaml fields
  • Adjusts tests

Screenshot

AWS package:

image

Agent YAML output:

image

@jen-huang
Support yaml var type:
fc6ef12 
* Change stream config model to save type and value, instead of just value
* Add code editor for configuring yaml vars
* Adjust tests
@jen-huang jen-huang added v8.0.0 release_note:skip Skip the PR/issue when compiling release notes Feature:EPM Fleet team's Elastic Package Manager (aka Integrations) project Feature:Fleet Fleet team's agent central management project v7.7.0 labels Mar 17, 2020
@jen-huang jen-huang requested review from ruflin and a team March 17, 2020 20:33
@jen-huang jen-huang self-assigned this Mar 17, 2020
@elasticmachine
Copy link
Contributor

Pinging @elastic/ingest-management (Feature:EPM)

@jen-huang jen-huang mentioned this pull request Mar 17, 2020
Merged
@kibanamachine
Copy link
Contributor

💚 Build Succeeded

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

ruflin
ruflin approved these changes Mar 18, 2020
View reviewed changes
Copy link
Member

@ruflin ruflin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested this on my end and works as expected. Interestingly it strips out the comments.

This is a very powerful feature we can use in many places as escape hatch to not have to build everything in the UI for experts. @ph Probably worth you know about this.

x-pack/plugins/ingest_manager/common/services/datasource_to_agent_datasource.ts
acc[configName] = yamlValue;
}
} catch (e) {
// Silently swallow parsing error
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this something we could send into debug logs (if there is such a thing?)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Kibana logging is only on server-side. We should validate that the yaml can be parsed before allowing the user to continue to review step to avoid this branch, I have that captured in #60446

jen-huang
jen-huang commented Mar 18, 2020
View reviewed changes
Copy link
Contributor Author

@jen-huang jen-huang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interestingly it strips out the comments.

Yep, we save the value as a simple string in the saved object, so if the user goes back to edit it, they will see the commented out YAML lines. But during conversion of saved object config -> agent config, it gets parsed as JSON (and there's no "comments" concept in JSON).

@jen-huang jen-huang merged commit 60d385e into elastic:master Mar 18, 2020
@jen-huang jen-huang deleted the ingest/yaml-ds branch March 18, 2020 22:59
@jen-huang jen-huang mentioned this pull request Mar 18, 2020
Merged
jen-huang added a commit that referenced this pull request Mar 19, 2020
@jen-huang
[Ingest] Add support for yaml field types (#60440) (#60579)
cd1e182 
* Support yaml var type:
* Change stream config model to save type and value, instead of just value
* Add code editor for configuring yaml vars
* Adjust tests

* Account for empty yaml value

* Better account for invalid yaml parsing
gmmorris added a commit to gmmorris/kibana that referenced this pull request Mar 19, 2020
@gmmorris
Merge branch 'master' into alerting/empty-list
2ef7c4f 
* master: (35 commits)
  [Alerting] Adds navigation by consumer and alert type to alerting (elastic#58997)
  Introduce search interceptor (elastic#60523)
  [ML] Add functional tests for file data visualizer (elastic#60413)
  [APM] Optimize service map query (elastic#60412)
  [SIEM][Detection Engine] Adds lists feature flag and list values to the REST interfaces
  Enhancement/update esdocs datasource (elastic#59512)
  [junit] only include stdout in report for failures (elastic#60530)
  Update dependency nock to v12 (elastic#60422)
  upgrade execa to get stdout/stderr in error messages (elastic#60537)
  skip flaky suite (elastic#60471)
  [Ingest] Agent Config Details - Data sources list ui (elastic#60429)
  [SIEM] Create ML Rules (elastic#58053)
  skip flaky suite (elastic#60559)
  fix agent type (elastic#60554)
  Fixed default message for index threshold includes both threshold values (elastic#60545)
  [Ingest] Add support for `yaml` field types (elastic#60440)
  Solved the issue for a GROUP BY expression validation (elastic#60558)
  [Maps] Mark instance state as readonly (elastic#60557)
  Move ui/indices into es_ui_shared plugin. (elastic#60186)
  ServiceNow action improvements (elastic#60052)
  ...
gmmorris added a commit to gmmorris/kibana that referenced this pull request Mar 19, 2020
@gmmorris
Merge branch 'master' into alerting/api-constant
8e2d333 
* master: (64 commits)
  [Alerting] Adds navigation by consumer and alert type to alerting (elastic#58997)
  Introduce search interceptor (elastic#60523)
  [ML] Add functional tests for file data visualizer (elastic#60413)
  [APM] Optimize service map query (elastic#60412)
  [SIEM][Detection Engine] Adds lists feature flag and list values to the REST interfaces
  Enhancement/update esdocs datasource (elastic#59512)
  [junit] only include stdout in report for failures (elastic#60530)
  Update dependency nock to v12 (elastic#60422)
  upgrade execa to get stdout/stderr in error messages (elastic#60537)
  skip flaky suite (elastic#60471)
  [Ingest] Agent Config Details - Data sources list ui (elastic#60429)
  [SIEM] Create ML Rules (elastic#58053)
  skip flaky suite (elastic#60559)
  fix agent type (elastic#60554)
  Fixed default message for index threshold includes both threshold values (elastic#60545)
  [Ingest] Add support for `yaml` field types (elastic#60440)
  Solved the issue for a GROUP BY expression validation (elastic#60558)
  [Maps] Mark instance state as readonly (elastic#60557)
  Move ui/indices into es_ui_shared plugin. (elastic#60186)
  ServiceNow action improvements (elastic#60052)
  ...
jloleysens added a commit to jloleysens/kibana that referenced this pull request Mar 19, 2020
@jloleysens
Merge branch 'master' of github.com:elastic/kibana into np-shim/cross…
e2f7a22 
…-cluster-replication

* 'master' of github.com:elastic/kibana: (89 commits)
  Sort by name when fetching alerts and connectors (elastic#60506)
  Make slack param validation handle empty messages (elastic#60468)
  [Alerting] Adds navigation by consumer and alert type to alerting (elastic#58997)
  Introduce search interceptor (elastic#60523)
  [ML] Add functional tests for file data visualizer (elastic#60413)
  [APM] Optimize service map query (elastic#60412)
  [SIEM][Detection Engine] Adds lists feature flag and list values to the REST interfaces
  Enhancement/update esdocs datasource (elastic#59512)
  [junit] only include stdout in report for failures (elastic#60530)
  Update dependency nock to v12 (elastic#60422)
  upgrade execa to get stdout/stderr in error messages (elastic#60537)
  skip flaky suite (elastic#60471)
  [Ingest] Agent Config Details - Data sources list ui (elastic#60429)
  [SIEM] Create ML Rules (elastic#58053)
  skip flaky suite (elastic#60559)
  fix agent type (elastic#60554)
  Fixed default message for index threshold includes both threshold values (elastic#60545)
  [Ingest] Add support for `yaml` field types (elastic#60440)
  Solved the issue for a GROUP BY expression validation (elastic#60558)
  [Maps] Mark instance state as readonly (elastic#60557)
  ...

# Conflicts:
#	x-pack/legacy/plugins/cross_cluster_replication/public/np_ready/app/components/auto_follow_pattern_form.js
#	x-pack/legacy/plugins/cross_cluster_replication/public/np_ready/app/components/follower_index_form/follower_index_form.js
#	x-pack/legacy/plugins/cross_cluster_replication/public/np_ready/app/components/follower_index_form/follower_index_form.test.js
#	x-pack/legacy/plugins/cross_cluster_replication/public/np_ready/app/services/auto_follow_pattern_validators.js
#	x-pack/legacy/plugins/cross_cluster_replication/public/np_ready/app/services/input_validation.js
jloleysens added a commit to jloleysens/kibana that referenced this pull request Mar 19, 2020
@jloleysens
Merge branch 'master' of github.com:elastic/kibana into refactor/cons…
3c1d020 
…ole/cleanup

* 'master' of github.com:elastic/kibana: (47 commits)
  [Remote clusters] Update copy (elastic#60382)
  Sort by name when fetching alerts and connectors (elastic#60506)
  Make slack param validation handle empty messages (elastic#60468)
  [Alerting] Adds navigation by consumer and alert type to alerting (elastic#58997)
  Introduce search interceptor (elastic#60523)
  [ML] Add functional tests for file data visualizer (elastic#60413)
  [APM] Optimize service map query (elastic#60412)
  [SIEM][Detection Engine] Adds lists feature flag and list values to the REST interfaces
  Enhancement/update esdocs datasource (elastic#59512)
  [junit] only include stdout in report for failures (elastic#60530)
  Update dependency nock to v12 (elastic#60422)
  upgrade execa to get stdout/stderr in error messages (elastic#60537)
  skip flaky suite (elastic#60471)
  [Ingest] Agent Config Details - Data sources list ui (elastic#60429)
  [SIEM] Create ML Rules (elastic#58053)
  skip flaky suite (elastic#60559)
  fix agent type (elastic#60554)
  Fixed default message for index threshold includes both threshold values (elastic#60545)
  [Ingest] Add support for `yaml` field types (elastic#60440)
  Solved the issue for a GROUP BY expression validation (elastic#60558)
  ...
gmmorris added a commit to gmmorris/kibana that referenced this pull request Mar 19, 2020
@gmmorris
Merge branch 'master' into alerting/tls-warning
18dc4ee 
* master: (39 commits)
  [SIEM][CASE]  Configuration page action bar (elastic#60608)
  [Remote clusters] Update copy (elastic#60382)
  Sort by name when fetching alerts and connectors (elastic#60506)
  Make slack param validation handle empty messages (elastic#60468)
  [Alerting] Adds navigation by consumer and alert type to alerting (elastic#58997)
  Introduce search interceptor (elastic#60523)
  [ML] Add functional tests for file data visualizer (elastic#60413)
  [APM] Optimize service map query (elastic#60412)
  [SIEM][Detection Engine] Adds lists feature flag and list values to the REST interfaces
  Enhancement/update esdocs datasource (elastic#59512)
  [junit] only include stdout in report for failures (elastic#60530)
  Update dependency nock to v12 (elastic#60422)
  upgrade execa to get stdout/stderr in error messages (elastic#60537)
  skip flaky suite (elastic#60471)
  [Ingest] Agent Config Details - Data sources list ui (elastic#60429)
  [SIEM] Create ML Rules (elastic#58053)
  skip flaky suite (elastic#60559)
  fix agent type (elastic#60554)
  Fixed default message for index threshold includes both threshold values (elastic#60545)
  [Ingest] Add support for `yaml` field types (elastic#60440)
  ...
@jen-huang jen-huang added the Team:Fleet Team label for Observability Data Collection Fleet team label Mar 26, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ruflin ruflin ruflin approved these changes

@paul-tavares paul-tavares paul-tavares approved these changes

Assignees

@jen-huang jen-huang

Labels
Feature:EPM Fleet team's Elastic Package Manager (aka Integrations) project Feature:Fleet Fleet team's agent central management project release_note:skip Skip the PR/issue when compiling release notes Team:Fleet Team label for Observability Data Collection Fleet team v7.7.0 v8.0.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@jen-huang @elasticmachine @kibanamachine @ruflin @paul-tavares