[Ingest Management] main branch uses epr-snapshot. Others production #73555

jfsiii · 2020-07-28T21:23:38Z

Summary

Updated getRegistryUrl based on elastic/package-storage#86 (comment)

master always points to the snapshot registry and everything else to production

Changes in `getRegistryUrl`

if (branch === 'master') {
  return SNAPSHOT_REGISTRY_URL_CDN;
} else {
  return PRODUCTION_REGISTRY_URL_CDN;
}

Changes in `AppContextService`

Use the version & branch from kibana's package.json as default values.

code

class AppContextService {
  private kibanaVersion: IngestManagerAppContext['kibanaVersion'] = packageJSON.version;
  private kibanaBranch: IngestManagerAppContext['kibanaBranch'] = packageJSON.branch;
}

More background in comments here & here

version & branch are available in package.json master, 7.9, and 7.8

master

kibana/package.json

Lines 14 to 15 in aa68e3b

    
           "version": "8.0.0", 
        
           "branch": "master",

7.9

kibana/package.json

Lines 14 to 15 in da1ae75

    
           "version": "7.9.0", 
        
           "branch": "7.9",

7.8

kibana/package.json

Lines 14 to 15 in f6f3f37

    
           "version": "7.8.2", 
        
           "branch": "7.8",

Checklist

Unit or functional tests were updated or added to match the most common scenarios

elasticmachine · 2020-07-28T21:23:42Z

Pinging @elastic/ingest-management (Team:Ingest Management)

ruflin · 2020-07-29T11:22:56Z

x-pack/plugins/ingest_manager/server/services/epm/registry/registry_url.ts

+
+const PRODUCTION_REGISTRY_URL_CDN = 'https://epr.elastic.co';
+// const STAGING_REGISTRY_URL_CDN = 'https://epr-staging.elastic.co';
+// const EXPERIMENTAL_REGISTRY_URL_CDN = 'https://epr-experimental.elastic.co/';


Experimental should not be here as this will never apply for master or anything coming out of master. Same on line 19.

ruflin · 2020-07-29T11:24:19Z

LGTM, one minor comment. Did not test it locally.

jfsiii · 2020-07-29T11:40:49Z

@elasticmachine merge upstream

x-pack/test/security_solution_endpoint/apps/endpoint/index.ts

skh

LGTM when CI is happy 👍

jfsiii · 2020-07-29T15:02:25Z

The test report shows no failing tests.

The pipeline steps only show 1 failure afaict which I hope is just an issue while cleaning up or some other transient/unrelated bit.

I'll update the function names and we'll see what the next run brings.

jfsiii · 2020-07-29T16:48:49Z

https://kibana-ci.elastic.co/job/elastic+kibana+pipeline-pull-request/65254/execution/node/316/ & https://kibana-ci.elastic.co/job/elastic+kibana+pipeline-pull-request/65254/execution/node/335/ show errors

Just gonna keep doing @elasticmachine merge upstream for a bit

jfsiii · 2020-07-29T16:48:53Z

@elasticmachine merge upstream

jonathan-buttner · 2020-07-29T17:00:54Z

@jfsiii Looks like the failure might be related to finding the kibana branch version?

nfo Running /dev/shm/workspace/kibana/x-pack/test/security_solution_endpoint/config.ts
12:57:53   │ info Config loaded
12:57:55   │ warn These tests are being run with an external package registry
12:57:55  
12:57:55  Failure to load testProvider[/dev/shm/workspace/kibana/x-pack/test/security_solution_endpoint/apps/endpoint] -- from testFile[/dev/shm/workspace/kibana/x-pack/test/security_solution_endpoint/apps/endpoint]
12:57:55  
12:57:55    Original Error: Error: Kibana branch is not set.
12:57:55      at AppContextService.getKibanaBranch (/dev/shm/workspace/kibana/x-pack/plugins/ingest_manager/server/services/app_context.ts:132:13)
12:57:55      at getDefaultRegistryUrl (/dev/shm/workspace/kibana/x-pack/plugins/ingest_manager/server/services/epm/registry/registry_url.ts:23:36)
12:57:55      at getRegistryUrl (/dev/shm/workspace/kibana/x-pack/plugins/ingest_manager/server/services/epm/registry/registry_url.ts:44:10)
12:57:55      at Suite.call (/dev/shm/workspace/kibana/x-pack/test/security_solution_endpoint/apps/endpoint/index.ts:25:56)
12:57:55      at Suite.argumentsList.(anonymous function) (/dev/shm/workspace/kibana/packages/kbn-test/src/functional_test_runner/lib/mocha/decorate_mocha_ui.js:73:20)
12:57:55      at Object.create (/dev/shm/workspace/kibana/node_modules/mocha/lib/interfaces/common.js:140:19)
12:57:55      at apply (/dev/shm/workspace/kibana/node_modules/mocha/lib/interfaces/bdd.js:42:27)
12:57:55      at Object.describe [as apply] (/dev/shm/workspace/kibana/packages/kbn-test/src/functional_test_runner/lib/mocha/wrap_function.js:55:24)
12:57:55      at provider (/dev/shm/workspace/kibana/x-pack/test/security_solution_endpoint/apps/endpoint/index.ts:16:3)
12:57:55      at load (/dev/shm/workspace/kibana/packages/kbn-test/src/functional_test_runner/lib/mocha/load_test_files.js:61:25)
12:57:55      at loadTracer (/dev/shm/workspace/kibana/packages/kbn-test/src/functional_test_runner/lib/load_tracer.ts:59:12)
12:57:55      at runTestProvider (/dev/shm/workspace/kibana/packages/kbn-test/src/functional_test_runner/lib/mocha/load_test_files.js:55:5)
12:57:55      at load (/dev/shm/workspace/kibana/packages/kbn-test/src/functional_test_runner/lib/mocha/load_test_files.js:46:7)
12:57:55      at loadTracer (/dev/shm/workspace/kibana/packages/kbn-test/src/functional_test_runner/lib/load_tracer.ts:59:12)
12:57:55      at forEach (/dev/shm/workspace/kibana/packages/kbn-test/src/functional_test_runner/lib/mocha/load_test_files.js:40:5)
12:57:55      at Array.forEach (<anonymous>)
12:57:55      at loadTestFiles (/dev/shm/workspace/kibana/packages/kbn-test/src/functional_test_runner/lib/mocha/load_test_files.js:80:9)
12:57:55      at setupMocha (/dev/shm/workspace/kibana/packages/kbn-test/src/functional_test_runner/lib/mocha/setup_mocha.js:49:3)
12:57:55      at process._tickCallback (internal/process/next_tick.js:68:7)
12:57:55      at Function.Module.runMain (internal/modules/cjs/loader.js:834:11)
12:57:55      at startup (internal/bootstrap/node.js:283:19)
12:57:55      at bootstrapNodeJSCore (internal/bootstrap/node.js:623:3)
12:57:55  
12:57:56  runbld>>> <<<<<<<<<<<< SCRIPT EXECUTION END <<<<<<<<<<<<

skh · 2020-07-29T21:06:23Z

x-pack/plugins/ingest_manager/server/services/epm/registry/registry_url.ts

+  let branch;
+  try {
+    branch = appContextService.getKibanaBranch();
+  } catch (error) {


Under what conditions can kibana branch not be set?

During the test:ftr Endpoint tests. The error seems in the ftr:runner portion.

On the phone right now, but check the stack trace in #73555 (comment)

I can add details about what I saw while debugging later

It was a good reminder that those methods throw :(

If getKibanaBranch() errors in this case, it might in other places where it is used, I think this should be checked before merging.

skh

I think I understand why the test was failing.

When getDefaultRegistryUrl() is called from the api integration tests, this happens in the test runner node process, and hence the ingest manager plugin hasn't been initialized, and the app context hasn't been populated with a value for kibanaBranch.

If the tests should use the registry that is defined for the branch, I think ~~the only~~ one option is to pick the branch from the top level package.json in the test code, and change getDefaultRegistryUrl() so that it accepts the branch as a parameter.

@jonathan-buttner what do you think?

jfsiii · 2020-07-30T11:48:13Z

@skh I think I will change appcontext branch name and version to use disk if it wasn't set to something else. That way we can use platform and/or override but it will be accurate and shouldn't be missing

jonathan-buttner · 2020-07-30T13:02:01Z

@skh I think I will change appcontext branch name and version to use disk if it wasn't set to something else. That way we can use platform and/or override but it will be accurate and shouldn't be missing

Ah ok cool. Yeah the code that uses that function is just logging it. The endpoint integration and functional tests fallback to using the default external registry when the docker environment variable and the override url aren't specified. This allows developers to continue running the tests even though they don't have docker installed.

jfsiii · 2020-07-30T14:33:20Z

@elasticmachine merge upstream

jfsiii · 2020-07-30T15:56:17Z

failed b/c of an missing i18n string (from APM, iiuc) https://github.com/elastic/kibana/pull/73555/checks?check_run_id=928358900

jfsiii · 2020-07-30T15:56:21Z

@elasticmachine merge upstream

jfsiii · 2020-07-30T18:27:58Z

now? 550d44e

jfsiii · 2020-07-30T18:28:09Z

@elasticmachine merge upstream

kibanamachine · 2020-07-30T20:14:33Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: 86ba25a

Build metrics

page load bundle size

id	value	diff	baseline
ingestManager	458.9KB	-146.0B	459.0KB

History

💔 Build #65569 failed 4c9f212
💔 Build #65541 failed 33a98da
💔 Build #65529 failed 421f3ac
💚 Build #65345 succeeded d49f968
💔 Build #65277 failed 65d2074

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

* master: (54 commits) [ML] Migrate to React BrowserRouter and Kibana provided History. (elastic#71941) [Discover] Improve saveSearch functional test handling (elastic#73626) [Metrics UI] Fix all threshold alert conditions disappearing due to alert prefill (elastic#73708) [Metrics UI] Fix alert previews of ungrouped alerts (elastic#73735) [SIEM] Fixes "include building block button" to operate (elastic#73900) [Metrics UI] Fix alert management to open without refresh (elastic#73739) [Security Solution][Lists] - Tests cleanup and remove unnecessary import (elastic#73865) [Ingest Management] main branch uses epr-snapshot. Others production (elastic#73555) [Canvas][tech-debt] Fix SVG not shrinking vertically properly (elastic#73867) [Maps] upgrade turf (elastic#73816) [Security Solution][Telemetry] Concurrent telemetry requests (elastic#73558) [Security Solution][Exceptions] - Update how nested entries are displayed in exceptions viewer (elastic#73745) [Security Solution][Exceptions] Adds autocomplete workaround for .text fields (elastic#73761) [Metrics UI] Fix previewing of No Data results (elastic#73753) Closes elastic#72914 by hiding anomaly detection settings links when the ml plugin is disabled. (elastic#73638) [Ingest Manager] Fix config selection in enrollment flyout from config list page (elastic#73833) [DOCS] Fixes typo in Alerting actions (elastic#73756) [APM] fixes linking errors to ML and Discover (elastic#73758) Handle promise rejections when building artifacts (elastic#73831) [Security Solution][Detections] Change from sha1 to sha256 (elastic#73741) ...

* master: (38 commits) [Discover] Context unskip date nanos functional tests (elastic#73781) [ML] Migrate to React BrowserRouter and Kibana provided History. (elastic#71941) [Discover] Improve saveSearch functional test handling (elastic#73626) [Metrics UI] Fix all threshold alert conditions disappearing due to alert prefill (elastic#73708) [Metrics UI] Fix alert previews of ungrouped alerts (elastic#73735) [SIEM] Fixes "include building block button" to operate (elastic#73900) [Metrics UI] Fix alert management to open without refresh (elastic#73739) [Security Solution][Lists] - Tests cleanup and remove unnecessary import (elastic#73865) [Ingest Management] main branch uses epr-snapshot. Others production (elastic#73555) [Canvas][tech-debt] Fix SVG not shrinking vertically properly (elastic#73867) [Maps] upgrade turf (elastic#73816) [Security Solution][Telemetry] Concurrent telemetry requests (elastic#73558) [Security Solution][Exceptions] - Update how nested entries are displayed in exceptions viewer (elastic#73745) [Security Solution][Exceptions] Adds autocomplete workaround for .text fields (elastic#73761) [Metrics UI] Fix previewing of No Data results (elastic#73753) Closes elastic#72914 by hiding anomaly detection settings links when the ml plugin is disabled. (elastic#73638) [Ingest Manager] Fix config selection in enrollment flyout from config list page (elastic#73833) [DOCS] Fixes typo in Alerting actions (elastic#73756) [APM] fixes linking errors to ML and Discover (elastic#73758) Handle promise rejections when building artifacts (elastic#73831) ...

jen-huang · 2020-08-03T15:47:42Z

@jfsiii Not sure where the backport missing bot is, but this still needs backporting :)

…lastic#73555) * Same behavior as now. Just refactored. * main branch uses epr-snapshot. Others use prod * Link some types vs repeating them * replace DEFAULT_REGISTRY_URL with getRegistryUrl in Endpoint tests * Make an Endpoint test helper name more clear * try/catch around getKibanaBranch * Use branch & version from package.json as fallback * No guards b/c kibana{Branch,Version} have defaults Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> # Conflicts: # x-pack/plugins/ingest_manager/common/constants/epm.ts

kibanamachine · 2020-08-03T23:18:45Z

Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync.

…73555) (#74122) * Same behavior as now. Just refactored. * main branch uses epr-snapshot. Others use prod * Link some types vs repeating them * replace DEFAULT_REGISTRY_URL with getRegistryUrl in Endpoint tests * Make an Endpoint test helper name more clear * try/catch around getKibanaBranch * Use branch & version from package.json as fallback * No guards b/c kibana{Branch,Version} have defaults Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com> # Conflicts: # x-pack/plugins/ingest_manager/common/constants/epm.ts

John Schulz added 2 commits July 28, 2020 16:50

Same behavior as now. Just refactored.

62ba64a

main branch uses epr-snapshot. Others use prod

0ddafdd

jfsiii requested a review from a team July 28, 2020 21:23

botelastic bot added the Team:Fleet Team label for Observability Data Collection Fleet team label Jul 28, 2020

jfsiii requested a review from ruflin July 28, 2020 21:23

Link some types vs repeating them

de149f8

jfsiii added release_note:skip Skip the PR/issue when compiling release notes v7.10.0 v8.0.0 labels Jul 28, 2020

jfsiii mentioned this pull request Jul 28, 2020

[Ingest Manager] Update registry URL to point to CDN #73533

Closed

replace DEFAULT_REGISTRY_URL with getRegistryUrl in Endpoint tests

35fedc2

jfsiii requested review from a team as code owners July 29, 2020 10:43

ruflin reviewed Jul 29, 2020

View reviewed changes

Merge branch 'master' into support-different-default-registries

381bbf7

skh reviewed Jul 29, 2020

View reviewed changes

x-pack/test/security_solution_endpoint/apps/endpoint/index.ts Outdated Show resolved Hide resolved

skh self-requested a review July 29, 2020 13:19

jonathan-buttner approved these changes Jul 29, 2020

View reviewed changes

x-pack/test/security_solution_endpoint/apps/endpoint/index.ts Outdated Show resolved Hide resolved

skh previously approved these changes Jul 29, 2020

View reviewed changes

Make an Endpoint test helper name more clear

4746210

Merge branch 'master' into support-different-default-registries

65d2074

try/catch around getKibanaBranch

d49f968

skh reviewed Jul 29, 2020

View reviewed changes

skh self-requested a review July 29, 2020 23:10

skh suggested changes Jul 30, 2020

View reviewed changes

Use branch & version from package.json as fallback

421f3ac

jfsiii requested a review from skh July 30, 2020 14:31

elasticmachine and others added 2 commits July 30, 2020 08:33

Merge branch 'master' into support-different-default-registries

61abba4

No guards b/c kibana{Branch,Version} have defaults

33a98da

Merge branch 'master' into support-different-default-registries

4c9f212

Merge branch 'master' into support-different-default-registries

86ba25a

skh approved these changes Jul 30, 2020

View reviewed changes

jfsiii merged commit 9c9080c into elastic:master Jul 30, 2020

jfsiii mentioned this pull request Aug 3, 2020

[7.x] [Ingest Management] main branch uses epr-snapshot. Others production (#73555) #74122

Merged

kibanamachine added backport missing Added to PRs automatically when the are determined to be missing a backport. and removed backport missing Added to PRs automatically when the are determined to be missing a backport. labels Aug 3, 2020

jfsiii self-assigned this Aug 10, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Ingest Management] main branch uses epr-snapshot. Others production #73555

[Ingest Management] main branch uses epr-snapshot. Others production #73555

jfsiii commented Jul 28, 2020 •

edited

Loading

elasticmachine commented Jul 28, 2020

ruflin Jul 29, 2020

ruflin commented Jul 29, 2020

jfsiii commented Jul 29, 2020

skh left a comment

jfsiii commented Jul 29, 2020

jfsiii commented Jul 29, 2020

jfsiii commented Jul 29, 2020

jonathan-buttner commented Jul 29, 2020

skh Jul 29, 2020

jfsiii Jul 29, 2020

skh left a comment •

edited

Loading

jfsiii commented Jul 30, 2020

jonathan-buttner commented Jul 30, 2020

jfsiii commented Jul 30, 2020

jfsiii commented Jul 30, 2020

jfsiii commented Jul 30, 2020

jfsiii commented Jul 30, 2020

jfsiii commented Jul 30, 2020

kibanamachine commented Jul 30, 2020

jen-huang commented Aug 3, 2020

kibanamachine commented Aug 3, 2020

[Ingest Management] main branch uses epr-snapshot. Others production #73555

[Ingest Management] main branch uses epr-snapshot. Others production #73555

Conversation

jfsiii commented Jul 28, 2020 • edited Loading

Summary

Changes in getRegistryUrl

Changes in AppContextService

Checklist

elasticmachine commented Jul 28, 2020

ruflin Jul 29, 2020

Choose a reason for hiding this comment

ruflin commented Jul 29, 2020

jfsiii commented Jul 29, 2020

skh left a comment

Choose a reason for hiding this comment

jfsiii commented Jul 29, 2020

jfsiii commented Jul 29, 2020

jfsiii commented Jul 29, 2020

jonathan-buttner commented Jul 29, 2020

skh Jul 29, 2020

Choose a reason for hiding this comment

jfsiii Jul 29, 2020

Choose a reason for hiding this comment

skh left a comment • edited Loading

Choose a reason for hiding this comment

jfsiii commented Jul 30, 2020

jonathan-buttner commented Jul 30, 2020

jfsiii commented Jul 30, 2020

jfsiii commented Jul 30, 2020

jfsiii commented Jul 30, 2020

jfsiii commented Jul 30, 2020

jfsiii commented Jul 30, 2020

kibanamachine commented Jul 30, 2020

💚 Build Succeeded

Build metrics

page load bundle size

History

jen-huang commented Aug 3, 2020

kibanamachine commented Aug 3, 2020

jfsiii commented Jul 28, 2020 •

edited

Loading

Changes in `getRegistryUrl`

Changes in `AppContextService`

skh left a comment •

edited

Loading