* Add beginning models and two routes for Ingest

* Update types & models per discussion w/Ruflin
Also reviewed data structures listed at https://docs.google.com/document/d/1IBR3f9dpHqJmXYEdg06WV34KSMd3g5k4aMGa4jde_Eg/edit#

* Update: /policies always returns array. /policy returns single policy

* Add pagination for /policy & /datasources. Uses per_page & page params

* Add API metadata. Standardize policy_id param name.

* Update descriptions to match Google Doc. Move use case to Policy.

Disabled the '@typescript-eslint/array-type' rule because it was going around in circles. It didn't like Datasource[] or Array<Datasource>

* Return to initial TS annotation for Arrays

Remove the line disabling @typescript-eslint/array-type now that it's behaving normally again 🤷

* Add directory structure for server/lib.

* 'tests' seems to be more common than 'test'

* Make CI happy

* [EPM] Add basic documentation directory

Having the doc directory around allows us to easily add docs from here on to document how EPM works.

To run the docs build, use the following command from the kibana directory:

```
../docs/build_docs --doc docs/epm/index.asciidoc --open
```

The above assumes that docs (https://github.com/elastic/docs) are checked out in the same directory as Kibana.

With this change, the EPM docs build is not included yet in the overall docs build. For this adjustments to https://github.com/elastic/docs/blob/master/conf.yaml must be made.

This PR adds the very basic index template we will use for the packages. It contains all the basic settings and some examples. The examples will be remove as soon as we have an actual implementation with packages but for now is convenient to see if it is a valid package.

This code is put into the lib directory as it does not tie directly into any handlers.

It also adds an functional tests for loading a template. This means we have a way to check if a template is valid in Elasticsearch. Based on this we can check in the future all our generated templates for validity with Elasticsearch.

To run the functional test, go to the Kibana x-pack directory. Start the first command:

```
node scripts/functional_tests_server.js --config test/epm_api_integration/config.ts
```

Keep the above running and switch to an other Terminal. Now run:

```
node scripts/functional_test_runner.js --config x-pack/test/epm_api_integration/config.ts
```

* Add directory structure for server/lib.

* 'tests' seems to be more common than 'test'

* Make CI happy

* Implement pipeline rewriting.

* Add more testcases

* For posterity (comment change)

* Allow beats-style template delimiters

* Be more succinct

* Document better

See #50609 (comment)

Discussed in Slack and agree to revert for now. Can track down the issues & restore later

…rations-manager

…rations-manager

…0609)

* cleanup assets, filter assets for those currently supported

* removed unused type

* fix type

* add comment for better type

* change type name to be more descriptive

…ana into feature-integrations-manager

* hardcode image width for ie11

* eslint

* improve comment

* add maxWidth

* plugin.start now does reactdom.render vs returning react element

export plugin function from public/index

* Move setClient call from plugin.start to plugin.setup

* Use `useUiSetting$` from `useKibana` hooks

Can't use useKibana outside a React component.

Reverting to prior approach since it's still NP. Can revisit context usage in a followup PR

* Support basic "click button -> show spinner -> installed" install flow

* Remove incorrect comments. Add TS return types to data functions.

* Use NP feature_catalogue.register

* Use type from NP plugin

…to feature-ingest

…ana into feature-ingest

This reverts commit d05f20d.

* Update (all remaining?) references from integrations_manager to EPM

* Update path in i18n file

* change min max version to single range value

* add elastic stack icon and change text

* remove badge from version and use code font

* remove euistyled

* add back version component lost in merge

* remove euiStyled

* remove old file

Temporary work around until we know how the stack wants to add the permissions we need. Either adding to the kibana user or creating a new user.

This reverts commit f1020e4.

    Temporary work around until we know how the stack wants to add the permissions we need. Either adding to the kibana user or creating a new user.

* Initialize es in test.

* Add it(), no es.init()

* Clean up.

* add confirmation modal, move install state to Header

* update callout to use title

* move components only used in detail view to detail dir

* use better variable names

* update to more descriptive  variable names

Discussed this with @skh #51112 (comment) & #51112 (comment) as well as in a video call

Also added some TS type annotations for data fetching functions to make the contracts more explicit

)

* API only shows installable assets. Server types to own file. Restore enums

* Fix type imports

* Only return installable asset types (kibana for now)

* server/types now only has code from hapi which shouldn't go to client

* Add more restricted TS types to DisplayAssets object

* Flip order of arguments to Extract

In these cases it still works the same, but looking at https://www.typescriptlang.org/docs/handbook/advanced-types.html the signature is

`Extract<T, U>` - Extract from `T` those types that are assignable to `U`

so the larger set should be first

This documentation is at the moment mainly for internal use. I found myself searching for this URL several times in the code or PRs so I thought I rather add it to the docs for now.

…s for a policy (#51673)

This is mainly for internal usage at the moment to look up.

* wip policy

* tweaks

* tweaks

* FIX TYPOS

* WIP move policy => agent config conversion to fleet, WIP policy changed method

* fix tests and bugs

* updates tests and snaps

* more fixes

* use AGENT_POLLING_INTERVAL

* cleanup and fix some formatting

* Update x-pack/legacy/plugins/ingest/server/libs/datasources.ts

Co-Authored-By: John Schulz <github.com@jfsiii.org>

* Update x-pack/legacy/plugins/ingest/server/libs/datasources.ts

Co-Authored-By: John Schulz <github.com@jfsiii.org>

* Update x-pack/legacy/plugins/ingest/server/libs/outputs.ts

Co-Authored-By: John Schulz <github.com@jfsiii.org>

* Update x-pack/legacy/plugins/fleet/server/libs/policy.ts

Co-Authored-By: John Schulz <github.com@jfsiii.org>

* fix things broken by PR review suggestions

* remove unused field

* fix types

* fix mappings

* add datasource mappings

* Fix mappings and remove get full policy from checkin

* Fix ingest api integration tests

* add notifications from core to plugin

* add package install state hook

* fix type error

* use toMountPoint helper to add jsx to notification

* add warning notification to failed install

* make notifications dependency explicit prop

* move PackageInstall provider lower

* add comment about InstallStatus type overlapping InstallationStatus

* use InstallStatus type in InstallationButton component

* fix type

* aadding config

* add working settings to the default output

* remove default username and password

* update libs

This contains the basic objects to setup ILM

* Create index and alias with a write index
* Get the policy

The code does not contain any functional tests yet as it is still open on how to do it best. I suggest to get this in as a foundation and then iterate on top of it.

## Summary

This mixes a few concerns but I think it's worth it to show the parts working together.

Take a look at the individual commits for a better separation of features.

This adds 
 - the `/datasource/install/{pkgkey}` endpoint which installs ingest pipelines from a package into ES and saves a reference to them in the EPM state Saved Object
 - Connects the "Add datasource" button in the successful installation Toast to the new API
 - Adds a toast notification to inform the user the datasource was added correctly
 - Adds a "Delete Package" button on the details page so we can uninstall a package while we're waiting for the separate view which allows deletes
 - b99eda6 Pushes logic that was in the detail view into `InstallationButton`. This consolidates the logic in one component (or one component & the existing hook) and, iiic, means we can put `<InstallationButton package={...} />` on any view and get the same behavior

I'm marking this as a normal PR so people can merge if they wish

![add-datasource-delete-package-small](https://user-images.githubusercontent.com/57655/69775686-7fb39280-1167-11ea-8d41-e2b8a02252a1.gif)

The fields.yml is used to generate the Elasticsearch template and Kibana index pattern. This PR adds a very basic implementation of processing the fields.yml and then create an Elasticsearch template out of it. The only fields that are supported at the moment are keyword fields, more will be added as a follow up.

The testing was implemented with a golden file. The output from the method is compared to a json file. If the input is changed or the method is changed, it is possible to regenerate the files with the `-generate` flag as following:

```
node scripts/jest  ./legacy/plugins/epm/server/lib/template/template.test.ts -generate
```

This will allow us to quickly test many inputs / outputs in the future, make adjustments to the existing files and generate the new outputs. We then can compare it in the diff it the changes make sense.

… manifest (#51803)

* [EPM] Create basic implementation to merge input template and dataset manifest

With this code it is possible to take an input template for the agent and merge it with the config variables from the dataset manifest file. Currently only the name and the default value are merged. Later on we must implement to be able to pass user configured variables to it and make decision based on OS selection.

Closing #51794

This refactors the structure of lib. As so far all the lib parts are related to assets in the package, it is organised the same way with the same structure. Each directory has its own tests directory if it needs one. This makes it possible to (almost) not need relative paths for tests.

This change allows to also extract files from the `fields` directory. Previously this was not possible because it always assumed a service must be there.

This installs the Elasticsearch index template for each dataset in a package. For now the names are hardcoded based on package key and dataset name but will be more dynamic later on when we pass the full dataset information.

The dataset extractions is a bit "hacky" at the moment and we should get a full implementation of dataset at a later stage and replace this code.

* Set up simple policies list view

* Adjust spec to return single policy

* Set up simple policy details page

* Add demo stats/chart to policy details

* Add description string

* Initial setup of policy form and create policy UI

* Policy create/edit form; integrate policy list api

* Integrate create policy api

* Integrate policy detail, agent status, and policy edit APIs; adjust policy list api integration in agent enrollment

* Fix edit policy mock meta

* Fix policy list search bar

* PR and linting fixes; use typings from ingest plugin

* Fix i18n

## Summary

This PR makes a few assumptions, and contains a lot of refactoring. It might be beneficial to look at the resulting directory structure under `server` first to get the (new) big picture.

Assumptions:
- our API deals with several concerns, for now these are packages and datasources
- we manage our own HTTP API endpoints for these concerns (in particular, don't use the ingest plugin for that)
- we manage (for now) the Kibana saved object in which datasources are saved. importing and calling methods from the ingest plugin to do that down the road will (hopefully) be a manageable change

This led to the following decisions:
- the code is separated into subdirectories by concern, containing all the route handlers and tightly coupled code
- for now, these directories are in `server/packages` and `server/datasources`. I'm tempted to move them into `server/api/{packages,datasources}` but wanted to limit the amount of refactoring in one PR
- shared code lives in `server/lib`
- some code from `server/packages` has been almost duplicated to handle saving to Datasource saved objects, some has been refactored and is used from both places. The deduplication needs further improvement
- maybe `server/registry` should also move under `server/lib` (but see above, I'm trying to not move everything around all at once)

Testing:

* Please note that this is a breaking change because the saved object type for package information
has also been renamed. You'll need to start with a fresh `.kibana-*` index. Restarting `yarn es snapshot` (withouth specifying a data directory) should do the trick.

* Package installation should still work, e.g. with a GET request to `http://localhost:5601/api/epm/package/coredns-1.0.1`. The saved objects for packages can be inspected with a GET request to `http://localhost:5601/api/saved_objects/epm-package/$PKG_KEY`, e.g. `http://localhost:5601/api/saved_objects/epm-package/coredns-1.0.1`
* Datasource creation should still work, e.g. with a GET request to `http://localhost:5601/api/epm/datasource/install/coredns-1.0.1`. The saved objects for datasources can be inspected with a GET to `http://localhost:5601/api/saved_objects/epm-datasource/$PKG_KEY`, e.g. `http://localhost:5601/api/saved_objects/epm-datasource/coredns-1.0.1`

The registry slightly changed the .tar.gz path because of download stats reason. This adjusts for it. See elastic/package-registry#169

* [EPM] Move template installation to lib and add asset helper

All the logic related to the installation of the templates for a package should be inside the template library folder. This moves the logic into this folder.

A few refactorings were made to simplify installation:

* Introduction of DataSet interface: This interface is needed to extract the data sets inside a package and install one template per data set.
* Pass package instead of package key to installation process: Passing the package instead of the package key means fetching of package information is decoupled from the installation process and abstracted. This separates the two concerns and should simplify testing.
* getAsssets method: The getAssets methods works on top of the package object to extract asset paths. It is inspired by get_objects methods but supports passing a package and a dataset.

Currently one problem with testing that exists is that to fetch the content of an asset is not decoupled yet.

Between Fleet / Ingest / EPM there had been several interface definitions of Datasource and the related types. This reduces it to one place for the definition. The same applies to the policy definition.

The goal of this is that from now on we all rely on the same definition. If we make changes, we make them in all parts of the code.

In this PR is only the minimal change needed to get us all on one interface. Further changes will be needed that we all rely on the same saved objects etc.

* add export command

* revert 2 more files to rely on export

* revert imports

* Fix types for Datasource Saved Object

* merge in master

* fix type check

Learned about it on Slack from https://twitter.com/ryanchenkie/status/1201883268527927301

Blog at https://code.visualstudio.com/updates/v1_23#_run-code-actions-on-save

Basically does the order we've been loosely following (3rd party, then relative) & alphabetic by location and variable name.

It's not customizable but it's reasonable and, afaict, consistent.

…52229)

* Move cache 'hack' into getAssetsData

* p -> pkg. package is reserved. pkgkey is used in many places

* Remove unnecessary type cast

* Clarify reasons behind asset path manipulation

* Return the Datasource; not the Saved Object.

* Use real values from package in fake datasource SO

* Error if /datasource/install before /package/install

```
> curl --user elastic:changeme localhost:5601/api/epm/datasource/install/coredns-1.0.1
{
  "statusCode": 403,
  "error": "Forbidden",
  "message": "coredns-1.0.1 is not installed"
}

> curl --user elastic:changeme localhost:5601/api/epm/install/coredns-1.0.1
[
  {
    "id": "53aa1f70-443e-11e9-8548-ab7fbe04f038",
    "type": "dashboard"
  },
  {
    "id": "Metricbeat-CoreDNS-Dashboard-ecs",
    "type": "dashboard"
  },
  {
    "id": "75743f70-443c-11e9-8548-ab7fbe04f038",
    "type": "visualization"
  },
  {
    "id": "36e08510-53c4-11e9-b466-9be470bbd327-ecs",
    "type": "visualization"
  },
  {
    "id": "277fc650-67a9-11e9-a534-715561d0bf42",
    "type": "visualization"
  },
  {
    "id": "cfde7fb0-443d-11e9-8548-ab7fbe04f038",
    "type": "visualization"
  },
  {
    "id": "a19df590-53c4-11e9-b466-9be470bbd327-ecs",
    "type": "visualization"
  },
  {
    "id": "a58345f0-7298-11e9-b0d0-414c3011ddbb",
    "type": "visualization"
  },
  {
    "id": "9dc640e0-4432-11e9-8548-ab7fbe04f038",
    "type": "visualization"
  },
  {
    "id": "3ad75810-4429-11e9-8548-ab7fbe04f038",
    "type": "visualization"
  },
  {
    "id": "57c74300-7308-11e9-b0d0-414c3011ddbb",
    "type": "visualization"
  },
  {
    "id": "27da53f0-53d5-11e9-b466-9be470bbd327-ecs",
    "type": "visualization"
  },
  {
    "id": "86177430-728d-11e9-b0d0-414c3011ddbb",
    "type": "visualization"
  },
  {
    "id": "4804eaa0-7315-11e9-b0d0-414c3011ddbb",
    "type": "visualization"
  }
]

> curl --user elastic:changeme localhost:5601/api/epm/datasource/install/coredns-1.0.1
[
  {
    "id": "coredns_1_0_1_dataset_log_elasticsearch_ingest_pipeline_pipeline_plaintext_json",
    "type": "ingest-pipeline"
  },
  {
    "id": "coredns_1_0_1_dataset_log_elasticsearch_ingest_pipeline_pipeline_json_json",
    "type": "ingest-pipeline"
  },
  {
    "id": "coredns_1_0_1_dataset_log_elasticsearch_ingest_pipeline_pipeline_entry_json",
    "type": "ingest-pipeline"
  }
]
```

* fix duplicated imports

* [EPM] Move golden files generation over to jest snapshot

I initially used my own implementation to write the generated files. It runs out jest has a feature to write snapshots which simplifies the code a lot.

I added a loop with an additional test file so in the future we can just keep adding test files without having to modify the test code.

To updated the snapshots, the param `-u` has to be used:

```
node scripts/jest legacy/plugins/epm/server/lib/fields/field.test.ts -u
```

This creates the very basic Kibana index patterns metrics-* and logs-* for Kibana. At the moment it is overwritten every time. We need to change this in the future to take the fields from all installed data sources and regenerate it.

Most of the Elasticsearch assets have the same base name. This creates a helper to get the base name for the assets. In case we decide to change the base name in the future, we can change it in one place.

* fix tests and destructing

…urces. (#52285)

* Update RegistryPackage type

* Use download key from EPR to fetch archive

* Fix errors caused by correcting the Registry types.

The issues were largely that some Registry types like `title, `datasets` and `assets` where marked as required, but are actually optional. This highlighted area in the code were we relied on them always being present.

We added to the issue by wrapping Registry types in `Required` which made those items which were correctly listed as optional, required for EPM code. Updated EPM types to reflect the largely pass-through nature of the EPM types.

There are two properties which we ensure are in every EPM response, those were put into their own (unexported) type.

Confirm by trying to add a datasource to a package which has no datasources, like apache-1.0.1 or system-2.0.1. In `feature-ingest` and the earlier version of this PR, the `/datasource/install` call returns a 500. In this PR it succeeds.

This creates two ILM policies: logs-default and metrics-default. These are the default ILM policies used. Currently the policy content is hardcoded in the code but should be fetched from the base package in the future. The setup happens as part of the datasource installation. When a data source is installed it is a good time to check if the assets are there but we might extract this to a better place in the future.

* Update RegistryPackage type

* add first page of add data source

* fix for ie11 flex min width bug

* remove toDetailViewRelative

* remove unneeded spread

 * `RegistryList -> RegistrySearchResults`
 * `RegistryListItem -> RegistrySearchResult`

Many of the changes from #52234 were lost. Presumably due to PR(s) merging which were based on branches which had the previous unsorted order.

 * PackageNotInstalledError -> packages/index.ts
 * pkgToPkgKey -> registry/index.ts (will convert existing `${name}-${version}` instances later)

* Replace export * from packages.

There's an argument that the import sites should be updated to import from `packages/get`, `packages/install`, etc but that can wait for a later PR.

* Delete existing Installation type. Rename InstallationAttributes to Installation

* Reduce usage of EPM SO. Add getInstallation().

Replaced two calls of getInstallationObject() with getInstallation().

Two less places with knowledge of SO internals.

Lots of potential improvements for EPM TS types remain (refactoring/removing Installable, etc), but this is a good incremental step, IMO

Without it, things break. I am surprised CI did not catch this.

Before the check for the ILM policy to exist triggered an exception. With this change it is a normal response also if the policy does not exist yet.

A follow up issue will be created in Elasticsearch to get a HEAD request for this available.

The old cluster with the registry will be removed as soon as this is merged.

This will make sure we have to pass much feature params and can fully rely on the datasource object to create names for assets.

* Add fixtures for data source integration test.

* Move test setup to beforeEach

* Add test for datasource creation

* Handle pipelines in yml format.

* Make integration test for adding a data source pass.

See #52626 (review)

* remove dupe type RegistryPackage

* change switches to checkboxes, use datasets to create checkboxes, add some local form state

* update types

* add callback after successful installation and redirect

* add temp data sources tab content to access add data source page

* remove assets tab for mvp

* hide data sources link and redirect from data sources tab if package not installed

* change callback name

* remove commented out assets logic

* add redirect to hook

* fix type

…ingest-fix

Incremental change. Uses HTTP API for datasource creation. Will do follow-up PR which uses JS function instead

* Remove duplicate fetchInfo & installTemplates

I think this was from a bad merge, but pretty sure we don't want these functions called twice in the same function

* WIP. Pushing so others can see

* Improve correctness/flexibility of absolute URL

* Disable datasource test & template installation

* Clean up ingest imports and remove unneeded mock_spec files

* Initial pass at datasources lib and API endpoints

* Add add/remove datasource to/from policy API endpoints

* Add datasource contract tests and related policy contract tests; update snapshots

* Fix tests

* Fix tests again

* Fix tests 3

* Adjust routes, PR feedback

* change epm/datasource/install/{pkg} to POST, send user data to endpoint, install pipelines and templates based on user selected datasets

* change test to post for installing a datasource

* change some names and types around

* delete request.headers['transfer-encoding'] being passed through from epm request

It's not shared between client & server so it doesn't need to be in common. Also, it imports server code which would try to bring server types to the client. It's types so they're compiled away but it's important to keep common to what's truly common. Breaking this separation is why we thought enums broke the client. A lint rule just landed in master to prevent this.

* Only add keyword type field to mappings.

* Index template installation

* Handle empty fields definition files
* Re-enable index template installation

* Add index files to export various modules; normalize imports

* Clean up unused files; extract datasources table component from policy details page

* Expose http client to frontend libs; remove unused types; import ES UI's useRequest lib

* Adjust shape of rest api adapter interface to better match with rest of kibana; remove unused node adapter; change per_page param to perPage in agent events route

* Initial pass at assign data sources flyout

* Initial pass at unassigning data sources from policy

* Make data sources table searchable by package values

* Fix enrollment key lib for rest adapter param changes

* Fix imports and types

* `yarn.lock` changes after bootstrapping