From 4b9a61ef2d22a6480959883bdfa8ae7875c349a5 Mon Sep 17 00:00:00 2001 From: Stacey Gammon Date: Mon, 13 Jul 2020 10:47:01 -0400 Subject: [PATCH] Improvements to our developer guide (#67764) * contributing guide -> asciidoc * Update docs/developer/contributing/index.asciidoc Co-authored-by: Peter Schretlen * Update CONTRIBUTING.md Co-authored-by: Peter Schretlen * Update docs/developer/best-practices/stability.asciidoc Co-authored-by: Peter Schretlen * Update docs/developer/contributing/index.asciidoc Co-authored-by: Peter Schretlen * address code review comments * Update docs/developer/contributing/development-documentation.asciidoc Co-authored-by: Peter Schretlen * review comment updates * fix bad ref Co-authored-by: Peter Schretlen # Conflicts: # CONTRIBUTING.md # docs/developer/contributing/development-functional-tests.asciidoc # docs/developer/contributing/development-unit-tests.asciidoc --- CONTRIBUTING.md | 727 +----------------- docs/developer/add-data-guide.asciidoc | 38 - .../advanced/development-basepath.asciidoc | 18 + .../development-es-snapshots.asciidoc | 22 +- docs/developer/advanced/index.asciidoc | 12 + .../advanced/running-elasticsearch.asciidoc | 118 +++ .../architecture/add-data-tutorials.asciidoc | 38 + .../development-visualize-index.asciidoc | 8 +- docs/developer/architecture/index.asciidoc | 25 + .../security/feature-registration.asciidoc} | 20 +- .../architecture/security/index.asciidoc | 12 + .../{ => architecture}/security/rbac.asciidoc | 10 +- docs/developer/best-practices/index.asciidoc | 136 ++++ .../best-practices/security.asciidoc | 55 ++ .../best-practices/stability.asciidoc | 66 ++ .../development-accessibility-tests.asciidoc | 23 + .../development-documentation.asciidoc | 34 + .../development-functional-tests.asciidoc | 48 +- .../contributing/development-github.asciidoc | 112 +++ .../development-pull-request.asciidoc | 32 + .../contributing/development-tests.asciidoc | 96 +++ .../development-unit-tests.asciidoc | 87 ++- docs/developer/contributing/index.asciidoc | 89 +++ .../interpreting-ci-failures.asciidoc | 10 +- .../kibana-issue-reporting.asciidoc | 46 ++ docs/developer/contributing/linting.asciidoc | 70 ++ .../{ => contributing}/pr-review.asciidoc | 28 +- docs/developer/core-development.asciidoc | 24 - .../core/development-basepath.asciidoc | 85 -- .../core/development-dependencies.asciidoc | 103 --- .../core/development-elasticsearch.asciidoc | 40 - .../core/development-modules.asciidoc | 63 -- .../getting-started/building-kibana.asciidoc | 39 + .../getting-started/debugging.asciidoc | 59 ++ .../development-plugin-resources.asciidoc | 52 +- docs/developer/getting-started/index.asciidoc | 144 ++++ .../running-kibana-advanced.asciidoc | 87 +++ .../getting-started/sample-data.asciidoc | 31 + docs/developer/getting-started/sass.asciidoc | 36 + docs/developer/index.asciidoc | 28 +- docs/developer/plugin-development.asciidoc | 24 - .../plugin/development-uiexports.asciidoc | 16 - ...external-plugin-functional-tests.asciidoc} | 16 +- ... => external-plugin-localization.asciidoc} | 22 +- docs/developer/plugin/index.asciidoc | 42 + docs/developer/security/index.asciidoc | 12 - 46 files changed, 1630 insertions(+), 1273 deletions(-) delete mode 100644 docs/developer/add-data-guide.asciidoc create mode 100644 docs/developer/advanced/development-basepath.asciidoc rename docs/developer/{core => advanced}/development-es-snapshots.asciidoc (90%) create mode 100644 docs/developer/advanced/index.asciidoc create mode 100644 docs/developer/advanced/running-elasticsearch.asciidoc create mode 100644 docs/developer/architecture/add-data-tutorials.asciidoc rename docs/developer/{visualize => architecture}/development-visualize-index.asciidoc (85%) create mode 100644 docs/developer/architecture/index.asciidoc rename docs/developer/{plugin/development-plugin-feature-registration.asciidoc => architecture/security/feature-registration.asciidoc} (96%) create mode 100644 docs/developer/architecture/security/index.asciidoc rename docs/developer/{ => architecture}/security/rbac.asciidoc (96%) create mode 100644 docs/developer/best-practices/index.asciidoc create mode 100644 docs/developer/best-practices/security.asciidoc create mode 100644 docs/developer/best-practices/stability.asciidoc create mode 100644 docs/developer/contributing/development-accessibility-tests.asciidoc create mode 100644 docs/developer/contributing/development-documentation.asciidoc rename docs/developer/{core => contributing}/development-functional-tests.asciidoc (90%) create mode 100644 docs/developer/contributing/development-github.asciidoc create mode 100644 docs/developer/contributing/development-pull-request.asciidoc create mode 100644 docs/developer/contributing/development-tests.asciidoc rename docs/developer/{core => contributing}/development-unit-tests.asciidoc (52%) create mode 100644 docs/developer/contributing/index.asciidoc rename docs/developer/{testing => contributing}/interpreting-ci-failures.asciidoc (87%) create mode 100644 docs/developer/contributing/kibana-issue-reporting.asciidoc create mode 100644 docs/developer/contributing/linting.asciidoc rename docs/developer/{ => contributing}/pr-review.asciidoc (90%) delete mode 100644 docs/developer/core-development.asciidoc delete mode 100644 docs/developer/core/development-basepath.asciidoc delete mode 100644 docs/developer/core/development-dependencies.asciidoc delete mode 100644 docs/developer/core/development-elasticsearch.asciidoc delete mode 100644 docs/developer/core/development-modules.asciidoc create mode 100644 docs/developer/getting-started/building-kibana.asciidoc create mode 100644 docs/developer/getting-started/debugging.asciidoc rename docs/developer/{plugin => getting-started}/development-plugin-resources.asciidoc (51%) create mode 100644 docs/developer/getting-started/index.asciidoc create mode 100644 docs/developer/getting-started/running-kibana-advanced.asciidoc create mode 100644 docs/developer/getting-started/sample-data.asciidoc create mode 100644 docs/developer/getting-started/sass.asciidoc delete mode 100644 docs/developer/plugin-development.asciidoc delete mode 100644 docs/developer/plugin/development-uiexports.asciidoc rename docs/developer/plugin/{development-plugin-functional-tests.asciidoc => external-plugin-functional-tests.asciidoc} (79%) rename docs/developer/plugin/{development-plugin-localization.asciidoc => external-plugin-localization.asciidoc} (87%) create mode 100644 docs/developer/plugin/index.asciidoc delete mode 100644 docs/developer/security/index.asciidoc diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 5b9185e7b608e..11c595a1ad983 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,728 +1,5 @@ # Contributing to Kibana -We understand that you may not have days at a time to work on Kibana. We ask that you read our contributing guidelines carefully so that you spend less time, overall, struggling to push your PR through our code review processes. +We understand that you may not have days at a time to work on Kibana. We ask that you read our [developer guide](https://www.elastic.co/guide/en/kibana/master/development.html) carefully so that you spend less time, overall, struggling to push your PR through our code review processes. -At the same time, reading the contributing guidelines will give you a better idea of how to post meaningful issues that will be more easily be parsed, considered, and resolved. A big win for everyone involved! :tada: - -## Table of Contents - -A high level overview of our contributing guidelines. - -- [Effective issue reporting in Kibana](#effective-issue-reporting-in-kibana) - - [Voicing the importance of an issue](#voicing-the-importance-of-an-issue) - - ["My issue isn't getting enough attention"](#my-issue-isnt-getting-enough-attention) - - ["I want to help!"](#i-want-to-help) -- [How We Use Git and GitHub](#how-we-use-git-and-github) - - [Forking](#forking) - - [Branching](#branching) - - [Commits and Merging](#commits-and-merging) - - [Rebasing and fixing merge conflicts](#rebasing-and-fixing-merge-conflicts) - - [What Goes Into a Pull Request](#what-goes-into-a-pull-request) -- [Contributing Code](#contributing-code) - - [Setting Up Your Development Environment](#setting-up-your-development-environment) - - [Increase node.js heap size](#increase-nodejs-heap-size) - - [Running Elasticsearch Locally](#running-elasticsearch-locally) - - [Nightly snapshot (recommended)](#nightly-snapshot-recommended) - - [Keeping data between snapshots](#keeping-data-between-snapshots) - - [Source](#source) - - [Archive](#archive) - - [Sample Data](#sample-data) - - [Running Elasticsearch Remotely](#running-elasticsearch-remotely) - - [Running remote clusters](#running-remote-clusters) - - [Running Kibana](#running-kibana) - - [Running Kibana in Open-Source mode](#running-kibana-in-open-source-mode) - - [Unsupported URL Type](#unsupported-url-type) - - [Customizing `config/kibana.dev.yml`](#customizing-configkibanadevyml) - - [Potential Optimization Pitfalls](#potential-optimization-pitfalls) - - [Setting Up SSL](#setting-up-ssl) - - [Linting](#linting) - - [Setup Guide for VS Code Users](#setup-guide-for-vs-code-users) - - [Internationalization](#internationalization) - - [Localization](#localization) - - [Styling with SASS](#styling-with-sass) - - [Testing and Building](#testing-and-building) - - [Debugging server code](#debugging-server-code) - - [Instrumenting with Elastic APM](#instrumenting-with-elastic-apm) - - [Unit testing frameworks](#unit-testing-frameworks) - - [Running specific Kibana tests](#running-specific-kibana-tests) - - [Debugging Unit Tests](#debugging-unit-tests) - - [Unit Testing Plugins](#unit-testing-plugins) - - [Automated Accessibility Testing](#automated-accessibility-testing) - - [Cross-browser compatibility](#cross-browser-compatibility) - - [Testing compatibility locally](#testing-compatibility-locally) - - [Running Browser Automation Tests](#running-browser-automation-tests) - - [Building OS packages](#building-os-packages) - - [Writing documentation](#writing-documentation) - - [Release Notes Process](#release-notes-process) -- [Signing the contributor license agreement](#signing-the-contributor-license-agreement) -- [Submitting a Pull Request](#submitting-a-pull-request) -- [Code Reviewing](#code-reviewing) - - [Getting to the Code Review Stage](#getting-to-the-code-review-stage) - - [Reviewing Pull Requests](#reviewing-pull-requests) - -Don't fret, it's not as daunting as the table of contents makes it out to be! - -## Effective issue reporting in Kibana - -### Voicing the importance of an issue - -We seriously appreciate thoughtful comments. If an issue is important to you, add a comment with a solid write up of your use case and explain why it's so important. Please avoid posting comments comprised solely of a thumbs up emoji 👍. - -Granted that you share your thoughts, we might even be able to come up with creative solutions to your specific problem. If everything you'd like to say has already been brought up but you'd still like to add a token of support, feel free to add a [👍 thumbs up reaction](https://github.com/blog/2119-add-reactions-to-pull-requests-issues-and-comments) on the issue itself and on the comment which best summarizes your thoughts. - -### "My issue isn't getting enough attention" - -First of all, **sorry about that!** We want you to have a great time with Kibana. - -There's hundreds of open issues and prioritizing what to work on is an important aspect of our daily jobs. We prioritize issues according to impact and difficulty, so some issues can be neglected while we work on more pressing issues. - -Feel free to bump your issues if you think they've been neglected for a prolonged period. - -### "I want to help!" - -**Now we're talking**. If you have a bug fix or new feature that you would like to contribute to Kibana, please **find or open an issue about it before you start working on it.** Talk about what you would like to do. It may be that somebody is already working on it, or that there are particular issues that you should know about before implementing the change. - -We enjoy working with contributors to get their code accepted. There are many approaches to fixing a problem and it is important to find the best approach before writing too much code. - -## How We Use Git and GitHub - -### Forking - -We follow the [GitHub forking model](https://help.github.com/articles/fork-a-repo/) for collaborating -on Kibana code. This model assumes that you have a remote called `upstream` which points to the -official Kibana repo, which we'll refer to in later code snippets. - -### Branching - -* All work on the next major release goes into master. -* Past major release branches are named `{majorVersion}.x`. They contain work that will go into the next minor release. For example, if the next minor release is `5.2.0`, work for it should go into the `5.x` branch. -* Past minor release branches are named `{majorVersion}.{minorVersion}`. They contain work that will go into the next patch release. For example, if the next patch release is `5.3.1`, work for it should go into the `5.3` branch. -* All work is done on feature branches and merged into one of these branches. -* Where appropriate, we'll backport changes into older release branches. - -### Commits and Merging - -* Feel free to make as many commits as you want, while working on a branch. -* When submitting a PR for review, please perform an interactive rebase to present a logical history that's easy for the reviewers to follow. -* Please use your commit messages to include helpful information on your changes, e.g. changes to APIs, UX changes, bugs fixed, and an explanation of *why* you made the changes that you did. -* Resolve merge conflicts by rebasing the target branch over your feature branch, and force-pushing (see below for instructions). -* When merging, we'll squash your commits into a single commit. - -#### Rebasing and fixing merge conflicts - -Rebasing can be tricky, and fixing merge conflicts can be even trickier because it involves force pushing. This is all compounded by the fact that attempting to push a rebased branch remotely will be rejected by git, and you'll be prompted to do a `pull`, which is not at all what you should do (this will really mess up your branch's history). - -Here's how you should rebase master onto your branch, and how to fix merge conflicts when they arise. - -First, make sure master is up-to-date. - -``` -git checkout master -git fetch upstream -git rebase upstream/master -``` - -Then, check out your branch and rebase master on top of it, which will apply all of the new commits on master to your branch, and then apply all of your branch's new commits after that. - -``` -git checkout name-of-your-branch -git rebase master -``` - -You want to make sure there are no merge conflicts. If there are merge conflicts, git will pause the rebase and allow you to fix the conflicts before continuing. - -You can use `git status` to see which files contain conflicts. They'll be the ones that aren't staged for commit. Open those files, and look for where git has marked the conflicts. Resolve the conflicts so that the changes you want to make to the code have been incorporated in a way that doesn't destroy work that's been done in master. Refer to master's commit history on GitHub if you need to gain a better understanding of how code is conflicting and how best to resolve it. - -Once you've resolved all of the merge conflicts, use `git add -A` to stage them to be committed, and then use `git rebase --continue` to tell git to continue the rebase. - -When the rebase has completed, you will need to force push your branch because the history is now completely different than what's on the remote. **This is potentially dangerous** because it will completely overwrite what you have on the remote, so you need to be sure that you haven't lost any work when resolving merge conflicts. (If there weren't any merge conflicts, then you can force push without having to worry about this.) - -``` -git push origin name-of-your-branch --force -``` - -This will overwrite the remote branch with what you have locally. You're done! - -**Note that you should not run `git pull`**, for example in response to a push rejection like this: - -``` -! [rejected] name-of-your-branch -> name-of-your-branch (non-fast-forward) -error: failed to push some refs to 'https://github.com/YourGitHubHandle/kibana.git' -hint: Updates were rejected because the tip of your current branch is behind -hint: its remote counterpart. Integrate the remote changes (e.g. -hint: 'git pull ...') before pushing again. -hint: See the 'Note about fast-forwards' in 'git push --help' for details. -``` - -Assuming you've successfully rebased and you're happy with the code, you should force push instead. - -### What Goes Into a Pull Request - -* Please include an explanation of your changes in your PR description. -* Links to relevant issues, external resources, or related PRs are very important and useful. -* Please update any tests that pertain to your code, and add new tests where appropriate. -* See [Submitting a Pull Request](#submitting-a-pull-request) for more info. - -## Contributing Code - -These guidelines will help you get your Pull Request into shape so that a code review can start as soon as possible. - -### Setting Up Your Development Environment - -Fork, then clone the `kibana` repo and change directory into it - -```bash -git clone https://github.com/[YOUR_USERNAME]/kibana.git kibana -cd kibana -``` - -Install the version of Node.js listed in the `.node-version` file. This can be automated with tools such as [nvm](https://github.com/creationix/nvm), [nvm-windows](https://github.com/coreybutler/nvm-windows) or [avn](https://github.com/wbyoung/avn). As we also include a `.nvmrc` file you can switch to the correct version when using nvm by running: - -```bash -nvm use -``` - -Install the latest version of [yarn](https://yarnpkg.com). - -Bootstrap Kibana and install all the dependencies - -```bash -yarn kbn bootstrap -``` - -> Node.js native modules could be in use and node-gyp is the tool used to build them. There are tools you need to install per platform and python versions you need to be using. Please see https://github.com/nodejs/node-gyp#installation and follow the guide according your platform. - -(You can also run `yarn kbn` to see the other available commands. For more info about this tool, see https://github.com/elastic/kibana/tree/master/packages/kbn-pm.) - -#### Increase node.js heap size - -Kibana is a big project and for some commands it can happen that the process hits the default heap limit and crashes with an out-of-memory error. If you run into this problem, you can increase maximum heap size by setting the `--max_old_space_size` option on the command line. To set the limit for all commands, simply add the following line to your shell config: `export NODE_OPTIONS="--max_old_space_size=2048"`. - -### Running Elasticsearch Locally - -There are a few options when it comes to running Elasticsearch locally: - -#### Nightly snapshot (recommended) - -These snapshots are built on a nightly basis which expire after a couple weeks. If running from an old, untracted branch this snapshot might not exist. In which case you might need to run from source or an archive. - -```bash -yarn es snapshot -``` - -##### Keeping data between snapshots - -If you want to keep the data inside your Elasticsearch between usages of this command, -you should use the following command, to keep your data folder outside the downloaded snapshot -folder: - -```bash -yarn es snapshot -E path.data=../data -``` - -The same parameter can be used with the source and archive command shown in the following -paragraphs. - -#### Source - -By default, it will reference an [elasticsearch](https://github.com/elastic/elasticsearch) checkout which is a sibling to the Kibana directory named `elasticsearch`. If you wish to use a checkout in another location you can provide that by supplying `--source-path` - -```bash -yarn es source -``` - -#### Archive - -Use this if you already have a distributable. For released versions, one can be obtained on the [Elasticsearch downloads](https://www.elastic.co/downloads/elasticsearch) page. - -```bash -yarn es archive -``` - -**Each of these will run Elasticsearch with a `basic` license. Additional options are available, pass `--help` for more information.** - -##### Sample Data - -If you're just getting started with Elasticsearch, you could use the following command to populate your instance with a few fake logs to hit the ground running. - -```bash -node scripts/makelogs --auth : -``` -> The default username and password combination are `elastic:changeme` - -> Make sure to execute `node scripts/makelogs` *after* elasticsearch is up and running! - -### Running Elasticsearch Remotely - -You can save some system resources, and the effort of generating sample data, if you have a remote Elasticsearch cluster to connect to. (**Elasticians: you do! Check with your team about where to find credentials**) - -You'll need to [create a `kibana.dev.yml`](#customizing-configkibanadevyml) and add the following to it: - -``` -elasticsearch.hosts: - - {{ url }} -elasticsearch.username: {{ username }} -elasticsearch.password: {{ password }} -elasticsearch.ssl.verificationMode: none -``` - -If many other users will be interacting with your remote cluster, you'll want to add the following to avoid causing conflicts: - -``` -kibana.index: '.{YourGitHubHandle}-kibana' -xpack.task_manager.index: '.{YourGitHubHandle}-task-manager-kibana' -``` - -### Running remote clusters -Setup remote clusters for cross cluster search (CCS) and cross cluster replication (CCR). - -Start your primary cluster by running: -```bash -yarn es snapshot -E path.data=../data_prod1 -``` - -Start your remote cluster by running: -```bash -yarn es snapshot -E transport.port=9500 -E http.port=9201 -E path.data=../data_prod2 -``` - -Once both clusters are running, start kibana. Kibana will connect to the primary cluster. - -Setup the remote cluster in Kibana from either `Management` -> `Elasticsearch` -> `Remote Clusters` UI or by running the following script in `Console`. -``` -PUT _cluster/settings -{ - "persistent": { - "cluster": { - "remote": { - "cluster_one": { - "seeds": [ - "localhost:9500" - ] - } - } - } - } -} -``` - -Follow the [cross-cluster search](https://www.elastic.co/guide/en/kibana/current/management-cross-cluster-search.html) instructions for setting up index patterns to search across clusters. - -### Running Kibana - -Start the development server. - -```bash -yarn start -``` - -> On Windows, you'll need to use Git Bash, Cygwin, or a similar shell that exposes the `sh` command. And to successfully build you'll need Cygwin optional packages zip, tar, and shasum. - -Now you can point your web browser to http://localhost:5601 and start using Kibana! When running `yarn start`, Kibana will also log that it is listening on port 5603 due to the base path proxy, but you should still access Kibana on port 5601. - -By default, you can log in with username `elastic` and password `changeme`. See the `--help` options on `yarn es ` if you'd like to configure a different password. - -#### Running Kibana in Open-Source mode - -If you're looking to only work with the open-source software, supply the license type to `yarn es`: - -```bash -yarn es snapshot --license oss -``` - -And start Kibana with only open-source code: - -```bash -yarn start --oss -``` - -#### Unsupported URL Type - -If you're installing dependencies and seeing an error that looks something like - -``` -Unsupported URL Type: link:packages/eslint-config-kibana -``` - -you're likely running `npm`. To install dependencies in Kibana you need to run `yarn kbn bootstrap`. For more info, see [Setting Up Your Development Environment](#setting-up-your-development-environment) above. - -#### Customizing `config/kibana.dev.yml` - -The `config/kibana.yml` file stores user configuration directives. Since this file is checked into source control, however, developer preferences can't be saved without the risk of accidentally committing the modified version. To make customizing configuration easier during development, the Kibana CLI will look for a `config/kibana.dev.yml` file if run with the `--dev` flag. This file behaves just like the non-dev version and accepts any of the [standard settings](https://www.elastic.co/guide/en/kibana/current/settings.html). - -#### Potential Optimization Pitfalls - - - Webpack is trying to include a file in the bundle that I deleted and is now complaining about it is missing - - A module id that used to resolve to a single file now resolves to a directory, but webpack isn't adapting - - (if you discover other scenarios, please send a PR!) - -#### Setting Up SSL - -Kibana includes self-signed certificates that can be used for development purposes in the browser and for communicating with Elasticsearch: `yarn start --ssl` & `yarn es snapshot --ssl`. - -### Linting - -A note about linting: We use [eslint](http://eslint.org) to check that the [styleguide](STYLEGUIDE.md) is being followed. It runs in a pre-commit hook and as a part of the tests, but most contributors integrate it with their code editors for real-time feedback. - -Here are some hints for getting eslint setup in your favorite editor: - -Editor | Plugin ------------|------------------------------------------------------------------------------- -Sublime | [SublimeLinter-eslint](https://github.com/roadhump/SublimeLinter-eslint#installation) -Atom | [linter-eslint](https://github.com/AtomLinter/linter-eslint#installation) -VSCode | [ESLint](https://marketplace.visualstudio.com/items?itemName=dbaeumer.vscode-eslint) -IntelliJ | Settings » Languages & Frameworks » JavaScript » Code Quality Tools » ESLint -`vi` | [scrooloose/syntastic](https://github.com/scrooloose/syntastic) - -Another tool we use for enforcing consistent coding style is EditorConfig, which can be set up by installing a plugin in your editor that dynamically updates its configuration. Take a look at the [EditorConfig](http://editorconfig.org/#download) site to find a plugin for your editor, and browse our [`.editorconfig`](https://github.com/elastic/kibana/blob/master/.editorconfig) file to see what config rules we set up. - -#### Setup Guide for VS Code Users - -Note that for VSCode, to enable "live" linting of TypeScript (and other) file types, you will need to modify your local settings, as shown below. The default for the ESLint extension is to only lint JavaScript file types. - -```json -"eslint.validate": [ - "javascript", - "javascriptreact", - { "language": "typescript", "autoFix": true }, - { "language": "typescriptreact", "autoFix": true } -] -``` - -`eslint` can automatically fix trivial lint errors when you save a file by adding this line in your setting. - -```json - "eslint.autoFixOnSave": true, -``` - -:warning: It is **not** recommended to use the [`Prettier` extension/IDE plugin](https://prettier.io/) while maintaining the Kibana project. Formatting and styling roles are set in the multiple `.eslintrc.js` files across the project and some of them use the [NPM version of Prettier](https://www.npmjs.com/package/prettier). Using the IDE extension might cause conflicts, applying the formatting to too many files that shouldn't be prettier-ized and/or highlighting errors that are actually OK. - -### Internationalization - -All user-facing labels and info texts in Kibana should be internationalized. Please take a look at the [readme](packages/kbn-i18n/README.md) and the [guideline](packages/kbn-i18n/GUIDELINE.md) of the i18n package on how to do so. - -In order to enable translations in the React parts of the application, the top most component of every `ReactDOM.render` call should be the `Context` component from the `i18n` core service: -```jsx -const I18nContext = coreStart.i18n.Context; - -ReactDOM.render( - - {myComponentTree} - , - container -); -``` - -There are a number of tools created to support internationalization in Kibana that would allow one to validate internationalized labels, -extract them to a `JSON` file or integrate translations back to Kibana. To know more, please read corresponding [readme](src/dev/i18n/README.md) file. - -### Localization - -We cannot support accepting contributions to the translations from any source other than the translators we have engaged to do the work. -We are still to develop a proper process to accept any contributed translations. We certainly appreciate that people care enough about the localization effort to want to help improve the quality. We aim to build out a more comprehensive localization process for the future and will notify you once contributions can be supported, but for the time being, we are not able to incorporate suggestions. - -### Styling with SASS - -When writing a new component, create a sibling SASS file of the same name and import directly into the JS/TS component file. Doing so ensures the styles are never separated or lost on import and allows for better modularization (smaller individual plugin asset footprint). - -All SASS (.scss) files will automatically build with the [EUI](https://elastic.github.io/eui/#/guidelines/sass) & Kibana invisibles (SASS variables, mixins, functions) from the [`globals_[theme].scss` file](src/legacy/ui/public/styles/_globals_v7light.scss). - -**Example:** - -```tsx -// component.tsx - -import './component.scss'; - -export const Component = () => { - return ( -
- ); -} -``` - -```scss -// component.scss - -.plgComponent { ... } -``` - -Do not use the underscore `_` SASS file naming pattern when importing directly into a javascript file. - -### Testing and Building - -To ensure that your changes will not break other functionality, please run the test suite and build process before submitting your Pull Request. - -Before running the tests you will need to install the projects dependencies as described above. - -Once that's done, just run: - -```bash -yarn test && yarn build --skip-os-packages -``` - -You can get all build options using the following command: - -```bash -yarn build --help -``` - -macOS users on a machine with a discrete graphics card may see significant speedups (up to 2x) when running tests by changing your terminal emulator's GPU settings. In iTerm2: -- Open Preferences (Command + ,) -- In the General tab, under the "Magic" section, ensure "GPU rendering" is checked -- Open "Advanced GPU Settings..." -- Uncheck the "Prefer integrated to discrete GPU" option -- Restart iTerm - -#### Debugging Server Code -`yarn debug` will start the server with Node's inspect flag. Kibana's development mode will start three processes on ports `9229`, `9230`, and `9231`. Chrome's developer tools need to be configured to connect to all three connections. Add `localhost:` for each Kibana process in Chrome's developer tools connection tab. - -#### Instrumenting with Elastic APM -Kibana ships with the [Elastic APM Node.js Agent](https://github.com/elastic/apm-agent-nodejs) built-in for debugging purposes. - -Its default configuration is meant to be used by core Kibana developers only, but it can easily be re-configured to your needs. -In its default configuration it's disabled and will, once enabled, send APM data to a centrally managed Elasticsearch cluster accessible only to Elastic employees. - -To change the location where data is sent, use the [`serverUrl`](https://www.elastic.co/guide/en/apm/agent/nodejs/current/configuration.html#server-url) APM config option. -To activate the APM agent, use the [`active`](https://www.elastic.co/guide/en/apm/agent/nodejs/current/configuration.html#active) APM config option. - -All config options can be set either via environment variables, or by creating an appropriate config file under `config/apm.dev.js`. -For more information about configuring the APM agent, please refer to [the documentation](https://www.elastic.co/guide/en/apm/agent/nodejs/current/configuring-the-agent.html). - -Example `config/apm.dev.js` file: - -```js -module.exports = { - active: true, -}; -``` - -APM [Real User Monitoring agent](https://www.elastic.co/guide/en/apm/agent/rum-js/current/index.html) is not available in the Kibana distributables, -however the agent can be enabled by setting `ELASTIC_APM_ACTIVE` to `true`. -flags -``` -ELASTIC_APM_ACTIVE=true yarn start -// activates both Node.js and RUM agent -``` - -Once the agent is active, it will trace all incoming HTTP requests to Kibana, monitor for errors, and collect process-level metrics. -The collected data will be sent to the APM Server and is viewable in the APM UI in Kibana. - -#### Unit testing frameworks -Kibana is migrating unit testing from Mocha to Jest. Legacy unit tests still -exist in Mocha but all new unit tests should be written in Jest. Mocha tests -are contained in `__tests__` directories. Whereas Jest tests are stored in -the same directory as source code files with the `.test.js` suffix. - -#### Running specific Kibana tests - -The following table outlines possible test file locations and how to invoke them: - -| Test runner | Test location | Runner command (working directory is kibana root) | -| ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- | -| Jest | `src/**/*.test.js`
`src/**/*.test.ts` | `yarn test:jest -t regexp [test path]` | -| Jest (integration) | `**/integration_tests/**/*.test.js` | `yarn test:jest_integration -t regexp [test path]` | -| Mocha | `src/**/__tests__/**/*.js`
`!src/**/public/__tests__/*.js`
`packages/kbn-datemath/test/**/*.js`
`packages/kbn-dev-utils/src/**/__tests__/**/*.js`
`tasks/**/__tests__/**/*.js` | `node scripts/mocha --grep=regexp [test path]` | -| Functional | `test/*integration/**/config.js`
`test/*functional/**/config.js`
`test/accessibility/config.js` | `yarn test:ftr:server --config test/[directory]/config.js`
`yarn test:ftr:runner --config test/[directory]/config.js --grep=regexp` | -| Karma | `src/**/public/__tests__/*.js` | `yarn test:karma:debug` | - -For X-Pack tests located in `x-pack/` see [X-Pack Testing](x-pack/README.md#testing) - -Test runner arguments: - - Where applicable, the optional arguments `-t=regexp` or `--grep=regexp` will only run tests or test suites whose descriptions matches the regular expression. - - `[test path]` is the relative path to the test file. - - Examples: - - Run the entire elasticsearch_service test suite: - ``` - yarn test:jest src/core/server/elasticsearch/elasticsearch_service.test.ts - ``` - - Run the jest test case whose description matches `stops both admin and data clients`: - ``` - yarn test:jest -t 'stops both admin and data clients' src/core/server/elasticsearch/elasticsearch_service.test.ts - ``` - - Run the api integration test case whose description matches the given string: - ``` - yarn test:ftr:server --config test/api_integration/config.js - yarn test:ftr:runner --config test/api_integration/config.js --grep='should return 404 if id does not match any sample data sets' - ``` - -#### Debugging Unit Tests - -The standard `yarn test` task runs several sub tasks and can take several minutes to complete, making debugging failures pretty painful. In order to ease the pain specialized tasks provide alternate methods for running the tests. - -You could also add the `--debug` option so that `node` is run using the `--debug-brk` flag. You'll need to connect a remote debugger such as [`node-inspector`](https://github.com/node-inspector/node-inspector) to proceed in this mode. - -```bash -node scripts/mocha --debug -``` - -With `yarn test:karma`, you can run only the browser tests. Coverage reports are available for browser tests by running `yarn test:coverage`. You can find the results under the `coverage/` directory that will be created upon completion. - -```bash -yarn test:karma -``` - -Using `yarn test:karma:debug` initializes an environment for debugging the browser tests. Includes an dedicated instance of the kibana server for building the test bundle, and a karma server. When running this task the build is optimized for the first time and then a karma-owned instance of the browser is opened. Click the "debug" button to open a new tab that executes the unit tests. - -```bash -yarn test:karma:debug -``` - -In the screenshot below, you'll notice the URL is `localhost:9876/debug.html`. You can append a `grep` query parameter to this URL and set it to a string value which will be used to exclude tests which don't match. For example, if you changed the URL to `localhost:9876/debug.html?query=my test` and then refreshed the browser, you'd only see tests run which contain "my test" in the test description. - - -![Browser test debugging](http://i.imgur.com/DwHxgfq.png) - -#### Unit Testing Plugins - -This should work super if you're using the [Kibana plugin generator](https://github.com/elastic/kibana/tree/master/packages/kbn-plugin-generator). If you're not using the generator, well, you're on your own. We suggest you look at how the generator works. - -To run the tests for just your particular plugin run the following command from your plugin: - -```bash -yarn test:mocha -yarn test:karma:debug # remove the debug flag to run them once and close -``` - -#### Automated Accessibility Testing - -To run the tests locally: - -1. In one terminal window run `node scripts/functional_tests_server --config test/accessibility/config.ts` -2. In another terminal window run `node scripts/functional_test_runner.js --config test/accessibility/config.ts` - -To run the x-pack tests, swap the config file out for `x-pack/test/accessibility/config.ts`. - -After the server is up, you can go to this instance of Kibana at `localhost:5620`. - -The testing is done using [axe](https://github.com/dequelabs/axe-core). The same thing that runs in CI, -can be run locally using their browser plugins: - -- [Chrome](https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US) -- [Firefox](https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/) - -#### Cross-browser Compatibility - -##### Testing Compatibility Locally - -###### Testing IE on OS X - -* [Download VMWare Fusion](http://www.vmware.com/products/fusion/fusion-evaluation.html). -* [Download IE virtual machines](https://developer.microsoft.com/en-us/microsoft-edge/tools/vms/#downloads) for VMWare. -* Open VMWare and go to Window > Virtual Machine Library. Unzip the virtual machine and drag the .vmx file into your Virtual Machine Library. -* Right-click on the virtual machine you just added to your library and select "Snapshots...", and then click the "Take" button in the modal that opens. You can roll back to this snapshot when the VM expires in 90 days. -* In System Preferences > Sharing, change your computer name to be something simple, e.g. "computer". -* Run Kibana with `yarn start --host=computer.local` (substituting your computer name). -* Now you can run your VM, open the browser, and navigate to `http://computer.local:5601` to test Kibana. -* Alternatively you can use browserstack - -##### Running Browser Automation Tests - -[Read about the `FunctionalTestRunner`](https://www.elastic.co/guide/en/kibana/current/development-functional-tests.html) to learn more about how you can run and develop functional tests for Kibana core and plugins. - -You can also look into the [Scripts README.md](./scripts/README.md) to learn more about using the node scripts we provide for building Kibana, running integration tests, and starting up Kibana and Elasticsearch while you develop. - -### Building OS packages - -Packages are built using fpm, dpkg, and rpm. Package building has only been tested on Linux and is not supported on any other platform. - -```bash -apt-get install ruby-dev rpm -gem install fpm -v 1.5.0 -yarn build --skip-archives -``` - -To specify a package to build you can add `rpm` or `deb` as an argument. - -```bash -yarn build --rpm -``` - -Distributable packages can be found in `target/` after the build completes. - -### Writing documentation - -Kibana documentation is written in [asciidoc](http://asciidoc.org/) format in -the `docs/` directory. - -To build the docs, clone the [elastic/docs](https://github.com/elastic/docs) -repo as a sibling of your Kibana repo. Follow the instructions in that project's -README for getting the docs tooling set up. - -**To build the Kibana docs and open them in your browser:** - -```bash -./docs/build_docs --doc kibana/docs/index.asciidoc --chunk 1 --open -``` -or - -```bash -node scripts/docs.js --open -``` - -### Release Notes process - -Part of this process only applies to maintainers, since it requires access to GitHub labels. - -Kibana publishes [Release Notes](https://www.elastic.co/guide/en/kibana/current/release-notes.html) for major and minor releases. The Release Notes summarize what the PRs accomplish in language that is meaningful to users. To generate the Release Notes, the team runs a script against this repo to collect the merged PRs against the release. - -#### Create the Release Notes text -The text that appears in the Release Notes is pulled directly from your PR title, or a single paragraph of text that you specify in the PR description. - -To use a single paragraph of text, enter `Release note:` or a `## Release note` header in the PR description, followed by your text. For example, refer to this [PR](https://github.com/elastic/kibana/pull/65796) that uses the `## Release note` header. - -When you create the Release Notes text, use the following best practices: -* Use present tense. -* Use sentence case. -* When you create a feature PR, start with `Adds`. -* When you create an enhancement PR, start with `Improves`. -* When you create a bug fix PR, start with `Fixes`. -* When you create a deprecation PR, start with `Deprecates`. - -#### Add your labels -1. Label the PR with the targeted version (ex: `v7.3.0`). -2. Label the PR with the appropriate GitHub labels: - * For a new feature or functionality, use `release_note:enhancement`. - * For an external-facing fix, use `release_note:fix`. We do not include docs, build, and test fixes in the Release Notes, or unreleased issues that are only on `master`. - * For a deprecated feature, use `release_note:deprecation`. - * For a breaking change, use `release_note:breaking`. - * To **NOT** include your changes in the Release Notes, use `release_note:skip`. - -We also produce a blog post that details more important breaking API changes in every major and minor release. When your PR includes a breaking API change, add the `release_note:dev_docs` label, and add a brief summary of the break at the bottom of the PR using the format below: - -``` -# Dev Docs - -## Name the feature with the break (ex: Visualize Loader) - -Summary of the change. Anything Under `#Dev Docs` is used in the blog. -``` - -## Signing the contributor license agreement - -Please make sure you have signed the [Contributor License Agreement](http://www.elastic.co/contributor-agreement/). We are not asking you to assign copyright to us, but to give us the right to distribute your code without restriction. We ask this of all contributors in order to assure our users of the origin and continuing existence of the code. You only need to sign the CLA once. - -## Submitting a Pull Request - -Push your local changes to your forked copy of the repository and submit a Pull Request. In the Pull Request, describe what your changes do and mention the number of the issue where discussion has taken place, e.g., “Closes #123″. - -Always submit your pull against `master` unless the bug is only present in an older version. If the bug affects both `master` and another branch say so in your pull. - -Then sit back and wait. There will probably be discussion about the Pull Request and, if any changes are needed, we'll work with you to get your Pull Request merged into Kibana. - -## Code Reviewing - -After a pull is submitted, it needs to get to review. If you have commit permission on the Kibana repo you will probably perform these steps while submitting your Pull Request. If not, a member of the Elastic organization will do them for you, though you can help by suggesting a reviewer for your changes if you've interacted with someone while working on the issue. - -### Getting to the Code Review Stage - -1. Assign the `review` label. This signals to the team that someone needs to give this attention. -1. Do **not** assign a version label. Someone from Elastic staff will assign a version label, if necessary, when your Pull Request is ready to be merged. -1. Find someone to review your pull. Don't just pick any yahoo, pick the right person. The right person might be the original reporter of the issue, but it might also be the person most familiar with the code you've changed. If neither of those things apply, or your change is small in scope, try to find someone on the Kibana team without a ton of existing reviews on their plate. As a rule, most pulls will require 2 reviewers, but the first reviewer will pick the 2nd. - -### Reviewing Pull Requests - -So, you've been assigned a pull to review. Check out our [pull request review guidelines](https://www.elastic.co/guide/en/kibana/master/pr-review.html) for our general philosophy for pull request reviewers. - -Thank you so much for reading our guidelines! :tada: +Our developer guide is written in asciidoc and located under [./docs/developer](./docs/developer) if you want to make edits or access it in raw form. diff --git a/docs/developer/add-data-guide.asciidoc b/docs/developer/add-data-guide.asciidoc deleted file mode 100644 index e00e46868bb2d..0000000000000 --- a/docs/developer/add-data-guide.asciidoc +++ /dev/null @@ -1,38 +0,0 @@ -[[add-data-guide]] -== Add Data Guide - -`Add Data` in the Kibana Home application contains tutorials for setting up data flows in the Elastic stack. - -Each tutorial contains three sets of instructions: - -* `On Premise.` Set up a data flow when both Kibana and Elasticsearch are running on premise. -* `On Premise Elastic Cloud.` Set up a data flow when Kibana is running on premise and Elasticsearch is running on Elastic Cloud. -* `Elastic Cloud.` Set up a data flow when both Kibana and Elasticsearch are running on Elastic Cloud. - -[float] -=== Creating a new tutorial -1. Create a new directory in the link:https://github.com/elastic/kibana/tree/master/src/plugins/home/server/tutorials[tutorials directory]. -2. In the new directory, create a file called `index.ts` that exports a function. -The function must return a function object that conforms to the `TutorialSchema` interface link:https://github.com/elastic/kibana/blob/master/src/plugins/home/server/services/tutorials/lib/tutorial_schema.ts[tutorial schema]. -3. Register the tutorial in link:https://github.com/elastic/kibana/blob/master/src/plugins/home/server/tutorials/register.ts[register.ts] by adding it to the `builtInTutorials`. -// TODO update path once assets are migrated -4. Add image assets to the link:https://github.com/elastic/kibana/tree/master/src/legacy/core_plugins/kibana/public/home/tutorial_resources[tutorial_resources directory]. -5. Run Kibana locally to preview the tutorial. -6. Create a PR and go through the review process to get the changes approved. - -If you are creating a new plugin and the tutorial is only related to that plugin, you can also place the `TutorialSchema` object into your plugin folder. Add `home` to the `requiredPlugins` list in your `kibana.json` file. -Then register the tutorial object by calling `home.tutorials.registerTutorial(tutorialObject)` in the `setup` lifecycle of your server plugin. - -[float] -==== Variables -String values can contain variables that are substituted when rendered. Variables are specified by `{}`. -For example: `{config.docs.version}` is rendered as `6.2` when running the tutorial in Kibana 6.2. - -link:https://github.com/elastic/kibana/blob/master/src/legacy/core_plugins/kibana/public/home/np_ready/components/tutorial/replace_template_strings.js#L23[Provided variables] - -[float] -==== Markdown -String values can contain limited Markdown syntax. - -link:https://github.com/elastic/kibana/blob/master/src/legacy/core_plugins/kibana/public/home/components/tutorial/content.js#L8[Enabled Markdown grammars] - diff --git a/docs/developer/advanced/development-basepath.asciidoc b/docs/developer/advanced/development-basepath.asciidoc new file mode 100644 index 0000000000000..f0b760a21ea0c --- /dev/null +++ b/docs/developer/advanced/development-basepath.asciidoc @@ -0,0 +1,18 @@ +[[development-basepath]] +=== Considerations for basepath + +In dev mode, {kib} by default runs behind a proxy which adds a random path component to its URL. + +You can set this explicitly using `server.basePath` in <>. + +Use the server.rewriteBasePath setting to tell {kib} if it should remove the basePath from requests it receives, and to prevent a deprecation warning at startup. This setting cannot end in a slash (/). + +If you want to turn off the basepath when in development mode, start {kib} with the `--no-basepath` flag + +[source,bash] +---- +yarn start --no-basepath +---- + + + diff --git a/docs/developer/core/development-es-snapshots.asciidoc b/docs/developer/advanced/development-es-snapshots.asciidoc similarity index 90% rename from docs/developer/core/development-es-snapshots.asciidoc rename to docs/developer/advanced/development-es-snapshots.asciidoc index 4cd4f31e582db..92fae7a241edf 100644 --- a/docs/developer/core/development-es-snapshots.asciidoc +++ b/docs/developer/advanced/development-es-snapshots.asciidoc @@ -1,7 +1,7 @@ [[development-es-snapshots]] === Daily Elasticsearch Snapshots -For local development and CI, Kibana, by default, uses Elasticsearch snapshots that are built daily when running tasks that require Elasticsearch (e.g. functional tests). +For local development and CI, {kib}, by default, uses Elasticsearch snapshots that are built daily when running tasks that require Elasticsearch (e.g. functional tests). A snapshot is just a group of tarballs, one for each supported distribution/architecture/os of Elasticsearch, and a JSON-based manifest file containing metadata about the distributions. @@ -9,13 +9,13 @@ https://ci.kibana.dev/es-snapshots[A dashboard] is available that shows the curr ==== Process Overview -1. Elasticsearch snapshots are built for each current tracked branch of Kibana. +1. Elasticsearch snapshots are built for each current tracked branch of {kib}. 2. Each snapshot is uploaded to a public Google Cloud Storage bucket, `kibana-ci-es-snapshots-daily`. ** At this point, the snapshot is not automatically used in CI or local development. It needs to be tested/verified first. -3. Each snapshot is tested with the latest commit of the corresponding Kibana branch, using the full CI suite. +3. Each snapshot is tested with the latest commit of the corresponding {kib} branch, using the full CI suite. 4. After CI ** If the snapshot passes, it is promoted and automatically used in CI and local development. -** If the snapshot fails, the issue must be investigated and resolved. A new incompatibility may exist between Elasticsearch and Kibana. +** If the snapshot fails, the issue must be investigated and resolved. A new incompatibility may exist between Elasticsearch and {kib}. ==== Using the latest snapshot @@ -39,7 +39,7 @@ KBN_ES_SNAPSHOT_USE_UNVERIFIED=true node scripts/functional_tests_server Currently, there is not a way to run your pull request with the latest unverified snapshot without a code change. You can, however, do it with a small code change. -1. Edit `Jenkinsfile` in the root of the Kibana repo +1. Edit `Jenkinsfile` in the root of the {kib} repo 2. Add `env.KBN_ES_SNAPSHOT_USE_UNVERIFIED = 'true'` at the top of the file. 3. Commit the change @@ -75,13 +75,13 @@ The file structure for this bucket looks like this: ==== How snapshots are built, tested, and promoted -Each day, a https://kibana-ci.elastic.co/job/elasticsearch+snapshots+trigger/[Jenkins job] runs that triggers Elasticsearch builds for each currently tracked branch/version. This job is automatically updated with the correct branches whenever we release new versions of Kibana. +Each day, a https://kibana-ci.elastic.co/job/elasticsearch+snapshots+trigger/[Jenkins job] runs that triggers Elasticsearch builds for each currently tracked branch/version. This job is automatically updated with the correct branches whenever we release new versions of {kib}. ===== Build https://kibana-ci.elastic.co/job/elasticsearch+snapshots+build/[This Jenkins job] builds the Elasticsearch snapshots and uploads them to GCS. -The Jenkins job pipeline definition is https://github.com/elastic/kibana/blob/master/.ci/es-snapshots/Jenkinsfile_build_es[in the kibana repo]. +The Jenkins job pipeline definition is https://github.com/elastic/kibana/blob/master/.ci/es-snapshots/Jenkinsfile_build_es[in the {kib} repo]. 1. Checkout Elasticsearch repo for the given branch/version. 2. Run `./gradlew -p distribution/archives assemble --parallel` to create all of the Elasticsearch distributions. @@ -91,15 +91,15 @@ The Jenkins job pipeline definition is https://github.com/elastic/kibana/blob/ma ** e.g. `/archives/` 6. Replace `/manifest-latest.json` in GCS with this newest manifest. ** This allows the `KBN_ES_SNAPSHOT_USE_UNVERIFIED` flag to work. -7. Trigger the verification job, to run the full Kibana CI test suite with this snapshot. +7. Trigger the verification job, to run the full {kib} CI test suite with this snapshot. ===== Verification and Promotion -https://kibana-ci.elastic.co/job/elasticsearch+snapshots+verify/[This Jenkins job] tests the latest Elasticsearch snapshot with the full Kibana CI pipeline, and promotes if it there are no test failures. +https://kibana-ci.elastic.co/job/elasticsearch+snapshots+verify/[This Jenkins job] tests the latest Elasticsearch snapshot with the full {kib} CI pipeline, and promotes if it there are no test failures. -The Jenkins job pipeline definition is https://github.com/elastic/kibana/blob/master/.ci/es-snapshots/Jenkinsfile_verify_es[in the kibana repo]. +The Jenkins job pipeline definition is https://github.com/elastic/kibana/blob/master/.ci/es-snapshots/Jenkinsfile_verify_es[in the {kib} repo]. -1. Checkout Kibana and set up CI environment as normal. +1. Checkout {kib} and set up CI environment as normal. 2. Set the `ES_SNAPSHOT_MANIFEST` env var to point to the latest snapshot manifest. 3. Run CI (functional tests, integration tests, etc). 4. After CI diff --git a/docs/developer/advanced/index.asciidoc b/docs/developer/advanced/index.asciidoc new file mode 100644 index 0000000000000..139940ee42fe2 --- /dev/null +++ b/docs/developer/advanced/index.asciidoc @@ -0,0 +1,12 @@ +[[advanced]] +== Advanced + +* <> +* <> +* <> + +include::development-es-snapshots.asciidoc[] + +include::running-elasticsearch.asciidoc[] + +include::development-basepath.asciidoc[] \ No newline at end of file diff --git a/docs/developer/advanced/running-elasticsearch.asciidoc b/docs/developer/advanced/running-elasticsearch.asciidoc new file mode 100644 index 0000000000000..b03c231678eee --- /dev/null +++ b/docs/developer/advanced/running-elasticsearch.asciidoc @@ -0,0 +1,118 @@ +[[running-elasticsearch]] +=== Running elasticsearch during development + +There are many ways to run Elasticsearch while you are developing. + +[float] + +==== By snapshot + +This will run a snapshot of elasticsearch that is usually built nightly. Read more about <>. + +[source,bash] +---- +yarn es snapshot +---- + +See all available options, like how to specify a specific license, with the `--help` flag. + +[source,bash] +---- +yarn es snapshot --help +---- + +`trial` will give you access to all capabilities. + +**Keeping data between snapshots** + +If you want to keep the data inside your Elasticsearch between usages of this command, you should use the following command, to keep your data folder outside the downloaded snapshot folder: + +[source,bash] +---- +yarn es snapshot -E path.data=../data +---- + +==== By source + +If you have the Elasticsearch repo checked out locally and wish to run against that, use `source`. By default, it will reference an elasticsearch checkout which is a sibling to the {kib} directory named elasticsearch. If you wish to use a checkout in another location you can provide that by supplying --source-path + +[source,bash] +---- +yarn es source +---- + +==== From an archive + +Use this if you already have a distributable. For released versions, one can be obtained on the Elasticsearch downloads page. + +[source,bash] +---- +yarn es archive +---- + +Each of these will run Elasticsearch with a basic license. Additional options are available, pass --help for more information. + +==== From a remote host + +You can save some system resources, and the effort of generating sample data, if you have a remote Elasticsearch cluster to connect to. (Elasticians: you do! Check with your team about where to find credentials) + +You'll need to create a kibana.dev.yml (<>) and add the following to it: + +[source,bash] +---- +elasticsearch.hosts: + - {{ url }} +elasticsearch.username: {{ username }} +elasticsearch.password: {{ password }} +elasticsearch.ssl.verificationMode: none +---- + +If many other users will be interacting with your remote cluster, you'll want to add the following to avoid causing conflicts: + +[source,bash] +---- +kibana.index: '.{YourGitHubHandle}-kibana' +xpack.task_manager.index: '.{YourGitHubHandle}-task-manager-kibana' +---- + +===== Running remote clusters + +Setup remote clusters for cross cluster search (CCS) and cross cluster replication (CCR). + +Start your primary cluster by running: + +[source,bash] +---- +yarn es snapshot -E path.data=../data_prod1 +---- + +Start your remote cluster by running: + +[source,bash] +---- +yarn es snapshot -E transport.port=9500 -E http.port=9201 -E path.data=../data_prod2 +---- + +Once both clusters are running, start {kib}. {kib} will connect to the primary cluster. + +Setup the remote cluster in {kib} from either Management -> Elasticsearch -> Remote Clusters UI or by running the following script in Console. + +[source,bash] +---- +PUT _cluster/settings +{ + "persistent": { + "cluster": { + "remote": { + "cluster_one": { + "seeds": [ + "localhost:9500" + ] + } + } + } + } +} +---- + +Follow the cross-cluster search instructions for setting up index patterns to search across clusters (<>). \ No newline at end of file diff --git a/docs/developer/architecture/add-data-tutorials.asciidoc b/docs/developer/architecture/add-data-tutorials.asciidoc new file mode 100644 index 0000000000000..e16b1bc039a10 --- /dev/null +++ b/docs/developer/architecture/add-data-tutorials.asciidoc @@ -0,0 +1,38 @@ +[[add-data-tutorials]] +=== Add data tutorials + +`Add Data` in the {kib} Home application contains tutorials for setting up data flows in the Elastic stack. + +Each tutorial contains three sets of instructions: + +* `On Premise.` Set up a data flow when both {kib} and Elasticsearch are running on premise. +* `On Premise Elastic Cloud.` Set up a data flow when {kib} is running on premise and Elasticsearch is running on Elastic Cloud. +* `Elastic Cloud.` Set up a data flow when both {kib} and Elasticsearch are running on Elastic Cloud. + +[float] +==== Creating a new tutorial +1. Create a new directory in the link:https://github.com/elastic/kibana/tree/master/src/plugins/home/server/tutorials[tutorials directory]. +2. In the new directory, create a file called `index.ts` that exports a function. +The function must return a function object that conforms to the `TutorialSchema` interface link:{kib-repo}tree/{branch}/src/plugins/home/server/services/tutorials/lib/tutorial_schema.ts[tutorial schema]. +3. Register the tutorial in link:{kib-repo}tree/{branch}/src/plugins/home/server/tutorials/register.ts[register.ts] by adding it to the `builtInTutorials`. +// TODO update path once assets are migrated +4. Add image assets to the link:{kib-repo}tree/{branch}/src/legacy/core_plugins/kibana/public/home/tutorial_resources[tutorial_resources directory]. +5. Run {kib} locally to preview the tutorial. +6. Create a PR and go through the review process to get the changes approved. + +If you are creating a new plugin and the tutorial is only related to that plugin, you can also place the `TutorialSchema` object into your plugin folder. Add `home` to the `requiredPlugins` list in your `kibana.json` file. +Then register the tutorial object by calling `home.tutorials.registerTutorial(tutorialObject)` in the `setup` lifecycle of your server plugin. + +[float] +===== Variables +String values can contain variables that are substituted when rendered. Variables are specified by `{}`. +For example: `{config.docs.version}` is rendered as `6.2` when running the tutorial in {kib} 6.2. + +link:{kib-repo}tree/{branch}/src/legacy/core_plugins/kibana/public/home/np_ready/components/tutorial/replace_template_strings.js#L23[Provided variables] + +[float] +===== Markdown +String values can contain limited Markdown syntax. + +link:{kib-repo}tree/{branch}/src/legacy/core_plugins/kibana/public/home/components/tutorial/content.js#L8[Enabled Markdown grammars] + diff --git a/docs/developer/visualize/development-visualize-index.asciidoc b/docs/developer/architecture/development-visualize-index.asciidoc similarity index 85% rename from docs/developer/visualize/development-visualize-index.asciidoc rename to docs/developer/architecture/development-visualize-index.asciidoc index ac824b4702a3c..551c41833fb72 100644 --- a/docs/developer/visualize/development-visualize-index.asciidoc +++ b/docs/developer/architecture/development-visualize-index.asciidoc @@ -1,13 +1,13 @@ [[development-visualize-index]] -== Developing Visualizations +=== Developing Visualizations [IMPORTANT] ============================================== -These pages document internal APIs and are not guaranteed to be supported across future versions of Kibana. +These pages document internal APIs and are not guaranteed to be supported across future versions of {kib}. ============================================== The internal APIs for creating custom visualizations are in a state of heavy churn as -they are being migrated to the new Kibana platform, and large refactorings have been +they are being migrated to the new {kib} platform, and large refactorings have been happening across minor releases in the `7.x` series. In particular, in `7.5` and later we have made significant changes to the legacy APIs as we work to gradually replace them. @@ -20,7 +20,7 @@ If you would like to keep up with progress on the visualizations plugin in the m here are a few resources: * The <> documentation, where we try to capture any changes to the APIs as they occur across minors. -* link:https://github.com/elastic/kibana/issues/44121[Meta issue] which is tracking the move of the plugin to the new Kibana platform +* link:https://github.com/elastic/kibana/issues/44121[Meta issue] which is tracking the move of the plugin to the new {kib} platform * Our link:https://www.elastic.co/blog/join-our-elastic-stack-workspace-on-slack[Elastic Stack workspace on Slack]. * The {kib-repo}blob/{branch}/src/plugins/visualizations[source code], which will continue to be the most accurate source of information. diff --git a/docs/developer/architecture/index.asciidoc b/docs/developer/architecture/index.asciidoc new file mode 100644 index 0000000000000..d726a8bd3642d --- /dev/null +++ b/docs/developer/architecture/index.asciidoc @@ -0,0 +1,25 @@ +[[kibana-architecture]] +== Architecture + +[IMPORTANT] +============================================== +{kib} developer services and apis are in a state of constant development. We cannot provide backwards compatibility at this time due to the high rate of change. +============================================== + +Our developer services are changing all the time. One of the best ways to discover and learn about them is to read the available +READMEs from all the plugins inside our {kib-repo}tree/{branch}/src/plugins[open source plugins folder] and our +{kib-repo}/tree/{branch}/x-pack/plugins[commercial plugins folder]. + +A few services also automatically generate api documentation which can be browsed inside the {kib-repo}tree/{branch}/docs/development[docs/development section of our repo] + +A few notable services are called out below. + +* <> +* <> +* <> + +include::add-data-tutorials.asciidoc[] + +include::development-visualize-index.asciidoc[] + +include::security/index.asciidoc[] diff --git a/docs/developer/plugin/development-plugin-feature-registration.asciidoc b/docs/developer/architecture/security/feature-registration.asciidoc similarity index 96% rename from docs/developer/plugin/development-plugin-feature-registration.asciidoc rename to docs/developer/architecture/security/feature-registration.asciidoc index 203cc201ee626..164f6d1cf9c74 100644 --- a/docs/developer/plugin/development-plugin-feature-registration.asciidoc +++ b/docs/developer/architecture/security/feature-registration.asciidoc @@ -1,13 +1,13 @@ [[development-plugin-feature-registration]] -=== Plugin feature registration +==== Plugin feature registration If your plugin will be used with {kib}'s default distribution, then you have the ability to register the features that your plugin provides. Features are typically apps in {kib}; once registered, you can toggle them via Spaces, and secure them via Roles when security is enabled. -==== UI Capabilities +===== UI Capabilities Registering features also gives your plugin access to “UI Capabilities”. These capabilities are boolean flags that you can use to conditionally render your interface, based on the current user's permissions. For example, you can hide or disable a Save button if the current user is not authorized. -==== Registering a feature +===== Registering a feature Feature registration is controlled via the built-in `xpack_main` plugin. To register a feature, call `xpack_main`'s `registerFeature` function from your plugin's `init` function, and provide the appropriate details: @@ -65,12 +65,12 @@ Registering a feature consists of the following fields. For more information, co |The ID of the navigation link associated with your feature. |=== -===== Privilege definition +====== Privilege definition The `privileges` section of feature registration allows plugins to implement read/write and read-only modes for their applications. For a full explanation of fields and options, consult the {kib-repo}blob/{branch}/x-pack/plugins/features/server/feature_registry.ts[feature registry interface]. -==== Using UI Capabilities +===== Using UI Capabilities UI Capabilities are available to your public (client) plugin code. These capabilities are read-only, and are used to inform the UI. This object is namespaced by feature id. For example, if your feature id is “foo”, then your UI Capabilities are stored at `uiCapabilities.foo`. To access capabilities, import them from `ui/capabilities`: @@ -86,7 +86,7 @@ if (canUserSave) { ----------- [[example-1-canvas]] -==== Example 1: Canvas Application +===== Example 1: Canvas Application ["source","javascript"] ----------- init(server) { @@ -118,13 +118,13 @@ init(server) { } ----------- -This shows how the Canvas application might register itself as a Kibana feature. +This shows how the Canvas application might register itself as a {kib} feature. Note that it specifies different `savedObject` access levels for each privilege: - Users with read/write access (`all` privilege) need to be able to read/write `canvas-workpad` saved objects, and they need read-only access to `index-pattern` saved objects. - Users with read-only access (`read` privilege) do not need to have read/write access to any saved objects, but instead get read-only access to `index-pattern` and `canvas-workpad` saved objects. -Additionally, Canvas registers the `canvas` UI app and `canvas` catalogue entry. This tells Kibana that these entities are available for users with either the `read` or `all` privilege. +Additionally, Canvas registers the `canvas` UI app and `canvas` catalogue entry. This tells {kib} that these entities are available for users with either the `read` or `all` privilege. The `all` privilege defines a single “save” UI Capability. To access this in the UI, Canvas could: @@ -141,7 +141,7 @@ if (canUserSave) { Because the `read` privilege does not define the `save` capability, users with read-only access will have their `uiCapabilities.canvas.save` flag set to `false`. [[example-2-dev-tools]] -==== Example 2: Dev Tools +===== Example 2: Dev Tools ["source","javascript"] ----------- @@ -199,7 +199,7 @@ server.route({ ----------- [[example-3-discover]] -==== Example 3: Discover +===== Example 3: Discover Discover takes advantage of subfeature privileges to allow fine-grained access control. In this example, a single "Create Short URLs" subfeature privilege is defined, which allows users to grant access to this feature without having to grant the `all` privilege to Discover. In other words, you can grant `read` access to Discover, and also grant the ability to create short URLs. diff --git a/docs/developer/architecture/security/index.asciidoc b/docs/developer/architecture/security/index.asciidoc new file mode 100644 index 0000000000000..55b2450caf7a7 --- /dev/null +++ b/docs/developer/architecture/security/index.asciidoc @@ -0,0 +1,12 @@ +[[development-security]] +=== Security + +{kib} has generally been able to implement security transparently to core and plugin developers, and this largely remains the case. {kib} on two methods that the elasticsearch `Cluster` provides: `callWithRequest` and `callWithInternalUser`. + +`callWithRequest` executes requests against Elasticsearch using the authentication credentials of the {kib} end-user. So, if you log into {kib} with the user of `foo` when `callWithRequest` is used, {kib} execute the request against Elasticsearch as the user `foo`. Historically, `callWithRequest` has been used extensively to perform actions that are initiated at the request of {kib} end-users. + +`callWithInternalUser` executes requests against Elasticsearch using the internal {kib} server user, and has historically been used for performing actions that aren't initiated by {kib} end users; for example, creating the initial `.kibana` index or performing health checks against Elasticsearch. + +However, with the changes that role-based access control (RBAC) introduces, this is no longer cut and dry. {kib} now requires all access to the `.kibana` index goes through the `SavedObjectsClient`. This used to be a best practice, as the `SavedObjectsClient` was responsible for translating the documents stored in Elasticsearch to and from Saved Objects, but RBAC is now taking advantage of this abstraction to implement access control and determine when to use `callWithRequest` versus `callWithInternalUser`. + +include::rbac.asciidoc[] diff --git a/docs/developer/security/rbac.asciidoc b/docs/developer/architecture/security/rbac.asciidoc similarity index 96% rename from docs/developer/security/rbac.asciidoc rename to docs/developer/architecture/security/rbac.asciidoc index 02b8233a9a3df..ae1979e856e23 100644 --- a/docs/developer/security/rbac.asciidoc +++ b/docs/developer/architecture/security/rbac.asciidoc @@ -1,5 +1,5 @@ [[development-security-rbac]] -=== Role-based access control +==== Role-based access control Role-based access control (RBAC) in {kib} relies upon the {ref}/security-privileges.html#application-privileges[application privileges] @@ -11,7 +11,7 @@ consumers when using `request.getSavedObjectsClient()` or `savedObjects.getScopedSavedObjectsClient()`. [[development-rbac-privileges]] -==== {kib} Privileges +===== {kib} Privileges When {kib} first starts up, it executes the following `POST` request against {es}. This synchronizes the definition of the privileges with various `actions` which are later used to authorize a user: @@ -19,7 +19,7 @@ When {kib} first starts up, it executes the following `POST` request against {es ---------------------------------- POST /_security/privilege Content-Type: application/json -Authorization: Basic kibana changeme +Authorization: Basic {kib} changeme { "kibana-.kibana":{ @@ -56,7 +56,7 @@ The application is created by concatenating the prefix of `kibana-` with the val ============================================== [[development-rbac-assigning-privileges]] -==== Assigning {kib} Privileges +===== Assigning {kib} Privileges {kib} privileges are assigned to specific roles using the `applications` element. For example, the following role assigns the <> privilege at `*` `resources` (which will in the future be used to secure spaces) to the default {kib} `application`: @@ -81,7 +81,7 @@ Roles that grant <> should be managed using the <> +* <> +* <> + +include::stability.asciidoc[] + +include::security.asciidoc[] diff --git a/docs/developer/best-practices/security.asciidoc b/docs/developer/best-practices/security.asciidoc new file mode 100644 index 0000000000000..26fcc73ce2b90 --- /dev/null +++ b/docs/developer/best-practices/security.asciidoc @@ -0,0 +1,55 @@ +[[security-best-practices]] +=== Security best practices + +* XSS +** Check for usages of `dangerouslySetInnerHtml`, `Element.innerHTML`, +`Element.outerHTML` +** Ensure all user input is properly escaped. +** Ensure any input in `$.html`, `$.append`, `$.appendTo`, +latexmath:[$.prepend`, `$].prependTo`is escaped. Instead use`$.text`, or +don’t use jQuery at all. +* CSRF +** Ensure all APIs are running inside the {kib} HTTP service. +* RCE +** Ensure no usages of `eval` +** Ensure no usages of dynamic requires +** Check for template injection +** Check for usages of templating libraries, including `_.template`, and +ensure that user provided input isn’t influencing the template and is +only used as data for rendering the template. +** Check for possible prototype pollution. +* Prototype Pollution +** Check for instances of `anObject[a][b] = c` where a, b, and c are +user defined. This includes code paths where the following logical code +steps could be performed in separate files by completely different +operations, or recursively using dynamic operations. +** Validate any user input, including API +url-parameters/query-parameters/payloads, preferable against a schema +which only allows specific keys/values. At a very minimum, black-list +`__proto__` and `prototype.constructor` for use within keys +** When calling APIs which spawn new processes or potentially perform +code generation from strings, defensively protect against Prototype +Pollution by checking `Object.hasOwnProperty` if the arguments to the +APIs originate from an Object. An example is the Code app’s +https://github.com/elastic/kibana/blob/b49192626a8528af5d888545fb14cd1ce66a72e7/x-pack/legacy/plugins/code/server/lsp/workspace_command.ts#L40-L44[spawnProcess]. +*** Common Node.js offenders: `child_process.spawn`, +`child_process.exec`, `eval`, `Function('some string')`, +`vm.runIn*Context(x)` +*** Common Client-side offenders: `eval`, `Function('some string')`, +`setTimeout('some string', num)`, `setInterval('some string', num)` +* Check for accidental reveal of sensitive information +** The biggest culprit is errors which contain stack traces or other +sensitive information which end up in the HTTP Response +* Checked for Mishandled API requests +** Ensure no sensitive cookies are forwarded to external resources. +** Ensure that all user controllable variables that are used in +constructing a URL are escaped properly. This is relevant when using +`transport.request` with the Elasticsearch client as no automatic +escaping is performed. +* Reverse tabnabbing - +https://github.com/OWASP/CheatSheetSeries/blob/master/cheatsheets/HTML5_Security_Cheat_Sheet.md#tabnabbing +** When there are user controllable links or hard-coded links to +third-party domains that specify target="_blank" or target="_window", the a tag should have the rel="noreferrer noopener" attribute specified. +Allowing users to input markdown is a common culprit, a custom link renderer should be used +* SSRF - https://www.owasp.org/index.php/Server_Side_Request_Forgery +All network requests made from the {kib} server should use an explicit configuration or white-list specified in the kibana.yml \ No newline at end of file diff --git a/docs/developer/best-practices/stability.asciidoc b/docs/developer/best-practices/stability.asciidoc new file mode 100644 index 0000000000000..68237a034be52 --- /dev/null +++ b/docs/developer/best-practices/stability.asciidoc @@ -0,0 +1,66 @@ +[[stability]] +=== Stability + +Ensure your feature will work under all possible {kib} scenarios. + +[float] +==== Environmental configuration scenarios + +* Cloud +** Does the feature work on *cloud environment*? +** Does it create a setting that needs to be exposed, or configured +differently than the default, on Cloud? (whitelisting of certain +settings/users? Ref: +https://www.elastic.co/guide/en/cloud/current/ec-add-user-settings.html +, +https://www.elastic.co/guide/en/cloud/current/ec-manage-kibana-settings.html) +** Is there a significant performance impact that may affect Cloud +{kib} instances? +** Does it need to be aware of running in a container? (for example +monitoring) +* Multiple {kib} instances +** Pointing to the same index +** Pointing to different indexes +*** Should make sure that the {kib} index is not hardcoded anywhere. +*** Should not be storing a bunch of stuff in {kib} memory. +*** Should emulate a high availability deployment. +*** Anticipating different timing related issues due to shared resource +access. +*** We need to make sure security is set up in a specific way for +non-standard {kib} indices. (create their own custom roles) +* {kib} running behind a reverse proxy or load balancer, without sticky +sessions. (we have had many discuss/SDH tickets around this) +* If a proxy/loadbalancer is running between ES and {kib} + +[float] +==== Kibana.yml settings + +* Using a custom {kib} index alias +* When optional dependencies are disabled +** Ensure all your required dependencies are listed in kibana.json +dependency list! + +[float] +==== Test coverage + +* Does the feature have sufficient unit test coverage? (does it handle +storeinSessions?) +* Does the feature have sufficient Functional UI test coverage? +* Does the feature have sufficient Rest API coverage test coverage? +* Does the feature have sufficient Integration test coverage? + +[float] +==== Browser coverage + +Refer to the list of browsers and OS {kib} supports +https://www.elastic.co/support/matrix + +Does the feature work efficiently on the list of supported browsers? + +[float] +==== Upgrade Scenarios - Migration scenarios- + +Does the feature affect old +indices, saved objects ? - Has the feature been tested with {kib} +aliases - Read/Write privileges of the indices before and after the +upgrade? diff --git a/docs/developer/contributing/development-accessibility-tests.asciidoc b/docs/developer/contributing/development-accessibility-tests.asciidoc new file mode 100644 index 0000000000000..a3ffefb94cd2a --- /dev/null +++ b/docs/developer/contributing/development-accessibility-tests.asciidoc @@ -0,0 +1,23 @@ +[[development-accessibility-tests]] +==== Automated Accessibility Testing + +To run the tests locally: + +[arabic] +. In one terminal window run +`node scripts/functional_tests_server --config test/accessibility/config.ts` +. In another terminal window run +`node scripts/functional_test_runner.js --config test/accessibility/config.ts` + +To run the x-pack tests, swap the config file out for +`x-pack/test/accessibility/config.ts`. + +After the server is up, you can go to this instance of {kib} at +`localhost:5620`. + +The testing is done using https://github.com/dequelabs/axe-core[axe]. +The same thing that runs in CI, can be run locally using their browser +plugins: + +* https://chrome.google.com/webstore/detail/axe-web-accessibility-tes/lhdoppojpmngadmnindnejefpokejbdd?hl=en-US[Chrome] +* https://addons.mozilla.org/en-US/firefox/addon/axe-devtools/[Firefox] \ No newline at end of file diff --git a/docs/developer/contributing/development-documentation.asciidoc b/docs/developer/contributing/development-documentation.asciidoc new file mode 100644 index 0000000000000..d9fae42eef87e --- /dev/null +++ b/docs/developer/contributing/development-documentation.asciidoc @@ -0,0 +1,34 @@ +[[development-documentation]] +=== Documentation during development + +Docs should be written during development and accompany PRs when relevant. There are multiple types of documentation, and different places to add each. + +[float] +==== Developer services documentation + +Documentation about specific services a plugin offers should be encapsulated in: + +* README.asciidoc at the base of the plugin folder. +* Typescript comments for all public services. + +[float] +==== End user documentation + +Documentation about user facing features should be written in http://asciidoc.org/[asciidoc] at +{kib-repo}/tree/master/docs[https://github.com/elastic/kibana/tree/master/docs] + +To build the docs, you must clone the https://github.com/elastic/docs[elastic/docs] +repo as a sibling of your {kib} repo. Follow the instructions in that project's +README for getting the docs tooling set up. + +**To build the docs:** + +```bash +node scripts/docs.js --open +``` + +[float] +==== General developer documentation and guidelines + +General developer guildlines and documentation, like this right here, should be written in http://asciidoc.org/[asciidoc] +at {kib-repo}/tree/master/docs/developer[https://github.com/elastic/kibana/tree/master/docs/developer] diff --git a/docs/developer/core/development-functional-tests.asciidoc b/docs/developer/contributing/development-functional-tests.asciidoc similarity index 90% rename from docs/developer/core/development-functional-tests.asciidoc rename to docs/developer/contributing/development-functional-tests.asciidoc index 130549348f475..7823654c24f61 100644 --- a/docs/developer/core/development-functional-tests.asciidoc +++ b/docs/developer/contributing/development-functional-tests.asciidoc @@ -1,38 +1,39 @@ [[development-functional-tests]] === Functional Testing -We use functional tests to make sure the Kibana UI works as expected. It replaces hours of manual testing by automating user interaction. To have better control over our functional test environment, and to make it more accessible to plugin authors, Kibana uses a tool called the `FunctionalTestRunner`. +We use functional tests to make sure the {kib} UI works as expected. It replaces hours of manual testing by automating user interaction. To have better control over our functional test environment, and to make it more accessible to plugin authors, {kib} uses a tool called the `FunctionalTestRunner`. [float] ==== Running functional tests -The `FunctionalTestRunner` is very bare bones and gets most of its functionality from its config file, located at {blob}test/functional/config.js[test/functional/config.js]. If you’re writing a plugin you will have your own config file. See <> for more info. +The `FunctionalTestRunner` is very bare bones and gets most of its functionality from its config file, located at {blob}test/functional/config.js[test/functional/config.js]. If you’re writing a plugin outside the {kib} repo, you will have your own config file. + See <> for more info. There are three ways to run the tests depending on your goals: 1. Easiest option: -** Description: Starts up Kibana & Elasticsearch servers, followed by running tests. This is much slower when running the tests multiple times because slow startup time for the servers. Recommended for single-runs. +** Description: Starts up {kib} & Elasticsearch servers, followed by running tests. This is much slower when running the tests multiple times because slow startup time for the servers. Recommended for single-runs. ** `node scripts/functional_tests` -*** does everything in a single command, including running Elasticsearch and Kibana locally +*** does everything in a single command, including running Elasticsearch and {kib} locally *** tears down everything after the tests run *** exit code reports success/failure of the tests 2. Best for development: -** Description: Two commands, run in separate terminals, separate the components that are long-running and slow from those that are ephemeral and fast. Tests can be re-run much faster, and this still runs Elasticsearch & Kibana locally. +** Description: Two commands, run in separate terminals, separate the components that are long-running and slow from those that are ephemeral and fast. Tests can be re-run much faster, and this still runs Elasticsearch & {kib} locally. ** `node scripts/functional_tests_server` -*** starts Elasticsearch and Kibana servers +*** starts Elasticsearch and {kib} servers *** slow to start *** can be reused for multiple executions of the tests, thereby saving some time when re-running tests -*** automatically restarts the Kibana server when relevant changes are detected +*** automatically restarts the {kib} server when relevant changes are detected ** `node scripts/functional_test_runner` -*** runs the tests against Kibana & Elasticsearch servers that were started by `node scripts/functional_tests_server` +*** runs the tests against {kib} & Elasticsearch servers that were started by `node scripts/functional_tests_server` *** exit code reports success or failure of the tests 3. Custom option: -** Description: Runs tests against instances of Elasticsearch & Kibana started some other way (like Elastic Cloud, or an instance you are managing in some other way). +** Description: Runs tests against instances of Elasticsearch & {kib} started some other way (like Elastic Cloud, or an instance you are managing in some other way). ** just executes the functional tests -** url, credentials, etc. for Elasticsearch and Kibana are specified via environment variables -** Here's an example that runs against an Elastic Cloud instance. Note that you must run the same branch of tests as the version of Kibana you're testing. +** url, credentials, etc. for Elasticsearch and {kib} are specified via environment variables +** Here's an example that runs against an Elastic Cloud instance. Note that you must run the same branch of tests as the version of {kib} you're testing. + ["source","shell"] ---------- @@ -95,9 +96,10 @@ node scripts/functional_test_runner --exclude-tag skipCloud When run without any arguments the `FunctionalTestRunner` automatically loads the configuration in the standard location, but you can override that behavior with the `--config` flag. List configs with multiple --config arguments. -* `--config test/functional/config.js` starts Elasticsearch and Kibana servers with the WebDriver tests configured to run in Chrome. -* `--config test/functional/config.firefox.js` starts Elasticsearch and Kibana servers with the WebDriver tests configured to run in Firefox. -* `--config test/api_integration/config.js` starts Elasticsearch and Kibana servers with the api integration tests configuration. +* `--config test/functional/config.js` starts Elasticsearch and {kib} servers with the WebDriver tests configured to run in Chrome. +* `--config test/functional/config.firefox.js` starts Elasticsearch and {kib} servers with the WebDriver tests configured to run in Firefox. +* `--config test/api_integration/config.js` starts Elasticsearch and {kib} servers with the api integration tests configuration. +* `--config test/accessibility/config.ts` starts Elasticsearch and {kib} servers with the WebDriver tests configured to run an accessibility audit using https://www.deque.com/axe/[axe]. There are also command line flags for `--bail` and `--grep`, which behave just like their mocha counterparts. For instance, use `--grep=foo` to run only tests that match a regular expression. @@ -116,7 +118,7 @@ The tests are written in https://mochajs.org[mocha] using https://github.com/ela We use https://www.w3.org/TR/webdriver1/[WebDriver Protocol] to run tests in both Chrome and Firefox with the help of https://sites.google.com/a/chromium.org/chromedriver/[chromedriver] and https://firefox-source-docs.mozilla.org/testing/geckodriver/[geckodriver]. When the `FunctionalTestRunner` launches, remote service creates a new webdriver session, which starts the driver and a stripped-down browser instance. We use `browser` service and `webElementWrapper` class to wrap up https://seleniumhq.github.io/selenium/docs/api/javascript/module/selenium-webdriver/[Webdriver API]. -The `FunctionalTestRunner` automatically transpiles functional tests using babel, so that tests can use the same ECMAScript features that Kibana source code uses. See {blob}style_guides/js_style_guide.md[style_guides/js_style_guide.md]. +The `FunctionalTestRunner` automatically transpiles functional tests using babel, so that tests can use the same ECMAScript features that {kib} source code uses. See {blob}style_guides/js_style_guide.md[style_guides/js_style_guide.md]. [float] ===== Definitions @@ -303,9 +305,9 @@ The `FunctionalTestRunner` comes with three built-in services: * Phases include: `beforeLoadTests`, `beforeTests`, `beforeEachTest`, `cleanup` [float] -===== Kibana Services +===== {kib} Services -The Kibana functional tests define the vast majority of the actual functionality used by tests. +The {kib} functional tests define the vast majority of the actual functionality used by tests. **browser**::: * Source: {blob}test/functional/services/browser.ts[test/functional/services/browser.ts] @@ -355,7 +357,7 @@ await testSubjects.click(‘containerButton’); **kibanaServer:**::: * Source: {blob}test/common/services/kibana_server/kibana_server.js[test/common/services/kibana_server/kibana_server.js] -* Helpers for interacting with Kibana's server +* Helpers for interacting with {kib}'s server * Commonly used methods: ** `kibanaServer.uiSettings.update()` ** `kibanaServer.version.get()` @@ -500,3 +502,13 @@ const log = getService(‘log’); // log.debug only writes when using the `--debug` or `--verbose` flag. log.debug(‘done clicking menu’); ----------- + +[float] +==== MacOS testing performance tip + +macOS users on a machine with a discrete graphics card may see significant speedups (up to 2x) when running tests by changing your terminal emulator's GPU settings. In iTerm2: +* Open Preferences (Command + ,) +* In the General tab, under the "Magic" section, ensure "GPU rendering" is checked +* Open "Advanced GPU Settings..." +* Uncheck the "Prefer integrated to discrete GPU" option +* Restart iTerm \ No newline at end of file diff --git a/docs/developer/contributing/development-github.asciidoc b/docs/developer/contributing/development-github.asciidoc new file mode 100644 index 0000000000000..027b4e73aa9de --- /dev/null +++ b/docs/developer/contributing/development-github.asciidoc @@ -0,0 +1,112 @@ +[[development-github]] +=== How we use git and github + +[float] +==== Forking + +We follow the https://help.github.com/articles/fork-a-repo/[GitHub +forking model] for collaborating on {kib} code. This model assumes that +you have a remote called `upstream` which points to the official {kib} +repo, which we'll refer to in later code snippets. + +[float] +==== Branching + +* All work on the next major release goes into master. +* Past major release branches are named `{majorVersion}.x`. They contain +work that will go into the next minor release. For example, if the next +minor release is `5.2.0`, work for it should go into the `5.x` branch. +* Past minor release branches are named `{majorVersion}.{minorVersion}`. +They contain work that will go into the next patch release. For example, +if the next patch release is `5.3.1`, work for it should go into the +`5.3` branch. +* All work is done on feature branches and merged into one of these +branches. +* Where appropriate, we'll backport changes into older release branches. + +[float] +==== Commits and Merging + +* Feel free to make as many commits as you want, while working on a +branch. +* When submitting a PR for review, please perform an interactive rebase +to present a logical history that's easy for the reviewers to follow. +* Please use your commit messages to include helpful information on your +changes, e.g. changes to APIs, UX changes, bugs fixed, and an +explanation of _why_ you made the changes that you did. +* Resolve merge conflicts by rebasing the target branch over your +feature branch, and force-pushing (see below for instructions). +* When merging, we'll squash your commits into a single commit. + +[float] +===== Rebasing and fixing merge conflicts + +Rebasing can be tricky, and fixing merge conflicts can be even trickier +because it involves force pushing. This is all compounded by the fact +that attempting to push a rebased branch remotely will be rejected by +git, and you'll be prompted to do a `pull`, which is not at all what you +should do (this will really mess up your branch's history). + +Here's how you should rebase master onto your branch, and how to fix +merge conflicts when they arise. + +First, make sure master is up-to-date. + +["source","shell"] +----------- +git checkout master +git fetch upstream +git rebase upstream/master +----------- + +Then, check out your branch and rebase master on top of it, which will +apply all of the new commits on master to your branch, and then apply +all of your branch's new commits after that. + +["source","shell"] +----------- +git checkout name-of-your-branch +git rebase master +----------- + +You want to make sure there are no merge conflicts. If there are merge +conflicts, git will pause the rebase and allow you to fix the conflicts +before continuing. + +You can use `git status` to see which files contain conflicts. They'll +be the ones that aren't staged for commit. Open those files, and look +for where git has marked the conflicts. Resolve the conflicts so that +the changes you want to make to the code have been incorporated in a way +that doesn't destroy work that's been done in master. Refer to master's +commit history on GitHub if you need to gain a better understanding of how code is conflicting and how best to resolve it. + +Once you've resolved all of the merge conflicts, use `git add -A` to stage them to be committed, and then use + `git rebase --continue` to tell git to continue the rebase. + +When the rebase has completed, you will need to force push your branch because the history is now completely different than what's on the remote. This is potentially dangerous because it will completely overwrite what you have on the remote, so you need to be sure that you haven't lost any work when resolving merge conflicts. (If there weren't any merge conflicts, then you can force push without having to worry about this.) + +["source","shell"] +----------- +git push origin name-of-your-branch --force +----------- + +This will overwrite the remote branch with what you have locally. You're done! + +**Note that you should not run git pull**, for example in response to a push rejection like this: + +["source","shell"] +----------- +! [rejected] name-of-your-branch -> name-of-your-branch (non-fast-forward) +error: failed to push some refs to 'https://github.com/YourGitHubHandle/kibana.git' +hint: Updates were rejected because the tip of your current branch is behind +hint: its remote counterpart. Integrate the remote changes (e.g. +hint: 'git pull ...') before pushing again. +hint: See the 'Note about fast-forwards' in 'git push --help' for details. +----------- + +Assuming you've successfully rebased and you're happy with the code, you should force push instead. + +[float] +==== Creating a pull request + +See <> for the next steps on getting your code changes merged into {kib}. \ No newline at end of file diff --git a/docs/developer/contributing/development-pull-request.asciidoc b/docs/developer/contributing/development-pull-request.asciidoc new file mode 100644 index 0000000000000..5d3c30fec7383 --- /dev/null +++ b/docs/developer/contributing/development-pull-request.asciidoc @@ -0,0 +1,32 @@ +[[development-pull-request]] +=== Submitting a pull request + +[float] +==== What Goes Into a Pull Request + +* Please include an explanation of your changes in your PR description. +* Links to relevant issues, external resources, or related PRs are very important and useful. +* Please update any tests that pertain to your code, and add new tests where appropriate. +* Update or add docs when appropriate. Read more about <>. + +[float] +==== Submitting a Pull Request + + 1. Push your local changes to your forked copy of the repository and submit a pull request. + 2. Describe what your changes do and mention the number of the issue where discussion has taken place, e.g., “Closes #123″. + 3. Assign the `review` and `💝community` label (assuming you are not a member of the Elastic organization). This signals to the team that someone needs to give this attention. + 4. Do *not* assign a version label. Someone from Elastic staff will assign a version label, if necessary, when your Pull Request is ready to be merged. + 5. If you would like someone specific to review your pull request, assign them. Otherwise an Elastic staff member will assign the appropriate person. + +Always submit your pull against master unless the bug is only present in an older version. If the bug affects both master and another branch say so in your pull. + +Then sit back and wait. There will probably be discussion about the Pull Request and, if any changes are needed, we'll work with you to get your Pull Request merged into {kib}. + +[float] +==== What to expect during the pull request review process + +Most PRs go through several iterations of feedback and updates. Depending on the scope and complexity of the PR, the process can take weeks. Please +be patient and understand we hold our code base to a high standard. + +Check out our <> for our general philosophy for pull request reviews. + diff --git a/docs/developer/contributing/development-tests.asciidoc b/docs/developer/contributing/development-tests.asciidoc new file mode 100644 index 0000000000000..b470ea61669b2 --- /dev/null +++ b/docs/developer/contributing/development-tests.asciidoc @@ -0,0 +1,96 @@ +[[development-tests]] +=== Testing + +To ensure that your changes will not break other functionality, please run the test suite and build (<>) before submitting your Pull Request. + +[float] +==== Running specific {kib} tests + +The following table outlines possible test file locations and how to +invoke them: + +[width="100%",cols="7%,59%,34%",options="header",] +|=== +|Test runner |Test location |Runner command (working directory is {kib} +root) +|Jest |`src/**/*.test.js` `src/**/*.test.ts` +|`yarn test:jest -t regexp [test path]` + +|Jest (integration) |`**/integration_tests/**/*.test.js` +|`yarn test:jest_integration -t regexp [test path]` + +|Mocha +|`src/**/__tests__/**/*.js` `!src/**/public/__tests__/*.js``packages/kbn-datemath/test/**/*.js` `packages/kbn-dev-utils/src/**/__tests__/**/*.js` `tasks/**/__tests__/**/*.js` +|`node scripts/mocha --grep=regexp [test path]` + +|Functional +|`test/*integration/**/config.js` `test/*functional/**/config.js` `test/accessibility/config.js` +|`yarn test:ftr:server --config test/[directory]/config.js``yarn test:ftr:runner --config test/[directory]/config.js --grep=regexp` + +|Karma |`src/**/public/__tests__/*.js` |`yarn test:karma:debug` +|=== + +For X-Pack tests located in `x-pack/` see +link:{kib-repo}tree/{branch}/x-pack/README.md#testing[X-Pack Testing] + +Test runner arguments: - Where applicable, the optional arguments +`-t=regexp` or `--grep=regexp` will only run tests or test suites +whose descriptions matches the regular expression. - `[test path]` is +the relative path to the test file. + +Examples: - Run the entire elasticsearch_service test suite: +`yarn test:jest src/core/server/elasticsearch/elasticsearch_service.test.ts` +- Run the jest test case whose description matches +`stops both admin and data clients`: +`yarn test:jest -t 'stops both admin and data clients' src/core/server/elasticsearch/elasticsearch_service.test.ts` +- Run the api integration test case whose description matches the given +string: ``` yarn test:ftr:server –config test/api_integration/config.js +yarn test:ftr:runner –config test/api_integration/config + +[float] +==== Cross-browser compatibility + +**Testing IE on OS X** + +* http://www.vmware.com/products/fusion/fusion-evaluation.html[Download +VMWare Fusion]. +* https://developer.microsoft.com/en-us/microsoft-edge/tools/vms/#downloads[Download +IE virtual machines] for VMWare. +* Open VMWare and go to Window > Virtual Machine Library. Unzip the +virtual machine and drag the .vmx file into your Virtual Machine +Library. +* Right-click on the virtual machine you just added to your library and +select "`Snapshots…`", and then click the "`Take`" button in the modal +that opens. You can roll back to this snapshot when the VM expires in 90 +days. +* In System Preferences > Sharing, change your computer name to be +something simple, e.g. "`computer`". +* Run {kib} with `yarn start --host=computer.local` (substituting +your computer name). +* Now you can run your VM, open the browser, and navigate to +`http://computer.local:5601` to test {kib}. +* Alternatively you can use browserstack + +[float] +==== Running browser automation tests + +Check out <> to learn more about how you can run +and develop functional tests for {kib} core and plugins. + +You can also look into the {kib-repo}tree/{branch}/scripts/README.md[Scripts README.md] +to learn more about using the node scripts we provide for building +{kib}, running integration tests, and starting up {kib} and +Elasticsearch while you develop. + +[float] +==== More testing information: + +* <> +* <> +* <> + +include::development-functional-tests.asciidoc[] + +include::development-unit-tests.asciidoc[] + +include::development-accessibility-tests.asciidoc[] \ No newline at end of file diff --git a/docs/developer/core/development-unit-tests.asciidoc b/docs/developer/contributing/development-unit-tests.asciidoc similarity index 52% rename from docs/developer/core/development-unit-tests.asciidoc rename to docs/developer/contributing/development-unit-tests.asciidoc index 266f9333f50a9..3e839f207bcce 100644 --- a/docs/developer/core/development-unit-tests.asciidoc +++ b/docs/developer/contributing/development-unit-tests.asciidoc @@ -1,15 +1,11 @@ [[development-unit-tests]] -=== Unit Testing +==== Unit testing frameworks -We use unit tests to make sure that individual software units of {kib} perform as they were designed to. +{kib} is migrating unit testing from `Mocha` to `Jest`. Legacy unit tests +still exist in Mocha but all new unit tests should be written in Jest. [float] -=== Current Frameworks - -{kib} is migrating unit testing from `Mocha` to `Jest`. Legacy unit tests still exist in `Mocha` but all new unit tests should be written in `Jest`. - -[float] -==== Mocha (legacy) +===== Mocha (legacy) Mocha tests are contained in `__tests__` directories. @@ -32,7 +28,7 @@ yarn test:jest ----------- [float] -===== Writing Jest Unit Tests +====== Writing Jest Unit Tests In order to write those tests there are two main things you need to be aware of. The first one is the different between `jest.mock` and `jest.doMock` @@ -42,7 +38,7 @@ specially for the tests implemented on Typescript in order to benefit from the auto-inference types feature. [float] -===== Jest.mock vs Jest.doMock +====== Jest.mock vs Jest.doMock Both methods are essentially the same on their roots however the `jest.mock` calls will get hoisted to the top of the file and can only reference variables @@ -52,7 +48,7 @@ variables are instantiated at the time we need them which lead us to the next section where we'll talk about our jest mock files pattern. [float] -===== Jest Mock Files Pattern +====== Jest Mock Files Pattern Specially on typescript it is pretty common to have in unit tests `jest.doMock` calls which reference for example imported types. Any error @@ -79,4 +75,71 @@ like: `import * as Mocks from './mymodule.test.mocks'`, `import { mockX } from './mymodule.test.mocks'` or just `import './mymodule.test.mocks'` if there isn't anything exported to be used. - + +[float] +[[debugging-unit-tests]] +===== Debugging Unit Tests + +The standard `yarn test` task runs several sub tasks and can take +several minutes to complete, making debugging failures pretty painful. +In order to ease the pain specialized tasks provide alternate methods +for running the tests. + +You could also add the `--debug` option so that `node` is run using +the `--debug-brk` flag. You’ll need to connect a remote debugger such +as https://github.com/node-inspector/node-inspector[`node-inspector`] +to proceed in this mode. + +[source,bash] +---- +node scripts/mocha --debug +---- + +With `yarn test:karma`, you can run only the browser tests. Coverage +reports are available for browser tests by running +`yarn test:coverage`. You can find the results under the `coverage/` +directory that will be created upon completion. + +[source,bash] +---- +yarn test:karma +---- + +Using `yarn test:karma:debug` initializes an environment for debugging +the browser tests. Includes an dedicated instance of the {kib} server +for building the test bundle, and a karma server. When running this task +the build is optimized for the first time and then a karma-owned +instance of the browser is opened. Click the "`debug`" button to open a +new tab that executes the unit tests. + +[source,bash] +---- +yarn test:karma:debug +---- + +In the screenshot below, you’ll notice the URL is +`localhost:9876/debug.html`. You can append a `grep` query parameter +to this URL and set it to a string value which will be used to exclude +tests which don’t match. For example, if you changed the URL to +`localhost:9876/debug.html?query=my test` and then refreshed the +browser, you’d only see tests run which contain "`my test`" in the test +description. + +image:http://i.imgur.com/DwHxgfq.png[Browser test debugging] + +[float] +===== Unit Testing Plugins + +This should work super if you’re using the +https://github.com/elastic/kibana/tree/master/packages/kbn-plugin-generator[Kibana +plugin generator]. If you’re not using the generator, well, you’re on +your own. We suggest you look at how the generator works. + +To run the tests for just your particular plugin run the following +command from your plugin: + +[source,bash] +---- +yarn test:mocha +yarn test:karma:debug # remove the debug flag to run them once and close +---- diff --git a/docs/developer/contributing/index.asciidoc b/docs/developer/contributing/index.asciidoc new file mode 100644 index 0000000000000..4f987f31cf1f6 --- /dev/null +++ b/docs/developer/contributing/index.asciidoc @@ -0,0 +1,89 @@ +[[contributing]] +== Contributing + +Whether you want to fix a bug, implement a feature, or add some other improvements or apis, the following sections will +guide you on the process. + +Read <> to get your environment up and running, then read <>. + +* <> +* <> +* <> +* <> +* <> +* <> +* <> +* <> +* <> +* <> + +[discrete] +[[signing-contributor-agreement]] +=== Signing the contributor license agreement + +Please make sure you have signed the [Contributor License Agreement](http://www.elastic.co/contributor-agreement/). We are not asking you to assign copyright to us, but to give us the right to distribute your code without restriction. We ask this of all contributors in order to assure our users of the origin and continuing existence of the code. You only need to sign the CLA once. + +[float] +[[kibana-localization]] +=== Localization + +Read <> for details on our localization practices. + +Note that we cannot support accepting contributions to the translations from any source other than the translators we have engaged to do the work. +We are still to develop a proper process to accept any contributed translations. We certainly appreciate that people care enough about the localization effort to want to help improve the quality. We aim to build out a more comprehensive localization process for the future and will notify you once contributions can be supported, but for the time being, we are not able to incorporate suggestions. + +[float] +[[kibana-release-notes-process]] +=== Release Notes Process + +Part of this process only applies to maintainers, since it requires +access to GitHub labels. + +{kib} publishes https://www.elastic.co/guide/en/kibana/current/release-notes.html[Release Notes] for major and minor releases. +The Release Notes summarize what the PRs accomplish in language that is meaningful to users. + To generate the Release Notes, the team runs a script against this repo to collect the merged PRs against the release. + +[float] +==== Create the Release Notes text + +The text that appears in the Release Notes is pulled directly from your PR title, or a single paragraph of text that you specify in the PR description. + +To use a single paragraph of text, enter `Release note:` or a `## Release note` header in the PR description, followed by your text. For example, refer to this https://github.com/elastic/kibana/pull/65796[PR] that uses the `## Release note` header. + +When you create the Release Notes text, use the following best practices: + +* Use present tense. +* Use sentence case. +* When you create a feature PR, start with `Adds`. +* When you create an enhancement PR, start with `Improves`. +* When you create a bug fix PR, start with `Fixes`. +* When you create a deprecation PR, start with `Deprecates`. + +[float] +==== Add your labels + +[arabic] +. Label the PR with the targeted version (ex: `v7.3.0`). +. Label the PR with the appropriate GitHub labels: + * For a new feature or functionality, use `release_note:enhancement`. + * For an external-facing fix, use `release_note:fix`. We do not include docs, build, and test fixes in the Release Notes, or unreleased issues that are only on `master`. + * For a deprecated feature, use `release_note:deprecation`. + * For a breaking change, use `release_note:breaking`. + * To **NOT** include your changes in the Release Notes, use `release_note:skip`. + + +include::development-github.asciidoc[] + +include::development-tests.asciidoc[] + +include::interpreting-ci-failures.asciidoc[] + +include::development-documentation.asciidoc[] + +include::development-pull-request.asciidoc[] + +include::kibana-issue-reporting.asciidoc[] + +include::pr-review.asciidoc[] + +include::linting.asciidoc[] diff --git a/docs/developer/testing/interpreting-ci-failures.asciidoc b/docs/developer/contributing/interpreting-ci-failures.asciidoc similarity index 87% rename from docs/developer/testing/interpreting-ci-failures.asciidoc rename to docs/developer/contributing/interpreting-ci-failures.asciidoc index bc237928cf5aa..86c4f22b4f894 100644 --- a/docs/developer/testing/interpreting-ci-failures.asciidoc +++ b/docs/developer/contributing/interpreting-ci-failures.asciidoc @@ -1,19 +1,19 @@ [[interpreting-ci-failures]] -== Interpreting CI Failures +=== Interpreting CI Failures -Kibana CI uses a Jenkins feature called "Pipelines" to automate testing of the code in pull requests and on tracked branches. Pipelines are defined within the repository via the `Jenkinsfile` at the root of the project. +{kib} CI uses a Jenkins feature called "Pipelines" to automate testing of the code in pull requests and on tracked branches. Pipelines are defined within the repository via the `Jenkinsfile` at the root of the project. More information about Jenkins Pipelines can be found link:https://jenkins.io/doc/book/pipeline/[in the Jenkins book]. [float] -=== Github Checks +==== Github Checks When a test fails it will be reported to Github via Github Checks. We currently bucket tests into several categories which run in parallel to make CI faster. Groups like `ciGroup{X}` get a single check in Github, and other tests like linting, or type checks, get their own checks. Clicking the link next to the check in the conversation tab of a pull request will take you to the log output from that section of the tests. If that log output is truncated, or doesn't clearly identify what happened, you can usually get more complete information by visiting Jenkins directly. [float] -=== Viewing Job Executions in Jenkins +==== Viewing Job Executions in Jenkins To view the results of a job execution in Jenkins, either click the link in the comment left by `@elasticmachine` or search for the `kibana-ci` check in the list at the bottom of the PR. This link will take you to the top-level page for the specific job execution that failed. @@ -25,7 +25,7 @@ image::images/jenkins/job_view.png[] 4. *Pipeline Steps:*: A breakdown of the pipline that was executed, along with individual log output for each step in the pipeline. [float] -=== Viewing ciGroup/test Logs +==== Viewing ciGroup/test Logs To view the logs for a failed specific ciGroup, jest, mocha, type checkers, linters, etc., click on the *Pipeline Steps* link in from the Job page. diff --git a/docs/developer/contributing/kibana-issue-reporting.asciidoc b/docs/developer/contributing/kibana-issue-reporting.asciidoc new file mode 100644 index 0000000000000..36c50b612d675 --- /dev/null +++ b/docs/developer/contributing/kibana-issue-reporting.asciidoc @@ -0,0 +1,46 @@ +[[kibana-issue-reporting]] +=== Effective issue reporting in {kib} + +[float] +==== Voicing the importance of an issue + +We seriously appreciate thoughtful comments. If an issue is important to +you, add a comment with a solid write up of your use case and explain +why it’s so important. Please avoid posting comments comprised solely of +a thumbs up emoji 👍. + +Granted that you share your thoughts, we might even be able to come up +with creative solutions to your specific problem. If everything you’d +like to say has already been brought up but you’d still like to add a +token of support, feel free to add a +https://github.com/blog/2119-add-reactions-to-pull-requests-issues-and-comments[👍 +thumbs up reaction] on the issue itself and on the comment which best +summarizes your thoughts. + +[float] +==== "`My issue isn’t getting enough attention`" + +First of all, *sorry about that!* We want you to have a great time with +{kib}. + +There’s hundreds of open issues and prioritizing what to work on is an +important aspect of our daily jobs. We prioritize issues according to +impact and difficulty, so some issues can be neglected while we work on +more pressing issues. + +Feel free to bump your issues if you think they’ve been neglected for a +prolonged period. + +[float] +==== "`I want to help!`" + +*Now we’re talking*. If you have a bug fix or new feature that you would +like to contribute to {kib}, please *find or open an issue about it +before you start working on it.* Talk about what you would like to do. +It may be that somebody is already working on it, or that there are +particular issues that you should know about before implementing the +change. + +We enjoy working with contributors to get their code accepted. There are +many approaches to fixing a problem and it is important to find the best +approach before writing too much code. \ No newline at end of file diff --git a/docs/developer/contributing/linting.asciidoc b/docs/developer/contributing/linting.asciidoc new file mode 100644 index 0000000000000..234bd90478907 --- /dev/null +++ b/docs/developer/contributing/linting.asciidoc @@ -0,0 +1,70 @@ +[[kibana-linting]] +=== Linting + +A note about linting: We use http://eslint.org[eslint] to check that the +link:STYLEGUIDE.md[styleguide] is being followed. It runs in a +pre-commit hook and as a part of the tests, but most contributors +integrate it with their code editors for real-time feedback. + +Here are some hints for getting eslint setup in your favorite editor: + +[width="100%",cols="13%,87%",options="header",] +|=== +|Editor |Plugin +|Sublime +|https://github.com/roadhump/SublimeLinter-eslint#installation[SublimeLinter-eslint] + +|Atom +|https://github.com/AtomLinter/linter-eslint#installation[linter-eslint] + +|VSCode +|https://marketplace.visualstudio.com/items?itemName=dbaeumer.vscode-eslint[ESLint] + +|IntelliJ |Settings » Languages & Frameworks » JavaScript » Code Quality +Tools » ESLint + +|`vi` |https://github.com/scrooloose/syntastic[scrooloose/syntastic] +|=== + +Another tool we use for enforcing consistent coding style is +EditorConfig, which can be set up by installing a plugin in your editor +that dynamically updates its configuration. Take a look at the +http://editorconfig.org/#download[EditorConfig] site to find a plugin +for your editor, and browse our +https://github.com/elastic/kibana/blob/master/.editorconfig[`.editorconfig`] +file to see what config rules we set up. + +[float] +==== Setup Guide for VS Code Users + +Note that for VSCode, to enable "`live`" linting of TypeScript (and +other) file types, you will need to modify your local settings, as shown +below. The default for the ESLint extension is to only lint JavaScript +file types. + +[source,json] +---- +"eslint.validate": [ + "javascript", + "javascriptreact", + { "language": "typescript", "autoFix": true }, + { "language": "typescriptreact", "autoFix": true } +] +---- + +`eslint` can automatically fix trivial lint errors when you save a +file by adding this line in your setting. + +[source,json] +---- + "eslint.autoFixOnSave": true, +---- + +:warning: It is *not* recommended to use the +https://prettier.io/[`Prettier` extension/IDE plugin] while +maintaining the {kib} project. Formatting and styling roles are set in +the multiple `.eslintrc.js` files across the project and some of them +use the https://www.npmjs.com/package/prettier[NPM version of Prettier]. +Using the IDE extension might cause conflicts, applying the formatting +to too many files that shouldn’t be prettier-ized and/or highlighting +errors that are actually OK. \ No newline at end of file diff --git a/docs/developer/pr-review.asciidoc b/docs/developer/contributing/pr-review.asciidoc similarity index 90% rename from docs/developer/pr-review.asciidoc rename to docs/developer/contributing/pr-review.asciidoc index 304718e437dc5..ebab3b24aaaee 100644 --- a/docs/developer/pr-review.asciidoc +++ b/docs/developer/contributing/pr-review.asciidoc @@ -1,7 +1,7 @@ [[pr-review]] -== Pull request review guidelines +=== Pull request review guidelines -Every change made to Kibana must be held to a high standard, and while the responsibility for quality in a pull request ultimately lies with the author, Kibana team members have the responsibility as reviewers to verify during their review process. +Every change made to {kib} must be held to a high standard, and while the responsibility for quality in a pull request ultimately lies with the author, {kib} team members have the responsibility as reviewers to verify during their review process. Frankly, it's impossible to build a concrete list of requirements that encompass all of the possible situations we'll encounter when reviewing pull requests, so instead this document tries to lay out a common set of the few obvious requirements while also outlining a general philosophy that we should have when approaching any PR review. @@ -11,15 +11,15 @@ While the review process is always done by Elastic staff members, these guidelin [float] -=== Target audience +==== Target audience -The target audience for this document are pull request reviewers. For Kibana maintainers, the PR review is the only part of the contributing process in which we have complete control. The author of any given pull request may not be up to speed on the latest expectations we have for pull requests, and they may have never read our guidelines at all. It's our responsibility as reviewers to guide folks through this process, but it's hard to do that consistently without a common set of documented principles. +The target audience for this document are pull request reviewers. For {kib} maintainers, the PR review is the only part of the contributing process in which we have complete control. The author of any given pull request may not be up to speed on the latest expectations we have for pull requests, and they may have never read our guidelines at all. It's our responsibility as reviewers to guide folks through this process, but it's hard to do that consistently without a common set of documented principles. Pull request authors can benefit from reading this document as well because it'll help establish a common set of expectations between authors and reviewers early. [float] -=== Reject fast +==== Reject fast Every pull request is different, and before reviewing any given PR, reviewers should consider the optimal way to approach the PR review so that if the change is ultimately rejected, it is done so as early in the process as possible. @@ -27,7 +27,7 @@ For example, a reviewer may want to do a product level review as early as possib [float] -=== The big three +==== The big three There are a lot of discrete requirements and guidelines we want to follow in all of our pull requests, but three things in particular stand out as important above all the rest. @@ -58,24 +58,24 @@ This isn't simply a question of enough test files. The code in the tests themsel All of our code should have unit tests that verify its behaviors, including not only the "happy path", but also edge cases, error handling, etc. When you change an existing API of a module, then there should always be at least one failing unit test, which in turn means we need to verify that all code consuming that API properly handles the change if necessary. For modules at a high enough level, this will mean we have breaking change in the product, which we'll need to handle accordingly. -In addition to extensive unit test coverage, PRs should include relevant functional and integration tests. In some cases, we may simply be testing a programmatic interface (e.g. a service) that is integrating with the file system, the network, Elasticsearch, etc. In other cases, we'll be testing REST APIs over HTTP or comparing screenshots/snapshots with prior known acceptable state. In the worst case, we are doing browser-based functional testing on a running instance of Kibana using selenium. +In addition to extensive unit test coverage, PRs should include relevant functional and integration tests. In some cases, we may simply be testing a programmatic interface (e.g. a service) that is integrating with the file system, the network, Elasticsearch, etc. In other cases, we'll be testing REST APIs over HTTP or comparing screenshots/snapshots with prior known acceptable state. In the worst case, we are doing browser-based functional testing on a running instance of {kib} using selenium. Enhancements are pretty much always going to have extensive unit tests as a base as well as functional and integration testing. Bug fixes should always include regression tests to ensure that same bug does not manifest again in the future. -- [float] -=== Product level review +==== Product level review Reviewers are not simply evaluating the code itself, they are also evaluating the quality of the user-facing change in the product. This generally means they need to check out the branch locally and "play around" with it. In addition to the "do we want this change in the product" details, the reviewer should be looking for bugs and evaluating how approachable and useful the feature is as implemented. Special attention should be given to error scenarios and edge cases to ensure they are all handled well within the product. [float] -=== Consistency, style, readability +==== Consistency, style, readability Having a relatively consistent codebase is an important part of us building a sustainable project. With dozens of active contributors at any given time, we rely on automation to help ensure consistency - we enforce a comprehensive set of linting rules through CI. We're also rolling out prettier to make this even more automatic. -For things that can't be easily automated, we maintain a link:https://github.com/elastic/kibana/blob/master/STYLEGUIDE.md[style guide] that authors should adhere to and reviewers should keep in mind when they review a pull request. +For things that can't be easily automated, we maintain a link:{kib-repo}tree/{branch}/STYLEGUIDE.md[style guide] that authors should adhere to and reviewers should keep in mind when they review a pull request. Beyond that, we're into subjective territory. Statements like "this isn't very readable" are hardly helpful since they can't be qualified, but that doesn't mean a reviewer should outright ignore code that is hard to understand due to how it is written. There isn't one definitively "best" way to write any particular code, so pursuing such shouldn't be our goal. Instead, reviewers and authors alike must accept that there are likely many different appropriate ways to accomplish the same thing with code, and so long as the contribution is utilizing one of those ways, then we're in good shape. @@ -87,7 +87,7 @@ There may also be times when a person is inspired by a particular contribution t [float] -=== Nitpicking +==== Nitpicking Nitpicking is when a reviewer identifies trivial and unimportant details in a pull request and asks the author to change them. This is a completely subjective category that is impossible to define universally, and it's equally impractical to define a blanket policy on nitpicking that everyone will be happy with. @@ -97,13 +97,13 @@ Often, reviewers have an opinion about whether the feedback they are about to gi [float] -=== Handling disagreements +==== Handling disagreements Conflicting opinions between reviewers and authors happen, and sometimes it is hard to reconcile those opinions. Ideally folks can work together in the spirit of these guidelines toward a consensus, but if that doesn't work out it may be best to bring a third person into the discussion. Our pull requests generally have two reviewers, so an appropriate third person may already be obvious. Otherwise, reach out to the functional area that is most appropriate or to technical leadership if an area isn't obvious. [float] -=== Inappropriate review feedback +==== Inappropriate review feedback Whether or not a bit of feedback is appropriate for a pull request is often dependent on the motivation for giving the feedback in the first place. @@ -113,7 +113,7 @@ Inflammatory feedback such as "this is crap" isn't feedback at all. It's both me [float] -=== A checklist +==== A checklist Establishing a comprehensive checklist for all of the things that should happen in all possible pull requests is impractical, but that doesn't mean we lack a concrete set of minimum requirements that we can enumerate. The following items should be double checked for any pull request: diff --git a/docs/developer/core-development.asciidoc b/docs/developer/core-development.asciidoc deleted file mode 100644 index 8f356abd095f2..0000000000000 --- a/docs/developer/core-development.asciidoc +++ /dev/null @@ -1,24 +0,0 @@ -[[core-development]] -== Core Development - -* <> -* <> -* <> -* <> -* <> -* <> -* <> - -include::core/development-basepath.asciidoc[] - -include::core/development-dependencies.asciidoc[] - -include::core/development-modules.asciidoc[] - -include::core/development-elasticsearch.asciidoc[] - -include::core/development-unit-tests.asciidoc[] - -include::core/development-functional-tests.asciidoc[] - -include::core/development-es-snapshots.asciidoc[] diff --git a/docs/developer/core/development-basepath.asciidoc b/docs/developer/core/development-basepath.asciidoc deleted file mode 100644 index d49dfe2938fad..0000000000000 --- a/docs/developer/core/development-basepath.asciidoc +++ /dev/null @@ -1,85 +0,0 @@ -[[development-basepath]] -=== Considerations for basePath - -All communication from the Kibana UI to the server needs to respect the -`server.basePath`. Here are the "blessed" strategies for dealing with this -based on the context: - -[float] -==== Getting a static asset url - -Use webpack to import the asset into the build. This will give you a URL in -JavaScript and gives webpack a chance to perform optimizations and -cache-busting. - -["source","shell"] ------------ -// in plugin/public/main.js -import uiChrome from 'ui/chrome'; -import logoUrl from 'plugins/facechimp/assets/banner.png'; - -uiChrome.setBrand({ - logo: `url(${logoUrl}) center no-repeat` -}); ------------ - -[float] -==== API requests from the front-end - -Use `chrome.addBasePath()` to append the basePath to the front of the url. - -["source","shell"] ------------ -import chrome from 'ui/chrome'; -$http.get(chrome.addBasePath('/api/plugin/things')); ------------ - -[float] -==== Server side - -Append `request.getBasePath()` to any absolute URL path. - -["source","shell"] ------------ -const basePath = server.config().get('server.basePath'); -server.route({ - path: '/redirect', - handler(request, h) { - return h.redirect(`${request.getBasePath()}/otherLocation`); - } -}); ------------ - -[float] -==== BasePathProxy in dev mode - -The Kibana dev server automatically runs behind a proxy with a random -`server.basePath`. This way developers will be constantly verifying that their -code works with basePath, while they write it. - -To accomplish this the `serve` task does a few things: - -1. change the port for the server to the `dev.basePathProxyTarget` setting (default `5603`) -2. start a `BasePathProxy` at `server.port` - - picks a random 3-letter value for `randomBasePath` - - redirects from `/` to `/{randomBasePath}` - - redirects from `/{any}/app/{appName}` to `/{randomBasePath}/app/{appName}` so that refreshes should work - - proxies all requests starting with `/{randomBasePath}/` to the Kibana server - -If you're writing scripts that interact with the Kibana API, the base path proxy will likely -make this difficult. To bypass the base path proxy for a single request, prefix urls with -`__UNSAFE_bypassBasePath` and the request will be routed to the development Kibana server. - -["source","shell"] ------------ -curl "http://elastic:changeme@localhost:5601/__UNSAFE_bypassBasePath/api/status" ------------ - -This proxy can sometimes have unintended side effects in development, so when -needed you can opt out by passing the `--no-base-path` flag to the `serve` task -or `yarn start`. - -["source","shell"] ------------ -yarn start --no-base-path ------------ diff --git a/docs/developer/core/development-dependencies.asciidoc b/docs/developer/core/development-dependencies.asciidoc deleted file mode 100644 index 285d338a23a0d..0000000000000 --- a/docs/developer/core/development-dependencies.asciidoc +++ /dev/null @@ -1,103 +0,0 @@ -[[development-dependencies]] -=== Managing Dependencies - -While developing plugins for use in the Kibana front-end environment you will -probably want to include a library or two (at least). While that should be -simple to do 90% of the time, there are always outliers, and some of those -outliers are very popular projects. - -Before you can use an external library with Kibana you have to install it. You -do that using... - -[float] -==== yarn (preferred method) - -Once you've http://npmsearch.com[found] a dependency you want to add, you can -install it like so: - -["source","shell"] ------------ -yarn add some-neat-library ------------ - -At the top of a javascript file, just import the library using it's name: - -["source","shell"] ------------ -import someNeatLibrary from 'some-neat-library'; ------------ - -Just like working in node.js, front-end code can require node modules installed -by yarn without any additional configuration. - -[float] -==== webpackShims - -When a library you want to use does use es6 or common.js modules but is not -available with yarn, you can copy the source of the library into a webpackShim. - -["source","shell"] ------------ -# create a directory for our new library to live -mkdir -p webpackShims/some-neat-library -# download the library you want to use into that directory -curl https://cdnjs.com/some-neat-library/library.js > webpackShims/some-neat-library/index.js ------------ - -Then include the library in your JavaScript code as you normally would: - -["source","shell"] ------------ -import someNeatLibrary from 'some-neat-library'; ------------ - -[float] -==== Shimming third party code - -Some JavaScript libraries do not declare their dependencies in a way that tools -like webpack can understand. It is also often the case that libraries do not -`export` their provided values, but simply write them to a global variable name -(or something to that effect). - -When pulling code like this into Kibana we need to write "shims" that will -adapt the third party code to work with our application, other libraries, and -module system. To do this we can utilize the `webpackShims` directory. - -The easiest way to explain how to write a shim is to show you some. Here is our -webpack shim for jQuery: - -["source","shell"] ------------ -// webpackShims/jquery.js - -module.exports = window.jQuery = window.$ = require('../node_modules/jquery/dist/jquery'); -require('ui/jquery/findTestSubject')(window.$); ------------ - -This shim is loaded up anytime an `import 'jquery';` statement is found by -webpack, because of the way that `webpackShims` behaves like `node_modules`. -When that happens, the shim does two things: - -. Assign the exported value of the actual jQuery module to the window at `$` and `jQuery`, allowing libraries like angular to detect that jQuery is available, and use it as the module's export value. -. Finally, a jQuery plugin that we wrote is included so that every time a file imports jQuery it will get both jQuery and the `$.findTestSubject` helper function. - -Here is what our webpack shim for angular looks like: - -["source","shell"] ------------ -// webpackShims/angular.js - -require('jquery'); -require('../node_modules/angular/angular'); -require('../node_modules/angular-elastic/elastic'); -require('ui/modules').get('kibana', ['monospaced.elastic']); -module.exports = window.angular; ------------ - -What this shim does is fairly simple if you go line by line: - -. makes sure that jQuery is loaded before angular (which actually runs the shim) -. load the angular.js file from the node_modules directory -. load the angular-elastic plugin, a plugin we want to always be included whenever we import angular -. use the `ui/modules` module to add the module exported by angular-elastic as a dependency to the `kibana` angular module -. finally, export the window.angular variable. This means that writing `import angular from 'angular';` will properly set the angular variable to the angular library, rather than undefined which is the default behavior. diff --git a/docs/developer/core/development-elasticsearch.asciidoc b/docs/developer/core/development-elasticsearch.asciidoc deleted file mode 100644 index 89f85cfc19fbf..0000000000000 --- a/docs/developer/core/development-elasticsearch.asciidoc +++ /dev/null @@ -1,40 +0,0 @@ -[[development-elasticsearch]] -=== Communicating with Elasticsearch - -Kibana exposes two clients on the server and browser for communicating with elasticsearch. -There is an 'admin' client which is used for managing Kibana's state, and a 'data' client for all -other requests. The clients use the {jsclient-current}/index.html[elasticsearch.js library]. - -[float] -[[client-server]] -=== Server clients - -Server clients are exposed through the elasticsearch plugin. -[source,javascript] ----- - const adminCluster = server.plugins.elasticsearch.getCluster('admin'); - const dataCluster = server.plugins.elasticsearch.getCluster('data'); - - //ping as the configured elasticsearch.user in kibana.yml - adminCluster.callWithInternalUser('ping'); - - //ping as the user specified in the current requests header - adminCluster.callWithRequest(req, 'ping'); ----- - -[float] -[[client-browser]] -=== Browser clients - -Browser clients are exposed through AngularJS services. - -[source,javascript] ----- -uiModules.get('kibana') -.run(function (es) { - es.ping() - .catch(err => { - console.log('error pinging servers'); - }); -}); ----- diff --git a/docs/developer/core/development-modules.asciidoc b/docs/developer/core/development-modules.asciidoc deleted file mode 100644 index cc5cd69ed8cb9..0000000000000 --- a/docs/developer/core/development-modules.asciidoc +++ /dev/null @@ -1,63 +0,0 @@ -[[development-modules]] -=== Modules and Autoloading - -[float] -==== Autoloading - -Because of the disconnect between JS modules and angular directives, filters, -and services it is difficult to know what you need to import. It is even more -difficult to know if you broke something by removing an import that looked -unused. - -To prevent this from being an issue the ui module provides "autoloading" -modules. The sole purpose of these modules is to extend the environment with -certain components. Here is a breakdown of those modules: - -- *`import 'ui/autoload/modules'`* - Imports angular and several ui services and "components" which Kibana - depends on without importing. The full list of imports is hard coded in the - module. Hopefully this list will shrink over time as we properly map out - the required modules and import them were they are actually necessary. - -- *`import 'ui/autoload/all'`* - Imports all of the modules - -[float] -==== Resolving Require Paths - -Kibana uses Webpack to bundle Kibana's dependencies. - -Here is how import/require statements are resolved to a file: - -. Check the beginning of the module path - * if the path starts with a '.' - ** append it the directory of the current file - ** proceed to *3* - * if the path starts with a '/' - ** search for this exact path - ** proceed to *3* - * proceed to *2* -. Search for a named module - * `moduleName` = dirname(require path)` - * match if `moduleName` is or starts with one of these aliases - ** replace the alias with the match and continue to ***3*** - * match when any of these conditions are met: - ** `./webpackShims/${moduleName}` is a directory - ** `./node_modules/${moduleName}` is a directory - * if no match was found - ** move to the parent directory - ** start again at *2.iii* until reaching the root directory or a match is found - * if a match was found - ** replace the `moduleName` prefix from the require statement with the full path of the match and proceed to *3* -. Search for a file - * the first of the following paths that resolves to a **file** is our match - ** path + '.js' - ** path + '.json' - ** path - ** path/${basename(path)} + '.js' - ** path/${basename(path)} + '.json' - ** path/${basename(path)} - ** path/index + '.js' - ** path/index + '.json' - ** path/index - * if none of the paths matches then an error is thrown diff --git a/docs/developer/getting-started/building-kibana.asciidoc b/docs/developer/getting-started/building-kibana.asciidoc new file mode 100644 index 0000000000000..e1f1ca336a5da --- /dev/null +++ b/docs/developer/getting-started/building-kibana.asciidoc @@ -0,0 +1,39 @@ +[[building-kibana]] +=== Building a {kib} distributable + +The following commands will build a {kib} production distributable. + +[source,bash] +---- +yarn build --skip-os-packages +---- + +You can get all build options using the following command: + +[source,bash] +---- +yarn build --help +---- + +[float] +==== Building OS packages + +Packages are built using fpm, dpkg, and rpm. Package building has only been tested on Linux and is not supported on any other platform. + + +[source,bash] +---- +apt-get install ruby-dev rpm +gem install fpm -v 1.5.0 +yarn build --skip-archives +---- + +To specify a package to build you can add `rpm` or `deb` as an argument. + + +[source,bash] +---- +yarn build --rpm +---- + +Distributable packages can be found in `target/` after the build completes. \ No newline at end of file diff --git a/docs/developer/getting-started/debugging.asciidoc b/docs/developer/getting-started/debugging.asciidoc new file mode 100644 index 0000000000000..b369dcda748af --- /dev/null +++ b/docs/developer/getting-started/debugging.asciidoc @@ -0,0 +1,59 @@ +[[kibana-debugging]] +=== Debugging {kib} + +For information about how to debug unit tests, refer to <>. + +[float] +==== Server Code + +`yarn debug` will start the server with Node's inspect flag. {kib}'s development mode will start three processes on ports `9229`, `9230`, and `9231`. Chrome's developer tools need to be configured to connect to all three connections. Add `localhost:` for each {kib} process in Chrome's developer tools connection tab. + +[float] +==== Instrumenting with Elastic APM + +{kib} ships with the +https://github.com/elastic/apm-agent-nodejs[Elastic APM Node.js Agent] +built-in for debugging purposes. + +Its default configuration is meant to be used by core {kib} developers +only, but it can easily be re-configured to your needs. In its default +configuration it’s disabled and will, once enabled, send APM data to a +centrally managed Elasticsearch cluster accessible only to Elastic +employees. + +To change the location where data is sent, use the +https://www.elastic.co/guide/en/apm/agent/nodejs/current/configuration.html#server-url[`serverUrl`] +APM config option. To activate the APM agent, use the +https://www.elastic.co/guide/en/apm/agent/nodejs/current/configuration.html#active[`active`] +APM config option. + +All config options can be set either via environment variables, or by +creating an appropriate config file under `config/apm.dev.js`. For +more information about configuring the APM agent, please refer to +https://www.elastic.co/guide/en/apm/agent/nodejs/current/configuring-the-agent.html[the +documentation]. + +Example `config/apm.dev.js` file: + +[source,js] +---- +module.exports = { + active: true, +}; +---- + +APM +https://www.elastic.co/guide/en/apm/agent/rum-js/current/index.html[Real +User Monitoring agent] is not available in the {kib} distributables, +however the agent can be enabled by setting `ELASTIC_APM_ACTIVE` to +`true`. flags + +.... +ELASTIC_APM_ACTIVE=true yarn start +// activates both Node.js and RUM agent +.... + +Once the agent is active, it will trace all incoming HTTP requests to +{kib}, monitor for errors, and collect process-level metrics. The +collected data will be sent to the APM Server and is viewable in the APM +UI in {kib}. \ No newline at end of file diff --git a/docs/developer/plugin/development-plugin-resources.asciidoc b/docs/developer/getting-started/development-plugin-resources.asciidoc similarity index 51% rename from docs/developer/plugin/development-plugin-resources.asciidoc rename to docs/developer/getting-started/development-plugin-resources.asciidoc index 3a32c49e40e0f..dfe8efc4fef57 100644 --- a/docs/developer/plugin/development-plugin-resources.asciidoc +++ b/docs/developer/getting-started/development-plugin-resources.asciidoc @@ -5,54 +5,35 @@ Here are some resources that are helpful for getting started with plugin develop [float] ==== Some light reading -Our {kib-repo}blob/master/CONTRIBUTING.md[contributing guide] can help you get a development environment going. +If you haven't already, start with <>. If you are planning to add your plugin to the {kib} repo, read the <> guide, if you are building a plugin externally, read <>. In both cases, read up on our recommended <>. [float] -==== Plugin Generator +==== Creating an empty plugin -We recommend that you kick-start your plugin by generating it with the {kib-repo}tree/{branch}/packages/kbn-plugin-generator[Kibana Plugin Generator]. Run the following in the Kibana repo, and you will be asked a couple questions, see some progress bars, and have a freshly generated plugin ready for you to play with in Kibana's `plugins` folder. +You can use the <> to get a basic structure for a new plugin. Plugins that are not part of the +{kib} repo should be developed inside the `plugins` folder. If you are building a new plugin to check in to the {kib} repo, +you will choose between a few locations: -["source","shell"] ------------ -node scripts/generate_plugin my_plugin_name # replace "my_plugin_name" with your desired plugin name ------------ - - -[float] -==== Directory structure for plugins - -The Kibana directory must be named `kibana`, and your plugin directory should be located in the root of `kibana` in a `plugins` directory, for example: - -["source","shell"] ----- -. -└── kibana - └── plugins - ├── foo-plugin - └── bar-plugin ----- - -[float] -==== References in the code - - {kib-repo}blob/{branch}/src/legacy/server/plugins/lib/plugin.js[Plugin class]: What options does the `kibana.Plugin` class accept? - - <>: What type of exports are available? + - {kib-repo}tree/{branch}/x-pack/plugins[x-pack/plugins] for commercially licensed plugins + - {kib-repo}tree/{branch}/src/plugins[src/plugins] for open source licensed plugins + - {kib-repo}tree/{branch}/examples[examples] for developer example plugins (these will not be included in the distributables) [float] ==== Elastic UI Framework If you're developing a plugin that has a user interface, take a look at our https://elastic.github.io/eui[Elastic UI Framework]. -It documents the CSS and React components we use to build Kibana's user interface. +It documents the CSS and React components we use to build {kib}'s user interface. You're welcome to use these components, but be aware that they are rapidly evolving, and we might introduce breaking changes that will disrupt your plugin's UI. [float] ==== TypeScript Support -Plugin code can be written in http://www.typescriptlang.org/[TypeScript] if desired. +We recommend your plugin code is written in http://www.typescriptlang.org/[TypeScript]. To enable TypeScript support, create a `tsconfig.json` file at the root of your plugin that looks something like this: ["source","js"] ----------- { - // extend Kibana's tsconfig, or use your own settings + // extend {kib}'s tsconfig, or use your own settings "extends": "../../kibana/tsconfig.json", // tell the TypeScript compiler where to find your source files @@ -64,10 +45,17 @@ To enable TypeScript support, create a `tsconfig.json` file at the root of your ----------- TypeScript code is automatically converted into JavaScript during development, -but not in the distributable version of Kibana. If you use the -{kib-repo}blob/{branch}/packages/kbn-plugin-helpers[@kbn/plugin-helpers] to build your plugin, then your `.ts` and `.tsx` files will be permanently transpiled before your plugin is archived. If you have your own build process, make sure to run the TypeScript compiler on your source files and ship the compilation output so that your plugin will work with the distributable version of Kibana. +but not in the distributable version of {kib}. If you use the +{kib-repo}blob/{branch}/packages/kbn-plugin-helpers[@kbn/plugin-helpers] to build your plugin, then your `.ts` and `.tsx` files will be permanently transpiled before your plugin is archived. If you have your own build process, make sure to run the TypeScript compiler on your source files and ship the compilation output so that your plugin will work with the distributable version of {kib}. +[float] ==== {kib} platform migration guide {kib-repo}blob/{branch}/src/core/MIGRATION.md#migrating-legacy-plugins-to-the-new-platform[This guide] provides an action plan for moving a legacy plugin to the new platform. + +[float] +==== Externally developed plugins + +If you are building a plugin outside of the {kib} repo, read <>. + diff --git a/docs/developer/getting-started/index.asciidoc b/docs/developer/getting-started/index.asciidoc new file mode 100644 index 0000000000000..ff1623e22f1eb --- /dev/null +++ b/docs/developer/getting-started/index.asciidoc @@ -0,0 +1,144 @@ +[[development-getting-started]] +== Getting started + +Get started building your own plugins, or contributing directly to the {kib} repo. + +[float] +[[get-kibana-code]] +=== Get the code + +https://help.github.com/en/github/getting-started-with-github/fork-a-repo[Fork], then https://help.github.com/en/github/getting-started-with-github/fork-a-repo#step-2-create-a-local-clone-of-your-fork[clone] the {kib-repo}[{kib} repo] and change directory into it: + +[source,bash] +---- +git clone https://github.com/[YOUR_USERNAME]/kibana.git kibana +cd kibana +---- + +[float] +=== Install dependencies + +Install the version of Node.js listed in the `.node-version` file. This +can be automated with tools such as +https://github.com/creationix/nvm[nvm], +https://github.com/coreybutler/nvm-windows[nvm-windows] or +https://github.com/wbyoung/avn[avn]. As we also include a `.nvmrc` file +you can switch to the correct version when using nvm by running: + +[source,bash] +---- +nvm use +---- + +Install the latest version of https://yarnpkg.com[yarn]. + +Bootstrap {kib} and install all the dependencies: + +[source,bash] +---- +yarn kbn bootstrap +---- + +____ +Node.js native modules could be in use and node-gyp is the tool used to +build them. There are tools you need to install per platform and python +versions you need to be using. Please see +https://github.com/nodejs/node-gyp#installation[https://github.com/nodejs/node-gyp#installation] +and follow the guide according your platform. +____ + +(You can also run `yarn kbn` to see the other available commands. For +more info about this tool, see +{kib-repo}tree/{branch}/packages/kbn-pm[{kib-repo}tree/{branch}packages/kbn-pm].) + +When switching branches which use different versions of npm packages you +may need to run: + +[source,bash] +---- +yarn kbn clean +---- + +If you have failures during `yarn kbn bootstrap` you may have some +corrupted packages in your yarn cache which you can clean with: + +[source,bash] +---- +yarn cache clean +---- + +[float] +=== Configure environmental settings + +[[increase-nodejs-heap-size]] +[float] +==== Increase node.js heap size + +{kib} is a big project and for some commands it can happen that the +process hits the default heap limit and crashes with an out-of-memory +error. If you run into this problem, you can increase maximum heap size +by setting the `--max_old_space_size` option on the command line. To set +the limit for all commands, simply add the following line to your shell +config: `export NODE_OPTIONS="--max_old_space_size=2048"`. + +[float] +=== Run Elasticsearch + +Run the latest Elasticsearch snapshot. Specify an optional license with the `--license` flag. + +[source,bash] +---- +yarn es snapshot --license trial +---- + +`trial` will give you access to all capabilities. + +Read about more options for <>, like connecting to a remote host, running from source, +preserving data inbetween runs, running remote cluster, etc. + +[float] +=== Run {kib} + +In another terminal window, start up {kib}. Include developer examples by adding an optional `--run-examples` flag. + +[source,bash] +---- +yarn start --run-examples +---- + +View all available options by running `yarn start --help` + +Read about more advanced options for <>. + +[float] +=== Code away! + +You are now ready to start developing. Changes to your files should be picked up automatically. Server side changes will +cause the {kib} server to reboot. + +[float] +=== More information + +* <> + +* <> + +* <> + +* <> + +* <> + +* <> + +include::running-kibana-advanced.asciidoc[] + +include::sample-data.asciidoc[] + +include::debugging.asciidoc[] + +include::sass.asciidoc[] + +include::building-kibana.asciidoc[] + +include::development-plugin-resources.asciidoc[] \ No newline at end of file diff --git a/docs/developer/getting-started/running-kibana-advanced.asciidoc b/docs/developer/getting-started/running-kibana-advanced.asciidoc new file mode 100644 index 0000000000000..e36f38de1b366 --- /dev/null +++ b/docs/developer/getting-started/running-kibana-advanced.asciidoc @@ -0,0 +1,87 @@ +[[running-kibana-advanced]] +=== Running {kib} + +Change to your local {kib} directory. Start the development server. + +[source,bash] +---- +yarn start +---- + +____ +On Windows, you’ll need to use Git Bash, Cygwin, or a similar shell that +exposes the `sh` command. And to successfully build you’ll need Cygwin +optional packages zip, tar, and shasum. +____ + +Now you can point your web browser to http://localhost:5601 and start +using {kib}! When running `yarn start`, {kib} will also log that it +is listening on port 5603 due to the base path proxy, but you should +still access {kib} on port 5601. + +By default, you can log in with username `elastic` and password +`changeme`. See the `--help` options on `yarn es ` if +you’d like to configure a different password. + +[float] +==== Running {kib} in Open-Source mode + +If you’re looking to only work with the open-source software, supply the +license type to `yarn es`: + +[source,bash] +---- +yarn es snapshot --license oss +---- + +And start {kib} with only open-source code: + +[source,bash] +---- +yarn start --oss +---- + +[float] +==== Unsupported URL Type + +If you’re installing dependencies and seeing an error that looks +something like + +.... +Unsupported URL Type: link:packages/eslint-config-kibana +.... + +you’re likely running `npm`. To install dependencies in {kib} you +need to run `yarn kbn bootstrap`. For more info, see +link:#setting-up-your-development-environment[Setting Up Your +Development Environment] above. + +[float] +[[customize-kibana-yml]] +==== Customizing `config/kibana.dev.yml` + +The `config/kibana.yml` file stores user configuration directives. +Since this file is checked into source control, however, developer +preferences can’t be saved without the risk of accidentally committing +the modified version. To make customizing configuration easier during +development, the {kib} CLI will look for a `config/kibana.dev.yml` +file if run with the `--dev` flag. This file behaves just like the +non-dev version and accepts any of the +https://www.elastic.co/guide/en/kibana/current/settings.html[standard +settings]. + +[float] +==== Potential Optimization Pitfalls + +* Webpack is trying to include a file in the bundle that I deleted and +is now complaining about it is missing +* A module id that used to resolve to a single file now resolves to a +directory, but webpack isn’t adapting +* (if you discover other scenarios, please send a PR!) + +[float] +==== Setting Up SSL + +{kib} includes self-signed certificates that can be used for +development purposes in the browser and for communicating with +Elasticsearch: `yarn start --ssl` & `yarn es snapshot --ssl`. \ No newline at end of file diff --git a/docs/developer/getting-started/sample-data.asciidoc b/docs/developer/getting-started/sample-data.asciidoc new file mode 100644 index 0000000000000..376211ceb2634 --- /dev/null +++ b/docs/developer/getting-started/sample-data.asciidoc @@ -0,0 +1,31 @@ +[[sample-data]] +=== Installing sample data + +There are a couple ways to easily get data ingested into Elasticsearch. + +[float] +==== Sample data packages available for one click installation + +The easiest is to install one or more of our vailable sample data packages. If you have no data, you should be +prompted to install when running {kib} for the first time. You can also access and install the sample data packages +by going to the home page and clicking "add sample data". + +[float] +==== makelogs script + +The provided `makelogs` script will generate sample data. + +[source,bash] +---- +node scripts/makelogs --auth : +---- + +The default username and password combination are `elastic:changeme` + +Make sure to execute `node scripts/makelogs` *after* elasticsearch is up and running! + +[float] +==== CSV upload + +If running with a platinum or trial license, you can also use the CSV uploader provided inside the Machine learning app. +Navigate to the Data visualizer to upload your data from a file. \ No newline at end of file diff --git a/docs/developer/getting-started/sass.asciidoc b/docs/developer/getting-started/sass.asciidoc new file mode 100644 index 0000000000000..194e001f642e1 --- /dev/null +++ b/docs/developer/getting-started/sass.asciidoc @@ -0,0 +1,36 @@ +[[kibana-sass]] +=== Styling with SASS + +When writing a new component, create a sibling SASS file of the same +name and import directly into the JS/TS component file. Doing so ensures +the styles are never separated or lost on import and allows for better +modularization (smaller individual plugin asset footprint). + +All SASS (.scss) files will automatically build with the +https://elastic.github.io/eui/#/guidelines/sass[EUI] & {kib} invisibles (SASS variables, mixins, functions) from +the {kib-repo}tree/{branch}/src/legacy/ui/public/styles/_globals_v7light.scss[globals_THEME.scss] file. + +*Example:* + +[source,tsx] +---- +// component.tsx + +import './component.scss'; + +export const Component = () => { + return ( +
+ ); +} +---- + +[source,scss] +---- +// component.scss + +.plgComponent { ... } +---- + +Do not use the underscore `_` SASS file naming pattern when importing +directly into a javascript file. \ No newline at end of file diff --git a/docs/developer/index.asciidoc b/docs/developer/index.asciidoc index 50e41a4e18207..db57815a1285a 100644 --- a/docs/developer/index.asciidoc +++ b/docs/developer/index.asciidoc @@ -3,25 +3,27 @@ [partintro] -- -Contributing to Kibana can be daunting at first, but it doesn't have to be. If -you're planning a pull request to the Kibana repository, you may want to start -with <>. +Contributing to {kib} can be daunting at first, but it doesn't have to be. The following sections should get you up and +running in no time. If you have any problems, file an issue in the https://github.com/elastic/kibana/issues[Kibana repo]. -If you'd prefer to use Kibana's internal plugin API, then check out -<>. --- +* <> +* <> +* <> +* <> +* <> +* <> -include::core-development.asciidoc[] +-- -include::plugin-development.asciidoc[] +include::getting-started/index.asciidoc[] -include::visualize/development-visualize-index.asciidoc[] +include::best-practices/index.asciidoc[] -include::add-data-guide.asciidoc[] +include::architecture/index.asciidoc[] -include::security/index.asciidoc[] +include::contributing/index.asciidoc[] -include::pr-review.asciidoc[] +include::plugin/index.asciidoc[] -include::testing/interpreting-ci-failures.asciidoc[] +include::advanced/index.asciidoc[] diff --git a/docs/developer/plugin-development.asciidoc b/docs/developer/plugin-development.asciidoc deleted file mode 100644 index 691fdb0412fd2..0000000000000 --- a/docs/developer/plugin-development.asciidoc +++ /dev/null @@ -1,24 +0,0 @@ -[[plugin-development]] -== Plugin Development - -[IMPORTANT] -============================================== -The Kibana plugin interfaces are in a state of constant development. We cannot provide backwards compatibility for plugins due to the high rate of change. Kibana enforces that the installed plugins match the version of Kibana itself. Plugin developers will have to release a new version of their plugin for each new Kibana release as a result. -============================================== - -* <> -* <> -* <> -* <> -* <> - -include::plugin/development-plugin-resources.asciidoc[] - -include::plugin/development-uiexports.asciidoc[] - -include::plugin/development-plugin-feature-registration.asciidoc[] - -include::plugin/development-plugin-functional-tests.asciidoc[] - -include::plugin/development-plugin-localization.asciidoc[] - diff --git a/docs/developer/plugin/development-uiexports.asciidoc b/docs/developer/plugin/development-uiexports.asciidoc deleted file mode 100644 index 18d326cbfb9c0..0000000000000 --- a/docs/developer/plugin/development-uiexports.asciidoc +++ /dev/null @@ -1,16 +0,0 @@ -[[development-uiexports]] -=== UI Exports - -An aggregate list of available UiExport types: - -[cols="> docs are the best place to +start. However, there are a few differences when developing plugins outside the {kib} repo. These differences are covered here. + +[float] +[[automatic-plugin-generator]] +==== Automatic plugin generator + +We recommend that you kick-start your plugin by generating it with the {kib-repo}tree/{branch}/packages/kbn-plugin-generator[Kibana Plugin Generator]. Run the following in the {kib} repo, and you will be asked a couple questions, see some progress bars, and have a freshly generated plugin ready for you to play with in {kib}'s `plugins` folder. + +["source","shell"] +----------- +node scripts/generate_plugin my_plugin_name # replace "my_plugin_name" with your desired plugin name +----------- + +[float] +=== Plugin location + +The {kib} directory must be named `kibana`, and your plugin directory should be located in the root of `kibana` in a `plugins` directory, for example: + +["source","shell"] +---- +. +└── kibana + └── plugins + ├── foo-plugin + └── bar-plugin +---- + +* <> +* <> + +include::external-plugin-functional-tests.asciidoc[] + +include::external-plugin-localization.asciidoc[] diff --git a/docs/developer/security/index.asciidoc b/docs/developer/security/index.asciidoc deleted file mode 100644 index e7ef0b85930e4..0000000000000 --- a/docs/developer/security/index.asciidoc +++ /dev/null @@ -1,12 +0,0 @@ -[[development-security]] -== Security - -Kibana has generally been able to implement security transparently to core and plugin developers, and this largely remains the case. {kib} on two methods that the <>'s `Cluster` provides: `callWithRequest` and `callWithInternalUser`. - -`callWithRequest` executes requests against Elasticsearch using the authentication credentials of the Kibana end-user. So, if you log into Kibana with the user of `foo` when `callWithRequest` is used, {kib} execute the request against Elasticsearch as the user `foo`. Historically, `callWithRequest` has been used extensively to perform actions that are initiated at the request of Kibana end-users. - -`callWithInternalUser` executes requests against Elasticsearch using the internal Kibana server user, and has historically been used for performing actions that aren't initiated by Kibana end users; for example, creating the initial `.kibana` index or performing health checks against Elasticsearch. - -However, with the changes that role-based access control (RBAC) introduces, this is no longer cut and dry. {kib} now requires all access to the `.kibana` index goes through the `SavedObjectsClient`. This used to be a best practice, as the `SavedObjectsClient` was responsible for translating the documents stored in Elasticsearch to and from Saved Objects, but RBAC is now taking advantage of this abstraction to implement access control and determine when to use `callWithRequest` versus `callWithInternalUser`. - -include::rbac.asciidoc[]