Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce cross-cluster replication API docs #34726

Merged
merged 17 commits into from
Oct 26, 2018

Conversation

jasontedor
Copy link
Member

This commit is our first introduction to cross-cluster replication docs. In this commit, we introduce the cross-cluster replication API docs. We also add skelton docs for additional content that will be added in a series of follow-up commits.

This commit is our first introduction to cross-cluster replication
docs. In this commit, we introduce the cross-cluster replication API
docs. We also add skelton docs for additional content that will be added
in a series of follow-up commits.
@jasontedor jasontedor added >docs General docs changes :Distributed/CCR Issues around the Cross Cluster State Replication features labels Oct 23, 2018
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed

@jasontedor
Copy link
Member Author

jasontedor commented Oct 23, 2018

This PR is the first in a series of PRs that will introduce docs for CCR. We breakdown the introduction of these docs into a series of PRs to ease reviewing. I am aware that there is work outstanding that will render some of these docs broken (that is, we are actively making breaking API changes). I want that work to go in first, and then I will keep this PR up-to-date with those changes. However, I want reviewing to get started, and I want to start opening up the follow-up PRs.

@jasontedor
Copy link
Member Author

jasontedor commented Oct 23, 2018

For reviewers not use to the workflow of working with the docs, here is the simplest way to read these docs:

$ git clone -o elastic git@github.com:elastic/docs.git && \
export GIT_HOME=. && \
source docs/doc_build_aliases.sh; \
git clone -o elastic git@github.com:elastic/elasticsearch.git && \
git -C elasticsearch remote add jasontedor git@github.com:jasontedor/elasticsearch.git && \
git -C elasticsearch fetch jasontedor ccr-docs && \
git -C elasticsearch checkout ccr-docs && \
docbldesx

Then you can open the resulting index HTML page, navigate to X-Pack APIs, and then navigate to cross-cluster replication (e.g., on macOS you could do open html_docs/index.html).

If you already have the elastic/docs and elastic/elasticsearch repositories cloned to the same parent directory, you can set GIT_HOME to that parent directory. Note that you will still need to add my remote.

@@ -149,7 +149,6 @@ buildRestTests.setups['host'] = '''
- do:
nodes.info:
metric: [ http, transport ]
- is_true: nodes.$master.http.publish_address
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is removed because it is now used in the setup section of some of the docs tests, and setup sections do not support is_true sections. While we could add support for is_true sections to setup section, this assertion is adding little value to begin with.

* master: (24 commits)
  ingest: better support for conditionals with simulate?verbose (elastic#34155)
  [Rollup] Job deletion should be invoked on the allocated task (elastic#34574)
  [DOCS] .Security index is never auto created (elastic#34589)
  CCR: Requires soft-deletes on the follower (elastic#34725)
  re-enable bwc tests (elastic#34743)
  Empty GetAliases authorization fix (elastic#34444)
  INGEST: Document Processor Conditional (elastic#33388)
  [CCR] Add total fetch time leader stat (elastic#34577)
  SQL: Support pattern against compatible indices (elastic#34718)
  [CCR] Auto follow pattern APIs adjustments (elastic#34518)
  [Test] Remove dead code from ExceptionSerializationTests (elastic#34713)
  A small typo in migration-assistance doc (elastic#34704)
  ingest: processor stats (elastic#34724)
  SQL: Implement IN(value1, value2, ...) expression. (elastic#34581)
  Tests: Add checks to GeoDistanceQueryBuilderTests (elastic#34273)
  INGEST: Rename Pipeline Processor Param. (elastic#34733)
  Core: Move IndexNameExpressionResolver to java time (elastic#34507)
  [DOCS] Force Merge: clarify execution and storage requirements (elastic#33882)
  TESTING.asciidoc fix examples using forbidden annotation (elastic#34515)
  SQL: Implement `CONVERT`, an alternative to `CAST` (elastic#34660)
  ...
@lcawl
Copy link
Contributor

lcawl commented Oct 23, 2018

@jasontedor I took a quick look and one thing I don't see in the APIs is an "Authorization" section, which lists what roles or privileges are required to run the API when security is enabled. Is this something you plan to add?

I noticed a few other sections missing, so I'll submit suggestions for your perusal.

// CONSOLE
// TEST[s/<follower_index>/follower_index/]

Here, `<follower_index>` is the name of the follower index to pause.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Here, `<follower_index>` is the name of the follower index to pause.
==== Path Parameters
`follower_id` (required)::
(string) Identifier for the follower index.

Here, the auto-follow patterns associated with the
<<modules-remote-clusters,remote cluster>> specified by `<leader_cluster>`
will be deleted.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
==== Path Parameters
`auto_follow_pattern_name` (required)::
(string) Specifies auto-follow patterns that you want to delete.


Here, the named auto-follow pattern collection specified by
`<auto_follow_pattern_name>` will be returned.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
==== Path Parameters
`auto_follow_pattern_name` ::
(string) Specifies the auto-follow patterns that you want to retrieve. If you do not specify any patterns, the API returns information for all patterns.

pattern to use to name the resulting follower indices. The
`<follow_index_pattern>` can use the template `{{leader_index}}` to refer to the
name of the leader index being followed.

Copy link
Contributor

@lcawl lcawl Oct 23, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
==== Path Parameters
`auto_follow_pattern_name` (required)::
(string) Identifier for the auto-follow patterns.

//////////////////////////

Here, `<auto_follow_pattern_name>` is a user-chosen name for the collection of
auto-follow patterns being created, `<leader_cluster>` is the
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are there rules for what this name can contain? For example, for job names, we say "This identifier can contain lowercase alphanumeric characters (a-z and 0-9), hyphens, and underscores. It must start and end with alphanumeric characters."

name of the leader index being followed.

==== Request Body
`leader_cluster`::
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
`leader_cluster`::
`leader_cluster` (required) ::


==== Request Body
`leader_cluster`::
(required string) the <<modules-remote-clusters,remote cluster>> containing the
Copy link
Contributor

@lcawl lcawl Oct 23, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
(required string) the <<modules-remote-clusters,remote cluster>> containing the
(string) The <<modules-remote-clusters,remote cluster>> containing the

leader indices to match against

`leader_index_patterns`::
(array) an array of simple index patterns to match against indices in the
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
(array) an array of simple index patterns to match against indices in the
(array) An array of simple index patterns to match against indices in the

remote cluster specified by the `leader_cluster` field

`follow_index_pattern`::
(string) the name of follower index; the template `{{leader_index}}` can be
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
(string) the name of follower index; the template `{{leader_index}}` can be
(string) The name of follower index; the template `{{leader_index}}` can be

@@ -0,0 +1,30 @@
`max_batch_operation_count`::
(integer) the maximum number of operations to pull per fetch from the remote
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
(integer) the maximum number of operations to pull per fetch from the remote
(integer) The maximum number of operations to pull per fetch from the remote

cluster

`max_concurrent_read_batches`::
(long) the maximum number of concurrent fetches from the remote cluster
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
(long) the maximum number of concurrent fetches from the remote cluster
(long) The maximum number of concurrent fetches from the remote cluster

(long) the maximum number of concurrent fetches from the remote cluster

`max_batch_size`::
(<<byte-units,byte value>>) the maximum size in bytes of batches pulled per
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
(<<byte-units,byte value>>) the maximum size in bytes of batches pulled per
(<<byte-units,byte value>>) The maximum size in bytes of batches pulled per


==== Example

The following is an example of using the pause follower API.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The following is an example of using the pause follower API.
This example pauses a follower index named `follower_index`:


==== Example

The following is an example of using the resume follower API.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The following is an example of using the resume follower API.
This example resumes a follower index named `follower_index`:


==== Example

The following is an example of using the unfollow API.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The following is an example of using the unfollow API.
This example converts `follower_index` from a follower index to a regular index:


==== Example

The following is an example of using the create follower API.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The following is an example of using the create follower API.
This example creates a follower index named `follower_index`:

Copy link
Member

@nik9000 nik9000 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I looked at the snippets and have two comments:

  1. You have a bunch of empty // TEST comments. // CONSOLE implies // TEST so you shouldn't need them but they don't hurt anything. // TEST exists primarily to send extra stuff to the test framework rather than the doc generation framework.
  2. In a few places you clean up after a test using // TEST[continued]. I think the // TEARDOWN annotation that you built is better for that. But, I wonder if these are things that should be cleaned up higher up, like we do with rollups and such. I'm don't think the answer to that question should block merging, but it is a thing to think about.

* master: (74 commits)
  XContent: Check for bad parsers (elastic#34561)
  Docs: Align prose with snippet (elastic#34839)
  document the search context is freed if the scroll is not extended (elastic#34739)
  Test: Lookup node versions on rest test start (elastic#34657)
  SQL: Return error with ORDER BY on non-grouped. (elastic#34855)
  Reduce channels in AbstractSimpleTransportTestCase (elastic#34863)
  [DOCS] Updates Elasticsearch monitoring tasks (elastic#34339)
  Check self references in metric agg after last doc collection (elastic#33593) (elastic#34001)
  [Docs] Add `indices.query.bool.max_clause_count` setting (elastic#34779)
  Add 6.6.0 version to master (elastic#34847)
  Test: ensure char[] doesn't being with prefix (elastic#34816)
  Remove static import from HLRC doc snippet (elastic#34834)
  Logging: server: clean up logging (elastic#34593)
  Logging: tests: clean up logging (elastic#34606)
  SQL: Fix edge case: `<field> IN (null)` (elastic#34802)
  [Test] Mute FullClusterRestartIT.testShrink() until test is fixed
  SQL: Introduce ODBC mode, similar to JDBC (elastic#34825)
  SQL: handle X-Pack or X-Pack SQL not being available in a more graceful way (elastic#34736)
  [Docs] Add explanation for code snippets line width (elastic#34796)
  CCR: Rename follow-task parameters and stats (elastic#34836)
  ...
This commit adds some fields that were missing from put follow, and
fixes a bug in resume follow.
* 'master' of github.com:elastic/elasticsearch:
  Fix line length for org.elasticsearch.common.* files (elastic#34888)
  [ML] Extract common native process base class (elastic#34856)
  Refactor children aggregator into a generic ParentJoinAggregator (elastic#34845)
  [Style] Fix line lengths in action.admin.indices (elastic#34890)
  HLRC - add support for source exists API (elastic#34519)
  [CCR] Retry when no index shard stats can be found (elastic#34852)
  [Docs] audit logfile structured format (elastic#34584)
  [Test] Fix FullClusterRestartIT.testShrink() with copy_settings param (elastic#34853)
  Fix LineLength Check Suppressions: index.fielddata (elastic#34891)
  TEST: Stablize Minio Free Port Search (elastic#34894)
  Delete flaky SettingsBasedHostProviderIT test (elastic#34813)
  [ML] Include message in field_stats for text log files (elastic#34861)
  [TEST] HLRC: Expand failure messages in API checks (elastic#34838)
  Lowercase static final DeprecationLogger instance names (elastic#34887)
* fix-put-resume-follow:
  Fix put/resume follow request parsing
* master:
  Fix put/resume follow request parsing (elastic#34913)
@jasontedor
Copy link
Member Author

Thanks @lcawl, I've incorporated your feedback.

@jasontedor jasontedor merged commit fdfdbe4 into elastic:master Oct 26, 2018
jasontedor added a commit that referenced this pull request Oct 26, 2018
This commit is our first introduction to cross-cluster replication
docs. In this commit, we introduce the cross-cluster replication API
docs. We also add skelton docs for additional content that will be added
in a series of follow-up commits.
jasontedor added a commit that referenced this pull request Oct 26, 2018
This commit is our first introduction to cross-cluster replication
docs. In this commit, we introduce the cross-cluster replication API
docs. We also add skelton docs for additional content that will be added
in a series of follow-up commits.
@jasontedor jasontedor deleted the ccr-docs branch October 26, 2018 15:28
jasontedor added a commit to martijnvg/elasticsearch that referenced this pull request Oct 26, 2018
* master:
  Introduce cross-cluster replication API docs (elastic#34726)
  Responses can use Writeable.Reader interface (elastic#34655)
  SQL: Provide null-safe scripts for Not and Neg (elastic#34877)
  Fix put/resume follow request parsing (elastic#34913)
  Fix line length for org.elasticsearch.common.* files (elastic#34888)
  [ML] Extract common native process base class (elastic#34856)
  Refactor children aggregator into a generic ParentJoinAggregator (elastic#34845)
  [Style] Fix line lengths in action.admin.indices (elastic#34890)
  HLRC - add support for source exists API (elastic#34519)
kcm pushed a commit that referenced this pull request Oct 30, 2018
This commit is our first introduction to cross-cluster replication
docs. In this commit, we introduce the cross-cluster replication API
docs. We also add skelton docs for additional content that will be added
in a series of follow-up commits.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Distributed/CCR Issues around the Cross Cluster State Replication features >docs General docs changes v6.5.0 v6.6.0 v7.0.0-beta1
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants