Merge branch 'master' into single-node-discovery

* master: (137 commits)
  CONSOLEify the "using scripts" documentation
  Adds tests for cardinality and filter aggregations
  Add Backoff policy to azure repository
  Revert "Adds tests for cardinality and filter aggregations (elastic#23826)"
  Adds tests for cardinality and filter aggregations (elastic#23826)
  mute testDifferentRolesMaintainPathOnRestart
  Replace custom sort field with SortedSetSortField and SortedNumericSortField when possible (elastic#23827)
  Prevent nodes from joining if newer indices exist in the cluster (elastic#23843)
  Synchronized CollapseTopFieldDocs with lucenes relatives (elastic#23854)
  CONSOLEify analysis docs
  Adapted search_shards rest test to work with Perl
  To examine an exception in rest tests, the exception should be caught, not ignored
  Fixed bad YAML in rest tests
  Fix BootstrapForTesting blowup
  Ban Boolean#getBoolean
  Fix language in some docs
  CONSOLEify lang-analyzer docs
  Stricter parsing of remote node attribute
  Fix cross-cluster remote node gateway attributes
  FieldCapabilitiesRequest should implements Replaceable since it accepts index patterns
  ...

.dir-locals.el

@@ -83,6 +83,6 @@
       ))
    (c-basic-offset . 4)
    (c-comment-only-line-offset . (0 . 0))
-   (fill-column . 140)
-   (fci-rule-column . 140)
+   (fill-column . 100)
+   (fci-rule-column . 100)
    (compile-command . "gradle compileTestJava"))))

.github/PULL_REQUEST_TEMPLATE.md

@@ -11,3 +11,4 @@ attention.
 - If submitting code, have you built your formula locally prior to submission with `gradle check`?
 - If submitting code, is your pull request against master? Unless there is a good reason otherwise, we prefer pull requests against master and will backport as needed.
 - If submitting code, have you checked that your submission is for an [OS that we support](https://www.elastic.co/support/matrix#show_os)?
+- If you are submitting this code for a class then read our [policy](https://github.com/elastic/elasticsearch/blob/master/CONTRIBUTING.md#contributing-as-part-of-a-class) for that.

CONTRIBUTING.md

@@ -88,8 +88,8 @@ Contributing to the Elasticsearch codebase
 **Repository:** [https://github.com/elastic/elasticsearch](https://github.com/elastic/elasticsearch)
 
 Make sure you have [Gradle](http://gradle.org) installed, as
-Elasticsearch uses it as its build system. Gradle must be version 2.13 _exactly_ in
-order to build successfully.
+Elasticsearch uses it as its build system. Gradle must be at least
+version 3.3 in order to build successfully.
 
 Eclipse users can automatically configure their IDE: `gradle eclipse`
 then `File: Import: Existing Projects into Workspace`. Select the
@@ -117,7 +117,7 @@ For Eclipse, go to `Preferences->Java->Installed JREs` and add `-ea` to
 Please follow these formatting guidelines:
 
 * Java indent is 4 spaces
-* Line width is 140 characters
+* Line width is 100 characters
 * The rest is left to Java coding standards
 * Disable “auto-format on save” to prevent unnecessary format changes. This makes reviews much harder as it generates unnecessary formatting changes. If your IDE supports formatting only modified chunks that is fine to do.
 * Wildcard imports (`import foo.bar.baz.*`) are forbidden and will cause the build to fail. Please attempt to tame your IDE so it doesn't make them and please send a PR against this document with instructions for your IDE if it doesn't contain them.
@@ -139,3 +139,32 @@ Before submitting your changes, run the test suite to make sure that nothing is
 ```sh
 gradle check
 ```
+
+Contributing as part of a class
+-------------------------------
+In general Elasticsearch is happy to accept contributions that were created as
+part of a class but strongly advise against making the contribution as part of
+the class. So if you have code you wrote for a class feel free to submit it.
+
+Please, please, please do not assign contributing to Elasticsearch as part of a
+class. If you really want to assign writing code for Elasticsearch as an
+assignment then the code contributions should be made to your private clone and
+opening PRs against the primary Elasticsearch clone must be optional, fully
+voluntary, not for a grade, and without any deadlines.
+
+Because:
+
+* While the code review process is likely very educational, it can take wildly
+varying amounts of time depending on who is available, where the change is, and
+how deep the change is. There is no way to predict how long it will take unless
+we rush.
+* We do not rush reviews without a very, very good reason. Class deadlines
+aren't a good enough reason for us to rush reviews.
+* We deeply discourage opening a PR you don't intend to work through the entire
+code review process because it wastes our time.
+* We don't have the capacity to absorb an entire class full of new contributors,
+especially when they are unlikely to become long time contributors.
+
+Finally, we require that you run `gradle check` before submitting a
+non-documentation contribution. This is mentioned above, but it is worth
+repeating in this section because it has come up in this context.

README.textile

@@ -200,7 +200,7 @@ We have just covered a very small portion of what Elasticsearch is all about. Fo
 
 h3. Building from Source
 
-Elasticsearch uses "Gradle":https://gradle.org for its build system. You'll need to have version 2.13 of Gradle installed.
+Elasticsearch uses "Gradle":https://gradle.org for its build system. You'll need to have at least version 3.3 of Gradle installed.
 
 In order to create a distribution, simply run the @gradle assemble@ command in the cloned directory.
 

build.gradle

@@ -18,15 +18,17 @@
  */
 
 import java.nio.file.Path
+import java.util.regex.Matcher
 import org.eclipse.jgit.lib.Repository
 import org.eclipse.jgit.lib.RepositoryBuilder
 import org.gradle.plugins.ide.eclipse.model.SourceFolder
 import org.apache.tools.ant.taskdefs.condition.Os
+import org.elasticsearch.gradle.VersionProperties
 
 // common maven publishing configuration
 subprojects {
   group = 'org.elasticsearch'
-  version = org.elasticsearch.gradle.VersionProperties.elasticsearch
+  version = VersionProperties.elasticsearch
   description = "Elasticsearch subproject ${project.path}"
 }
 
@@ -59,12 +61,26 @@ configure(subprojects.findAll { it.projectDir.toPath().startsWith(rootPath) }) {
   }
 }
 
+int prevMajor = Integer.parseInt(VersionProperties.elasticsearch.split('\\.')[0]) - 1
+String prevSnapshot = VersionProperties.elasticsearch.contains('alpha') ? '-SNAPSHOT' : ''
+File versionFile = file('core/src/main/java/org/elasticsearch/Version.java')
+List<String> versionLines = versionFile.readLines('UTF-8')
+int prevMinor = 0
+for (String line : versionLines) {
+  Matcher match = line =~ /\W+public static final Version V_${prevMajor}_(\d+)_.*/
+  if (match.matches()) {
+    prevMinor = Math.max(Integer.parseInt(match.group(1)), prevMinor)
+  }
+}
+
+// injecting groovy property variables into all projects
 allprojects {
-  // injecting groovy property variables into all projects
   project.ext {
     // for ide hacks...
     isEclipse = System.getProperty("eclipse.launcher") != null || gradle.startParameter.taskNames.contains('eclipse') || gradle.startParameter.taskNames.contains('cleanEclipse')
     isIdea = System.getProperty("idea.active") != null || gradle.startParameter.taskNames.contains('idea') || gradle.startParameter.taskNames.contains('cleanIdea')
+    // for backcompat testing
+    bwcVersion = "${prevMajor}.${prevMinor}.0${prevSnapshot}"
   }
 }
 
@@ -112,6 +128,7 @@ subprojects {
     "org.elasticsearch.client:transport:${version}": ':client:transport',
     "org.elasticsearch.test:framework:${version}": ':test:framework',
     "org.elasticsearch.distribution.integ-test-zip:elasticsearch:${version}": ':distribution:integ-test-zip',
+    "org.elasticsearch.distribution.zip:elasticsearch:${bwcVersion}": ':distribution:bwc-zip',
     "org.elasticsearch.distribution.zip:elasticsearch:${version}": ':distribution:zip',
     "org.elasticsearch.distribution.tar:elasticsearch:${version}": ':distribution:tar',
     "org.elasticsearch.distribution.rpm:elasticsearch:${version}": ':distribution:rpm',
@@ -123,10 +140,12 @@ subprojects {
     "org.elasticsearch.plugin:lang-mustache-client:${version}": ':modules:lang-mustache',
     "org.elasticsearch.plugin:percolator-client:${version}": ':modules:percolator',
   ]
-  configurations.all {
-    resolutionStrategy.dependencySubstitution { DependencySubstitutions subs ->
-      projectSubstitutions.each { k,v ->
-        subs.substitute(subs.module(k)).with(subs.project(v))
+  project.afterEvaluate {
+    configurations.all {
+      resolutionStrategy.dependencySubstitution { DependencySubstitutions subs ->
+        projectSubstitutions.each { k,v ->
+          subs.substitute(subs.module(k)).with(subs.project(v))
+        }
       }
     }
   }

buildSrc/build.gradle

@@ -23,8 +23,8 @@ apply plugin: 'groovy'
 
 group = 'org.elasticsearch.gradle'
 
-if (GradleVersion.current() < GradleVersion.version('2.13')) {
-  throw new GradleException('Gradle 2.13+ is required to build elasticsearch')
+if (GradleVersion.current() < GradleVersion.version('3.3')) {
+  throw new GradleException('Gradle 3.3+ is required to build elasticsearch')
 }
 
 if (JavaVersion.current() < JavaVersion.VERSION_1_8) {
@@ -96,23 +96,12 @@ dependencies {
   compile 'org.apache.rat:apache-rat:0.11'
 }
 
-// Gradle version-specific options (allows build to run with Gradle 2.13 as well as 2.14+/3.+)
-if (GradleVersion.current() == GradleVersion.version("2.13")) {
-  // ProgressLogger(-Factory) classes are part of the public Gradle API
-  sourceSets.main.groovy.srcDir 'src/main/gradle-2.13-groovy'
+// Gradle 2.14+ removed ProgressLogger(-Factory) classes from the public APIs
+// Use logging dependency instead
 
-  dependencies {
-    compile 'ru.vyarus:gradle-animalsniffer-plugin:1.0.1' // last version compatible with Gradle 2.13
-  }
-} else {
-  // Gradle 2.14+ removed ProgressLogger(-Factory) classes from the public APIs
-  // Use logging dependency instead
-  sourceSets.main.groovy.srcDir 'src/main/gradle-2.14-groovy'
-
-  dependencies {
-    compileOnly "org.gradle:gradle-logging:${GradleVersion.current().getVersion()}"
-    compile 'ru.vyarus:gradle-animalsniffer-plugin:1.2.0' // Gradle 2.14 requires a version > 1.0.1
-  }
+dependencies {
+  compileOnly "org.gradle:gradle-logging:${GradleVersion.current().getVersion()}"
+  compile 'ru.vyarus:gradle-animalsniffer-plugin:1.2.0' // Gradle 2.14 requires a version > 1.0.1
 }
 
 /*****************************************************************************

buildSrc/src/main/groovy/com/carrotsearch/gradle/junit4/RandomizedTestingTask.groovy

@@ -8,7 +8,6 @@ import org.apache.tools.ant.BuildException
 import org.apache.tools.ant.DefaultLogger
 import org.apache.tools.ant.RuntimeConfigurable
 import org.apache.tools.ant.UnknownElement
-import org.elasticsearch.gradle.ProgressLoggerFactoryInjection
 import org.gradle.api.DefaultTask
 import org.gradle.api.file.FileCollection
 import org.gradle.api.file.FileTreeElement
@@ -20,9 +19,12 @@ import org.gradle.api.tasks.Optional
 import org.gradle.api.tasks.TaskAction
 import org.gradle.api.tasks.util.PatternFilterable
 import org.gradle.api.tasks.util.PatternSet
+import org.gradle.internal.logging.progress.ProgressLoggerFactory
 import org.gradle.util.ConfigureUtil
 
-class RandomizedTestingTask extends DefaultTask implements ProgressLoggerFactoryInjection {
+import javax.inject.Inject
+
+class RandomizedTestingTask extends DefaultTask {
 
     // TODO: change to "executable" to match gradle test params?
     @Optional
@@ -92,6 +94,11 @@ class RandomizedTestingTask extends DefaultTask implements ProgressLoggerFactory
         listenersConfig.listeners.add(new TestReportLogger(logger: logger, config: testLoggingConfig))
     }
 
+    @Inject
+    ProgressLoggerFactory getProgressLoggerFactory() {
+        throw new UnsupportedOperationException()
+    }
+
     void jvmArgs(Iterable<String> arguments) {
         jvmArgs.addAll(arguments)
     }

buildSrc/src/main/groovy/com/carrotsearch/gradle/junit4/TestProgressLogger.groovy

@@ -25,7 +25,8 @@ import com.carrotsearch.ant.tasks.junit4.events.aggregated.AggregatedStartEvent
 import com.carrotsearch.ant.tasks.junit4.events.aggregated.AggregatedSuiteResultEvent
 import com.carrotsearch.ant.tasks.junit4.events.aggregated.AggregatedTestResultEvent
 import com.carrotsearch.ant.tasks.junit4.listeners.AggregatedEventListener
-import org.elasticsearch.gradle.ProgressLogger
+import org.gradle.internal.logging.progress.ProgressLogger
+import org.gradle.internal.logging.progress.ProgressLoggerFactory
 
 import static com.carrotsearch.ant.tasks.junit4.FormattingUtils.formatDurationInSeconds
 import static com.carrotsearch.ant.tasks.junit4.events.aggregated.TestStatus.ERROR
@@ -51,6 +52,8 @@ import static java.lang.Math.max
  * quick.
  */
 class TestProgressLogger implements AggregatedEventListener {
+    /** Factory to build a progress logger when testing starts */
+    ProgressLoggerFactory factory
     ProgressLogger progressLogger
     int totalSuites
     int totalSlaves
@@ -74,17 +77,14 @@ class TestProgressLogger implements AggregatedEventListener {
     /** Have we finished a whole suite yet? */
     volatile boolean suiteFinished = false
     /* Note that we probably overuse volatile here but it isn't hurting us and
-      lets us move things around without worying about breaking things. */
-
-    TestProgressLogger(Map args) {
-        progressLogger = new ProgressLogger(args.factory.newOperation(TestProgressLogger))
-        progressLogger.setDescription('Randomized test runner')
-    }
+       lets us move things around without worrying about breaking things. */
 
     @Subscribe
     void onStart(AggregatedStartEvent e) throws IOException {
         totalSuites = e.suiteCount
         totalSlaves = e.slaveCount
+        progressLogger = factory.newOperation(TestProgressLogger)
+        progressLogger.setDescription('Randomized test runner')
         progressLogger.started()
         progressLogger.progress(
             "Starting JUnit4 for ${totalSuites} suites on ${totalSlaves} jvms")