diff --git a/README.md b/README.md index 4878a47e..079ec6f2 100644 --- a/README.md +++ b/README.md @@ -23,8 +23,10 @@ please refer to the [**Azure Marketplace and ARM template documentation**](https This repository consists of: -* [src/mainTemplate.json](src/mainTemplate.json) - The main Azure Resource Management (ARM) template. The template itself is composed of many nested linked templates with the main template acting as the entry point. -* [src/createUiDefinition](src/createUiDefinition.json) - UI definition file for our Azure Marketplace offering. This file produces an output JSON that the ARM template can accept as input parameters. +* [src/mainTemplate.json](src/mainTemplate.json) - The main Azure Resource Management (ARM) template. +The template itself is composed of many nested linked templates, with the main template acting as the entry point. +* [src/createUiDefinition](src/createUiDefinition.json) - UI definition file for our Azure Marketplace offering. +This file produces an output JSON that the ARM template can accept as input parameters. ## Building @@ -47,7 +49,7 @@ For more details around developing the template, take a look at the [Development The [Azure Marketplace Elastic Stack offering](https://azuremarketplace.microsoft.com/en-us/marketplace/apps/elastic.elasticsearch) offers a simplified UI and installation experience over the full power of the ARM template. -It will always bootstrap an Elasticsearch cluster complete with a trial license of the [Elastic Stack's commercial features](https://www.elastic.co/products/stack). +It will always bootstrap an Elasticsearch cluster complete with a trial license of the [Elastic Stack's platinum features](https://www.elastic.co/products/stack). Deploying through the Marketplace is great and easy way to get your feet wet for the first time with Elasticsearch on Azure, but in the long run, you'll want to deploy the templates directly from GitHub using the Azure CLI or PowerShell SDKs. Check out the CLI examples. @@ -74,7 +76,6 @@ not exposed within the Marketplace UI, such as configuring * Azure Storage account to use with Azure Repository plugin for Snapshot/Restore * Application Gateway to use for SSL/TLS and SSL offload -* The number and size of disks to attach to each data node VM Check out our [**examples repository**](https://github.com/elastic/azure-marketplace-examples) for examples of common scenarios and also take a look at the following blog @@ -142,12 +143,12 @@ value defined in the template. load balancer.

If you are setting up Elasticsearch or Kibana on a publicly available IP address, it is highly recommended to secure access to the cluster with a product like - X-Pack Security, in addition to configuring SSL/TLS.

+ Elastic Stack Security, in addition to configuring SSL/TLS.

internal xpackPluginsstring - Either Yes or No to install a trial license of the commercial X-Pack - features such as Monitoring, Security, Alerting, Graph, Machine Learning (5.5.0+) and SQL. If also installing Kibana, it will have Reporting and Profiler installed. + Either Yes or No to install a trial license of the Elastic Stack features (formerly X-Pack) + such as Monitoring, Security, Alerting, Graph, Machine Learning (5.5.0+) and SQL. If also installing Kibana, it will have Reporting and Profiler installed.

A value of No for Elasticsearch and Kibana prior to 6.3.0, will include only the Open Source features. @@ -186,33 +187,33 @@ value defined in the template. 0 esHttpCertBlobstring - A Base-64 encoded form of the PKCS#12 archive (.p12/.pfx) containing the certificate and key to secure communication for HTTP layer to Elasticsearch. X-Pack plugin must be installed + A Base-64 encoded form of the PKCS#12 archive (.p12/.pfx) containing the certificate and key to secure communication for HTTP layer to Elasticsearch. xpackPlugins must be Yes, or esVersion must be 6.8.0 or above (and less than 7.0.0) or 7.1.0 and above. "" esHttpCertPasswordsecurestring The password for the PKCS#12 archive (.p12/.pfx) containing the certificate and key to secure communication for HTTP layer to Elasticsearch. Optional as the archive may not be protected with a password.

If using esHttpCaCertBlob, this password will be used to protect the generated PKCS#12 archive on each node. - X-Pack plugin must be installed + xpackPlugins must be Yes, or esVersion must be 6.8.0 or above (and less than 7.0.0) or 7.1.0 and above. "" esHttpCaCertBlobstring - A Base-64 encoded form of a PKCS#12 archive (.p12/.pfx) containing the Certificate Authority (CA) certificate and key to use to generate certificates on each Elasticsearch node, to secure communication for the HTTP layer to Elasticsearch. X-Pack plugin must be installed + A Base-64 encoded form of a PKCS#12 archive (.p12/.pfx) containing the Certificate Authority (CA) certificate and key to use to generate certificates on each Elasticsearch node, to secure communication for the HTTP layer to Elasticsearch. xpackPlugins must be Yes, or esVersion must be 6.8.0 or above (and less than 7.0.0) or 7.1.0 and above. "" esHttpCaCertPasswordsecurestring - The password for the PKCS#12 archive (.p12/.pfx) containing the Certificate Authority (CA) certificate and key to secure communication for HTTP layer to Elasticsearch. Optional as the archive may not be be protected with a password. X-Pack plugin must be installed + The password for the PKCS#12 archive (.p12/.pfx) containing the Certificate Authority (CA) certificate and key to secure communication for HTTP layer to Elasticsearch. Optional as the archive may not be be protected with a password. xpackPlugins must be Yes, or esVersion must be 6.8.0 or above (and less than 7.0.0) or 7.1.0 and above. "" esTransportCaCertBlobstring - A Base-64 encoded form of a PKCS#12 archive (.p12/.pfx) containing the Certificate Authority (CA) certificate and key to use to generate certificates on each Elasticsearch node, to secure communication for Transport layer to Elasticsearch. X-Pack plugin must be installed + A Base-64 encoded form of a PKCS#12 archive (.p12/.pfx) containing the Certificate Authority (CA) certificate and key to use to generate certificates on each Elasticsearch node, to secure communication for Transport layer to Elasticsearch. xpackPlugins must be Yes, or esVersion must be 6.8.0 or above (and less than 7.0.0) or 7.1.0 and above. "" esTransportCaCertPasswordsecurestring - The password for the PKCS#12 archive (.p12/.pfx) containing the Certificate Authority (CA) certificate and key to secure communication for Transport layer to Elasticsearch. Optional as the archive may not be be protected with a password. X-Pack plugin must be installed + The password for the PKCS#12 archive (.p12/.pfx) containing the Certificate Authority (CA) certificate and key to secure communication for Transport layer to Elasticsearch. Optional as the archive may not be be protected with a password. xpackPlugins must be Yes, or esVersion must be 6.8.0 or above (and less than 7.0.0) or 7.1.0 and above. "" esTransportCertPasswordsecurestring - The password to protect the generated PKCS#12 archive on each node. X-Pack plugin must be installed + The password to protect the generated PKCS#12 archive on each node. xpackPlugins must be Yes, or esVersion must be 6.8.0 or above (and less than 7.0.0) or 7.1.0 and above. "" samlMetadataUristring @@ -392,8 +393,7 @@ value defined in the template. Kibana related settings kibanastring - Either Yes or No to provision a machine with Kibana installed and a public IP address to access it. If you have opted to also install the X-Pack plugins using xpackPlugins, - a trial license of the commercial Kibana features will be applied and activated. + Either Yes or No to provision a machine with Kibana installed and a public IP address to access it. Yes vmSizeKibanastring @@ -426,8 +426,7 @@ value defined in the template. Logstash related settings logstashstring - Either Yes or No to provision a machine with Logstash installed. If you have opted to also install the X-Pack plugins using xpackPlugins, - a trial license for the commercial Logstash features will be applied and activated. + Either Yes or No to provision a machine with Logstash installed. No vmSizeLogstashstring @@ -471,7 +470,7 @@ value defined in the template. jumpboxstring Either Yes or No to optionally add a virtual machine with a public IP to the deployment, which you can use to connect and manage virtual machines on the internal network. NOTE: If you are deploying Kibana, the Kibana VM can act - as a jumpbox. + as a jumpbox, so a separate jumpbox VM is not needed. No Virtual network related settings @@ -629,7 +628,7 @@ where `` refers to the resource group you just created. ```powershell $clusterParameters = @{ "artifactsBaseUrl"="https://raw.githubusercontent.com/elastic/azure-marketplace/master/src" - "esVersion" = "6.7.0" + "esVersion" = "7.1.1" "esClusterName" = "elasticsearch" "loadBalancerType" = "internal" "vmDataDiskCount" = 1 @@ -664,7 +663,7 @@ the artifactsBaseUrl parameter of the template to point to a specific tagged rel **Targeting a specific template version is recommended for repeatable production deployments.** -For example, to target the [`7.0.0` tag release with PowerShell](https://github.com/elastic/azure-marketplace/tree/6.6.1) +For example, to target the [`7.0.0` tag release with PowerShell](https://github.com/elastic/azure-marketplace/tree/7.0.0) ```powershell $templateVersion = "7.0.0" @@ -701,9 +700,11 @@ Role Based Access control, and Transport Layer Security (TLS) can be configured for both Elasticsearch and Kibana. For more details, please refer to [the Security documentation](https://www.elastic.co/guide/en/elastic-stack-deploy/current/azure-arm-template-security.html). -The Elastic Stack security features require a license level higher than basic. -They can be configured with a trial license, which provides access to the -security features for 30 days. +For Elasticsearch versions 6.8.0+ (and less than 7.0.0), and 7.1.0+, the Elastic Stack security features +that allow configuring TLS and role based access control are available in the free basic license level. +For all other versions, the Elastic Stack security +features require a license level higher than basic; They can be configured with a trial license, +which provides access to the security features for 30 days. ### TLS for Kibana @@ -715,7 +716,7 @@ a certificate and private key in PEM format with `kibanaCertBlob` and You can secure communication between nodes in the cluster with TLS on the Transport layer. Configuring TLS for the Transport layer requires -`xPackPlugins` be set to `Yes`. +`xPackPlugins` be set to `Yes`, or an Elasticsearch version 6.8.0+ (and less than 7.0.0) or 7.1.0+. You must supply a PKCS#12 archive with the `esTransportCaCertBlob` parameter (and optional passphrase with `esTransportCaCertPassword`) containing the CA cert which should be used to generate @@ -737,7 +738,7 @@ and follow the instructions. You can secure external access to the cluster with TLS with an external loadbalancer or Application Gateway. Configuring TLS for the HTTP layer requires -`xPackPlugins` be set to `Yes`. +`xPackPlugins` be set to `Yes`, or an Elasticsearch version 6.8.0+ (and less than 7.0.0) or 7.1.0+. #### External load balancer diff --git a/build/allowedValues.json b/build/allowedValues.json index 098009c2..d8bd35a6 100644 --- a/build/allowedValues.json +++ b/build/allowedValues.json @@ -1,15 +1,15 @@ { "versions": [ "5.6.15", - "6.0.1", - "6.1.3", "6.2.4", "6.3.2", "6.4.3", "6.5.4", "6.6.2", - "6.7.1", - "7.0.0" + "6.7.2", + "6.8.0", + "7.0.1", + "7.1.1" ], "numberOfDataNodes" : 50, "numberOfClientNodes" : 20, diff --git a/build/arm-tests/1d-0m-0c-basic-ext-tls-klp.json b/build/arm-tests/1d-0m-0c-basic-ext-tls-klp.json new file mode 100644 index 00000000..cae951fe --- /dev/null +++ b/build/arm-tests/1d-0m-0c-basic-ext-tls-klp.json @@ -0,0 +1,34 @@ +{ + "description": "1 data node cluster with basic license and TLS", + "condition" : { + "range": ">=7.1.0 || >=6.8.0 <7.0.0", + "reason": "the use of TLS with a basic license requires 6.8.0+ (and less than 7.0.0), or 7.1.0+" + }, + "isValid" : true, + "deploy" : true, + "why" : "", + "location" : "westeurope", + "parameters" : { + "loadBalancerType":{"value":"external"}, + "esAdditionalYaml":{"value":""}, + "xpackPlugins":{"value":"No"}, + "esHttpCaCertBlob":{"value":"certs/ca-cert-with-password.pfx"}, + "esHttpCaCertPassword":{"value":"Password123"}, + "esTransportCaCertBlob":{"value":"certs/ca-cert-with-password.pfx"}, + "esTransportCaCertPassword":{"value":"Password123"}, + "esTransportCertPassword":{"value":"Password1234"}, + "kibana":{"value":"Yes"}, + "vmSizeKibana":{"value":"Standard_DS1_v2"}, + "logstash":{"value":"Yes"}, + "vmSizeLogstash":{"value":"Standard_DS1_v2"}, + "logstashAdditionalPlugins":{"value":"logstash-input-heartbeat"}, + "logstashConf":{"value":"conf/logstash-tls.conf"}, + "vmSizeDataNodes":{"value":"Standard_DS1_v2"}, + "vmDataNodeCount":{"value":1}, + "vmDataDiskCount":{"value":1}, + "vmDataDiskSize":{"value":"32GiB"}, + "storageAccountType":{"value":"Default"}, + "dataNodesAreMasterEligible":{"value":"Yes"}, + "authenticationType":{"value":"password"} + } +} diff --git a/build/tasks/arm-validator.js b/build/tasks/arm-validator.js index 3c5dbd47..42b4aa2a 100644 --- a/build/tasks/arm-validator.js +++ b/build/tasks/arm-validator.js @@ -129,8 +129,12 @@ var bootstrap = (cb) => { var login = (cb) => { var version = [ '--version' ]; az(version, (error, stdout, stderr) => { - if (error || stderr) return bailOut(error || new Error(stderr)); - log(`Using ${stdout.split('\n')[0]}` ); + // ignore stderr if it's simply a warning about an older version of Azure CLI + if (error || (stderr && !/^WARNING: You have \d+ updates available/.test(stderr))) { + return bailOut(error || new Error(stderr)); + } + + log(`Using ${stdout.split('\n')[0].replace('*', '').replace(/\s\s+/g, ' ')}` ); var login = [ 'login', '--service-principal', diff --git a/docs/azure-arm-template.asciidoc b/docs/azure-arm-template.asciidoc index 069f5a79..7ef4e8bd 100644 --- a/docs/azure-arm-template.asciidoc +++ b/docs/azure-arm-template.asciidoc @@ -1,8 +1,8 @@ :marketplace: https://azuremarketplace.microsoft.com/en-au/marketplace/apps/elastic.elasticsearch :portal: https://portal.azure.com :github: https://github.com/elastic/azure-marketplace -:current: 7.0 -:version: 7.0.0 +:current: 7.1 +:version: 7.1.1 :register: https://register.elastic.co :elasticguide: https://www.elastic.co/guide/en/elasticsearch :elasticdocs: {elasticguide}/reference/{current} @@ -12,6 +12,7 @@ :bootstrappassword: : {stackdocs}/built-in-users.html#bootstrap-elastic-passwords :licenseexpiration: {stackdocs}/license-expiration.html :microsoftdocs: https://docs.microsoft.com +:azuredocs: https://azure.microsoft.com :azurecli: {microsoftdocs}/cli/azure/?view=azure-cli-latest :azurepowershell: {microsoftdocs}/powershell/azure/overview :subscriptions: https://www.elastic.co/subscriptions @@ -19,7 +20,7 @@ :resourcegroup: {microsoftdocs}/azure/azure-resource-manager/resource-group-portal :incrementalmode: {microsoftdocs}/azure/azure-resource-manager/resource-group-template-deploy#incremental-and-complete-deployments :vms: {microsoftdocs}/azure/virtual-machines/linux/sizes -:azurelocations: https://azure.microsoft.com/en-au/global-infrastructure/locations/ +:azurelocations: {azuredocs}/en-au/global-infrastructure/locations/ :availabilitysets: {microsoftdocs}/azure/virtual-machines/linux/manage-availability :availabilityzones: {microsoftdocs}/azure/availability-zones/az-overview :manageddisks: {microsoftdocs}/azure/virtual-machines/linux/managed-disks-overview @@ -33,9 +34,7 @@ :yamllint: http://www.yamllint.com/ :openssl: https://www.openssl.org/docs/man1.0.2/apps/openssl.html :base64: https://linux.die.net/man/1/base64 -:acceleratednetworking: https://azure.microsoft.com/en-us/blog/maximize-your-vm-s-performance-with-accelerated-networking-now-generally-available-for-both-windows-and-linux/ - - +:acceleratednetworking: {azuredocs}/en-us/blog/maximize-your-vm-s-performance-with-accelerated-networking-now-generally-available-for-both-windows-and-linux/ [[azure-arm-template]] == Azure Resource Manager (ARM) template @@ -1018,6 +1017,8 @@ cluster. include::trial-license-warning.asciidoc[] +include::basic-security.asciidoc[] + The following parameters are used to configure initial user accounts `securityBootstrapPassword`:: @@ -1065,7 +1066,7 @@ Valid only for Elasticsearch 6.5.0+ It is recommended after deployment to use the `elastic` superuser account to create the individual user accounts that will be needed for the users and applications -that will interact with Elasticsearch and Kibana, then use those accounts going +that will interact with Elasticsearch and Kibana, then to use these accounts going forward. [[saml-single-sign-on]] @@ -1518,11 +1519,11 @@ can be downloaded from a public URI at deployment time. === Elastic Stack features (formerly X-Pack) A trial license for the {subscriptions}[platinum features] of the Elastic Stack can be deployed for -Elasticsearch, and Kibana if also deployed, using the following parameter +Elasticsearch, and Kibana and Logstash if also deployed, using the following parameter `xPackPlugins`:: -Select `Yes` to install a trial license of the platinum features. A trial license -provides access to platinum features for 30 days. a value of `No` +Select `Yes` to install a trial license of the Elastic Stack platinum features. A trial license +provides access to {subscriptions}[platinum features] for 30 days. a value of `No` + . for Elasticsearch and Kibana 6.3.0+ will deploy with a free perpetual basic license providing access to the {subscriptions}[basic features]. These versions use a distribution with @@ -1622,4 +1623,4 @@ semicolon. For example, Any additional plugins installed are added to the {elasticguide}/plugins/{current}/_plugins_directory.html[`plugin.mandatory` setting in elasticsearch.yml configuration], to ensure that a node will start only when -it has all mandatory plugins. \ No newline at end of file +it has all mandatory plugins. diff --git a/docs/basic-security.asciidoc b/docs/basic-security.asciidoc new file mode 100644 index 00000000..2359583e --- /dev/null +++ b/docs/basic-security.asciidoc @@ -0,0 +1,14 @@ +:security-blog-post: https://www.elastic.co/blog/security-for-elasticsearch-is-now-free + +[IMPORTANT] +-- +Starting with Elastic Stack 6.8.0 and onwards in the 6.x major version lineage, and +7.1.0 onwards, Security features including Basic Authentication and Transport Layer +Security are enabled for the free basic license level. + +What this means is that if you deploy one of these versions of the Elastic Stack, +Basic Authentication will be enabled and configured, and if certificates are supplied +for Elasticsearch HTTP and Transport layers, and Kibana, <> will be configured. + +Please read the {security-blog-post}[Security for Elasticsearch is now free] blog post for more details. +-- \ No newline at end of file diff --git a/docs/trial-license-warning.asciidoc b/docs/trial-license-warning.asciidoc index 42ac6a62..20d2d0d3 100644 --- a/docs/trial-license-warning.asciidoc +++ b/docs/trial-license-warning.asciidoc @@ -1,4 +1,4 @@ -:current: 7.0 +:current: 7.1 :register: https://register.elastic.co :elasticdocs: https://www.elastic.co/guide/en/elasticsearch/reference/{current} :licenseexpiration: {stackdocs}/license-expiration.html diff --git a/src/createUiDefinition.json b/src/createUiDefinition.json index 6f05f4ef..94280cee 100644 --- a/src/createUiDefinition.json +++ b/src/createUiDefinition.json @@ -54,7 +54,7 @@ "name": "esVersion", "type": "Microsoft.Common.DropDown", "label": "Elasticsearch version", - "defaultValue": "v7.0.0", + "defaultValue": "v7.1.1", "toolTip": "Choose a version of Elasticsearch.", "constraints": { "allowedValues": [ @@ -62,14 +62,6 @@ "label": "v5.6.15", "value": "5.6.15" }, - { - "label": "v6.0.1", - "value": "6.0.1" - }, - { - "label": "v6.1.3", - "value": "6.1.3" - }, { "label": "v6.2.4", "value": "6.2.4" @@ -91,12 +83,20 @@ "value": "6.6.2" }, { - "label": "v6.7.1", - "value": "6.7.1" + "label": "v6.7.2", + "value": "6.7.2" + }, + { + "label": "v6.8.0", + "value": "6.8.0" + }, + { + "label": "v7.0.1", + "value": "7.0.1" }, { - "label": "v7.0.0", - "value": "7.0.0" + "label": "v7.1.1", + "value": "7.1.1" } ] } diff --git a/src/mainTemplate.json b/src/mainTemplate.json index 00a9c411..e14af064 100644 --- a/src/mainTemplate.json +++ b/src/mainTemplate.json @@ -21,21 +21,21 @@ }, "esVersion": { "type": "string", - "defaultValue": "7.0.0", + "defaultValue": "7.1.1", "allowedValues": [ "5.6.15", - "6.0.1", - "6.1.3", "6.2.4", "6.3.2", "6.4.3", "6.5.4", "6.6.2", - "6.7.1", - "7.0.0" + "6.7.2", + "6.8.0", + "7.0.1", + "7.1.1" ], "metadata": { - "description": "Elasticsearch version to install" + "description": "Elastic Stack version to install" } }, "esClusterName": { @@ -54,7 +54,7 @@ "gateway" ], "metadata": { - "description": "Set up an internal or external load balancer, or use Application Gateway (gateway) for load balancing and SSL offload. If you are setting up Elasticsearch on a publicly available endpoint, it is *strongly recommended* to secure your nodes with a product like Elastic's X-Pack Security" + "description": "Set up an internal or external load balancer, or use Application Gateway (gateway) for load balancing and SSL offload. If you are setting up Elasticsearch on a publicly available endpoint, it is *strongly recommended* to secure your nodes with a product like the Elastic Stack's Security features" } }, "azureCloudPlugin": { @@ -92,7 +92,7 @@ ], "defaultValue": "Yes", "metadata": { - "description": "Install the Commercial X-Pack Plugins - Monitoring, Security, Alerting, Graph, Machine Learning* (Elasticsearch 5.5.0+), and if installing Kibana, Reporting and Profiler" + "description": "Install a trial license to enable access to the Elastic Stack platinum features for 30 days. For Elastisearch less than version 6.3.0, a value of 'Yes' enables these features by installing the X-Pack plugin into each deployed Elastic Stack product. For Elastisearch less than version 6.3.0, a value of 'No' does not install the X-Pack plugin and the Elastic Stack is deployed with features available under OSS. For Elasticsearch 6.3.0+, a value of 'No' deploys the Elastic Stack with the basic license level features available." } }, "esAdditionalPlugins": { @@ -120,49 +120,49 @@ "type": "string", "defaultValue": "", "metadata": { - "description": "A Base-64 encoded form of the PKCS#12 archive (.p12/.pfx) containing the key and certificate used to secure HTTP layer of Elasticsearch" + "description": "A Base-64 encoded form of the PKCS#12 archive (.p12/.pfx) containing the key and certificate used to secure HTTP layer of Elasticsearch. Requires xpackPlugins be set to 'Yes' or esVersion to be >= 6.8.0 and < 7.0.0, or >= 7.1.0" } }, "esHttpCertPassword": { "type": "securestring", "defaultValue": "", "metadata": { - "description": "The password for the PKCS#12 archive (.p12/.pfx) containing the key and certificate used to secure HTTP layer of Elasticsearch" + "description": "The password for the PKCS#12 archive (.p12/.pfx) containing the key and certificate used to secure HTTP layer of Elasticsearch. Requires xpackPlugins be set to 'Yes' or esVersion to be >= 6.8.0 and < 7.0.0, or >= 7.1.0" } }, "esHttpCaCertBlob": { "type": "string", "defaultValue": "", "metadata": { - "description": "A Base-64 encoded form of the PKCS#12 archive (.p12/.pfx) containing the CA key and certificate used to secure HTTP layer of Elasticsearch" + "description": "A Base-64 encoded form of the PKCS#12 archive (.p12/.pfx) containing the CA key and certificate used to secure HTTP layer of Elasticsearch. Requires xpackPlugins be set to 'Yes' or esVersion to be >= 6.8.0 and < 7.0.0, or >= 7.1.0" } }, "esHttpCaCertPassword": { "type": "securestring", "defaultValue": "", "metadata": { - "description": "The password for the PKCS#12 archive (.p12/.pfx) containing the CA key and certificate used to secure HTTP layer of Elasticsearch" + "description": "The password for the PKCS#12 archive (.p12/.pfx) containing the CA key and certificate used to secure HTTP layer of Elasticsearch. Requires xpackPlugins be set to 'Yes' or esVersion to be >= 6.8.0 and < 7.0.0, or >= 7.1.0" } }, "esTransportCaCertBlob": { "type": "string", "defaultValue": "", "metadata": { - "description": "A Base-64 encoded form of the PKCS#12 archive (.p12/.pfx) containing the CA key and certificate used to secure Transport layer of Elasticsearch." + "description": "A Base-64 encoded form of the PKCS#12 archive (.p12/.pfx) containing the CA key and certificate used to secure Transport layer of Elasticsearch. Requires xpackPlugins be set to 'Yes' or esVersion to be >= 6.8.0 and < 7.0.0, or >= 7.1.0" } }, "esTransportCaCertPassword": { "type": "securestring", "defaultValue": "", "metadata": { - "description": "The password for the PKCS#12 archive (.p12/.pfx) containing the CA key and certificate used to secure Transport layer of Elasticsearch" + "description": "The password for the PKCS#12 archive (.p12/.pfx) containing the CA key and certificate used to secure Transport layer of Elasticsearch. Requires xpackPlugins be set to 'Yes' or esVersion to be >= 6.8.0 and < 7.0.0, or >= 7.1.0" } }, "esTransportCertPassword": { "type": "securestring", "defaultValue": "", "metadata": { - "description": "The password for the generated certificate used to secure Transport layer of Elasticsearch" + "description": "The password for the generated certificate used to secure Transport layer of Elasticsearch. Requires xpackPlugins be set to 'Yes' or esVersion to be >= 6.8.0 and < 7.0.0, or >= 7.1.0" } }, "samlMetadataUri": { @@ -1296,6 +1296,8 @@ } }, "variables": { + "esVersionMajor": "[int(split(parameters('esVersion'), '.')[0])]", + "esVersionMinor": "[int(split(parameters('esVersion'), '.')[1])]", "templateBaseUrl": "[concat(parameters('artifactsBaseUrl'), '/')]", "sharedTemplateUrl": "[concat(variables('templateBaseUrl'), 'partials/shared-resources.json')]", "networkTemplateUrl": "[concat(variables('templateBaseUrl'), 'networks/virtual-network-resources.json')]", @@ -1937,7 +1939,7 @@ "resourceGroup": "[variables('networkResourceGroupMap')[parameters('vNetNewOrExisting')]]", "location": "[variables('location')]", "addressPrefix": "[parameters('vNetNewAddressPrefix')]", - "https": "[if(and(or(greater(length(parameters('esHttpCertBlob')), 0), greater(length(parameters('esHttpCaCertBlob')), 0)), equals(parameters('xpackPlugins'), 'Yes')), 'Yes', 'No')]", + "https": "[if(and(or(greater(length(parameters('esHttpCertBlob')), 0), greater(length(parameters('esHttpCaCertBlob')), 0)), or(equals(parameters('xpackPlugins'), 'Yes'), and(greaterOrEquals(variables('esVersionMajor'), 7), greaterOrEquals(variables('esVersionMinor'), 1)), and(equals(variables('esVersionMajor'), 6), greaterOrEquals(variables('esVersionMinor'), 8)))), 'Yes', 'No')]", "subnet": { "name": "[parameters('vNetClusterSubnetName')]", "addressPrefix": "[parameters('vNetNewClusterSubnetAddressPrefix')]", diff --git a/src/scripts/elasticsearch-install.sh b/src/scripts/elasticsearch-install.sh index edd03b9e..2bad23bb 100644 --- a/src/scripts/elasticsearch-install.sh +++ b/src/scripts/elasticsearch-install.sh @@ -106,6 +106,7 @@ NAMESPACE_PREFIX="" ES_VERSION="6.4.1" ES_HEAP=0 INSTALL_XPACK=0 +BASIC_SECURITY=0 INSTALL_ADDITIONAL_PLUGINS="" YAML_CONFIGURATION="" MANDATORY_PLUGINS="" @@ -267,6 +268,11 @@ done # Parameter state changes ######################### +# supports security features with a basic license +if [[ $(dpkg --compare-versions "$ES_VERSION" "ge" "7.1.0"; echo $?) -eq 0 || ($(dpkg --compare-versions "$ES_VERSION" "ge" "6.8.0"; echo $?) -eq 0 && $(dpkg --compare-versions "$ES_VERSION" "lt" "7.0.0"; echo $?) -eq 0) ]]; then + BASIC_SECURITY=1 +fi + # zen2 should emit the ports from hosts if dpkg --compare-versions "$ES_VERSION" "ge" "7.0.0"; then UNICAST_HOST_PORT="" @@ -284,7 +290,7 @@ else UNICAST_HOSTS="${UNICAST_HOSTS%?}]" fi -if [[ $(dpkg --compare-versions "$ES_VERSION" "ge" "6.0.0"; echo $?) -eq 0 && ${INSTALL_XPACK} -ne 0 ]]; then +if [[ $(dpkg --compare-versions "$ES_VERSION" "ge" "6.0.0"; echo $?) -eq 0 && (${INSTALL_XPACK} -ne 0 || ${BASIC_SECURITY} -ne 0) ]]; then log "using bootstrap password as the seed password" SEED_PASSWORD="$BOOTSTRAP_PASSWORD" fi @@ -292,6 +298,7 @@ fi log "bootstrapping an Elasticsearch $ES_VERSION cluster named '$CLUSTER_NAME' with minimum_master_nodes set to $MINIMUM_MASTER_NODES" log "cluster uses dedicated master nodes is set to $CLUSTER_USES_DEDICATED_MASTERS and unicast goes to $UNICAST_HOSTS" log "cluster install X-Pack plugin is set to $INSTALL_XPACK" +log "cluster basic security is set to $BASIC_SECURITY" ######################### # Installation steps as functions @@ -1047,7 +1054,7 @@ configure_elasticsearch_yaml() fi fi - if [ ${INSTALL_XPACK} -ne 0 ]; then + if [[ ${INSTALL_XPACK} -ne 0 || ${BASIC_SECURITY} -ne 0 ]]; then if dpkg --compare-versions "$ES_VERSION" "ge" "6.3.0"; then log "[configure_elasticsearch_yaml] Set generated license type to trial" echo "xpack.license.self_generated.type: trial" >> $ES_CONF @@ -1099,13 +1106,18 @@ configure_elasticsearch_yaml() log "[configure_elasticsearch_yaml] setting bootstrap.memory_lock: true" echo "bootstrap.memory_lock: true" >> $ES_CONF + local INSTALL_CERTS=0 + if [[ ${INSTALL_XPACK} -ne 0 || ${BASIC_SECURITY} -ne 0 ]]; then + INSTALL_CERTS=1 + fi + # Configure SSL/TLS for HTTP layer - if [[ -n "${HTTP_CERT}" || -n "$HTTP_CACERT" && ${INSTALL_XPACK} -ne 0 ]]; then + if [[ -n "${HTTP_CERT}" || -n "$HTTP_CACERT" && ${INSTALL_CERTS} -ne 0 ]]; then configure_http_tls $ES_CONF fi # Configure TLS for Transport layer - if [[ -n "${TRANSPORT_CACERT}" && ${INSTALL_XPACK} -ne 0 ]]; then + if [[ -n "${TRANSPORT_CACERT}" && ${INSTALL_CERTS} -ne 0 ]]; then configure_transport_tls $ES_CONF fi @@ -1282,7 +1294,7 @@ install_es setup_data_disk -if [ ${INSTALL_XPACK} -ne 0 ]; then +if [[ ${INSTALL_XPACK} -ne 0 || ${BASIC_SECURITY} -ne 0 ]]; then install_xpack # in 6.x + we need to set up the bootstrap.password in the keystore to use when setting up users if dpkg --compare-versions "$ES_VERSION" "ge" "6.0.0"; then @@ -1310,7 +1322,7 @@ port_forward start_systemd # patch roles and users through the REST API which is a tad trickier -if [[ ${INSTALL_XPACK} -ne 0 ]]; then +if [[ ${INSTALL_XPACK} -ne 0 || ${BASIC_SECURITY} -ne 0 ]]; then wait_for_started apply_security_settings fi diff --git a/src/scripts/kibana-install.sh b/src/scripts/kibana-install.sh index 178329e2..5ae69230 100644 --- a/src/scripts/kibana-install.sh +++ b/src/scripts/kibana-install.sh @@ -67,6 +67,7 @@ KIBANA_VERSION="6.4.1" #Default internal load balancer ip ELASTICSEARCH_URL="http://10.0.0.4:9200" INSTALL_XPACK=0 +BASIC_SECURITY=0 USER_KIBANA_PWD="changeme" SSL_CERT="" SSL_KEY="" @@ -140,8 +141,14 @@ done # Parameter state changes ######################### -log "Installing Kibana $KIBANA_VERSION for Elasticsearch cluster: $CLUSTER_NAME" -log "Installing X-Pack plugins is set to: $INSTALL_XPACK" +# supports security features with a basic license +if [[ $(dpkg --compare-versions "$KIBANA_VERSION" "ge" "7.1.0"; echo $?) -eq 0 || ($(dpkg --compare-versions "$KIBANA_VERSION" "ge" "6.8.0"; echo $?) -eq 0 && $(dpkg --compare-versions "$KIBANA_VERSION" "lt" "7.0.0"; echo $?) -eq 0) ]]; then + BASIC_SECURITY=1 +fi + +log "installing Kibana $KIBANA_VERSION for Elasticsearch cluster: $CLUSTER_NAME" +log "installing X-Pack plugins is set to: $INSTALL_XPACK" +log "basic security is set to: $BASIC_SECURITY" log "Kibana will talk to Elasticsearch over $ELASTICSEARCH_URL" ######################### @@ -210,7 +217,7 @@ configure_kibana_yaml() echo "elasticsearch.hosts: [\"$ELASTICSEARCH_URL\"]" >> $KIBANA_CONF fi - echo "server.host:" $(hostname -I) >> $KIBANA_CONF + echo "server.host: $(hostname -i)" >> $KIBANA_CONF # specify kibana log location echo "logging.dest: /var/log/kibana.log" >> $KIBANA_CONF touch /var/log/kibana.log @@ -219,24 +226,30 @@ configure_kibana_yaml() # set logging to silent by default echo "logging.silent: true" >> $KIBANA_CONF - # install x-pack - if [ ${INSTALL_XPACK} -ne 0 ]; then + # configure security + local ENCRYPTION_KEY + + if [[ ${INSTALL_XPACK} -ne 0 || ${BASIC_SECURITY} -ne 0 ]]; then echo "elasticsearch.username: kibana" >> $KIBANA_CONF echo "elasticsearch.password: \"$USER_KIBANA_PWD\"" >> $KIBANA_CONF install_pwgen - local ENCRYPTION_KEY=$(pwgen 64 1) + ENCRYPTION_KEY=$(pwgen 64 1) echo "xpack.security.encryptionKey: \"$ENCRYPTION_KEY\"" >> $KIBANA_CONF log "[configure_kibana_yaml] X-Pack Security encryption key generated" - ENCRYPTION_KEY=$(pwgen 64 1) - echo "xpack.reporting.encryptionKey: \"$ENCRYPTION_KEY\"" >> $KIBANA_CONF - log "[configure_kibana_yaml] X-Pack Reporting encryption key generated" + fi + # install x-pack + if [ ${INSTALL_XPACK} -ne 0 ]; then if dpkg --compare-versions "$KIBANA_VERSION" "lt" "6.3.0"; then log "[configure_kibana_yaml] Installing X-Pack plugin" /usr/share/kibana/bin/kibana-plugin install x-pack log "[configure_kibana_yaml] Installed X-Pack plugin" fi + + ENCRYPTION_KEY=$(pwgen 64 1) + echo "xpack.reporting.encryptionKey: \"$ENCRYPTION_KEY\"" >> $KIBANA_CONF + log "[configure_kibana_yaml] X-Pack Reporting encryption key generated" fi # configure HTTPS if cert and private key supplied @@ -259,7 +272,12 @@ configure_kibana_yaml() # configure HTTPS communication with Elasticsearch if cert supplied and x-pack installed. # Kibana x-pack installed implies it's also installed for Elasticsearch - if [[ -n "${HTTP_CERT}" || -n "${HTTP_CACERT}" && ${INSTALL_XPACK} -ne 0 ]]; then + local INSTALL_CERTS=0 + if [[ ${INSTALL_XPACK} -ne 0 || ${BASIC_SECURITY} -ne 0 ]]; then + INSTALL_CERTS=1 + fi + + if [[ -n "${HTTP_CERT}" || -n "${HTTP_CACERT}" && ${INSTALL_CERTS} -ne 0 ]]; then [ -d $SSL_PATH ] || mkdir -p $SSL_PATH if [[ -n "${HTTP_CERT}" ]]; then @@ -432,4 +450,4 @@ start_systemd ELAPSED_TIME=$(($SECONDS - $START_TIME)) PRETTY=$(printf '%dh:%dm:%ds\n' $(($ELAPSED_TIME/3600)) $(($ELAPSED_TIME%3600/60)) $(($ELAPSED_TIME%60))) -log "End execution of Kibana script extension in ${PRETTY}" +log "End execution of Kibana script extension in ${PRETTY}" \ No newline at end of file diff --git a/src/scripts/logstash-install.sh b/src/scripts/logstash-install.sh index fef20555..caa4bd2b 100644 --- a/src/scripts/logstash-install.sh +++ b/src/scripts/logstash-install.sh @@ -66,6 +66,7 @@ LOGSTASH_VERSION="6.4.0" LOGSTASH_HEAP=0 ELASTICSEARCH_URL="http://10.0.0.4:9200" INSTALL_XPACK=0 +BASIC_SECURITY=0 INSTALL_ADDITIONAL_PLUGINS="" USER_LOGSTASH_PWD="changeme" LOGSTASH_KEYSTORE_PWD="changeme" @@ -131,6 +132,15 @@ while getopts :v:m:u:S:H:G:V:J:L:c:K:Y:lh optname; do esac done +######################### +# Parameter state changes +######################### + +# supports security features with a basic license +if [[ $(dpkg --compare-versions "$LOGSTASH_VERSION" "ge" "7.1.0"; echo $?) -eq 0 || ($(dpkg --compare-versions "$LOGSTASH_VERSION" "ge" "6.8.0"; echo $?) -eq 0 && $(dpkg --compare-versions "$LOGSTASH_VERSION" "lt" "7.0.0"; echo $?) -eq 0) ]]; then + BASIC_SECURITY=1 +fi + ######################### # Installation steps as functions ######################### @@ -309,7 +319,12 @@ configure_logstash_yaml() # Make the HTTP CA cert for communication with Elasticsearch available to # Logstash conf files through ${ELASTICSEARCH_CACERT} - if [[ -n "${HTTP_CERT}" || -n "${HTTP_CACERT}" && ${INSTALL_XPACK} -ne 0 ]]; then + local INSTALL_CERTS=0 + if [[ ${INSTALL_XPACK} -ne 0 || ${BASIC_SECURITY} -ne 0 ]]; then + INSTALL_CERTS=1 + fi + + if [[ -n "${HTTP_CERT}" || -n "${HTTP_CACERT}" && ${INSTALL_CERTS} -ne 0 ]]; then MONITORING='false' @@ -486,6 +501,7 @@ fi log "installing logstash $LOGSTASH_VERSION" log "installing X-Pack plugins is set to: $INSTALL_XPACK" +log "basic security it set to: $BASIC_SECURITY" log "[apt-get] updating apt-get" (apt-get -y update || (sleep 15; apt-get -y update)) > /dev/null log "[apt-get] updated apt-get" @@ -507,4 +523,4 @@ start_systemd ELAPSED_TIME=$(($SECONDS - $START_TIME)) PRETTY=$(printf '%dh:%dm:%ds\n' $(($ELAPSED_TIME/3600)) $(($ELAPSED_TIME%3600/60)) $(($ELAPSED_TIME%60))) -log "End execution of Logstash script extension in ${PRETTY}" +log "End execution of Logstash script extension in ${PRETTY}" \ No newline at end of file