Azure-keystore credentials not set

[okke-formsma]

Hi,

It seems that the azure-keystore credentials are not set even though the 'elasticsearch-keystore add' commands have executed successfully.

[16042018-09:46:57] [configure_elasticsearch_yaml] Configure storage for Azure Cloud in keystore

I think this is caused by a missing call to elastic-keystore create before the add commands.

azure-marketplace/src/scripts/elasticsearch-ubuntu-install.sh

Line 704 in acd6ae2

log "[configure_elasticsearch_yaml] Configure storage for Azure Cloud in keystore"

https://www.elastic.co/guide/en/elasticsearch/reference/current/secure-settings.html

[russcam]

Hey @okke-formsma, I just ran a deployment to test this. The values were inserted into the elasticsearch-keystore correctly

root@data-0:/# /usr/share/elasticsearch/bin/elasticsearch-keystore list
azure.client.default.account
azure.client.default.key
bootstrap.password
keystore.seed

However, when trying to create a repository immediately, the following error is returned:

PUT _snapshot/backup_1
{
    "type": "azure"
}

----

{
  "error": {
    "root_cause": [
      {
        "type": "repository_verification_exception",
        "reason": "[backup_1] path  is not accessible on master node"
      }
    ],
    "type": "repository_verification_exception",
    "reason": "[backup_1] path  is not accessible on master node",
    "caused_by": {
      "type": "i_o_exception",
      "reason": "Can not write blob master.dat-temp",
      "caused_by": {
        "type": "storage_exception",
        "reason": "storage_exception: The specified container does not exist."
      }
    }
  },
  "status": 500
}

According to the azure-repository documentation, the container needs to be created in the storage account before creating a repository. The default container name is elasticsearch-snapshots. Once the container is created though, the repository creation request succeeds. Creating with Azure PowerShell

Set-AzureRmCurrentStorageAccount -ResourceGroupName "<resource group>" -Name "<storage account name>"

New-AzureStorageContainer -Name "elasticsearch-snapshots"

then

PUT _snapshot/backup_1
{
    "type": "azure"
}

----

{
  "acknowledged": true
}

and

PUT /_snapshot/backup_1/snapshot_1?wait_for_completion=true

----

{
  "snapshot": {
    "snapshot": "snapshot_1",
    "uuid": "hv5cp2e2TAqvC9RV29RsKQ",
    "version_id": 6020399,
    "version": "6.2.3",
    "indices": [
      ".watcher-history-7-2018.04.18",
      ".triggered_watches",
      ".security-6",
      ".monitoring-kibana-6-2018.04.18",
      ".monitoring-es-6-2018.04.18",
      ".watches",
      ".monitoring-alerts-6"
    ],
    "include_global_state": true,
    "state": "SUCCESS",
    "start_time": "2018-04-18T04:51:49.107Z",
    "start_time_in_millis": 1524027109107,
    "end_time": "2018-04-18T04:51:51.516Z",
    "end_time_in_millis": 1524027111516,
    "duration_in_millis": 2409,
    "failures": [],
    "shards": {
      "total": 7,
      "failed": 0,
      "successful": 7
    }
  }
}

Would you be able to try creating the container in the storage account, to see if this addresses the issue?

[okke-formsma]

@russcam Thanks for investigating. I'll try a new deployment to investigate later today.

[russcam]

thanks @okke-formsma - you'll need to target master branch (commit e89baf2) as it contains a fix for the oracle-java8-installer apt package

[okke-formsma]

@russcam I ran with the latest commit but still have the same issue.

When is the elasticsearch-keystore create command executed? I can't find it in the azure-marketplace code anywhere. All manual commands I run require a keystore to be present before running 'add' commands.

Maybe the x-pack or kibana plugins create the keystore in your case? I'm running with those turned off.

root@el6stdata-0:/var/log# grep -R configure_elasticsearch_yaml .
./arm-install.log:[18042018-13:02:22] [configure_elasticsearch_yaml] Update configuration with data path list of /datadisks/disk1/elasticsearch/data
./arm-install.log:[18042018-13:02:22] [configure_elasticsearch_yaml] Update configuration with hosts configuration of ["el6stdata-0:9300"]
./arm-install.log:[18042018-13:02:22] [configure_elasticsearch_yaml] Configure master/client/data node type flags only master-0 only data-0
./arm-install.log:[18042018-13:02:22] [configure_elasticsearch_yaml] Configure node as master and data
./arm-install.log:[18042018-13:02:22] [configure_elasticsearch_yaml] Configure storage for Azure Cloud in keystore
./arm-install.log:[18042018-13:02:26] [configure_elasticsearch_yaml] Setting bootstrap.memory_lock: true

root@el6stdata-0:/etc/elasticsearch# echo "test" | /usr/share/elasticsearch/bin/elasticsearch-keystore add azure.client.default.account -xf
ERROR: Elasticsearch keystore not found. Use 'create' command to create one.

root@el6stdata-0:/etc/elasticsearch# ls /etc/elasticsearch/
elasticsearch.yml  elasticsearch.yml.bak  jvm.options  log4j2.properties

[russcam]

What version of Elasticsearch are you deploying?

[okke-formsma]

Actually we run a slightly modified version which installs 6.2.3 (6.2.2 has some issues with restoring backups from azure). I tested the elastic-keystore commands on 6.2.2 and these behaved identically. (both 6.2.2 and 6.2.3 don't create a keystore unless the 'create' command is issued)

[russcam]

@okke-formsma do you also install X-Pack plugin in your template, and therefore set a bootstrap password?

[okke-formsma]

We do not install xpack or kibana.

Configuration template follows below, we install using the following command;

az group deployment create --resource-group staging-elastic6 --name staging-elastic6 --template-file ./azure-marketplace/src/mainTemplate.json --parameters=@azure-template-staging-6.json

{
  "artifactsBaseUrl": {"value": "https://raw.githubusercontent.com/elastic/azure-marketplace/6.2.2/src"},

  "esVersion": {"value": "6.2.3"},
  "adminUsername": {"value": "snip"},
  "adminPassword": {"value": "snip"},
  "securityAdminPassword": {"value": "snip"},
  "securityReadPassword": {"value": "snip"},
  "securityKibanaPassword": {"value": "snip"},
  "securityBootstrapPassword": {"value": "snip"},
  "securityLogstashPassword": {"value": "snip"},

  "esClusterName": {"value": "staging-elastic6"},
  "vmHostNamePrefix": {"value": "el6st"},
  "vmSizeDataNodes": {"value": "Standard_A2"},
  "vmDataDiskCount": {"value": 1},
  "vmDataDiskSize": {"value": "Medium"},
  "vmDataNodeCount": {"value": 1},
  "dataNodesAreMasterEligible": {"value": "Yes"},

  "vNetNewOrExisting": {"value": "existing"},
  "vNetName": {"value": "snip"},
  "vNetExistingResourceGroup": {"value": "snip"},
  "vNetClusterSubnetName": {"value": "STAGING-ELASTIC6"},
  "vNetLoadBalancerIp": {"value": "snip"},
  "loadBalancerType": {"value": "internal"},

  "xpackPlugins": {"value": "No"},
  "kibana": {"value": "No"},
  "azureCloudPlugin": {"value": "Yes"},
  "azureCloudStorageAccountName": {"value": "snip"},
  "azureCloudStorageAccountKey": {"value": "snip"}
}

[russcam]

Running a test without installing X-Pack plugin now

[russcam]

Can replicate - will push a fix now

[russcam]

@okke-formsma I've opened #184. Would you mind trying it from the fix/create-keystore branch?

[okke-formsma]

@russcam Thank you for fixing, I have tested and it works perfectly now.