Help with dynamic yaml file

[neelasha-09]

Hi,

I have file as below:

com:
  amd:
    platform:
      servicebroker:
        elasticsearch:
          clusterid1:
            urls: es-test-es-http.platform.svc.cluster.local:9200
            sslEnabled: false
            secMode: tls
            clusterCredentials:
              username: elastic
              password: 9RC0Tz2z85n8bMmI2qc944Rq

I'm trying to fetch the value for username, pasword and urls
I can achive using ./dasel -f ~/secret1.yaml com.amdocs.platform.servicebroker.elasticsearch.clusterid.urls'
& /dasel -f ~/secret1.yaml 'com.amdocs.platform.servicebroker.elasticsearch.clusterid.cluserCredentails.username'
but the value clusterid1 is dyanmic

Can you please help how this can be achieved ?
Also is there a way to store the output in a variable ?

[TomWright]

If you mean that you don't know what clusterid1 will be and you just want to get the username regardless, you can use the [*] selector to retrieve the username for all elasticsearch clusters:

dasel -f ~/secret1.yaml -m 'com.amd.platform.servicebroker.elasticsearch.[*].clusterCredentials.username'
elastic

If you know what clusterid1 will be, but it's not always the same value then it's up to you to create the correct selector string and pass that to dasel.

Here's an example bash script that will create the correct selector string based on the cluster name/id and store the results in variables:

#!/bin/bash

CLUSTER_ID=$1
FILE="secret1.yaml"

get_cluster_info() {
  CLUSTER_ID=$1
  PROPERTY=$2

  SELECTOR="com.amd.platform.servicebroker.elasticsearch.${CLUSTER_ID}.${PROPERTY}"
  COMMAND="dasel -f ${FILE} ${SELECTOR}"

  echo $($COMMAND)
}

USERNAME=$(get_cluster_info "${CLUSTER_ID}" "clusterCredentials.username")
PASSWORD=$(get_cluster_info "${CLUSTER_ID}" "clusterCredentials.password")
URLS=$(get_cluster_info "${CLUSTER_ID}" "urls")

echo "Username: ${USERNAME}"
echo "Password: ${PASSWORD}"
echo "Urls: ${URLS}"

Usage:

$ ./eg.sh clusterid1
Username: elastic
Password: 9RC0Tz2z85n8bMmI2qc944Rq
Urls: es-test-es-http.platform.svc.cluster.local:9200

[neelasha-09]

Hi tom,

I had tried the command suggested by you, before raising the ticket. Howvever i could see below error:

Command:
dasel -f ~/secret1.yaml -m 'com.amd.platform.servicebroker.elasticsearch.[*].clusterCredentials.username'

Error:

Error: could not query multiple node: could not find multiple value: no value found for selector: .amd: map[amdocs:map[platform:map[servicebroker:map[elasticsearch:map[clusterid1:map[clusterCredentials:map[password:9RC0Tz2z85n8bMmI2qc944Rq username:elastic] secMode:tls sslEnabled:false urls:es-test-es-http.platform.svc.cluster.local:9200]]]]]]`

Am I missing something here?

[neelasha-09]

Please ignore the previous comment. It works fine :)

However when i use the same for below format of the file, it gives error:

com:
  amd:
    platform:
      servicebroker:
        couchbase:
          bucketName:nfsanitybucket
          nfsanitybucket:
            username: nfsanitybucket
            password: D63\3-nN:S

Command:
./dasel -f secret2.yaml -m "com.amd.platform.servicebroker.couchbase.[*].username"

Error: could not load input: could not unmarshal data: yaml: line 7: mapping values are not allowed in this context

In addition to this, I'm trying to add the data in TOML file which has below format:
Basially i have a complete TOML and I want to replace the content of the host and username

1. TOML File

[elasticsearch]
  hosts = ['localhost']
  username = 'xxxxx'
  pathToPassword = 'secrets/elasticsearch-password.toml'
  secureConnection = false

When i use below command,
./dasel put object -f default-connector.toml -p toml -t string '.elasticsearch' username="$Ela_User" , it replaces the entire content under [elasticsearch] but I only want the username value to be replaced .

Also I'm trying to replace the content of content of host
./dasel put object -f default-connector.toml -p toml -t string '.elasticsearch' hosts[0]="$Ela_host"

but the output is not in format as expected:

[elasticsearch]
  "hosts[0]" = "es-test-es-http.platform.svc.cluster.local:9200"

2. TOML file format:

password = 'password'

I'm trying to replace the content of password with below command but it adds the line :(

./dasel put object -f password.toml -p toml -t string password = "$Ela_password"

[password]
  abcd = ""

[TomWright]

Hey @neelasha-09,

However when i use the same for below format of the file, it gives error:

I can't actually reproduce this error. Here's an example on the playground: https://dasel.tomwright.me/s/7edd6069-057e-470b-be96-fc46801ef40e

I can only get that error if I try to use the toml parser on yaml content.

Another error is returned when toml parser is used, and that is because it is a select statement, and not all items contain a username attribute.

it replaces the entire content under [elasticsearch] but I only want the username value to be replaced .

The put object command is designed to overwrite whatever is at that key with the given object, so this is expected.

If you only want to overwrite a single property then go for put string and target the username in the selector instead.

Also I'm trying to replace the content of content of host

put object only accepts object keys + values at the moment, not selectors or arrays.

I would use the following instead:

./dasel put string -f default-connector.toml -p toml ​'.elasticsearch.hosts.[0]' "$Ela_host"

If you want to replace the entire array you could instead do this:

./dasel put document -f default-connector.toml -p toml -d json '.elasticsearch.hosts' '["x"]'

I'm trying to replace the content of password with below command but it adds the line

This is happening because of 2 things:

• You haven't specified a selector flag so dasel assumes the first argument is the selector.
• You have too much spacing when defining password=x

Here's a correct example:

echo '' | dasel put object -p toml -t string . "password=x"

password = "x"

[neelasha-09]

I would use the following instead:

./dasel put string -f default-connector.toml -p toml ​'.elasticsearch.hosts.[0]' "$Ela_host"
If you want to replace the entire array you could instead do this:

./dasel put document -f default-connector.toml -p toml -d json '.elasticsearch.hosts' '["x"]':

I'm looking just to replace the value of elasticsearch.host with variable "Ela_host"

The command doesn't work with variable however,
./dasel put document -f default-connector.toml -p toml -d json '.elasticsearch.hosts' '[$Ela_host]'
Error: could not parse document: could not unmarshal data: invalid character '$' looking for beginning of value

[neelasha-09]

Nevermind, it worked.
I tried in a different way :)

Thanks a lot for your help & support

[neelasha-09]

Hi Tom,

With respect to the TOML configuration mentioned in the previous comment where I'm looking to replace just the value of the parameters, I tried few things and below is the observation:

ORIGINAL FILE:

[couchbase]
username = "nfsanitybucket"
network = "auto"
pathToPassword = "secrets/couchbase-password.toml"
bucket = "nfsanitybucket"
collections = []
hostnameVerification = true
metadataBucket = ""
hosts = [ "platform-couchbase-cluster-srv",]
secureConnection = true
metadataCollection = ""
scope = ""


[couchbase.clientCertificate]
path = "path/to/couchbase-client-cert.p12"
use = false
pathToPassword = "secrets/couchbase-client-cert-password.toml"

[couchbase.dcp]
flowControlBuffer = "128mb"
compression = true
persistencePollingInterval = "100ms"

[elasticsearch]
username = "es_clusterid1"
pathToPassword = "secrets/elasticsearch-password.toml"
hosts = [ "elastic-operator-user-es-http.ocp-install1-pgoprtest.svc.cluster.local:9200",]
secureConnection = true

[[elasticsearch.type]]
ignore = true
prefix = "_sync:"

[[elasticsearch.type]]
index = "airlines"
prefix = "airline_"
pipeline = ""

[[elasticsearch.type]]
regex = ".*port_.*"
index = "airports"

[[elasticsearch.type]]
regex = "(?<index>.+?)::.*"

[[elasticsearch.type]]
index = "airlines"
prefix = "route_"
ignoreDeletes = true
routing = "/airlineid"

[[elasticsearch.type]]
matchOnQualifiedKey = true
regex = "(?<index>[^.]+.[^.]+).*"

[[elasticsearch.type]]
matchOnQualifiedKey = true
regex = "[^.]+.(?<index>[^.]+).*"

[[elasticsearch.type]]
index = "etc"
prefix = ""

[elasticsearch.clientCertificate]
path = "path/to/elasticsearch-client-cert.p12"
use = false
pathToPassword = "secrets/elasticsearch-client-cert-password.toml"


[elasticsearch.typeDefaults]
index = ""
pipeline = ""
ignore = false
typeName = "_doc"
ignoreDeletes = false
matchOnQualifiedKey = false

[elasticsearch.docStructure]
documentContentAtTopLevel = false
metadataFieldName = "meta"
wrapCounters = false

[elasticsearch.aws]
region = ""

[elasticsearch.rejectionLog]
index = "cbes-rejects"
typeName = "_doc"


[elasticsearch.bulkRequestLimits]
bytes = "10mb"
timeout = "1m"
actions = 1000
concurrentRequests = 2

For the section [couchbase], my script is trying to replace only the username & bucketname in TOML

Since the Command1 replaces everything under [couchbase], I tried to use the Commnd2 which also adds the original content under [couchbase]
Command1:
./dasel put object -f default-connector.toml -p toml -t string -t string '.couchbase' bucket="$Couch_Bucket" username="$Couch_User"

Command2:
./dasel put object -f default-connector.toml -p toml -t string -t string -t string -t string -t string -t string -t bool -t bool -t string '.couchbase' bucket="$Couch_Bucket" username="$Couch_User" network="auto" metadataBucket='' metadataCollection='' pathToPassword='secrets/couchbase-password.toml' secureConnection=false hostnameVerification=true scope=''

But the output has only the [couchbase] section and other [couchbase.clientCertificate] [couchbase.dcp] sections are overwritten
Similar behaviour is for [elasticsearch]section.

The output is now as below, the other sections of [couchbase] & [elasticsearch] are eliminated.
Is there a way to preserve the original file and just replace the required contents

[couchbase]
  bucket = "nfsanitybucket"
  hostnameVerification = true
  metadataBucket = ""
  metadataCollection = ""
  network = "auto"
  pathToPassword = "secrets/couchbase-password.toml"
  scope = ""
  secureConnection = false
  username = "nfsanitybucket"

[elasticsearch]
  pathToPassword = "secrets/elasticsearch-password.toml"
  secureConnection = false
  username = "es_clusterid1"

Thanks

[TomWright]

Hey @neelasha-09,

As mentioned above the put object command is designed to completely overwrite the object at the given selector.

Since couchbase.clientCertificate and couchbase.dcp are children underneath couchbase, when you overwrite the couchbase object you are removing the clientCertificate and dcp.

The best approach for you here is to pipe multiple dasel commands together.

Example:

dasel put string -f myFile.toml -o stdout '.couchbase.username' 'New username' |
dasel put string -p toml -o myFile.toml '.couchbase.bucket' 'New bucket name'

Notice that the first dasel command adds the -o stdout flag to write output to stdout, and the last dasel command specifies the -o myFile.toml flag to write data back into the original file.

Before

$ cat myFile.toml 
[couchbase]
username = "nfsanitybucket"
network = "auto"
pathToPassword = "secrets/couchbase-password.toml"
bucket = "nfsanitybucket"
collections = []
hostnameVerification = true
metadataBucket = ""
hosts = [ "platform-couchbase-cluster-srv",]
secureConnection = true
metadataCollection = ""
scope = ""


[couchbase.clientCertificate]
path = "path/to/couchbase-client-cert.p12"
use = false
pathToPassword = "secrets/couchbase-client-cert-password.toml"

[couchbase.dcp]
flowControlBuffer = "128mb"
compression = true
persistencePollingInterval = "100ms"

[elasticsearch]
username = "es_clusterid1"
pathToPassword = "secrets/elasticsearch-password.toml"
hosts = [ "elastic-operator-user-es-http.ocp-install1-pgoprtest.svc.cluster.local:9200",]
secureConnection = true

[[elasticsearch.type]]
ignore = true
prefix = "_sync:"

[[elasticsearch.type]]
index = "airlines"
prefix = "airline_"
pipeline = ""

[[elasticsearch.type]]
regex = ".*port_.*"
index = "airports"

[[elasticsearch.type]]
regex = "(?<index>.+?)::.*"

[[elasticsearch.type]]
index = "airlines"
prefix = "route_"
ignoreDeletes = true
routing = "/airlineid"

[[elasticsearch.type]]
matchOnQualifiedKey = true
regex = "(?<index>[^.]+.[^.]+).*"

[[elasticsearch.type]]
matchOnQualifiedKey = true
regex = "[^.]+.(?<index>[^.]+).*"

[[elasticsearch.type]]
index = "etc"
prefix = ""

[elasticsearch.clientCertificate]
path = "path/to/elasticsearch-client-cert.p12"
use = false
pathToPassword = "secrets/elasticsearch-client-cert-password.toml"


[elasticsearch.typeDefaults]
index = ""
pipeline = ""
ignore = false
typeName = "_doc"
ignoreDeletes = false
matchOnQualifiedKey = false

[elasticsearch.docStructure]
documentContentAtTopLevel = false
metadataFieldName = "meta"
wrapCounters = false

[elasticsearch.aws]
region = ""

[elasticsearch.rejectionLog]
index = "cbes-rejects"
typeName = "_doc"


[elasticsearch.bulkRequestLimits]
bytes = "10mb"
timeout = "1m"
actions = 1000
concurrentRequests = 2

After

$ cat myFile.toml                                                                 

[couchbase]
  bucket = "New bucket name"
  collections = []
  hostnameVerification = true
  hosts = ["platform-couchbase-cluster-srv"]
  metadataBucket = ""
  metadataCollection = ""
  network = "auto"
  pathToPassword = "secrets/couchbase-password.toml"
  scope = ""
  secureConnection = true
  username = "New username"

  [couchbase.clientCertificate]
    path = "path/to/couchbase-client-cert.p12"
    pathToPassword = "secrets/couchbase-client-cert-password.toml"
    use = false

  [couchbase.dcp]
    compression = true
    flowControlBuffer = "128mb"
    persistencePollingInterval = "100ms"

[elasticsearch]
  hosts = ["elastic-operator-user-es-http.ocp-install1-pgoprtest.svc.cluster.local:9200"]
  pathToPassword = "secrets/elasticsearch-password.toml"
  secureConnection = true
  username = "es_clusterid1"

  [elasticsearch.aws]
    region = ""

  [elasticsearch.bulkRequestLimits]
    actions = 1000
    bytes = "10mb"
    concurrentRequests = 2
    timeout = "1m"

  [elasticsearch.clientCertificate]
    path = "path/to/elasticsearch-client-cert.p12"
    pathToPassword = "secrets/elasticsearch-client-cert-password.toml"
    use = false

  [elasticsearch.docStructure]
    documentContentAtTopLevel = false
    metadataFieldName = "meta"
    wrapCounters = false

  [elasticsearch.rejectionLog]
    index = "cbes-rejects"
    typeName = "_doc"

  [[elasticsearch.type]]
    ignore = true
    prefix = "_sync:"

  [[elasticsearch.type]]
    index = "airlines"
    pipeline = ""
    prefix = "airline_"

  [[elasticsearch.type]]
    index = "airports"
    regex = ".*port_.*"

  [[elasticsearch.type]]
    regex = "(?<index>.+?)::.*"

  [[elasticsearch.type]]
    ignoreDeletes = true
    index = "airlines"
    prefix = "route_"
    routing = "/airlineid"

  [[elasticsearch.type]]
    matchOnQualifiedKey = true
    regex = "(?<index>[^.]+.[^.]+).*"

  [[elasticsearch.type]]
    matchOnQualifiedKey = true
    regex = "[^.]+.(?<index>[^.]+).*"

  [[elasticsearch.type]]
    index = "etc"
    prefix = ""

  [elasticsearch.typeDefaults]
    ignore = false
    ignoreDeletes = false
    index = ""
    matchOnQualifiedKey = false
    pipeline = ""
    typeName = "_doc"

[TomWright]

This does highlight the potential for a new command that will allow you to put multiple values at the same time, but for simplicity piping together commands achieves the same outcome for now.

[neelasha-09]

@TomWright Thanks, it works :)