sam deploy --guided doesn't escape quotation marks for parameters written to samconfig.toml

[senorkrabs]

Description

When using sam deploy --guided, parameters with quotation marks " are written to parameter_overrides in samconfig.toml without proper escaping. As a result, the parameter isn't correctly populated the next time sam deploy is called.

Steps to reproduce

Run sam deploy --guided and use these parameters:

        Stack Name [sftp-lambda]: 
        AWS Region [us-east-1]: 
        Parameter FilesListJSON []: [ { "SrcFile": "/athena/CUR2/CUR2/year=2020/month=6/*.snappy.parquet",  "TargetPath": "/sftp-lambda/test"}, { "SrcFile": "/athena/CUR2/CUR2/year=2020/month=5/*.snappy.parquet",  "TargetPath": "/sftp-lambda/test"}]
        #Shows you resources changes to be deployed and require a 'Y' to initiate deploy
        Confirm changes before deploy [Y/n]: Y
        #SAM needs permission to be able to create roles to connect to the resources in your template
        Allow SAM CLI IAM role creation [Y/n]: Y
        Save arguments to samconfig.toml [Y/n]: Y

Observed result

Written to samconfig.toml:

parameter_overrides = "FilesListJSON=\"[ { \"SrcFile\": \"/athena/CUR2/CUR2/year=2020/month=6/*.snappy.parquet\",  \"TargetPath\": \"/sftp-lambda/test\"}, { \"SrcFile\": \"/athena/CUR2/CUR2/year=2020/month=5/*.snappy.parquet\",  \"TargetPath\": \"/sftp-lambda/test\"}]\""

What happens when sam deploy is called again:

$ sam deploy

        Deploying with following values
        ===============================
        Stack name                 : sftp-lambda
        Region                     : us-east-1
        Confirm changeset          : True
        Deployment s3 bucket       : aws-sam-cli-managed-default-samclisourcebucket-
        Capabilities               : ["CAPABILITY_IAM"]
        Parameter overrides        : {'FilesListJSON': '[ { '}

(Note that Parameter overrides is incorrect)

Please provide command output with --debug flag set.

Expected result

Written to samconfig.toml:

parameter_overrides = "FilesListJSON=\"[ { \\\"SrcFile\\\": \\\"/athena/CUR2/CUR2/year=2020/month=6/*.snappy.parquet\\\",  \\\"TargetPath\\\": \\\"/sftp-lambda/test\\\"}, { \\\"SrcFile\\\": \\\"/athena/CUR2/CUR2/year=2020/month=5/*.snappy.parquet\\\",  \\\"TargetPath\\\": \\\"/sftp-lambda/test\\\"}]\""

$ sam deploy

        Deploying with following values
        ===============================
        Stack name                 : sftp-lambda
        Region                     : us-east-1
        Confirm changeset          : True
        Deployment s3 bucket       : aws-sam-cli-managed-default-samclisourcebucket-
        Capabilities               : ["CAPABILITY_IAM"]
        Parameter overrides        : {'FilesListJSON': '[ { "SrcFile": "/athena/CUR2/CUR2/year=2020/month=6/*.snappy.parquet",  "TargetPath": "/sftp-lambda/test"}, { "SrcFile": "/athena/CUR2/CUR2/year=2020/month=5/*.snappy.parquet"'}

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: Amazon Linux
2. sam --version: SAM CLI, version 0.52.0

Add --debug flag to command you are running

[wchengru]

Hi @senorkrabs,
For now, we only support wrapping strings with double quotes \", the escape sequence needs to be \\\" inside double quotes. I have created a pr to support single quoting parameter overrides. You can simply use \" inside pair of single quotes after this pr is released.

samdev deploy --parameter-overrides "FilesListJSON='[ { \"SrcFile\": \"/athena/CUR2/CUR2/year=2020/month=6/*.snappy.parquet\",  \"TargetPath\": \"/sftp-lambda/test\"}, { \"SrcFile\": \"/athena/CUR2/CUR2/year=2020/month=5/*.snappy.parquet\",  \"TargetPath\": \"/sftp-lambda/test\"}]'"

	Deploying with following values
	===============================
	Stack name                 : sam-app
	Region                     : us-east-1
	Confirm changeset          : False
	Deployment s3 bucket       : aws-sam-cli-managed-default-samclisourcebucket-6wsl7tf21dut
	Capabilities               : ["CAPABILITY_IAM"]
	Parameter overrides        : {'FilesListJSON': '[ { "SrcFile": "/athena/CUR2/CUR2/year=2020/month=6/*.snappy.parquet",  "TargetPath": "/sftp-lambda/test"}, { "SrcFile": "/athena/CUR2/CUR2/year=2020/month=5/*.snappy.parquet",  "TargetPath": "/sftp-lambda/test"}]'}

[wchengru]

PR is released.

[specious]

This issue should be reopened.

Specifying parameter values containing double quotes" records them as escaped as \", which then is not properly distinguished from the exact same escape sequence enclosing the value (\"value\") and on the next run the saved values are not parsed correctly.

So, if a value is "" it will be recorded as key=\"\"\"\" and will be interpreted as key=\"\".

For example:

$ sam --version
SAM CLI, version 1.29.0

$ sam deploy -g --confirm-changeset

Configuring SAM deploy
======================

        Looking for config file [samconfig.toml] :  Found
        Reading default arguments  :  Success

        Setting default arguments for 'sam deploy'
        =========================================
        Stack Name [myapp-backend]:
        AWS Region [us-east-1]:
        Parameter DBInstanceClass [db.t3.micro]:
        Parameter DBName [mydb]:
        Parameter DBAdminUser [admin]: 
        Parameter DBAdminPassword []: "
        Parameter DBWriterPassword []: \
        Parameter DBReaderPassword []: '

On the subsequent run, the values are:

$ sam deploy -g --confirm-changeset

Configuring SAM deploy
======================

        Looking for config file [samconfig.toml] :  Found
        Reading default arguments  :  Success

        Setting default arguments for 'sam deploy'
        =========================================
        Stack Name [myapp-backend]:
        AWS Region [us-east-1]:
        Parameter DBInstanceClass [db.t3.micro]:
        Parameter DBName [mydb]:
        Parameter DBAdminUser [admin]:
        Parameter DBAdminPassword []:
        Parameter DBWriterPassword [" DBReaderPassword=]:
        Parameter DBReaderPassword []: ^CAborted!

$ grep parameter_overrides samconfig.toml       
parameter_overrides = "DBInstanceClass=\"db.t3.micro\" DBName=\"mydb\" DBAdminUser=\"admin\" DBAdminPassword=\"\"\" DBWriterPassword=\"\\\" DBReaderPassword=\"'\""

I've got a use case where I'd like to use strong passwords generated by aws secretsmanager get-random-password --output=text as parameter values for password parameters.

I've got a script that pre-populates the password values in samconfig.toml. Escaping \ as \\ works as intended, but I haven't found a working solution for escaping " characters.

Escaping " as \\\" does not work. The \\ is interpreted as a slash \ and then the \" is interpreted as terminating the value field.

[specious]

Also, in reference to #2139, single quotes ('value') can be used when pre-populating the values, but after running sam deploy -g even if unchanged the values will be saved with double quotes (\"value\").

That means a value pre-populated as '\"' will be correctly interpreted as ". However, it will then be saved as \"\"\" and will be read as an empty value the next time.

[specious]

My current working solution is to escape \ as \\ and replace " with ':

#!/usr/bin/env sh

echo -n "Generating strong passwords for database access and administration"

function pw () {
  # Escape \ as \\ and replace " with '
  echo $(aws secretsmanager get-random-password --output=text | sed 's/\\/\\\\/g' | sed "s/\"/'/g")
}

p1="DBAdminPassword=\\\"$(pw)\\\""  && echo -n .
p2="DBWriterPassword=\\\"$(pw)\\\"" && echo -n .
p3="DBReaderPassword=\\\"$(pw)\\\"" && echo -n .

echo parameter_overrides = \"$p1 $p2 $p3\" >> samconfig.toml
echo \ done

Which works well:

$ cat samconfig.toml
version = 0.1
[default]
[default.deploy]
[default.deploy.parameters]
stack_name = "myapp-backend"

$ ./prepare
Generating strong passwords for database access and administration... done

$ sam deploy -g --confirm-changeset

Configuring SAM deploy
======================

        Looking for config file [samconfig.toml] :  Found
        Reading default arguments  :  Success

        Setting default arguments for 'sam deploy'
        =========================================
        Stack Name [myapp-backend]:
        AWS Region [us-east-1]:
        Parameter DBInstanceClass [db.t3.micro]:
        Parameter DBName [mydb]:
        Parameter DBAdminUser [admin]:
        Parameter DBAdminPassword [@Tv+lFiB~N7fSUj(7l6?pLkG#Nh:?Z8`]:
        Parameter DBWriterPassword [|fS';v,5HA`89{`Bfx),MS9%ZXR(90Y@]:
        Parameter DBReaderPassword [SgZ^TVh[d&>xvKmM)L4Nn^yOR/9b>r4O]:

However, I would like to be able to have double quotes as well.

[specious]

@wchengru, I saw what you did in #2139, however I'm having some trouble seeing how to properly approach fixing this issue. Would you be able to suggest a general course of action?

[awood45]

Pending release.