Version 3.7.0 and duplicated keys in a YAML file

[gasmick]

In contrast to Version 3.6.1,
in Version 3.7.0 on both Linux and Windows, when I encrypt a YAML file that accidentally has a duplicated key,
I cannot decrypt it afterwards.

Reproduce with sops test.yaml , duplicate the first line, and save.
Then try to decrypt, and get:

Error unmarshalling input yaml: yaml: unmarshal errors:
  line 2: mapping key "hello" already defined at line 1

The only way I found to recover the file would be to delete the offending key from the encrypted file
hello: ENC[AES256_GCM,data:nvp56OqVzo33WWLSyJubbsx4peUHNd/VQ3N4KXq2DRs4o5pf/3V3Hj8vg7DQUg==,iv:23E6rL8i155Ne1/SeGvsuc5OHku09sK9mhJv44Ky0gI=,tag:T0Dv5t+h3FMDYP7d2x01uw==,type:str]
and decrypt with sops --ignore-mac test.yaml

[felixfontein]

3.7.0 uses a new YAML parser (see the changelog), that's why the behavior is different from 3.6.x or earlier.

I'm a bit surprised that it is possible to encrypt a file with duplicate keys. I would expect it to fail with the same error.

[felixfontein]

I've looked at this a bit. There are two parts:

1. The YAML to sops.TreeBranches and back conversion code does not mind duplicate keys.
2. The code which loads the sops metadata (LoadEncryptedFile() in stores/yaml/store.go) uses yaml.Unmarshal(in, &metadataHolder) to parse the metadata, and that produces the error.

In other words: when using yaml.v3 to deserialize (or serialize from) yaml.Node, it does not care about duplicate keys. But when using yaml.v3 to deserialize into Go structs, it does mind.

I'm not sure what's the best way to proceed here. We can add code to prevent to parse YAML files with duplicate keys in all cases (but we'd have to do that manually), or we can try to work around the issue with yaml.v3's Go struct deserialization to be able to handle YAML files with duplicate keys in all cases.

@autrilla @ajvb what do you think should the behavior be?

[jaythamke]

I also found this error message in version 3.7.1. I expect sops editor should fail the file encryption at the first step.

Bdw, is there a new release/patch with fix? Thanks

[Docteur-RS]

Got the error in 3.7.1. It's really problematic. Can't read the file anymore...

[VanCoding]

How to recover from this?

edit

I just found out:
Edit the encrypted yaml file and remove the duplicate key. Then run sops --ignore-mac path/to/file.yaml

[jaythamke]

How to recover from this?

edit

I just found out: Edit the encrypted yaml file and remove the duplicate key. Then run sops --ignore-mac path/to/file.yaml

Ignore message authentication code (mac) with --ignore-mac option while decrypting the secret.

[felixfontein]

This should get fixed by #1203.