"keys" JSON format described in spec does not match reference implementation output

[nmiyake]

Overview

Section 4.2 of the spec states that:

The KEYID of a key is the hexadecimal encoding of the SHA-256 hash of the canonical JSON form of the key, where the "private" object key is excluded.

The spec also describes that the currently supported key types are "rsa", "ed25519", and "ecdsa", and describes their formats -- for example, for RSA, it states "PUBLIC and PRIVATE are in PEM format and are strings. All RSA keys must be at least 2048 bits" and for ESA it states "PUBLIC and PRIVATE are both 32-byte (256-bit) strings."

In order for different implementations of the spec to produce the same output, it's necessary for the JSON form of the keys to match. However, the reference implementation key objects contain fields not specified in the spec and also support key formats not described in the spec (for example, GPG keys).

It would help to clarify which scenario best describes the desired state:

• The spec is general, and different implementations aren't necessarily expected to interoperate with each other -- the spec describes the concepts of things like "KEYID" and the "KEY" structure, but different implementations may derive/represent them in their own way within their own ecosystem
• The spec should prescribe a particular set of supported keys and formats such that all in-toto implementations should generate the same output (in this case, spec and reference implementation should be in sync for GPG keys)
• Some other state?

I believe this is also relevant to the DSSE work, as the "key ID" concept exists there as well and the manner in which that MAY, SHOULD, or MUST line up with the notion of key IDs in the spec is related to this.

Details

In section 4.2 of the spec, it states that all keys have the format:

 { "keytype" : "<KEYTYPE>",
    "scheme" : "<SCHEME>",
    "keyval" : "<KEYVAL>" }

However, keys produced by the reference implementation also include values such as "keyid" and "keyid_hash_algorithms":

  "keys": {
   "2f89b9272acfc8f4a0a0f094d789fdb0ba798b0fe41f2f5f417c12f0085ff498": {
    "keyid": "2f89b9272acfc8f4a0a0f094d789fdb0ba798b0fe41f2f5f417c12f0085ff498",
    "keyid_hash_algorithms": [
     "sha256",
     "sha512"
    ],
    "keytype": "rsa",
    "keyval": {
     "private": "",
     "public": "-----BEGIN PUBLIC KEY-----\nMIIBojANBgkqhkiG9w0BAQEFAAOCAY8AMIIBigKCAYEAzgLBsMFSgwBiWTBmVsyW\n5KbJwLFSodAzdUhU2Bq6SdRz/W6UOBGdojZXibxupjRtAaEQW/eXDe+1CbKg6ENZ\nGt2D9HGFCQZgQS8ONgNDQGiNxgApMA0T21AaUhru0vEofzdN1DfEF4CAGv5AkcgK\nsalhTyONervFIjFEdXGelFZ7dVMV3Pp5WkZPG0jFQWjnmDZhUrtSxEtqbVghc3kK\nAUj9Ll/3jyi2wS92Z1j5ueN8X62hWX2xBqQ6nViOMzdujkoiYCRSwuMLRqzW2CbT\nL8hF1+S5KWKFzxl5sCVfpPe7V5HkgEHjwCILXTbCn2fCMKlaSbJ/MG2lW7qSY2Ro\nwVXWkp1wDrsJ6Ii9f2dErv9vJeOVZeO9DsooQ5EuzLCfQLEU5mn7ul7bU7rFsb8J\nxYOeudkNBatnNCgVMAkmDPiNA7E33bmL5ARRwU0iZicsqLQR32pmwdap8PjofxqQ\nk7Gtvz/iYzaLrZv33cFWWTsEOqK1gKqigSqgW9T26wO9AgMBAAE=\n-----END PUBLIC KEY-----"
    },
    "scheme": "rsassa-pss-sha256"
   },
...

The spec also notes:

We define three key types at present: "rsa", "ed25519", and "ecdsa".

However, the reference implementation has support for adding GPG keys to the layout, and when doing so the key type is noted as "rsa" but contains many more fields than documented in the spec:

   "aad6ec15d80aca160e1a0e7041fc235573127eb3": {
    "creation_time": 1633371753,
    "hashes": [
     "pgp+SHA2"
    ],
    "keyid": "aad6ec15d80aca160e1a0e7041fc235573127eb3",
    "keyval": {
     "private": "",
     "public": {
      "e": "010001",
      "n": "dc4bc6109e950fe1e68f69421bacf4786d91703be656a5cbc0d0abfe45763a2a86109dda3aed11da0b3d14d4d01304618d8b11c77ae22f1c52f18f4a637c8564211041338089c249b22b3f20c45ed9f6e0ed780acc2b6adb39f283a18e28bc7cc28a11fcee90a6aad765d6c7bca0d219f51b16fcfa65b922dd6cfdd78f7feff5366d07573a5ad8b6c314bbe9936586d3ee9edf49ce33a71ff26a62bc8d48484c6bdfe803017d9dc73ac4fd3bfce3ec4be6ab49781623e9e158e16a2b3d13701590e8dbb901fa4497233c858c1503c06fd963276f952e6f452536be37f99f3e13910f75fbbbae2a01307512961f76e0786c8877c047d7393db97d8a9175e2b3de90cd368e7907f9201064da1abd95fe766c61c972833f344a0e1d40af5ef14550d6185042d375e1e37ea25e0a042d3fa9f94ad631d4387b0927d7f66a10e464a5742695437e4c57186786559c78f790e0599a7f7bb5ae9466904f56459e0a91b1a66b9deb1c1d4a551c931734be22e850f6e3b628959c588c432dd504865503bdb286b368c9e6c4dfa46ead3edb5dac0675e736806efe9305974c69377ebf7b731763d01ab5d1738964ad3c6a388dd394cd80b318f09fefb3d44f10c06e2823221eee7938c63e247111c533d7a274d282845b9a69e062b6c6dedb87dcdaa10d7e8b8fe0766b3b323bf81497b5563051cf4e338a9d9b923b1cb3fd7d7408b89917"
     }
    },
    "method": "pgp+rsa-pkcsv1.5",
    "subkeys": {
     "236add4ccccd28f775ba491487649e16938531ef": {
      "creation_time": 1633371753,
      "hashes": [
       "pgp+SHA2"
      ],
      "keyid": "236add4ccccd28f775ba491487649e16938531ef",
      "keyval": {
       "private": "",
       "public": {
        "e": "010001",
        "n": "e0f7c91be8b8c0fcfabf322e347219b9866fd49f359f47f58c99b83dc0cd9a77a7c6780326b0dd2be404c2061e6f6e61169b8095bf5c7027209c35ec004e69b97269fdc123720be23772dc830121bbfcab3bc82cbe31bd59c163e1c74ff1f6f8a31b7f137253cb9bd27c3a31f167d26708e1775d641a4989272fcc74e7cc1d7a4d581cd2ae8837a819685026a40e635b0487bc74dfade968f66768b0d87c1677bb737d3642e5908ac643fb61b1687ba7c4ab117e00057ae84b2abca0351cbb55ff25752edb7aa5e955236afbb9350cfc894efccd471450214cefbd4a1a04af06c114f77bd17b4c95b49723aa0dde0bcfa2f9579498f538f7794f6a9a51e5c7640f15e687c16bedcc70767491b10765cfc7eae8b212e010119cef2b0ffee9d1eddd5e8d39d46fbad04941db4f5ae4b1a1a4a14a9e34108cde01ef0502870113aef9c69c7fa47c3080ecb79491466640d8702e3a979f344d55df8ee0613f23db4f51f17ae50f6488fa461125641778f46e273f5e667826043f3636b2c4d4f892a480229098f9df47d500f987f0976baf902234002bb22c6743be335a32076e8aa9b425e9a8fe2ddcc40784ce06baa2cc8374600441d1340675ceda1df03f6837afc0bcb70facf2329ef278a1b7684904411c2140414ae151c8c712df03bdd23862ed5f96507f3dcb9f0cacb8955a520acc703248aa40cdc8825cabc48ebd900ce3"
       }
      },
      "method": "pgp+rsa-pkcsv1.5",
      "type": "rsa",
      "validity_period": 94608000
     },
     "74201f27437bc868067ed70a2443912b98592890": {
      "creation_time": 1633371753,
      "hashes": [
       "pgp+SHA2"
      ],
      "keyid": "74201f27437bc868067ed70a2443912b98592890",
      "keyval": {
       "private": "",
       "public": {
        "e": "010001",
        "n": "ce3feeda9a87e1abe4bdb822e79b4cd5de931f3add989243e711ff807908bc8d7a5653746f38e81747507ea778ac46fb503dc42ad1b2f47744a53ab577df2e6da1c0ff9476c4c28bcf13997aa46e8bf989147fa53b85f8b71dfb24476ecb9b2ae043619831692249cd97704e7a2a50338f422ee7dd7cc9489ebc52c2a3cb014e7d2bb6dc00b028f931799cd6ad6380951b1d6090af5439e43f1bfc8f318f8ba1302437ee22d8696e877332b427d68fa443e50cdc87348217a568c2753ce5c777460ab351bbda21569f47603c273678f2ac59ebffa32b0dee1a21509e0d6983bec0b87c9db93f4031b9f733ec53a44bc8436e8e98ae25cb88149c6e9645496a7df9c606642a4958861dd2a71fe04ff9d7296ada176d08b38d63a601d098d68eed44dad6a736b1839e487e716fb5c1d8f93faafe13002f7128bd003e3dbbf03b69949485b63627f9aa4fd4f8316f77a53af6bf83c52ea1c7f96577c547b3b9de53634187f136f8fca9b8aef31338a0d6ec3417734f80205ed08f5027c47b29c91556be1a0ae2d660b6532adde67034e97eac3f0b00dbe0f86ccef30ba460b4a1c31fc7dccabe8a193c6d55995e5e07562b809992b74f2657cd940d86eceb78c2daf3a7fad3d80b070488c8955380853556ecb16d9aa586198d7bfd0e4090d0c999f01e8eb19ef1731791f97f875316394883ce2ef843d0d54632bfd873b09f4979"
       }
      },
      "method": "pgp+rsa-pkcsv1.5",
      "type": "rsa",
      "validity_period": 94608000
     }
    },
    "type": "rsa",
    "validity_period": 94608000
   }

(The above comes from a portion of the layout file produced by adding the line layout.add_functionary_key_from_gpg_keyid(gpg_keyid="AAD6EC15D80ACA160E1A0E7041FC235573127EB3") to https://github.com/in-toto/demo/blob/master/owner_alice/create_layout.py#L87 and running the layout creation logic)

This is relevant because the spec notes that:

The KEYID of a key is the hexadecimal encoding of the SHA-256 hash of the canonical JSON form of the key, where the "private" object key is excluded.

Thus, in order for "KEYID" to be determined in a canonical manner, the JSON form of the key must also be consistent/canonical.

[nmiyake]

Realized that this is a dupe of #33 -- closing this issue in favor of that one.