[BUG][OpenSearch] Can't use my own certificates mounted as secrets in the opensearch.yml

[Oznup]

Hello,

First of all, congratulations for this beautiful fork ^^
I've tried it with the demo configuration and everything is working well.
Now, I try to use a more production ready configuration, and then use the certificates registered in secrets (created by cert-manager).

To do so, I use extraVolumes and extraVolumesMounts (I've tried secretMounts, but I don't have better results) to mount my certificates this way :

• tls.crt -> /usr/share/opensearch/config/opensearch-cert.pem
• tls.key -> /usr/share/opensearch/config/opensearch-key.pem
• ca.crt ->/usr/share/opensearch/config/truststore.pem

On startup, opensearch crashes with the following stacktrace :

[2021-12-14T22:06:56,916][ERROR][o.o.s.s.DefaultSecurityKeyStore] [opensearch-cluster-master-0] Your keystore or PEM does not contain a key. If you specified a key password, try removing it. If you did not specify a key password, perhaps you need to if the key is in fact password-protected. Maybe you just confused keys and certificates.
[2021-12-14T22:06:57,009][ERROR][o.o.b.OpenSearchUncaughtExceptionHandler] [opensearch-cluster-master-0] uncaught exception in thread [main]
org.opensearch.bootstrap.StartupException: java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
        at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:182) ~[opensearch-1.2.1.jar:1.2.1]
        [...]
Caused by: java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
        at org.opensearch.plugins.PluginsService.loadPlugin(PluginsService.java:790) ~[opensearch-1.2.1.jar:1.2.1]
        [...]
        ... 6 more
Caused by: java.lang.reflect.InvocationTargetException
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
        [...]
        ... 6 more
Caused by: org.opensearch.OpenSearchSecurityException: Error while initializing transport SSL layer from PEM: java.lang.IllegalArgumentException: File does not contain valid private key: /usr/share/opensearch/config/opensearch-key.pem
        at org.opensearch.security.ssl.DefaultSecurityKeyStore.initTransportSSLConfig(DefaultSecurityKeyStore.java:419) ~[?:?]
        [...]
        ... 6 more
Caused by: java.lang.IllegalArgumentException: File does not contain valid private key: /usr/share/opensearch/config/opensearch-key.pem
        at io.netty.handler.ssl.SslContextBuilder.keyManager(SslContextBuilder.java:386) ~[?:?]
        [...]
uncaught exception in thread [main]
        at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
        [...]
        ... 6 more
Caused by: java.security.spec.InvalidKeySpecException: Neither RSA, DSA nor EC worked
        at io.netty.handler.ssl.SslContext.getPrivateKeyFromByteBuffer(SslContext.java:1155) ~[?:?]
        [...]
        ... 6 more
Caused by: java.security.spec.InvalidKeySpecException: java.security.InvalidKeyException: IOException : algid parse error, not a sequence
        at sun.security.ec.ECKeyFactory.engineGeneratePrivate(ECKeyFactory.java:169) ~[?:?]
        [...]
        ... 6 more
Caused by: java.security.InvalidKeyException: IOException : algid parse error, not a sequence
        at sun.security.pkcs.PKCS8Key.decode(PKCS8Key.java:133) ~[?:?]
        [...]
java.lang.IllegalStateException: failed to load plugin class [org.opensearch.security.OpenSearchSecurityPlugin]
Likely root cause: java.security.InvalidKeyException: IOException : algid parse error, not a sequence
        at java.base/sun.security.pkcs.PKCS8Key.decode(PKCS8Key.java:133)
        [...]

If I cat the file in the container by typing
kubectl exec opensearch-cluster-master-0 -- cat /usr/share/opensearch/config/opensearch-key.pem
I get the RSA private key which looks like this

-----BEGIN RSA PRIVATE KEY-----
[Masked content]
-----END RSA PRIVATE KEY-----

And if I compare it to the key in my cert-manager built secret :
kubectl get secret opensearch-secret -o jsonpath="{.data['tls\.key']}" | base64 -d
I get the same.

Here is my opensearch-values.yaml file :

image: "opensearchproject/opensearch"
tag: "1.2.1"

clusterName: "opensearch-cluster"
nodeGroup: "master"
opensearchHome: /usr/share/opensearch

replicas: 1
opensearchJavaOpts: "-Xmx2g -Xms2g"

persistence:
  enabled: false

extraVolumeMounts:
- name: cert
  mountPath: "/usr/share/opensearch/config/opensearch-cert.pem"
  subPath: "opensearch-cert.pem"
- name: key
  mountPath: "/usr/share/opensearch/config/opensearch-key.pem"
  subPath: "opensearch-key.pem"
- name: ca
  mountPath: "/usr/share/opensearch/config/truststore.pem"
  subPath: "truststore.pem"

extraVolumes:
- name: cert
  secret:
    secretName: opensearch-secret
    items:
    - key: "tls.crt"
      path: "opensearch-cert.pem"
- name: key
  secret:
    secretName: opensearch-secret
    items:
    - key: "tls.key"
      path: "opensearch-key.pem"
- name: ca
  secret:
    secretName: opensearch-secret
    items:
    - key: "ca.crt"
      path: "truststore.pem"

config:
  opensearch.yml: |
    cluster.name: opensearch-cluster
    network.host: 0.0.0.0
    plugins:
      security:
        ssl:
          transport:
            pemcert_filepath: opensearch-cert.pem
            pemkey_filepath: opensearch-key.pem
            pemtrustedcas_filepath: truststore.pem
            enforce_hostname_verification: false
          http:
            enabled: true
            pemcert_filepath: opensearch-cert.pem
            pemkey_filepath: opensearch-key.pem
            pemtrustedcas_filepath: truststore.pem
        allow_unsafe_democertificates: false
        allow_default_init_securityindex: true
        authcz:
          admin_dn:
            - CN=kibana.root
        audit.type: internal_opensearch
        enable_snapshot_restore_privilege: true
        check_snapshot_restore_write_privileges: true
        restapi:
          roles_enabled: ["all_access", "security_rest_api_access"]
        system_indices:
          enabled: true
          indices:
            [
              ".opendistro-alerting-config",
              ".opendistro-alerting-alert*",
              ".opendistro-anomaly-results*",
              ".opendistro-anomaly-detector*",
              ".opendistro-anomaly-checkpoints",
              ".opendistro-anomaly-detection-state",
              ".opendistro-reports-*",
              ".opendistro-notifications-*",
              ".opendistro-notebooks",
              ".opendistro-asynchronous-search-response*",
            ]

What am I missing ?

Thanks in advance for support :)

[life-learner1]

Is that solved @Oznup ,could you check if there is '\n' at the end of the pem file?

[peterzhuamazon]

Close this for now as there is no response for some time.
Please feel free to re-open if you still have questions.

Thanks.

[kevinlmadison]

We're having the same issue.

[smlx]

You need to convert the key to pkcs8 format in order for Opensearch to be able to load it.

This is covered in the documentation here https://opensearch.org/docs/latest/security-plugin/configuration/generate-certificates/

[peterzhuamazon]

@smlx I believe with these PRs we can also use pkcs1:

opensearch-project/OpenSearch#4893
netty/netty#12670

[Citrullin]

You need to convert the key to pkcs8 format in order for Opensearch to be able to load it.

This is covered in the documentation here https://opensearch.org/docs/latest/security-plugin/configuration/generate-certificates/

Does that mean I can't use elliptic curve certificates?

[flavienbwk]

For those who need it. I had to use the following conf in cert-manager:

spec:
  privateKey:
    algorithm: RSA
    size: 2048
    encoding: PKCS8
    rotationPolicy: Always

[Citrullin]

For those who need it. I had to use the following conf in cert-manager:

spec:
  privateKey:
    algorithm: RSA
    size: 2048
    encoding: PKCS8
    rotationPolicy: Always

smh, I figured it only can do RSA. What a shame.