Sensu Go + etcd authentication (grant-permission)

[tarcinil]

As an engineer,
I would like to ensure that my backend-etcd user has the correct permissions when etcd authentication is enabled.

Feature Suggestion

Documentation on how to run Sensu Go while utilizing etcd authentication. This will further enhance how to run Sensu Go securely.

Possible Implementation

Add documentation to the follow page + h2
https://docs.sensu.io/sensu-go/latest/guides/securing-sensu/#etcd-user-role

Context

Using etcd 3.4 with passwordless user account via mTLS with authentication enabled, I was unable to get Sensu Go backend init to run properly.

{"level":"warn","ts":"2020-02-13T15:31:24.552-0600","caller":"clientv3/retry_interceptor.go:61","msg":"retrying of unary invoker failed","target":"endpoint://client-424eebe0-8294-4eb0-b146-7c456157de7c/REDACTED:2379","attempt":0,"error":"rpc error: code = PermissionDenied desc = etcdserver: permission denied"}

The concerning part is that I had scoped quite a bit into the user/role in etcd.

[root@REDACTED ~]# /opt/etcd/etcdctl user get REDACTED --detail
User: REDACTED

Role sensu_readwrite
KV Read:
	/
	[/, 0) (prefix /)
	[/sensu.io, /sensu.ip) (prefix /sensu.io)
	[/sensu.io/, /sensu.io0) (prefix /sensu.io/)
KV Write:
	/
	[/, 0) (prefix /)
	[/sensu.io, /sensu.ip) (prefix /sensu.io)
	[/sensu.io/, /sensu.io0) (prefix /sensu.io/)

As a test, I disabled etcd authentication and it was able to initialize the backend. This means that this is definitely related to Sensu Go + permissions needed in etcd.

I will continue working on getting them working in my environment and contribute as I can.

[tarcinil]

I was able to get it working by recreating a wide-open grant.

/opt/etcd/etcdctl role grant-permission sensu_readwrite readwrite --from-key ''

This is not ideal of course so I continued to create grants that sounded ideal.

I have have created open-ended grants for the EtcdRoot, but they have not worked for init but seem to allow the process to work as expected.

/opt/etcd/etcdctl role grant-permission sensu_readwrite readwrite --from-key '/sensu.io/'

Does init require anything outside of EtcdRoot?

[tarcinil]

There is an .initialized key that does not sit in the / or /sensu.io/ space. This is what is causing init to fail because it was not in the --from-keys space that was expected.

The etcd role grants were as follows:

User: REDACTED

Role sensu_readwrite
KV Read:
	[.initialized, <open ended>
	[/sensu.io/, <open ended>
KV Write:
	[.initialized, <open ended>
	[/sensu.io/, <open ended>

This allows everything in /sensu.io, .initialized, and .initialized.lock.

[echlebek]

Since this is a documentation issue, I've moved it to sensu-docs.

Now that the key named .initialized has been moved under /sensu.io, we can document that users will need to RW access to that key space.

User: REDACTED

Role sensu_readwrite
KV Read:
	[/sensu.io/, <open ended>
KV Write:
	[/sensu.io/, <open ended>

/opt/etcd/etcdctl role grant-permission sensu_readwrite readwrite --from-key '/sensu.io/'

[hillaryfraley]

Determine best fit: Secure Sensu or a doc that discusses external etcd.
Edit: I think it makes sense to put this information here: https://docs.sensu.io/sensu-go/5.18/guides/clustering/#use-an-external-etcd-cluster

Need to test before publishing.

[bitnik]

Now that the key named .initialized has been moved under /sensu.io, we can document that users will need to RW access to that key space.

@echlebek is it so? Where is it documented? Using sensu 6.10.0 I still have to grant access to .initialized too.