[Extensions] Authenticate and authorize from service account token

[stephen-crawford]

As a part of handling service accounts, we first refactored how internal users were represented in the code and then added a mechanism to generate an auth token representing the Basic Auth credentials of a service account. The flow diagram can be found here.

As the next step in this flow, we need to be able to authenticate and authorize based on these tokens. In the diagram, we can see that the request is expected to return to the security plugin attempting to execute its operation. To handle this, code will need to be added in the security plugin to parse the token received as a part of the request. We will then need to verify it is the correct token by decrypting it for the encoded service account name and password. The password can then be hashed and compared to the stored hash of the service account.

This flow is almost identical to a normal basic auth flow but we use the token to pass the credentials instead of encoding the information as part of the URI.

Code changes for this issue can be added to the normal REST request handler but should look for a token which matches the specification of the tokens created in #2697.

Completion of this issue will look like a PR which introduces the code changes to add this logic to the security plugin as well as associated tests. Tests can make use of the previous test format added in the UserAPITest class.

[MaciejMierzwa]

Hi @scrawfor99 ,
I see that SecurityFilter uses PrivilegesEvaluators evaluate() method. From what I gathered this filtering will be applied to every rest request that's coming to the cluster. How could I make sure that only service account requests are evaluated at this point? Something similar to this: InternalUsersApiAction?
Another question about authentication/authorization. I see that as some point before calling evaluate() method there's a call to get user passed in the request in a form of header under a name "_opendistro_security_user" SecurityFilter. It's fetched from threadContext which I assume stores HTTP request headers and would represent logged-in user. Will this part of code be obsolete in case of changes? I assume that we want to get JWT, and confirm its validity. After that we could authz user (this part is already there, in evaluate() method. Is it correct?

[MaciejMierzwa]

@cwperks, @peternied, @scrawfor99 bump. Also, I've found this PR by Ryan which seems to touch on similar topics: #2672 and I got a feeling that this task might be dependent on Ryan's PR.
Sorry, but I have another 2 questions:

1. What's the difference between this task and that one: Authorizing User against requested extension endpoint #2622?
   2. What's the reason for separating core/security layers as in the tasks diagrams? Afaik we could implement the authz/authn logic somewhat similar to: SecurityRestFilter Since this component is intercepting all of the rest requests that get to the node what's the reason of additional layer, why not leave out the core?