[BUG] Can't create reporting indices due to permission issue

[maxemontio]

Describe the bug

Hi everyone!

I am facing problems with creating reports. Logs say that i have permissions errors, but it happens even with all_access role mapped, which makes no sense.
We use oidc to access dashboards.

This is the role for regular users:

{
  "osd-reader": {
    "reserved": false,
    "hidden": false,
    "cluster_permissions": [
      "cluster:admin/opensearch/observability/get",
      "cluster:admin/opendistro/reports/definition/list",
      "indices:data/read/mget*",
      "cluster:admin/opensearch/ql/datasources/read",
      "indices:data/write/bulk",
      "cluster:admin/opendistro/reports/menu/download",
      "cluster:admin/opendistro/reports/instance/list",
      "cluster:admin/opendistro/reports/definition/create",
      "cluster:admin/opendistro/reports/definition/update",
      "cluster:admin/opendistro/reports/definition/on_demand",
      "cluster:admin/opendistro/reports/definition/get",
      "cluster:admin/opendistro/reports/instance/get",
      "cluster:admin/opendistro/reports/definition/delete"
    ],
    "index_permissions": [
      {
        "index_patterns": [
          ".kibana_*"
        ],
        "dls": "",
        "fls": [],
        "masked_fields": [],
        "allowed_actions": [
          "write",
          "read",
          "indices:admin/create"
        ]
      },
      {
        "index_patterns": [
          "*"
        ],
        "dls": "",
        "fls": [],
        "masked_fields": [],
        "allowed_actions": [
          "read",
          "search",
          "get",
          "indices:admin/create"
        ]
      }
    ],
    "tenant_permissions": [],
    "static": false
  }
}

and it's rolemapping:

{
  "osd-reader": {
    "hosts": [],
    "users": [
      "osd-reader"
    ],
    "reserved": false,
    "hidden": false,
    "backend_roles": [
      "osd-reader"
    ],
    "and_backend_roles": []
  }
}

When i go to the saved searches, then to Reporting and there Generate CSV i get Insufficient permissions error in UI and that log on a master node:

[2024-05-06T08:58:18,251][INFO ][o.o.s.p.SecurityIndexAccessEvaluator] [master-1] No indices:admin/create permission for user roles roles=[
  osd-reader
    ipatterns=[
        indexPattern=.kibana_*
          dlsQuery=
          fls=[]
          perms=[indices:admin/mapping/put, indices:admin/resolve/index, indices:data/write*, indices:admin/create, indices:admin/mappings/fields/get*, indices:data/read*],
        indexPattern=*
          dlsQuery=
          fls=[]
          perms=[indices:data/read/msearch*, indices:data/read/get*, indices:admin/resolve/index, indices:data/read/search*, indices:data/read/suggest*, indices:data/read/mget*, indices:admin/create, indices:admin/mappings/fields/get*, indices:data/read*]]
    clusterPerms=[cluster:admin/opendistro/reports/definition/create, cluster:admin/opensearch/observability/get, cluster:admin/opensearch/ql/datasources/read, cluster:admin/opendistro/reports/definition/list, indices:data/read/mget*, cluster:admin/opendistro/reports/instance/list, cluster:admin/opendistro/reports/definition/on_demand, cluster:admin/opendistro/reports/definition/update, cluster:admin/opendistro/reports/definition/delete, cluster:admin/opendistro/reports/menu/download, cluster:admin/opendistro/reports/definition/get, indices:data/write/bulk, cluster:admin/opendistro/reports/instance/get]] to System Indices .opendistro-reports-instances
[2024-05-06T08:58:18,253][WARN ][o.o.r.a.PluginBaseAction ] [master-1] reports:OpenSearchStatusException: message:no permissions for [] and User [name=max, backend_roles=[osd-reader], requestedTenant=null]

The same happens with the all_access role, we just map it with our backend_role for admins:

  "all_access": {
    "hosts": [],
    "users": [
      "admin" 
    ],
    "reserved": false,
    "hidden": false,
    "backend_roles": [
      "osd-admin"
    ],
    "and_backend_roles": []
  }

The error:

[2024-05-06T09:09:39,849][INFO ][o.o.s.p.SecurityIndexAccessEvaluator] [master-1] No indices:admin/create permission for user roles roles=[
  all_access
    ipatterns=[
        indexPattern=*
          dlsQuery=null
          fls=[]
          perms=[*]]
    clusterPerms=*,
  osd-reader
    ipatterns=[
        indexPattern=.kibana_*
          dlsQuery=
          fls=[]
          perms=[indices:admin/mapping/put, indices:admin/resolve/index, indices:data/write*, indices:admin/create, indices:admin/mappings/fields/get*, indices:data/read*],
        indexPattern=*
          dlsQuery=
          fls=[]
          perms=[indices:data/read/msearch*, indices:data/read/get*, indices:admin/resolve/index, indices:data/read/search*, indices:data/read/suggest*, indices:data/read/mget*, indices:admin/create, indices:admin/mappings/fields/get*, indices:data/read*]]
    clusterPerms=[cluster:admin/opendistro/reports/definition/create, cluster:admin/opensearch/observability/get, cluster:admin/opensearch/ql/datasources/read, cluster:admin/opendistro/reports/definition/list, indices:data/read/mget*, cluster:admin/opendistro/reports/instance/list, cluster:admin/opendistro/reports/definition/on_demand, cluster:admin/opendistro/reports/definition/update, cluster:admin/opendistro/reports/definition/delete, cluster:admin/opendistro/reports/menu/download, cluster:admin/opendistro/reports/definition/get, indices:data/write/bulk, cluster:admin/opendistro/reports/instance/get]] to System Indices .opendistro-reports-instances
[2024-05-06T09:09:39,850][WARN ][o.o.r.a.PluginBaseAction ] [master-1] reports:OpenSearchStatusException: message:no permissions for [] and User [name=max, backend_roles=[osd-admin, osd-reader], requestedTenant=null]

Parts with No indices:admin/create permission and no permissions for [] completely baffle me.

1. indices:admin/create is present everywhere but not considered?
2. How come that there is an empty tuple ([]) with permissions?

You can see the same behaviour when you go to Reporting directly from the left panel and click refresh.

Related component

Plugins

To Reproduce

There are two ways:

1. Go to 'Discover -> Open -> some saved search -> Reporting'
2. Click on 'Generate CSV'
3. See error

---

1. Go to 'Reporting'
2. Click on 'Refresh'
3. See error

Expected behavior

Reports are generated and downloaded.

Additional Details

Plugins

opensearch-alerting                  2.12.0.0
opensearch-anomaly-detection         2.12.0.0
opensearch-asynchronous-search       2.12.0.0
opensearch-cross-cluster-replication 2.12.0.0
opensearch-custom-codecs             2.12.0.0
opensearch-flow-framework            2.12.0.0
opensearch-geospatial                2.12.0.0
opensearch-index-management          2.12.0.0
opensearch-job-scheduler             2.12.0.0
opensearch-knn                       2.12.0.0
opensearch-ml                        2.12.0.0
opensearch-neural-search             2.12.0.0
opensearch-notifications             2.12.0.0
opensearch-notifications-core        2.12.0.0
opensearch-observability             2.12.0.0
opensearch-performance-analyzer      2.12.0.0
opensearch-reports-scheduler         2.12.0.0
opensearch-security                  2.12.0.0
opensearch-security-analytics        2.12.0.0
opensearch-skills                    2.12.0.0
opensearch-sql                       2.12.0.0

Host/Environment:

• Docker image: opensearchproject/opensearch:2.12.0
• OS: Ubuntu 20.04.6 LTS

[dblock]

This may not be a bug, try asking on the public slack @maxemontio?

Catch All Triage - 1 2 3 4 5 6

[igor-nikiforov]

@dblock we're have the same issue in 2.14.0 and this is definitely the bug. Could you please advise what's minimum permission required for "Generate CSV"?

[joshuali925]

One possible causes we saw before is that it happens because the reporting plugin needs to use some indexes to store metadata, but user doesn't have permission to create index. see #309

A workaround is to use admin to go to the reporting plugin page once, it will automatically create the indexes, then have users to create report

Although I'm not sure why it also happens with all_access, this might be different

[cwperks]

it happens because the reporting plugin needs to use some indexes to store metadata, but user doesn't have permission to create index.

A workaround is to use admin to go to the reporting plugin page once, it will automatically create the indexes, then have users to create report

@joshuali925 which indices? System indices?

If reporting needs to write to a system index, then make sure to surround the calls with:

try (ThreadContext.StoredContext ctx = threadContext.stashContext()) {
    // Perform index to system index here.
}

[joshuali925]

@cwperks yes it's system indices, but it's already removing thread context

reporting/src/main/kotlin/org/opensearch/reportsscheduler/index/ReportInstancesIndex.kt

Line 54 in fb95c0e

this.client = SecureIndexClient(client)

reporting/src/main/kotlin/org/opensearch/reportsscheduler/util/SecureIndexClient.kt

Lines 43 to 56 in fb95c0e

	/**
	* Wrapper class on [Client] with security context removed.
	*/
	@Suppress("TooManyFunctions")
	internal class SecureIndexClient(private val client: Client) : Client by client {
	/**
	* {@inheritDoc}
	*/
	override fun <Request : ActionRequest, Response : ActionResponse> execute(
	action: ActionType<Response>,
	request: Request
	): ActionFuture<Response> {
	client.threadPool().threadContext.stashContext().use { return client.execute(action, request) }
	}

[cwperks]

Is it using the SecureIndexClient in this case? I'm not an expert in this plugin.

From a quick analysis, I see that its restoring and then calling executeRequest here.

Curious, why is restore called the first line after stashing? Will scope.launch create a new thread and its populating the context in the new thread?

[joshuali925]

Is it using the SecureIndexClient in this case?

yes

why is restore called the first line after stashing?

The code came from opensearch-project/notifications#274, unfortunately i don't remember much besides the PR description

[cwperks]

w/o diving deeper into the plugin, could the issue potentially be that there is no wrapper for index creation in https://github.com/opensearch-project/reporting/blob/main/src/main/kotlin/org/opensearch/reportsscheduler/util/SecureIndexClient.kt?

[cwperks]

FYI I think you can make that class more generic and stash the context for any action.

Something like this (this is Java code):

public class SystemIndexClient extends FilterClient {

    private static final Logger logger = LogManager.getLogger(SystemIndexClient.class);

    public SystemIndexClient(Client delegate) {
        super(delegate);
    }

    @Override
    protected <Request extends ActionRequest, Response extends ActionResponse> void doExecute(
        ActionType<Response> action,
        Request request,
        ActionListener<Response> listener
    ) {
          try (ThreadContext.StoredContext ctx = threadContext.stashContext()) {
              super.doExecute(action, request, ActionListener.runBefore(listener, ctx::restore));
          }
    }
}

[joshuali925]

thanks it's good to know about FilterClient.

could the issue potentially be that there is no wrapper for index creation

I think I'd need to look into how it has been working. The code didn't change for a few years and this is the only case having permission issue with all_access