Operator fails to enable authentication for Solr version 6.6.6, 6.6.6-slim

[mogul]

Summary:

The operator documentation says that it works with all official Solr images. However, the operator is not properly configuring authentication for solrImageTag 6.6.6-slim or 6.6.6.

We are dependent on Solr version 6.6.6 since that is the most recent Solr version supported by CKAN, which is the basis for our application (data.gov).

What I observed:

When I enter the service URL in my browser, I see the Solr Admin UI.

What I expected:

When I enter the service URL in my browser, I am prompted for authentication.

How to reproduce:

$ kubectl create -f https://solr.apache.org/operator/downloads/crds/v0.4.0/all-with-dependencies.yaml
$ helm install --namespace kube-system --version 0.4.0 solr solr-operator
# [Wait until the operator is ready...]
$ helm install example-solr apache-solr/solr --version 0.4.0 \
  --set image.tag="6.6.6-slim" \
  --set solrOptions.javaMemory="-Xms300m -Xmx300m" \
  --set solrOptions.security.authenticationType="Basic" \
  --set addressability.external.method=Ingress \
  --set addressability.external.domainName="ing.local.domain" \
  --set addressability.external.useExternalAddress="true"
$ kubectl get solrclouds -w  # Until the nodes are all ready
$ output=$(curl -s -o /dev/null -L -w "%{http_code}" "http://default-example-solrcloud.ing.local.domain/solr/#/~cloud?view=nodes")
$ [ $output -eq "401" ] || echo "Authentication is not enabled"

You will see "Authentication is not enabled" on the final line of output.

You get the same result if you use image.tag 6.6.6.

Observations

We don't know a lot about how the operator does its thing, but we tried querying the ZK instance for /security.json and got {} back. From inspection of the code, we think this might be an indicator that the setup-zk initContainer is not doing its job correctly.

Here's what we see in the solr-operator logs:

2021-11-09T23:55:14.703Z	INFO	controllers.SolrCloud	Updating SolrCloud Status	{"namespace": "default", "solrCloud": "solr-ae12422a8985749e", "status": {"solrNodes":[],"replicas":0,"podSelector":"solr-cloud=solr-ae12422a8985749e,technology=solr-cloud","readyReplicas":0,"upToDateNodes":0,"version":"6.6.6","internalCommonAddress":"http://solr-ae12422a8985749e-solrcloud-common.default","externalCommonAddress":"http://default-solr-ae12422a8985749e-solrcloud.ing.local.domain","zookeeperConnectionInfo":{"internalConnectionString":"solr-ae12422a8985749e-solrcloud-zookeeper-0.solr-ae12422a8985749e-solrcloud-zookeeper-headless.default.svc.cluster.local:2181,solr-ae12422a8985749e-solrcloud-zookeeper-1.solr-ae12422a8985749e-solrcloud-zookeeper-headless.default.svc.cluster.local:2181,solr-ae12422a8985749e-solrcloud-zookeeper-2.solr-ae12422a8985749e-solrcloud-zookeeper-headless.default.svc.cluster.local:2181","chroot":"/"},"backupRestoreReady":false}}
2021-11-09T23:55:14.721Z	ERROR	controller	Reconciler error	{"reconcilerGroup": "solr.apache.org", "reconcilerKind": "SolrCloud", "controller": "solrcloud", "name": "solr-ae12422a8985749e", "namespace": "default", "error": "Operation cannot be fulfilled on solrclouds.solr.apache.org \"solr-ae12422a8985749e\": the object has been modified; please apply your changes to the latest version and try again"}
github.com/go-logr/zapr.(*zapLogger).Error
	/go/pkg/mod/github.com/go-logr/zapr@v0.2.0/zapr.go:132
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.5/pkg/internal/controller/controller.go:246
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.5/pkg/internal/controller/controller.go:218
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.5/pkg/internal/controller/controller.go:197
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
	/go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
	/go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
	/go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.Until
	/go/pkg/mod/k8s.io/apimachinery@v0.20.4/pkg/util/wait/wait.go:90
2021-11-09T23:55:15.725Z	INFO	controllers.SolrCloud	Updating SolrCloud Status	{"namespace": "default", "solrCloud": "solr-ae12422a8985749e", "status": {"solrNodes":[{"name":"solr-ae12422a8985749e-solrcloud-0","nodeName":"datagov-broker-test-worker3","internalAddress":"http://solr-ae12422a8985749e-solrcloud-0.default","externalAddress":"http://default-solr-ae12422a8985749e-solrcloud-0.ing.local.domain","ready":false,"version":"6.6.6","specUpToDate":true}],"replicas":1,"podSelector":"solr-cloud=solr-ae12422a8985749e,technology=solr-cloud","readyReplicas":0,"upToDateNodes":1,"version":"6.6.6","internalCommonAddress":"http://solr-ae12422a8985749e-solrcloud-common.default","externalCommonAddress":"http://default-solr-ae12422a8985749e-solrcloud.ing.local.domain","zookeeperConnectionInfo":{"internalConnectionString":"solr-ae12422a8985749e-solrcloud-zookeeper-0.solr-ae12422a8985749e-solrcloud-zookeeper-headless.default.svc.cluster.local:2181,solr-ae12422a8985749e-solrcloud-zookeeper-1.solr-ae12422a8985749e-solrcloud-zookeeper-headless.default.svc.cluster.local:2181,solr-ae12422a8985749e-solrcloud-zookeeper-2.solr-ae12422a8985749e-solrcloud-zookeeper-headless.default.svc.cluster.local:2181","chroot":"/"},"backupRestoreReady":false}}
2021-11-09T23:55:15.754Z	DEBUG	controller	Successfully Reconciled	{"reconcilerGroup": "solr.apache.org", "reconcilerKind": "SolrCloud", "controller": "solrcloud", "name": "solr-ae12422a8985749e", "namespace": "default"}
2021-11-09T23:55:15.755Z	DEBUG	controller	Successfully Reconciled	{"reconcilerGroup": "solr.apache.org", "reconcilerKind": "SolrCloud", "controller": "solrcloud", "name": "solr-ae12422a8985749e", "namespace": "default"}
2021-11-09T23:55:15.811Z	DEBUG	controller	Successfully Reconciled	{"reconcilerGroup": "solr.apache.org", "reconcilerKind": "SolrCloud", "controller": "solrcloud", "name": "solr-ae12422a8985749e", "namespace": "default"}
2021-11-09T23:55:15.855Z	INFO	controllers.SolrCloud	Updating SolrCloud Status	{"namespace": "default", "solrCloud": "solr-ae12422a8985749e", "status": {"solrNodes":[{"name":"solr-ae12422a8985749e-solrcloud-0","nodeName":"datagov-broker-test-worker3","internalAddress":"http://solr-ae12422a8985749e-solrcloud-0.default","externalAddress":"http://default-solr-ae12422a8985749e-solrcloud-0.ing.local.domain","ready":false,"version":"6.6.6","specUpToDate":true}],"replicas":1,"podSelector":"solr-cloud=solr-ae12422a8985749e,technology=solr-cloud","readyReplicas":0,"upToDateNodes":1,"version":"6.6.6","internalCommonAddress":"http://solr-ae12422a8985749e-solrcloud-common.default","externalCommonAddress":"http://default-solr-ae12422a8985749e-solrcloud.ing.local.domain","zookeeperConnectionInfo":{"internalConnectionString":"solr-ae12422a8985749e-solrcloud-zookeeper-0.solr-ae12422a8985749e-solrcloud-zookeeper-headless.default.svc.cluster.local:2181,solr-ae12422a8985749e-solrcloud-zookeeper-1.solr-ae12422a8985749e-solrcloud-zookeeper-headless.default.svc.cluster.local:2181,solr-ae12422a8985749e-solrcloud-zookeeper-2.solr-ae12422a8985749e-solrcloud-zookeeper-headless.default.svc.cluster.local:2181","externalConnectionString":"N/A","chroot":"/"},"backupRestoreReady":false}}

[HoustonPutman]

Sorry about this.

At this point Solr 6.6 is over 4 years old and it's very hard to keep compatibility between 3-4 major versions of Solr.
I think the best task for us to do, especially a day or so before the next release, is to make the documentation more clear and give better expectations about version compatibility. We already have a Version Compatibility Matrix that shows 6.6 is in a grey area of probably working for a lot of tasks, but not necessarily supported for everything.

I've introduced a PR to point this matrix out instead of saying that all Docker images work. It also tempers expectations that not all Solr Operator functions are guaranteed to work for older versions.

Anyways, I would recommend upgrading to Solr 7.x (or even 8.x, but that might be more of a stretch). CKAN requires 6.6, but since you are managing how Solr itself runs, is it ok to use 7.7 instead? Is CKAN just updating/querying, or is it adding configSets, and doing other admin tasks? If it's just using Solr as a black box (updating/querying), then the APIs are almost identical (if not identical) between 6.6 and 7.7. If it is actually doing configSet management, then yeah you might be stuck.

[mogul]

Thanks for the quick response and definitive answer. We figured it was only a matter of time before CKAN's dependence on 6.6 became a showstopping liability.

Thanks also for the suggestion to examining the compatibility restriction more closely. We are considering our options.