Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

US-436846: BackingServices>Elasticsearch cluster-Encrypt internode communication enabled #337

Merged
merged 16 commits into from
Oct 7, 2021
Merged
Show file tree
Hide file tree
Changes from 12 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 24 additions & 0 deletions charts/backingservices/Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
default: es-prerequisite

NAMESPACE := <YOUR_NAMESPACE>
ELASTICSEARCH_IMAGE := docker.elastic.co/elasticsearch/elasticsearch:7.9.3

es-prerequisite: es-secrets

es-secrets:
docker run --name elastic-helm-charts-certs -i -w /tmp \
$(ELASTICSEARCH_IMAGE) \
/bin/sh -c " \
elasticsearch-certutil ca --out /tmp/elastic-stack-ca.p12 --pass '' && \
elasticsearch-certutil cert --name security-master --dns security-master --ca /tmp/elastic-stack-ca.p12 --pass '' --ca-pass '' --out /tmp/elastic-certificates.p12" && \
docker cp elastic-helm-charts-certs:/tmp/elastic-certificates.p12 ./ && \
docker rm -f elastic-helm-charts-certs && \
openssl pkcs12 -nodes -passin pass:'' -in elastic-certificates.p12 -out elastic-certificate.pem && \
openssl x509 -outform der -in elastic-certificate.pem -out elastic-certificate.crt && \
kubectl create secret generic elastic-certificates --from-file=elastic-certificates.p12 --namespace=$(NAMESPACE) && \
kubectl create secret generic elastic-certificate-pem --from-file=elastic-certificate.pem --namespace=$(NAMESPACE) && \
kubectl create secret generic elastic-certificate-crt --from-file=elastic-certificate.crt --namespace=$(NAMESPACE)
rm -f elastic-certificates.p12 elastic-certificate.pem elastic-certificate.crt elastic-stack-ca.p12

purge-es-secrets:
kubectl delete secrets elastic-certificates elastic-certificate-pem elastic-certificate-crt --namespace=$(NAMESPACE) || true
4 changes: 2 additions & 2 deletions charts/backingservices/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ The service deployment provisions runtime service pods along with a dependency o
Pega Infinity version | SRS version | ElasticSearch version | Description
--- | --- | --- | ---
< 8.6 | NA | NA | SRS can be used with Pega Infinity 8.6 and later
\>= 8.6 | \>= 1.12.0 | 7.9.3 | Pega Infinity 8.6 and later supports using a Pega-provided platform-services/search-n-reporting-service Docker Image that is tagged with version 1.12.0 and later. Current SRS versions are certified to support Elasticsearch version 7.9.3.
\>= 8.6 | \>=1.12.0 \<=1.16.1 | 7.9.3 | Pega Infinity 8.6 and later supports using a Pega-provided platform-services/search-n-reporting-service Docker Image that is tagged with version 1.12.0 or later (Pega recommends using 1.16.1 or later). All SRS Docker image versions are certified to support Elasticsearch version 7.9.3.

### SRS runtime configuration

Expand All @@ -42,7 +42,7 @@ Configuration | Usage
`deploymentName` | Specify the name of your SRS cluster. Your deployment creates resources prefixed with this string. This is also the service name for the SRS.
`srsRuntime` | Use this section to define specific resource configuration options like image, replica count, cpu and memory resource settings in the SRS.
`elasticsearch` | Define the elasticsearch cluster configurations. The [Elasticsearch](https://github.com/helm/charts/tree/master/stable/elasticsearch/values.yaml) chart defines the values for elasticsearch provisioning in the cluster.
`srsStorage.provisionInternalESCluster` | <ul><li>Enable this parameter to provision an internally managed and secured Elasticsearch cluster to be used with SRS.</li><li>Disable this parameter to use your own Elasticsearch service with SRS. Use the uncommented section to specify the connection details for your existing external ElasticSearch cluster that is available to your deployment. For Elasticsearch clusters secured with Basic Authentication, use the commented section 'srsStorage.basicAuthentication' to provide access credentials. For AWS Elasticsearch clusters secured with IAM role based authentication, use 'srsStorage.awsIAM' to set the aws region where AWS Elasticsearch cluster is hosted.</li></ul>
`srsStorage.provisionInternalESCluster` | <ul><li>Enable this parameter to provision an internally managed and secured Elasticsearch cluster to be used with the SRS (Requires you to download the `Makefile` file from the [repository](https://github.com/pegasystems/pega-helm-charts) (path from root : `pega-helm-charts/charts/backingservices/Makefile`), edit the file to replace `<YOUR_NAMESPACE>` with `namespace` used for the deployment, and then run `$ make es-prerequisite`).</li><li>Disable this parameter to use your own Elasticsearch service with the SRS.</li></ul>

Example:

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -69,4 +69,31 @@ spec:
value: "{{ .Values.srsRuntime.env.PublicKeyURL }}"
resources:
{{- toYaml .Values.srsRuntime.resources | nindent 12 }}
{{ end }}
{{- if .Values.srsStorage.provisionInternalESCluster }}
initContainers:
- name: wait-for-internal-es-cluster
image: busybox:1.31.0
Anand-Kotamsetti marked this conversation as resolved.
Show resolved Hide resolved
args:
- until $(wget -q -S --spider --timeout=2 -O /dev/null $ELASTICSEARCH_PROTO://$ELASTICSEARCH_USERNAME:$ELASTICSEARCH_PASSWORD@$ELASTICSEARCH_HOST:$ELASTICSEARCH_PORT ); do echo Waiting for Elasticsearch cluster to become live...; sleep 10; done;
command:
- sh
- -c
env:
- name: ELASTICSEARCH_PROTO
value: {{ template "elasticsearch.protocol" . }}
- name: ELASTICSEARCH_HOST
value: {{ template "elasticsearch.domain" . }}
- name: ELASTICSEARCH_PORT
value: {{ template "elasticsearch.port" . }}
- name: ELASTICSEARCH_USERNAME
valueFrom:
secretKeyRef:
name: srs-elastic-credentials
key: username
- name: ELASTICSEARCH_PASSWORD
valueFrom:
secretKeyRef:
name: srs-elastic-credentials
key: password
{{ end }}
{{ end }}
19 changes: 18 additions & 1 deletion charts/backingservices/values.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -58,11 +58,14 @@ elasticsearch:
esJavaOpts: "-Xmx1024m -Xms1024m"
# Allocate smaller chunks of memory per pod.
protocol: http
# This section specifies the elasticsearch cluster configuration for authentication.
# This section specifies the elasticsearch cluster configuration for authentication and TLS.
esConfig:
elasticsearch.yml: |
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: /usr/share/elasticsearch/config/certs/elastic-certificates.p12
# Use this section to include additional, supported environmental variables for Elasticsearch basic authentication.
# The parameter values can be read from a specified secrets file.
extraEnvs:
Expand All @@ -89,3 +92,17 @@ elasticsearch:
requests:
# configure volume size of the elasticsearch nodes based on search data storage requirements. The default storage size from elasticsearch is 30Gi.
storage: 30Gi
# elasticsearch.secretMounts will help reading certificates from elastic-certificates secret.
secretMounts:
- name: elastic-certificates
secretName: elastic-certificates
path: /usr/share/elasticsearch/config/certs
# For Openshift deployments, you must enable the following custom values. For details
# refer to https://github.com/elastic/helm-charts/tree/master/elasticsearch/examples/openshift.
# securityContext:
# runAsUser: null
# podSecurityContext:
# fsGroup: null
# runAsUser: null
# sysctlInitContainer:
# enabled: false
3 changes: 1 addition & 2 deletions charts/pega/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -640,8 +640,7 @@ dds:

## Search deployment

Use the `pegasearch` section to configure a deployment of ElasticSearch for searching Rules and Work within Pega. This deployment is used exclusively for Pega search, and is not the same ElasticSearch deployment used by the EFK stack or any other dedicated service such as Pega BI.

Use the `pegasearch` section to configure the source ElasticSearch service that the Pega Platform deployment uses for searching Rules and Work within Pega. The ElasticSearch service defined here is not related to the ElasticSearch deployment if you also define an EFK stack for logging and monitoring in your Pega Platform deployment.
### For Pega Platform 8.6 and later:

Pega recommends using the chart ['backingservices'](../backingservices) to enable Pega Infinity backing service and to deploy the latest generation of search and reporting capabilities to your Pega applications that run independently on nodes provisioned exclusively to run these services.
Expand Down
28 changes: 7 additions & 21 deletions docs/Deploying-Pega-on-AKS.md
Original file line number Diff line number Diff line change
Expand Up @@ -421,11 +421,14 @@ To configure the parameters in the backingservices.yaml file, download the file
| global.imageCredentials.registry: username: password: | Include the URL of your Docker registry along with the registry “username” and “password” credentials. | <ul><li>url: “\<URL of your registry>” </li><li>username: "\<Registry account username\>"</li><li> password: "\<Registry account password\>"</li></ul> |
| srs.deploymentName: | Specify unique name for the deployment based on org app and/or srs applicable environment name. | deploymentName: "acme-demo-dev-srs" |
| srs.srsRuntime.srsImage: | Specify the Pega-provided srs-service image, services/search-n-reporting-service:dockerTag that you downloaded and pushed to your Docker registry. | srs.srsRuntime.srsImage: "platform-services/search-n-reporting-service:<srs-version>". For `<srs-version>` tag (refer [compatibility matrix](../charts/backingservices/README.md#srs-version-compatibility-matrix)) |
| srs.srsStorage.provisionInternalESCluster: | Enabled by default to provision an Elasticsearch cluster. | <ul><li>Set srs.srsStorage.provisionInternalESCluster:`true` , download `Makefile` file from the [repository](https://github.com/pegasystems/pega-helm-charts) ( path from root : `pega-helm-charts/charts/backingservices/Makefile`), replace `<YOUR_NAMESPACE>` with the `namespace` you used for the deployment, and then run `$ make es-prerequisite `</li><li>Set srs.srsStorage.provisionInternalESCluster:`false` if you want to use an existing, externally provisioned ElasticSearch cluster. </li></ul> |
Anand-Kotamsetti marked this conversation as resolved.
Show resolved Hide resolved
| srs.srsStorage.domain: port: protocol: basicAuthentication: awsIAM: requireInternetAccess: | Disabled by default. Enable only when srs.srsStorage.provisionInternalESCluster is false and you want to configure SRS to use an existing, externally provisioned Elasticsearch cluster. For an Elasticsearch cluster secured with Basic Authentication, use `srs.srsStorage.basicAuthentication` section to provide access credentials. For an AWS Elasticsearch cluster secured with IAM role based authentication, use `srs.srsStorage.awsIAM` section to set the aws region where AWS Elasticsearch cluster is hosted. For unsecured managed ElasticSearch cluster do not configure these options. | <ul><li>srs.srsStorage.domain: "\<external-es domain name\>"</li> <li>srs.srsStorage.port: "\<external es port\>"</li> <li>srs.srsStorage.protocol: "\<external es http protocol, `http` or `https`\>"</li> <li>srs.srsStorage.basicAuthentication.username: "\<external es `basic Authentication username`\>"</li> <li>srs.srsStorage.basicAuthentication.password: "\<external es `basic Authentication password`\>"</li> <li>srs.srsStorage.awsIAM.region: "\<external AWS es cluster hosted `region`\>"</li><li> srs.srsStorage.requireInternetAccess: "\<set to `true` if you host your external Elasticsearch cluster outside of the current network and the deployment must access it over the internet.\>"</li></ul> |
| elasticsearch: volumeClaimTemplate: resources: requests: storage: | Specify the Elasticsearch cluster disk volume size. Default is 30Gi, set this value to at least three times the size of your estimated search data size | <ul><li>elasticsearch: volumeClaimTemplate: resources: requests: storage: "\<30Gi>” </li></ul> |

3. Save the file.

4. To use an internal Elasticsearch cluster (srs.srsStorage.requireInternetAccess:true) for your deployment, you must download the Makefile file from the repository (path from root : pega-helm-charts/charts/backingservices/Makefile), replace `<YOUR_NAMESPACE>` with the namespace you used for the deployment, and then run `$ make es-prerequisite`.

#### Add supported custom settings for Pega to your deployment

The Pega deployment model supports advanced configurations to fit most existing
Expand Down Expand Up @@ -455,6 +458,8 @@ Configure the following parameters so the pega.yaml Helm chart matches your depl

- Access your Azure SQL database.

- Access your ElasticSearch service (For 8.6 and later, Pega recommends deploying your service using an SRS cluster).

- Install the version of Pega Platform that you built into your Docker installation image.

- Specify host names for your web and stream tiers and import and use any required SSL certificates for your web tiers.
Expand All @@ -481,30 +486,11 @@ helm inspect values pega/pega > <local filepath>/aks-demo/pega.yaml
| docker.pega.image: | Specify the Pega-provided `Pega` image you downloaded and pushed to your Docker registry. | Image: "\<Registry host name:Port\>/my-pega:\<Pega Platform version>" |
| tier.name: ”web” tier.ingress.domain:| Set a host name for the pega-web service of the DNS zone. | <ul><li>domain: "\<the host name for your web service tier\>" </li><li>tier.ingress.tls: set to `true` to support HTTPS in the ingress and pass the SSL certificate in the cluster using a secret. For details, see step 12 in the section, **Deploying Pega Platform using the command line**.</li></ul>|
| tier.name: ”stream” tier.ingress.domain: | Set the host name for the pega-stream service of the DNS zone. | <ul><li>domain: "\<the host name for your stream service tier\>" </li><li>Your stream tier host name should comply with your networking standards.</li><li>tier.ingress.tls: set to `true` to support HTTPS in the ingress and pass the SSL certificate in the cluster using a secret. For details, see step 12 in the section, **Deploying Pega Platform using the command line**.</li><li>To remove the exposure of a stream from external network traffic, delete the `service` and `ingress` blocks in the tier.</li></ul>|
| pegasearch.image: | Specify the Pega-provided Docker `search` image that you downloaded and pushed to your Docker registry. | Image: "\<Registry host name:Port>/my-pega-search:\<Pega Platform version>"
| pegasearch: | For Pega Platform 8.6 and later, Pega recommends using the chart 'backingservices' to enable Pega SRS. To use this service, you must enable its use and provide the SRS URL for your Pega Infinity deployment. | <ul><li>externalSearchService: true</li><li>externalURL: pegasearch.externalURL For example, http://srs-service.mypega-aks-demo.svc.cluster.local </li></ul>
| installer.image: | Specify the Pega-provided Docker `installer` image that you downloaded and pushed to your Docker registry. | Image: "\<Registry host name:Port>/my-pega-installer:\<Pega Platform version>" |
| installer. adminPassword: | Specify an initial administrator@pega.com password for your installation. This will need to be changed at first login. The adminPassword value cannot start with "@". | adminPassword: "\<initial password\>" |

3. [For Pega Platform 8.6 and later] Applicable only when backingservices chart is configured.

For Pega Platform 8.6 and later installations in which you are configuring the backingservices Search and Reporting Service in your deployment, use a text editor to open the `pega.yaml` and update the following parameters in the chart based on your backing service configuration.

Chart parameter name | Purpose | Your setting
--- | --- | ---
`pegasearch.externalSearchService` | Set the `pegasearch.externalSearchService` as true to use Search and Reporting service as the search functionality provider to the Pega platform | true
`pegasearch.externalURL` | Set the `pegasearch.externalURL` value to the Search and Reporting Service endpoint url | `"http://<srs.deploymentName>.<namespace>.svc.cluster.local"` or `"http://<srs.deploymentName>.<namespace>"`

Example:

When backingservice is deployed into `mypega` namespace and `pegasearch.externalSearchService` value is "srs-service", configure the `pegasearch` section in pega.yaml as below:

```yaml
pegasearch:
externalSearchService: true
externalURL: "http://srs-service.mypega.svc.cluster.local"
```

4. Save the file.
3. Save the file.

### Deploying Pega Platform using the command line

Expand Down
Loading