Backport of Docs for dataplane upgrade on k8s into release/1.16.x (#1…

…8105)

* backport of commit 0d7bee8

* backport of commit 408cbe8

* backport of commit a085478

* backport of commit 71c4c65

* backport of commit 0c060fa

---------

Co-authored-by: Luke Kysow <1034429+lkysow@users.noreply.github.com>
Co-authored-by: David Yu <dyu@hashicorp.com>

website/content/docs/k8s/upgrade/index.mdx

@@ -219,25 +219,23 @@ In earlier versions, Consul on Kubernetes used client agents in its deployments.
 
 If you upgrade Consul from a version that uses client agents to a version the uses dataplanes, complete the following steps to upgrade your deployment safely and without downtime.
 
-1. Before you upgrade, edit your Helm chart configuration to enable Consul client agents by setting `client.enabled` and `client.updateStrategy`:
+1. If ACLs are enabled, you must first upgrade to consul-k8s 0.49.8 or above. These versions expose the setting `connectInject.prepareDataplanesUpgrade`
+   which is required for no-downtime upgrades when ACLs are enabled.
+
+   Set `connectInject.prepareDataplanesUpgrade` to `true` and then perform the upgrade to 0.49.8 or above (whichever is the latest in the 0.49.x series)
 
   ```yaml filename="values.yaml"
-  client:
-      enabled: true
-      updateStrategy: |
-        type: OnDelete
+  connectInject:
+    prepareDataplanesUpgrade: true
   ```
 
-1. Update the `connect-injector` to not log out on restart
-to make sure that the ACL tokens used by existing services are still valid during the migration to `consul-dataplane`.
-Note that you must remove the token manually after completing the migration.
+1. Consul dataplanes disables Consul clients by default, but during an upgrade you need to ensure Consul clients continue to run. Edit your Helm chart configuration and set the [`client.enabled`](/consul/docs/k8s/helm#v-client-enabled) field to `true` and specify an action for Consul to take during the upgrade process in the [`client.updateStrategy`](/consul/docs/k8s/helm#v-client-updatestrategy) field:
 
-  The following command triggers the deployment rollout. Wait for the rollout to complete before proceeding to next step.
-
-  ```bash
-  kubectl config set-context --current --namespace=<consul installation namespace>
-  INJECTOR_DEPLOYMENT=$(kubectl get deploy -l "component=connect-injector" -o=jsonpath='{.items[0].metadata.name}')
-  kubectl patch deploy $INJECTOR_DEPLOYMENT --type='json' -p='[{"op": "remove", "path": "/spec/template/spec/containers/0/lifecycle"}]'
+  ```yaml filename="values.yaml"
+  client:
+    enabled: true
+    updateStrategy: |
+      type: OnDelete
   ```
 
 1. Follow our [recommended procedures to upgrade servers](#upgrade-consul-servers) on Kubernetes deployments to upgrade Helm values for the new version of Consul. The latest version of consul-k8s components may be in a CrashLoopBackoff state during the performance of the server upgrade from versions <1.14.x until all Consul servers are on versions >=1.14.x. Components in CrashLoopBackoff will not negatively affect the cluster because older versioned components will still be operating. Once all servers have been fully upgraded, the latest consul-k8s components will automatically restore from CrashLoopBackoff and older component versions will be spun down.
@@ -246,7 +244,14 @@ Note that you must remove the token manually after completing the migration.
 
 1. Restart all gateways in your service mesh.
 
-1. Disable client agents in your Helm chart by deleting the `client` stanza or setting `client.enabled` to `false` and running a `consul-k8s` or Helm upgrade.
+1. Now that all services and gateways are using Consul dataplanes, disable client agents in your Helm chart by deleting the `client` stanza or setting `client.enabled` to `false` and running a `consul-k8s` or Helm upgrade.
+
+1. If ACLs are enabled, outdated ACL tokens will persist a result of the upgrade. You can manually delete the tokens to declutter your Consul environment.
+
+   Outdated connect-injector tokens have the following description: `token created via login: {"component":"connect-injector"}`. Do not delete
+   the tokens that have a description where `pod` is a key, for example `token created via login: {"component":"connect-injector","pod":"default/consul-connect-injector-576b65747c-9547x"}`). The dataplane-enabled connect inject pods use these tokens.    
+
+   You can also review the creation date for the tokens and only delete the injector tokens created before your upgrade, but do not delete all old tokens without considering if they are still in use. Some tokens, such as the server tokens, are still necessary.
 
 ## Configuring TLS on an existing cluster