Updated important note point with CNI env set

[itsLucario]

Issue: aws/amazon-vpc-cni-k8s#641 (comment) #202

Description of changes:
Updated important note for users who are upgrading the CNI with Custom Networking.
If users miss changing the CNI custom networking environment variable to true in case, they are using it. It can cause the whole cluster networking go down.

This is one important instruction which has to be included as an upgrade note.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

[jimdial-aws]

I don't understand your concern. If you're using custom networking, this value is already set to true (See step 3 - https://docs.aws.amazon.com/eks/latest/userguide/cni-custom-network.html), so why is it necessary to mention it in the upgrade document?

[itsLucario]

@jimdial-aws
When we do custom networking, we manual patch the environmental variable to true.
But when we do an upgrade the manifest will have environment variable false, which cause issues if we forget to update the value to true before upgrade.

So, this is an important note which has to be considered while upgrading.
Cause we faced this issue resulted in entire cluster chaos and the cluster networking went down.

[itsLucario]

Hi, If you think this is not required. I can close the PR.

[jimdial-aws]

OK, I get it, thanks, but this warning would apply to any changes made to your CNI config, not only custom networking, so I think we should broaden the warning. So, in the step where (today) it just tells you to apply the manifest, maybe there should be an important note there that says something like "If you've made any changes to your CNI configuration from the default configuration, then download the manifest and make the same changes to it before applying it. Does that sound useful?

[jimdial-aws]

Thanks for the submission. Closing because I added an important note cautioning about needing to re-make any changes to default settings in the manifest, rather than calling out custom networking specifically.