Set LimitNOFILE to 1048576 instead of infinity

[dims]

When we set LimitNOFILE to infinity things break .. for example the nfs based tests in kubernetes harness:
https://testgrid.k8s.io/amazon-ec2-al2023#ci-kubernetes-e2e-al2023-aws-canary&width=20

Also see containerd/containerd#9660 which points to more things that are likely to break with infinity containerd/containerd#8924 (comment)

[zetaab]

/lgtm

infinity should be same as 1048576

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: zetaab

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [zetaab]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[dims]

@zetaab thank you, back on its feet https://testgrid.k8s.io/sig-cluster-lifecycle-kops#ci-kubernetes-e2e-al2023-aws-canary&width=20

[zetaab]

I executed prometheus query max(max_over_time(process_open_fds[2w])) to all of our clusters and at least in our case 1048576 is going to be enough. However, I am not 100% sure is this the case in case of all kOps users. Our maximum is currently ~44k

[polarathene]

However, I am not 100% sure is this the case in case of all kOps users.

You honestly shouldn't need to do this if software affected properly raised their soft limit at runtime.

• The default soft limit should be 1024 for compatibility. It's per process, so the FD count is not cumulative.
• A hard limit could be set high and should be ok, it just defines the ceiling for how high a soft limit may be raised.
• I know some enterprise grade deployments have workloads that exceed 2^20 (1048576) limit, but if software relies on the soft limit being raised for it instead when it needs this much, it's negatively impacting other software in the container that regresses.

I encourage you to identify the software that hits those issues and push for them to fix it on their end. It's a couple lines to raise the soft limit to the max hard limit. It's not a difficult request to upstream. Why projects like Envoy are refusing to do this is beyond me.