Configuring cluster to increase the max number of files that can be open (needed by Spark during large shuffles)

[jbherman]

i saw you encountered this problem during big shuffles. here is how i fixed it. hope it helps.
Note: this is on amazon linux.

1. copy os's sysctl.conf
2. append "fs.file-max = 100000" to sysctl.conf
3. copy-file bigCluster /Users/jason/projects-misc/sysctl.conf /home/ec2-user/sysctl.conf
4. run-command bigCluster "sudo cp /home/ec2-user/sysctl.conf /etc/sysctl.conf"
5. run-command bigCluster "sudo shutdown -r now"
6. cat < limits.conf

• soft nproc 65535
• hard nproc 65535
• soft nofile 65535
• hard no file 65535
  EOF
  7) copy-file bigCluster /Users/jason/projects-misc/limits.conf /home/ec2-user/limits.conf
  8)run-command bigCluster "sudo cp /home/ec2-user/limits.conf /etc/security/limits.conf"
  9)run-command bigCluster "sudo shutdown -r now"

thanks for your great work.
-jason

[nchammas]

Hi Jason, and thank you for the kind words!

Is this in reference to this comment?

[jbherman]

yes it's a reference to that comment