Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cmd/compile: too many open files #21621

Closed
caglar10ur opened this issue Aug 25, 2017 · 20 comments
Closed

cmd/compile: too many open files #21621

caglar10ur opened this issue Aug 25, 2017 · 20 comments
Labels
CherryPickApproved Used during the release process for point releases FrozenDueToAge release-blocker
Milestone
Go1.9.2

Comments

@caglar10ur
Copy link
Contributor

caglar10ur commented Aug 25, 2017 •
edited
Loading

What version of Go are you using (go version)?

1.9

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/opt/go"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build185756508=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"

What did you do?

  • clone github.com/caglar10ur/vic
  • since the original repository is under vmware org., rename the parent directory as vmware (mv $GOPATH/src/github.com/caglar10ur $GOPATH/src/github.com/vmware)
  • switch to go19 branch
  • run sudo -E make docker-engine-api

Please note the make target install swagger binary on your $GOPATH/bin so if you don't want that please use a vm/container or a temporary GOPATH

What did you expect to see?

No errors

What did you see instead?

Build fails with "too many open files", system uses default limits provided by ubuntu 16.04.

Our build step contains huge number of autogenerated files. Example; https://10ur.org/golang19.txt

[vagrant@devbox:/opt/go/src/github.com/vmware/vic(go19)] sudo -E make docker-engine-api
Generating dependency set for cmd/imagec/
Generating dependency set for cmd/vic-machine/
Generating dependency set for cmd/vicadmin/
Generating dependency set for cmd/port-layer-server/
Generating dependency set for cmd/vic-dns/
Generating dependency set for cmd/tether/
Generating dependency set for cmd/vic-ui/
Generating dependency set for cmd/docker/
Generating dependency set for cmd/vic-init/
Generating dependency set for cmd/rpctool/
Generating dependency set for cmd/gandalf/
regenerating swagger models and operations for Portlayer API client...
done regenerating swagger models and operations for Portlayer API client...
regenerating swagger models and operations for Admiral API client...
done regenerating swagger models and operations for Admiral API client...
Building docker-engine-api server...
# github.com/vmware/vic/lib/config/dynamic/admiral/client/operations
lib/config/dynamic/admiral/client/operations/patch_resources_container_control_loop_id_parameters.go:0:0: open lib/config/dynamic/admiral/client/operations/patch_resources_container_control_loop_id_parameters.go: too many open files
Makefile:316: recipe for target 'bin/docker-engine-server' failed
make: *** [bin/docker-engine-server] Error 2

[vagrant@devbox:/opt/go/src/github.com/vmware/vic(go19)] ulimit -n
1024

Bumping the number of open files limit to 2048 makes error go away.

[vagrant@devbox:/opt/go/src/github.com/vmware/vic(go19)] ulimit -n 2048

[vagrant@devbox:/opt/go/src/github.com/vmware/vic(go19)] ulimit -n
2048

[vagrant@devbox:/opt/go/src/github.com/vmware/vic(go19)] sudo -E make docker-engine-api
Generating dependency set for cmd/imagec/
Generating dependency set for cmd/vic-machine/
Generating dependency set for cmd/vicadmin/
Generating dependency set for cmd/port-layer-server/
Generating dependency set for cmd/vic-dns/
Generating dependency set for cmd/tether/
Generating dependency set for cmd/vic-ui/
Generating dependency set for cmd/docker/
Generating dependency set for cmd/vic-init/
Generating dependency set for cmd/rpctool/
Generating dependency set for cmd/gandalf/
regenerating swagger models and operations for Portlayer API client...
done regenerating swagger models and operations for Portlayer API client...
regenerating swagger models and operations for Admiral API client...
done regenerating swagger models and operations for Admiral API client...
Building docker-engine-api server...
[vagrant@devbox:/opt/go/src/github.com/vmware/vic(go19)]

It works fine with go1.8
@caglar10ur caglar10ur mentioned this issue Aug 25, 2017
Closed
@ALTree
Copy link
Member

ALTree commented Aug 25, 2017

Dup of #21550 ?

@ianlancetaylor ianlancetaylor changed the title [go19] too many open files cmd/compile: too many open files Aug 25, 2017
@ianlancetaylor
Copy link
Member

Can you run the relevant go build or go install command with the -x option and show us the output? I'd like to see precisely which tool is generating the error. Thanks.

@ianlancetaylor ianlancetaylor added this to the Go1.9.1 milestone Aug 25, 2017
@caglar10ur
Copy link
Contributor Author

Sure, @ianlancetaylor please see below

...
mkdir -p $WORK/github.com/vmware/vic/lib/config/dynamic/admiral/client/resources_compute/_obj/
cd /opt/go/src/github.com/vmware/vic/lib/config/dynamic/admiral/client/resources_compute
/usr/local/go/pkg/tool/linux_amd64/compile -o $WORK/github.com/vmware/vic/lib/config/dynamic/admiral/client/resources_compute.a -trimpath $WORK -goversion go1.9 -p github.com/vmware/vic/lib/config/dynamic/admiral/client/resources_compute -complete -buildid ec1fb3ca006c02f81994384699d724ddb8479fdc -importmap github.com/go-openapi/errors=github.com/vmware/vic/vendor/github.com/go-openapi/errors -importmap github.com/go-openapi
/runtime=github.com/vmware/vic/vendor/github.com/go-openapi/runtime -importmap github.com/go-openapi/runtime/client=github.com/vmware/vic/vendor/github.com/go-openapi/runtime/client -importmap github.com/go-openapi/strfmt=github.com/vmware/vic/vendor/github.com/go-openapi/strfmt -importmap golang.org/x/net/context=github.com/vmware/vic/vendor/golang.org/x/net/context -D _/opt/go/src/github.com/vmware/vic/lib/config/dynamic/a
dmiral/client/resources_compute -I $WORK -I /opt/go/pkg/linux_amd64 -pack ./get_resources_compute_id_parameters.go ./get_resources_compute_id_responses.go ./get_resources_compute_parameters.go ./get_resources_compute_responses.go ./patch_resources_compute_id_parameters.go ./patch_resources_compute_id_responses.go ./post_resources_compute_id_parameters.go ./post_resources_compute_id_responses.go ./post_resources_compute_param
eters.go ./post_resources_compute_responses.go ./put_resources_compute_id_parameters.go ./put_resources_compute_id_responses.go ./resources_compute_client.go
# github.com/vmware/vic/lib/config/dynamic/admiral/client/operations
lib/config/dynamic/admiral/client/operations/patch_config_event_topic_id_parameters.go:0:0: open lib/config/dynamic/admiral/client/operations/patch_config_event_topic_id_parameters.go: too many open files
...

The full log can be found at https://10ur.org/build.log

@valyala valyala mentioned this issue Aug 27, 2017
Closed
@davecheney
Copy link
Contributor

davecheney commented Aug 27, 2017 via email

@davecheney
Copy link
Contributor

davecheney commented Aug 27, 2017 via email

@caglar10ur
Copy link
Contributor Author

caglar10ur commented Aug 27, 2017 •
edited
Loading

@davecheney ah yeah, the original repo is under vmware not under caglar10ur :/ An ugly workaround like following should do trick;

mv /home/dfc/src/github.com/caglar10ur/ /home/dfc/src/github.com/vmware/

Edit: Updated the original report to reflect this.

@davecheney
Copy link
Contributor

davecheney commented Aug 27, 2017 via email

@caglar10ur
Copy link
Contributor Author

Unfortunately go build -x -i ./cmd/docker will work only if you run swagger to generate those files before, that's why I suggested using make target as it does that for you.

One thing that I realized just now is, the make target also tries to install swagger on your GOPATH so if you don't want that then I suggest using a vm/container or a temporary GOPATH

@caglar10ur
Copy link
Contributor Author

@davecheney the same set of files are also visible on the build.log that I provided (#21621 (comment)) and unless I made a mistake to extract them via grep, I believe we are talking about 1834 files.

[conur@vir:~/Documents] curl https://10ur.org/build.log 2> /dev/null | grep put_resources_sub_network | grep pack | awk -F"-pack" '{print $2}' | gsed -e 's/\s\+/\n/g' | wc -l
    1834

@davecheney
Copy link
Contributor

davecheney commented Aug 28, 2017 via email

@dcheney-atlassian
Copy link

Here is a reproduction which fails reliably with go 1.9 and appears to pass reliably in go 1.8
bug.tar.gz

deadwood(~/src/github.com/davecheney/bug) % go build .
# github.com/davecheney/bug
./alec.go:0:0: open ./alec.go: too many open files
deadwood(~/src/github.com/davecheney/bug) % go build .
# github.com/davecheney/bug
./apology.go:0:0: open ./apology.go: too many open files
deadwood(~/src/github.com/davecheney/bug) % go build .
# github.com/davecheney/bug
./adjourning.go:0:0: open ./adjourning.go: too many open files
deadwood(~/src/github.com/davecheney/bug) % go1.8  build .
deadwood(~/src/github.com/davecheney/bug) % go1.8  build .
deadwood(~/src/github.com/davecheney/bug) % go1.8 build .
deadwood(~/src/github.com/davecheney/bug) %

@valyala
Copy link
Contributor

valyala commented Aug 29, 2017 •
edited
Loading

The odd thing is I couldn't trigger the error consistently. I could trigger
it enough to convince myself that there is a regression, but I'm wondering
why sometimes the build works, and other times it fails for that package.
It would appear the issue is more complicated than just opening all the
files passed to the compiler at once, then processing them.

This is a simple race between concurrently running goroutines that parse distinct files. Sometimes they open too many files (I suppose the probability increases on cold file cache), sometimes they manage to parse and close old files before other goroutines start opening new files.

Try the patch https://go-review.googlesource.com/c/go/+/57751 that limits the number of concurrently open files during package parsing to GOMAXPROCS.

@mvdan
Copy link
Member

mvdan commented Aug 29, 2017

Should we perhaps close this in favor of #21550, or close that one in favor of the discussion here? Seems to me like they're both about the same underlying issue.

@ianlancetaylor ianlancetaylor mentioned this issue Aug 29, 2017
Closed
@ianlancetaylor
Copy link
Member

I think https://golang.org/cl/57751 is the kind of CL we should try to get into the 1.9.1 release.

@davecheney You have a -2 on that CL pending discussion; are you OK with dropping that?

@gopherbot
Copy link
Contributor

Change https://golang.org/cl/57751 mentions this issue: cmd/compile: Limit the number of simultaneously opened files to avoid EMFILE/ENFILE errors

@dcheney-atlassian dcheney-atlassian mentioned this issue Sep 1, 2017
Closed
@ianlancetaylor
Copy link
Member

Reopen for merge to 1.9.1.

@gopherbot
Copy link
Contributor

Change https://golang.org/cl/63752 mentions this issue: [release-branch.go1.9] cmd/compile: limit the number of simultaneously opened files to avoid EMFILE/ENFILE errors

@rsc rsc modified the milestones: Go1.9.1, Go1.9.2 Oct 4, 2017
@rsc rsc reopened this Oct 13, 2017
@rsc
Copy link
Contributor

rsc commented Oct 13, 2017

If the hope is that CL 63752 is going to fix #21713 as well, the sem <- struct{}{} should be before the goroutine creation, not inside the new goroutine. Otherwise looks good for Go 1.9.2.

@rsc
Copy link
Contributor

rsc commented Oct 13, 2017 •
edited
Loading

CL 63752 OK for Go 1.9.2. (cherry-pick, want to record original if it applies cleanly)
CL 57751 OK for Go 1.9.2.

@rsc rsc added the CherryPickApproved Used during the release process for point releases label Oct 14, 2017
@rsc rsc mentioned this issue Oct 15, 2017
Closed
gopherbot pushed a commit that referenced this issue Oct 25, 2017
@chenzhongtao @rsc
[release-branch.go1.9] cmd/compile: limit the number of simultaneousl…
5c48811 
…y opened files to avoid EMFILE/ENFILE errors

If the Go packages with enough source files,it will cause EMFILE/ENFILE error,
Fix this by limiting the number of simultaneously opened files.

Fixes #21621

Change-Id: I8555d79242d2f90771e37e073b7540fc7194a64a
Reviewed-on: https://go-review.googlesource.com/57751
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Reviewed-on: https://go-review.googlesource.com/63752
Run-TryBot: Russ Cox <rsc@golang.org>
@rsc
Copy link
Contributor

rsc commented Oct 26, 2017

go1.9.2 has been packaged and includes:

The release is posted at golang.org/dl.

— golang.org/x/build/cmd/releasebot, Oct 26 21:09:09 UTC

@rsc rsc closed this as completed Oct 26, 2017
@golang golang locked and limited conversation to collaborators Oct 26, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
CherryPickApproved Used during the release process for point releases FrozenDueToAge release-blocker
Projects
None yet
Milestone
Go1.9.2
Development

No branches or pull requests
9 participants
@davecheney @rsc @caglar10ur @valyala @ianlancetaylor @mvdan @ALTree @gopherbot @dcheney-atlassian