Skip to content

Commit

Permalink
perf: skip walking unnecessary dirs when reading previous generation
Browse files Browse the repository at this point in the history
Noticed an error like this after running truss with a flattened service
path.
```
truss --svcout . service.proto
FATA[0000] cannot parse input: cannot read previously generated files: cannot fully walk directory /home/zaq/projects/go-mod/src/github.com/<redacted>: cannot read file: /home/zaq/projects/go-mod/src/github.com/<redacted>/.git/refs/remotes/origin/<redacted>: open /home/zaq/projects/go-mod/src/github.com/<redacted>/.git/refs/remotes/origin/<some_branch>: too many open files
```
NOTE: was using default nofiles limit on linux
```
$ ulimit -Sn
1024
```
When using `--svcout .`, `cmd`, `svc`, and `handlers` folder will
be created in the current directory. Which is really nice if you want a
flatter service structure. However, this has the unintented consequence
of opening significantly more files when truss is building context on
it's previous generation. This change cuts down the files read to only
those in the handlers dir as these are the only files whose previous
generation is leveraged

Ref: `gengokit/generator/gen.go:generateResponseFile`
  • Loading branch information
zaquestion committed Aug 18, 2020
1 parent c48ffb9 commit 2941cff
Showing 1 changed file with 10 additions and 2 deletions.
12 changes: 10 additions & 2 deletions cmd/truss/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -359,15 +359,23 @@ func readPreviousGeneration(serviceDir string) (map[string]io.Reader, error) {
return nil, nil
}

const handlersDirName = "handlers"
files := make(map[string]io.Reader)

addFileToFiles := func(path string, info os.FileInfo, err error) error {
if info.IsDir() {
return nil
switch info.Name() {
// Only files within the handlers dir are used to
// support regeneration.
// See `gengokit/generator/gen.go:generateResponseFile`
case filepath.Base(serviceDir), handlersDirName:
return nil
default:
return filepath.SkipDir
}
}

file, ioErr := os.Open(path)

if ioErr != nil {
return errors.Wrapf(ioErr, "cannot read file: %v", path)
}
Expand Down

0 comments on commit 2941cff

Please sign in to comment.