Extended globbing support #225

Wicketd · 2019-06-29T15:27:40Z

Task version	OS
master	macOS Mojave 10.14.5

Using extglob patterns in sources entries makes Task panic. I tried using this to exclude some files from being picked up.

Given the following Taskfile.yml:

version: '2'

tasks:
  foo:
    cmds:
      - printf %s Test
    sources:
      - ./!(vendor)/**/*.go
    method: checksum

Task panics after running task foo:

panic: unhandled word part: *syntax.ExtGlob

goroutine 1 [running]:
github.com/go-task/task/vendor/mvdan.cc/sh/expand.(*Config).wordFields(0xc00019e140, 0xc00015a540, 0x2, 0x2, 0x1, 0x1, 0x203000, 0x0, 0xc0001a0010)
    /Users/thierry/go/src/github.com/go-task/task/vendor/mvdan.cc/sh/expand/expand.go:546 +0x1292
github.com/go-task/task/vendor/mvdan.cc/sh/expand.Fields(0xc00019e140, 0xc0001a0010, 0x1, 0x1, 0x0, 0x0, 0x1, 0x0, 0xc000194020)
    /Users/thierry/go/src/github.com/go-task/task/vendor/mvdan.cc/sh/expand/expand.go:323 +0x19a
github.com/go-task/task/vendor/mvdan.cc/sh/shell.Fields(0xc0001940a0, 0x11, 0x0, 0x1, 0x13852c8, 0x2, 0xffffffffffffffff, 0xc0001940a0)
    /Users/thierry/go/src/github.com/go-task/task/vendor/mvdan.cc/sh/shell/expand.go:62 +0x162
github.com/go-task/task/internal/execext.Expand(0xc0001940a0, 0x11, 0x2, 0xc0001940a0, 0x11, 0x10e1548)
    /Users/thierry/go/src/github.com/go-task/task/internal/execext/exec.go:78 +0x8e
github.com/go-task/task/internal/status.glob(0x0, 0x0, 0xc000146010, 0x1, 0x1, 0x21, 0x0, 0x2341460, 0x0, 0x0)
    /Users/thierry/go/src/github.com/go-task/task/internal/status/glob.go:17 +0xca
github.com/go-task/task/internal/status.(*Checksum).IsUpToDate(0xc000144280, 0xc00015c000, 0x13f5520, 0xc000144280)
    /Users/thierry/go/src/github.com/go-task/task/internal/status/checksum.go:30 +0x10b
github.com/go-task/task.(*Executor).isTaskUpToDate(0xc0000b34a0, 0x13f9a20, 0xc000144000, 0xc00015c000, 0x1, 0x0, 0x0)
    /Users/thierry/go/src/github.com/go-task/task/status.go:40 +0x7f
github.com/go-task/task.(*Executor).RunTask(0xc0000b34a0, 0x13f9a20, 0xc000144000, 0x7ffeefbff38d, 0x3, 0x0, 0xc0000a25a0, 0x13f5d20)
    /Users/thierry/go/src/github.com/go-task/task/task.go:218 +0x60f
github.com/go-task/task.(*Executor).Run(0xc0000b34a0, 0x13f9a20, 0xc000144000, 0xc000090ca0, 0x1, 0x1, 0x0, 0x1394802)
    /Users/thierry/go/src/github.com/go-task/task/task.go:79 +0x139
main.main()
    /Users/thierry/go/src/github.com/go-task/task/cmd/task/task.go:141 +0xa8d

The text was updated successfully, but these errors were encountered:

andreynering · 2019-06-30T13:24:58Z

Hi @Yottum,

I think this just isn't supported by https://github.com/mattn/go-zglob.

Can you check if you glob use cases work on Gosh?

You can do that by installing Gosh and running:

gosh -c "echo ./!(vendor)/**/*.go"

Or by running it in Taskfile like this:

tasks:
  glob:
    - echo ./!(vendor)/**/*.go

If it works, we could maybe replace zglob with Gosh for globbing in Task.

andreynering · 2019-07-13T19:56:42Z

Ping @Yottum

Wicketd · 2019-07-14T14:01:55Z

Woops, sorry! Had totally forgotten I opened this issue. Thanks for replying.

No avail, sadly:

$ gosh -c 'echo ./!(vendor)/**/*.go'
panic: unhandled word part: *syntax.ExtGlob

goroutine 1 [running]:
mvdan.cc/sh/v3/expand.(*Config).wordFields(0xc0000f2000, 0xc0000bc100, 0x3, 0x4, 0xc0000c4028, 0x4, 0x0, 0x0, 0x0)
	/Users/thierry/go/pkg/mod/mvdan.cc/sh/v3@v3.0.0-alpha2/expand/expand.go:584 +0x1292
mvdan.cc/sh/v3/expand.Fields(0xc0000f2000, 0xc0000daab0, 0x2, 0x4, 0x20300000000000, 0x163ffff, 0x400, 0x1447900, 0x1115b38)
	/Users/thierry/go/pkg/mod/mvdan.cc/sh/v3@v3.0.0-alpha2/expand/expand.go:373 +0x15b
mvdan.cc/sh/v3/interp.(*Runner).fields(0xc0000dc000, 0xc0000daab0, 0x2, 0x4, 0x163ffff, 0x400, 0x20300000000000)
	/Users/thierry/go/pkg/mod/mvdan.cc/sh/v3@v3.0.0-alpha2/interp/interp.go:131 +0x5a
mvdan.cc/sh/v3/interp.(*Runner).cmd(0xc0000dc000, 0x11a3a60, 0xc0000c4000, 0x11a31a0, 0xc0000daa80)
	/Users/thierry/go/pkg/mod/mvdan.cc/sh/v3@v3.0.0-alpha2/interp/interp.go:782 +0x1114
mvdan.cc/sh/v3/interp.(*Runner).stmtSync(0xc0000dc000, 0x11a3a60, 0xc0000c4000, 0xc0000e2000)
	/Users/thierry/go/pkg/mod/mvdan.cc/sh/v3@v3.0.0-alpha2/interp/interp.go:721 +0x2d9
mvdan.cc/sh/v3/interp.(*Runner).stmt(0xc0000dc000, 0x11a3a60, 0xc0000c4000, 0xc0000e2000)
	/Users/thierry/go/pkg/mod/mvdan.cc/sh/v3@v3.0.0-alpha2/interp/interp.go:702 +0x1ce
mvdan.cc/sh/v3/interp.(*Runner).stmts(0xc0000dc000, 0x11a3a60, 0xc0000c4000, 0xc0000ee000, 0x1, 0x4)
	/Users/thierry/go/pkg/mod/mvdan.cc/sh/v3@v3.0.0-alpha2/interp/interp.go:1037 +0x56
mvdan.cc/sh/v3/interp.(*Runner).Run(0xc0000dc000, 0x11a3a60, 0xc0000c4000, 0x11a1e20, 0xc0000bc0c0, 0xc0000bc0c0, 0x0)
	/Users/thierry/go/pkg/mod/mvdan.cc/sh/v3@v3.0.0-alpha2/interp/interp.go:650 +0x172
main.run(0x11a1920, 0xc0000de060, 0x0, 0x0, 0x0, 0xc000044f28)
	/Users/thierry/go/pkg/mod/mvdan.cc/sh/v3@v3.0.0-alpha2/cmd/gosh/main.go:76 +0xd8
main.runAll(0xc0000b00c0, 0xc0000c0010)
	/Users/thierry/go/pkg/mod/mvdan.cc/sh/v3@v3.0.0-alpha2/cmd/gosh/main.go:44 +0x200
main.main()
	/Users/thierry/go/pkg/mod/mvdan.cc/sh/v3@v3.0.0-alpha2/cmd/gosh/main.go:30 +0x7c

leefernandes · 2019-09-03T19:37:54Z

I'm not seeing many go implementations that support negation syntax.

https://github.com/gobwas/glob doesn't directly implement filesystem search.

https://godoc.org/gopkg.in/godo.v2/glob appears to support negation, but is a bit stale.

andreynering · 2019-09-07T16:23:17Z

I just opened an issue on the glob lib we currently use: mattn/go-zglob#26.

I wouldn't expect that to be implemented soon, though.

leefernandes · 2019-09-09T15:27:49Z

also, perhaps another approach is an explicit exclusion glob similar to CompileDaemon

https://github.com/githubnemo/CompileDaemon/blob/master/daemon.go#L45

andreynering · 2019-09-14T19:51:07Z

also, perhaps another approach is an explicit exclusion glob similar to CompileDaemon

https://github.com/githubnemo/CompileDaemon/blob/master/daemon.go#L45

This also sounds like a good idea

lunny · 2020-03-20T08:46:14Z

See #309

marco-m · 2021-04-01T14:30:22Z

I just stumbled upon https://github.com/bmatcuk/doublestar that supports negation of character classes.

marco-m · 2021-08-13T06:14:22Z

Update: mattn/go-zglob#26 (Add glob negation support) has just been implemented!

Porges · 2021-08-17T21:24:17Z

@andreynering could we get a new release with updated zglob? I’d love to use this.

pd93 · 2022-04-06T12:05:47Z

Dug into this a little bit as I also wanted to exclude the vendor directory from a pattern. However, it looks like Taskfile is already using a version of zglob that supports negating globs.

The stacktrace I get (which is the same as OP) is actually pointing to an error caused by mvdan/sh. It looks like this issue is what we need for this to work in Taskfile.

ghostsquad · 2022-04-06T17:43:33Z

I'm working on this in the V4 Discussion #694 by turning the sources to act more or less like an ignore file, such that every item in the list is "combined" and "and'd" together. This would allow you to write:

/**/*.go
!/vendor

ghostsquad · 2022-04-06T17:44:52Z

Dug into this a little bit as I also wanted to exclude the vendor directory from a pattern. However, it looks like Taskfile is already using a version of zglob that supports negating globs.

The stacktrace I get (which is the same as OP) is actually pointing to an error caused by mvdan/sh. It looks like this issue is what we need for this to work in Taskfile.

I'm a bit curious how sh comes into play for sources in task. I haven't yet looked at the code path for this yet. This is one thing that I definitely would not shell out to handle. This can be handled natively in Go.

ghostsquad · 2022-04-06T17:48:22Z

I'm working on this in the V4 Discussion #694 by turning the sources to act more or less like an ignore file, such that every item in the list is "combined" and "and'd" together. This would allow you to write:
/**/*.go
!/vendor

Oh, I wanted to add one more thing, that I was also thinking about adding support to automatically read in a .gitignore file, and use that as an baseline for what files are applicable for matching. This would be an inline feature flag that can be disabled.

kirb · 2022-11-14T00:52:07Z

FYI the negation feature just got tagged: https://github.com/mattn/go-zglob/releases/tag/v0.0.4

aminya · 2022-11-16T21:48:07Z

#924

the globbing library that task uses fails to check some patterns.

The following checks the files inside the dev folder

"{dev/**/*,something}"

But this doesn't!

"{something,dev/**/*}"

Or for example, ** doesn't check all the files of the current project.

aminya · 2023-01-02T09:12:41Z

Another issue with the go-zglob library is that it is very slow. It still tries to read the excluded directory using the fastwalk function.

    sources:
      ["node_modules/**/*", "**/*.js"]

I have done some profiling on my fork, and here is the result. As you see, it reads the whole node_modules directory, which takes 70 seconds.

krystian-panek-vmltech · 2023-06-14T19:32:56Z

I really need this negation support in my beloved tool "task" ;)

I have some generated sources under some directory that usually holds only regular sources and I need to exclude them without listing all dirs separately as sources to cover that. I know that it is ugly but currently, the only way to make build caching work. To do it elegantly I need to have it implemented in Task for sure. Unfortunately, I am unable to reconfigure the build to move these generated sources somewhere else.

Could we somehow push this feature forward? May I help somehow?

thenbe · 2023-08-01T15:59:52Z

One suggestion would be to support a shell command that produces the necessary watched paths. Then, users can use a tool like fd to implement whatever watched logic they desire.

tasks:
  foo:
    sources_command: "fd -lp --exclude='**/*.cache.*' --glob '**/packages/{site,backend}/**/*.{go,js,ts,json}'"

I'm using fd as an example because it:

supports deterministic sort
very fast
powerful matching capabilities (globs, regex, negation, etc)
very actively maintained (to be fair towards go-zglob, there are no outlying bug reports. But it only has a single code commit/new feature during the past few years)

krystian-panek-vmltech · 2023-09-07T08:35:30Z

@thenbe I like the idea because currently as far as I investigated there is no nice Go library to support exclusions with the glob patterns that we use currently (without introducing backward incompatible change). Your extension to the task tool will be backward-compatible. However a little bit annoying then will be a need to provide fd binary for each host machine manually. If the task tool could embed fd somehow as well then that other problem will be gone.

I am using the task tool intensively and adopting it in my company however the lack of exclusions in sources is really annoying. Instead of having a single source pattern with one negation, I need to list many paths separately (to avoid treat generated files as sources) which is very error-prone as these path structures may change/are under active development.

If I could somehow help in pushing that topic forward, please let me know. I could contribute but at least I need to be sure which kind of improvements will be accepted and merged into the tool.

@andreynering, @pd93 maybe?

pd93 · 2023-09-07T11:26:41Z

@krystian-panek-wttech I think it's very unlikely that Task will ever embed another binary. If you need the functionality of fd, then you should add a precondition to you Taskfile that checks if it is installed and ask users to go and install it. Also fd is written in Rust and so there is no way for us to import its code either.

With that said, this issue has been bothering me for a long time. zglob supposedly added support for negation a while ago, but I've never been able to get it to work. Also, the syntax for it isn't great. I've opened a PR instead that will add native support for negating patterns in Task. This seems like a better solution to me. Please feel free to check it out and give it a try. Feedback always welcome.

krystian-panek-vmltech · 2023-09-07T13:34:49Z

I will give it a try and let you know.

the implementation looks good enough / promising. thank you very much for your time! ;)

andreynering added the type: bug Something not working as intended. label Jun 30, 2019

andreynering added type: enhancement A change to an existing feature or functionality. type: feature A new feature or functionality. and removed type: bug Something not working as intended. labels Sep 7, 2019

marco-m mentioned this issue Aug 29, 2020

Add support for exclude rules for sources #309

Closed

deviantintegral mentioned this issue Jun 24, 2021

Create system to define includes / excludes in artifact archives Lullabot/drainpipe#6

Closed

ghost mentioned this issue Oct 27, 2022

Add support for working with shell options #908

Closed

aminya mentioned this issue Jan 2, 2023

fastwalk function is slow because of walking the excluded directories mattn/go-zglob#40

Open

krystian-panek-vmltech mentioned this issue Feb 28, 2023

AEM build caching by Task tool wttech/aemc#87

Merged

StefanBRas mentioned this issue Sep 1, 2023

--list-all spends time globbing for sources. #1322

Open

pd93 mentioned this issue Sep 7, 2023

feat: support negative globs #1324

Merged

andreynering closed this as completed in #1324 Nov 30, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Extended globbing support #225

Extended globbing support #225

Wicketd commented Jun 29, 2019 •

edited

Loading

andreynering commented Jun 30, 2019

andreynering commented Jul 13, 2019

Wicketd commented Jul 14, 2019 •

edited

Loading

leefernandes commented Sep 3, 2019

andreynering commented Sep 7, 2019

leefernandes commented Sep 9, 2019

andreynering commented Sep 14, 2019

lunny commented Mar 20, 2020

marco-m commented Apr 1, 2021

marco-m commented Aug 13, 2021

Porges commented Aug 17, 2021

pd93 commented Apr 6, 2022

ghostsquad commented Apr 6, 2022

ghostsquad commented Apr 6, 2022

ghostsquad commented Apr 6, 2022

kirb commented Nov 14, 2022

aminya commented Nov 16, 2022

aminya commented Jan 2, 2023 •

edited

Loading

krystian-panek-vmltech commented Jun 14, 2023

thenbe commented Aug 1, 2023 •

edited

Loading

krystian-panek-vmltech commented Sep 7, 2023 •

edited

Loading

pd93 commented Sep 7, 2023

krystian-panek-vmltech commented Sep 7, 2023 •

edited

Loading

Extended globbing support #225

Extended globbing support #225

Comments

Wicketd commented Jun 29, 2019 • edited Loading

andreynering commented Jun 30, 2019

andreynering commented Jul 13, 2019

Wicketd commented Jul 14, 2019 • edited Loading

leefernandes commented Sep 3, 2019

andreynering commented Sep 7, 2019

leefernandes commented Sep 9, 2019

andreynering commented Sep 14, 2019

lunny commented Mar 20, 2020

marco-m commented Apr 1, 2021

marco-m commented Aug 13, 2021

Porges commented Aug 17, 2021

pd93 commented Apr 6, 2022

ghostsquad commented Apr 6, 2022

ghostsquad commented Apr 6, 2022

ghostsquad commented Apr 6, 2022

kirb commented Nov 14, 2022

aminya commented Nov 16, 2022

aminya commented Jan 2, 2023 • edited Loading

krystian-panek-vmltech commented Jun 14, 2023

thenbe commented Aug 1, 2023 • edited Loading

krystian-panek-vmltech commented Sep 7, 2023 • edited Loading

pd93 commented Sep 7, 2023

krystian-panek-vmltech commented Sep 7, 2023 • edited Loading

Wicketd commented Jun 29, 2019 •

edited

Loading

Wicketd commented Jul 14, 2019 •

edited

Loading

aminya commented Jan 2, 2023 •

edited

Loading

thenbe commented Aug 1, 2023 •

edited

Loading

krystian-panek-vmltech commented Sep 7, 2023 •

edited

Loading

krystian-panek-vmltech commented Sep 7, 2023 •

edited

Loading