Debian build is broken

[raoulb]

What version of Hugo are you using (hugo version)?

$ hugo version
Hugo Static Site Generator v0.80.0/extended linux/amd64 BuildDate: 2021-01-01T02:12:40Z

Does this issue reproduce with the latest release?

Yes.

Issues

Javascript building/bundling fails with sources and dependencies all in assets/js. Here is a minimal example I constructed based on the test project linked in the Hugo documentation.

With Hugo 0.80.0 the build fails:

$ hugo version
Hugo Static Site Generator v0.80.0/extended linux/amd64 BuildDate: 2021-01-01T02:12:40Z

$ hugo
Start building sites … 
Total in 12 ms
Error: Error building site: JSBUILD: failed to transform "js/main.js" (application/javascript): "/home/r/hugoTestProjectJSModImports/assets/js/main.js:1:23": Could not resolve "./lib"

While with Hugo 0.79.1 it works as expected:

$ hugo version
Hugo Static Site Generator v0.79.1/extended linux/amd64 BuildDate: 2020-12-22T10:11:47Z

$ hugo
Start building sites … 

                   | EN  
-------------------+-----
  Pages            |  2  
  Paginator pages  |  0  
  Non-page files   |  0  
  Static files     |  0  
  Processed images |  0  
  Aliases          |  0  
  Sitemaps         |  1  
  Cleaned          |  0  

Total in 11 ms

If this is not a regression but intended new behaviour, then the documentation might need further clarification.

BTW: Calling esbuild manually works on the same sources:

esbuild --version
0.8.28

esbuild --bundle --minify --outfile=out.js main.js 

cat out.js 
(()=>{function o(){return"Hello from lib in the main project!!"}console.log("Hello from main entry point.");function l(){console.log(o())}window.hello=l;})();

[bep]

I will have a look at it (I esp. appreciate the example repo).

The only related change in 0.80 was a upgrade from Esbuild 0.8.15 to 0.8.17. I do have some (I think) pretty good tests in this area (also a manual repo), so I'm a little surprised. A side not is that I tried to pull in the lates ESBuild (he works fast), and that broke my test repo.

[bep]

I cannot reproduce your issue (on MacOS, but I don't see how that can differ) -- the error you get looks like the error I get when I pull in the latest esbuild.

[moorereason]

@raoulb,
I'm unable to reproduce this on my system:

$ hugo-v0.80.0-extended env
Hugo Static Site Generator v0.80.0-792EF0F4/extended linux/amd64 BuildDate: 2020-12-31T13:46:18Z
GOOS="linux"
GOARCH="amd64"
GOVERSION="go1.15.1"

$ hugo-v0.80.0-extended
Start building sites …

                   | EN
-------------------+-----
  Pages            |  2
  Paginator pages  |  0
  Non-page files   |  0
  Static files     |  0
  Processed images |  0
  Aliases          |  0
  Sitemaps         |  1
  Cleaned          |  0

Total in 9 ms

$ hugo-v0.79.1-extended env
Hugo Static Site Generator v0.79.1-EDB9248D/extended linux/amd64 BuildDate: 2020-12-19T15:50:19Z
GOOS="linux"
GOARCH="amd64"
GOVERSION="go1.15.1"

$ hugo-v0.79.1-extended
Start building sites …

                   | EN
-------------------+-----
  Pages            |  2
  Paginator pages  |  0
  Non-page files   |  0
  Static files     |  0
  Processed images |  0
  Aliases          |  0
  Sitemaps         |  1
  Cleaned          |  0

Total in 15 ms

It's interesting to me that your version string is different from mine. I downloaded the tar.gz extended releases from GH. Where are you getting your hugo binaries? Can you try the ones in the tar.gz on GH?

[bep]

OK, then I guess that the version @raoulb is built without the versioning of Go Modules. Which I hope means that he built it himself?

[raoulb]

I use the official packages from Debian sid/unstable from here.

[moorereason]

Cc: @anthonyfok

[raoulb]

@moorereason

I'm unable to reproduce this on my system

Oh, interesting and kind of good news.

Where are you getting your hugo binaries? Can you try the ones in the tar.gz on GH?

Ok. I tried the both versions for the tar.gz files and they work.

$ ~/hugo_0.80.0_Linux-64bit/hugo version
Hugo Static Site Generator v0.80.0-792EF0F4 linux/amd64 BuildDate: 2020-12-31T13:37:58Z

$ ~/hugo_0.80.0_Linux-64bit/hugo 
Start building sites … 

                   | EN  
-------------------+-----
  Pages            |  2  
  Paginator pages  |  0  
  Non-page files   |  0  
  Static files     |  0  
  Processed images |  0  
  Aliases          |  0  
  Sitemaps         |  1  
  Cleaned          |  0  

Total in 10 ms

and

$ ~/hugo_extended_0.80.0_Linux-64bit/hugo version
Hugo Static Site Generator v0.80.0-792EF0F4/extended linux/amd64 BuildDate: 2020-12-31T13:46:18Z

$ ~/hugo_extended_0.80.0_Linux-64bit/hugo 
Start building sites … 

                   | EN  
-------------------+-----
  Pages            |  2  
  Paginator pages  |  0  
  Non-page files   |  0  
  Static files     |  0  
  Processed images |  0  
  Aliases          |  0  
  Sitemaps         |  1  
  Cleaned          |  0  

Total in 9 ms

Now I could try the deb packages provided here on GH if this helps.

[raoulb]

I cannot reproduce your issue (on MacOS, but I don't see how that can differ) -- the error you get looks like the error I get when I pull in the latest esbuild.

Oh, good point. Maybe the Debian package builds against an own, 'external' version of ESBuild.

At least this is what I see here with the build time dependency:

adep: golang-github-evanw-esbuild-dev (>= 0.8.17)
    extremely fast JavaScript bundler and minifier (library) 

if I'm reading this correctly.

[raoulb]

@moorereason @anthonyfok

Now I could try the deb packages provided here on GH if this helps.

Ok, the package hugo_extended_0.80.0_Linux-64bit.deb from GH also works fine.
So this is actually a pure Debian issue it seems. Should I open another issue there?

[moorereason]

We've pinned esbuild at v0.8.17, but the esbuild package from Debian is v0.8.28.

[moorereason]

Tested the latest esbuild and can confirm that it fails to build your test site with the error message in the OP: Could not resolve "./lib".

[moorereason]

Last update 😄

The test site in the OP starts failing with esbuild v0.8.21. If I change the import path to ./assets/js/lib, it builds successfully.

[bep]

@moorereason this issue is not about ESBuild (which is a known issue -- we had a buildbot PR that luckily failed in test some weeks ago). This is about the broken Debian build.

[anthonyfok]

Dear all,

@raoulb, Thank you for reporting the issue, especially timely because Debian 11 (bullseye) will go into freezing soon, and it is good that you let us know of the issue now instead of later.

@bep and @moorereason, thank you both for your excellent investigative work. Indeed, it was my laziness and oversight when updating Debian's copy of esbuild to a too-new version that led to this issue. Now that @moorereason has found out which version of esbuild (v0.8.21) has the behaviour change, I can start looking into a fix for my Debian build later today. Thank you @moorereason for saving me a lot of time.

[bep]

@anthonyfok are you somehow running with different versions of the dependencies in the Debian build (I assume because they're shared by others?)? Do the tests pass?

[anthonyfok]

Hi @bep, sorry for the late reply.

are you somehow running with different versions of the dependencies in the Debian build

Yes. I had stupidly upgraded golang-github-evanw-esbuild from 0.8.15-1 to 0.8.28-1. Usually, I follow the versions listed in Hugo's go.mod quite strictly, but this last time, I was not able to upgrade to 0.8.17 directly with the usual command because there had been too many newer esbuild releases that 0.8.17 got pushed from https://github.com/evanw/esbuild/releases to the "Next" page, and the pattern matching in debian/watch file did not work. There are ways around that, but I either forgot how or got too lazy, and decided to just go ahead and package 0.8.28, and that's how I got the Debian package into trouble.

(I assume because they're shared by others?)

At the moment, golang-github-evanw-esbuild-dev is not used by other packages. Hugo is currently the only Debian package that depends on it.

Do the tests pass?

Yes, they do, but the Debian build isn't running all the tests, but go test -vet=off -v -p 4 -test.short -timeout 120m

dh_auto_test -- -test.short -timeout 120m
	cd _build && go test -vet=off -v -p 4 -test.short -timeout 120m github.com/gohugoio/hugo github.com/gohugoio/hugo/bufferpool github.com/gohugoio/hugo/cache/filecache github.com/gohugoio/hugo/cache/namedmemcache github.com/gohugoio/hugo/codegen github.com/gohugoio/hugo/commands github.com/gohugoio/hugo/common/collections github.com/gohugoio/hugo/common/constants github.com/gohugoio/hugo/common/herrors github.com/gohugoio/hugo/common/hexec github.com/gohugoio/hugo/common/hreflect github.com/gohugoio/hugo/common/hugio github.com/gohugoio/hugo/common/hugo github.com/gohugoio/hugo/common/loggers github.com/gohugoio/hugo/common/maps github.com/gohugoio/hugo/common/math github.com/gohugoio/hugo/common/para github.com/gohugoio/hugo/common/terminal github.com/gohugoio/hugo/common/text github.com/gohugoio/hugo/common/types github.com/gohugoio/hugo/common/urls github.com/gohugoio/hugo/compare github.com/gohugoio/hugo/config github.com/gohugoio/hugo/config/privacy github.com/gohugoio/hugo/config/services github.com/gohugoio/hugo/create github.com/gohugoio/hugo/deploy github.com/gohugoio/hugo/deps github.com/gohugoio/hugo/docshelper github.com/gohugoio/hugo/helpers github.com/gohugoio/hugo/htesting github.com/gohugoio/hugo/htesting/hqt github.com/gohugoio/hugo/hugofs github.com/gohugoio/hugo/hugofs/files github.com/gohugoio/hugo/hugofs/glob github.com/gohugoio/hugo/hugolib github.com/gohugoio/hugo/hugolib/filesystems github.com/gohugoio/hugo/hugolib/paths github.com/gohugoio/hugo/identity github.com/gohugoio/hugo/langs github.com/gohugoio/hugo/langs/i18n github.com/gohugoio/hugo/lazy github.com/gohugoio/hugo/livereload github.com/gohugoio/hugo/markup github.com/gohugoio/hugo/markup/asciidocext github.com/gohugoio/hugo/markup/asciidocext/asciidocext_config github.com/gohugoio/hugo/markup/blackfriday github.com/gohugoio/hugo/markup/blackfriday/blackfriday_config github.com/gohugoio/hugo/markup/converter github.com/gohugoio/hugo/markup/converter/hooks github.com/gohugoio/hugo/markup/goldmark github.com/gohugoio/hugo/markup/goldmark/goldmark_config github.com/gohugoio/hugo/markup/highlight github.com/gohugoio/hugo/markup/internal github.com/gohugoio/hugo/markup/markup_config github.com/gohugoio/hugo/markup/mmark github.com/gohugoio/hugo/markup/org github.com/gohugoio/hugo/markup/pandoc github.com/gohugoio/hugo/markup/rst github.com/gohugoio/hugo/markup/tableofcontents github.com/gohugoio/hugo/media github.com/gohugoio/hugo/metrics github.com/gohugoio/hugo/minifiers github.com/gohugoio/hugo/modules github.com/gohugoio/hugo/modules/npm github.com/gohugoio/hugo/navigation github.com/gohugoio/hugo/output github.com/gohugoio/hugo/parser github.com/gohugoio/hugo/parser/metadecoders github.com/gohugoio/hugo/parser/pageparser github.com/gohugoio/hugo/publisher github.com/gohugoio/hugo/related github.com/gohugoio/hugo/releaser github.com/gohugoio/hugo/resources github.com/gohugoio/hugo/resources/images github.com/gohugoio/hugo/resources/images/exif github.com/gohugoio/hugo/resources/internal github.com/gohugoio/hugo/resources/jsconfig github.com/gohugoio/hugo/resources/page github.com/gohugoio/hugo/resources/page/page_generate github.com/gohugoio/hugo/resources/page/pagemeta github.com/gohugoio/hugo/resources/postpub github.com/gohugoio/hugo/resources/resource github.com/gohugoio/hugo/resources/resource_factories/bundler github.com/gohugoio/hugo/resources/resource_factories/create github.com/gohugoio/hugo/resources/resource_transformers/babel github.com/gohugoio/hugo/resources/resource_transformers/htesting github.com/gohugoio/hugo/resources/resource_transformers/integrity github.com/gohugoio/hugo/resources/resource_transformers/js github.com/gohugoio/hugo/resources/resource_transformers/minifier github.com/gohugoio/hugo/resources/resource_transformers/postcss github.com/gohugoio/hugo/resources/resource_transformers/templates github.com/gohugoio/hugo/resources/resource_transformers/tocss/dartsass github.com/gohugoio/hugo/resources/resource_transformers/tocss/scss github.com/gohugoio/hugo/source github.com/gohugoio/hugo/tpl github.com/gohugoio/hugo/tpl/cast github.com/gohugoio/hugo/tpl/collections github.com/gohugoio/hugo/tpl/compare github.com/gohugoio/hugo/tpl/crypto github.com/gohugoio/hugo/tpl/data github.com/gohugoio/hugo/tpl/debug github.com/gohugoio/hugo/tpl/encoding github.com/gohugoio/hugo/tpl/fmt github.com/gohugoio/hugo/tpl/hugo github.com/gohugoio/hugo/tpl/images github.com/gohugoio/hugo/tpl/inflect github.com/gohugoio/hugo/tpl/internal github.com/gohugoio/hugo/tpl/internal/go_templates/cfg github.com/gohugoio/hugo/tpl/internal/go_templates/fmtsort github.com/gohugoio/hugo/tpl/internal/go_templates/htmltemplate github.com/gohugoio/hugo/tpl/internal/go_templates/testenv github.com/gohugoio/hugo/tpl/internal/go_templates/texttemplate github.com/gohugoio/hugo/tpl/internal/go_templates/texttemplate/parse github.com/gohugoio/hugo/tpl/internal/resourcehelpers github.com/gohugoio/hugo/tpl/js github.com/gohugoio/hugo/tpl/lang github.com/gohugoio/hugo/tpl/math github.com/gohugoio/hugo/tpl/openapi/openapi3 github.com/gohugoio/hugo/tpl/os github.com/gohugoio/hugo/tpl/partials github.com/gohugoio/hugo/tpl/path github.com/gohugoio/hugo/tpl/reflect github.com/gohugoio/hugo/tpl/resources github.com/gohugoio/hugo/tpl/safe github.com/gohugoio/hugo/tpl/site github.com/gohugoio/hugo/tpl/strings github.com/gohugoio/hugo/tpl/templates github.com/gohugoio/hugo/tpl/time github.com/gohugoio/hugo/tpl/tplimpl github.com/gohugoio/hugo/tpl/tplimpl/embedded github.com/gohugoio/hugo/tpl/transform github.com/gohugoio/hugo/tpl/urls github.com/gohugoio/hugo/transform github.com/gohugoio/hugo/transform/livereloadinject github.com/gohugoio/hugo/transform/metainject github.com/gohugoio/hugo/transform/urlreplacers github.com/gohugoio/hugo/watcher

See https://buildd.debian.org/status/fetch.php?pkg=hugo&arch=amd64&ver=0.80.0-1&stamp=1609753546&raw=0 for the full build (and test) log.

I have decided to fix the bug by including a local copy of esbuild 0.8.17 into the Hugo Debian package for the time being. The fix will appear in hugo_0.80.0-2 soon.

[eclipseo]

Any plans on making it work with the latest ESBuild?

[anthonyfok]

Hello @raoulb and all:

Here is an update:

1. The incompatibility wtih ESBuild >= v0.8.21 was thankfully caught by this test (GitHub Actions): https://github.com/gohugoio/hugo/runs/1627365570?check_suite_focus=true (titled "Do not merge: deps: Update to ESBuild v0.8.27"). The tests that failed are TestJSBuildWithNPM and TestJSBuild.

2. As @moorereason reported, the incompatibility started with ESBuild v0.8.21, most likely due to Plugins don't seem to apply to entrypoints evanw/esbuild#546 "Plugins don't seem to apply to entrypoints" and its fix evanw/esbuild@05eaca4.

3. As @eclipseo suggested, I agree that making Hugo work with the latest ESBuild is a possibility. Let's open a new issue for that, so I can close this issue regarding version mismatch in Debian packaging.

4. The error reported in this issue was not detected during Debian build because:

   1. TestJSBuildWithNPM and TestJSBuild are run only when CI=true (or non-empty), but CI was not set in recent Debian packaging.
   2. With CI=true set, TestJSBuildWithNPM and TestJSBuild would still have failed during Debian build because Internet access during build is disallowed.

5. Fortunately, Debian does run autopkgtest CI where Internet access is allowed, so the latest upload to Debian takes advantage of this to hopefully catch errors like this. Here is the most recent changelog entries:

---

hugo (0.80.0-3) unstable; urgency=medium

• Replace local copy of github.com/cli/safeexec
  with dependency on golang-github-cli-safeexec-dev (>= 1.0.0)
• Replace local copy of github.com/bep/godartsass
  with dependency on golang-github-bep-godartsass-dev (>= 0.11.0)
• debian/rules: Restore environment variable CI=true for dh_auto_test
  so that tests that would have caught errors such as
  Debian build is broken #8141 would not be missed
• Remove obsolete 0006-skip-TestHugoModulesTargetInSubFolder.patch
• Merge patches which skip tests that access the Internet
  into 0005-skip-tests-that-access-the-internet.patch,
  and allow those tests to be run during autopkgtest.
• Add debian/tests/autopkgtest-pkg-go.conf
  to install git and npm as extra dependencies for autopkgtest
• debian/rules: Build non-extended version (without LibSass) on riscv64
  as go1.15 does not yet support cgo on riscv64, and extended build FTBFS.
  TODO: Revisit this when go1.16 is released.
  See all: add cgo support to the riscv port golang/go#36641

-- Anthony Fok foka@debian.org Wed, 27 Jan 2021 06:21:10 -0700

hugo (0.80.0-2) unstable; urgency=medium

• Include a local copy of github.com/evanw/esbuild v0.8.17
  because golang-github-evanw-esbuild 0.8.28-1 which I prematurely
  upgraded to is incompatible with Hugo 0.80.0.
  Fixes Debian build is broken #8141

-- Anthony Fok foka@debian.org Sat, 16 Jan 2021 04:28:36 -0700

---

@raoulb, thank you again for reporting this issue.

I believe the issue is resolved in the Debian package hugo_0.80.0-2, and made a bit more futureproof in 0.80.0-3, hence I'm closing this issue.

[github-actions]

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.