Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fail the build if apt-get or curl errors #79

Merged
merged 1 commit into from
Jan 15, 2021
Merged

Conversation

edmorley
Copy link
Member

@edmorley edmorley commented Jan 15, 2021

Enables the bash pipefail mode, which ensures that a failure in a command prior to a pipe correctly causes the script to exit 1.

Without this, failures during the apt-get and curl invocations were ignored and the compile marked as a success. At best this leads to confusing errors in later buildpacks (if build time dependencies are missing), and at worst this could cause runtime failures for packages not used during the build, but required by the app at runtime.

Enabling pipefail mode required a change to the custom repositories feature, to prevent the build exiting 1 when grep -s -e "^:repo:" found no matches (ie when no custom repositories are specified).

In addition, the --show-error and --fail flags have been added to the curl call, otherwise non-HTTP 200 exit codes are ignored and the compile similarly marked as successful when it should not have been.

Fixes #47.
Fixes W-8722791.

@edmorley edmorley self-assigned this Jan 15, 2021
@edmorley edmorley requested a review from a team January 15, 2021 18:22
Base automatically changed from edmorley/add-changelog to master January 15, 2021 18:33
Enables the bash `pipefail` mode, which ensures that a failure in a
command prior to a pipe correctly causes the script to exit 1.

Without this, failures during the `apt-get` and `curl` invocations were
ignored and the compile marked as a success. At best this leads to
confusing errors in later buildpacks (if build time dependencies are
missing), and at worst this could cause runtime failures for packages
not used during the build, but required by the app at runtime.

Enabling `pipefail` mode required a change to the custom repositories
feature, to prevent the build exiting 1 when `grep -s -e "^:repo:"`
found no matches (ie when no custom repositories are specified).

In addition, the `--show-error` and `--fail` flags have been added to
the `curl` call, otherwise non-HTTP 200 exit codes are ignored and the
compile similarly marked as successful when it should not have been.

Fixes #47.
Fixes W-8722791.
@edmorley
Copy link
Member Author

Correctly fails the build early for non-existent packages:

$ mkdir testapp-apt && cd $_ && git init && h create
...
$ h buildpacks:add https://github.com/heroku/heroku-buildpack-apt#edmorley/bash-strict
...
$ echo 'mysql-client-5.7' > Aptfile
$ git add -A && git commit -m '.' && git push heroku main
...
remote: -----> Apt app detected
remote: -----> Detected Aptfile or Stack changes, flushing cache
remote: -----> Updating apt caches
remote:        Get:1 http://archive.ubuntu.com/ubuntu focal InRelease [265 kB]
remote:        Get:2 http://apt.postgresql.org/pub/repos/apt focal-pgdg InRelease [81.6 kB]
remote:        Get:3 http://archive.ubuntu.com/ubuntu focal-security InRelease [109 kB]
remote:        Get:4 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]
remote:        Get:5 http://apt.postgresql.org/pub/repos/apt focal-pgdg/main amd64 Packages [308 kB]
remote:        Get:6 http://archive.ubuntu.com/ubuntu focal/universe amd64 Packages [11.3 MB]
remote:        Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages [1,275 kB]
remote:        Get:8 http://archive.ubuntu.com/ubuntu focal-security/main amd64 Packages [560 kB]
remote:        Get:9 http://archive.ubuntu.com/ubuntu focal-security/universe amd64 Packages [650 kB]
remote:        Get:10 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [905 kB]
remote:        Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [961 kB]
remote:        Fetched 16.6 MB in 2s (7,239 kB/s)
remote:        Reading package lists...
remote: -----> Fetching .debs for mysql-client-5.7
remote:        Reading package lists...
remote:        Building dependency tree...
remote:        Package mysql-client-5.7 is not available, but is referred to by another package.
remote:        This may mean that the package is missing, has been obsoleted, or
remote:        is only available from another source
remote:        However the following packages replace it:
remote:          mariadb-server-core-10.3 mariadb-server-10.3 mariadb-client-10.3
remote:
remote: E: Package 'mysql-client-5.7' has no installation candidate
remote:  !     Push rejected, failed to compile Apt app.
remote:
remote:  !     Push failed

Succeeds for a valid package:

$ echo 'mysql-client-8.0' > Aptfile
$ git add -A; git commit -m '.' --allow-empty && git push heroku main
...
remote: -----> Apt app detected
remote: -----> Detected Aptfile or Stack changes, flushing cache
remote: -----> Updating apt caches
remote:        Get:1 http://archive.ubuntu.com/ubuntu focal InRelease [265 kB]
remote:        Get:2 http://archive.ubuntu.com/ubuntu focal-security InRelease [109 kB]
remote:        Get:3 http://apt.postgresql.org/pub/repos/apt focal-pgdg InRelease [81.6 kB]
remote:        Get:4 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB]
remote:        Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 Packages [1,275 kB]
remote:        Get:6 http://archive.ubuntu.com/ubuntu focal/universe amd64 Packages [11.3 MB]
remote:        Get:7 http://apt.postgresql.org/pub/repos/apt focal-pgdg/main amd64 Packages [308 kB]
remote:        Get:8 http://archive.ubuntu.com/ubuntu focal-security/universe amd64 Packages [650 kB]
remote:        Get:9 http://archive.ubuntu.com/ubuntu focal-security/main amd64 Packages [560 kB]
remote:        Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [961 kB]
remote:        Get:11 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [905 kB]
remote:        Fetched 16.6 MB in 2s (7,193 kB/s)
remote:        Reading package lists...
remote: -----> Fetching .debs for mysql-client-8.0
remote:        Reading package lists...
remote:        Building dependency tree...
remote:        The following additional packages will be installed:
remote:          mysql-client-core-8.0
remote:        The following NEW packages will be installed:
remote:          mysql-client-8.0 mysql-client-core-8.0
remote:        0 upgraded, 2 newly installed, 0 to remove and 3 not upgraded.
remote:        Need to get 4,222 kB of archives.
remote:        After this operation, 64.9 MB of additional disk space will be used.
remote:        Get:1 http://archive.ubuntu.com/ubuntu focal-security/main amd64 mysql-client-core-8.0 amd64 8.0.22-0ubuntu0.20.04.3 [4,200 kB]
remote:        Get:2 http://archive.ubuntu.com/ubuntu focal-security/main amd64 mysql-client-8.0 amd64 8.0.22-0ubuntu0.20.04.3 [22.0 kB]
remote:        Fetched 4,222 kB in 1s (4,948 kB/s)
remote:        Download complete and in download only mode
remote: -----> Installing mysql-client-8.0_8.0.22-0ubuntu0.20.04.3_amd64.deb
remote: -----> Installing mysql-client-core-8.0_8.0.22-0ubuntu0.20.04.3_amd64.deb
remote: -----> Writing profile script
remote: -----> Rewrite package-config files
remote: -----> Discovering process types
remote:        Procfile declares types -> (none)
remote:
remote: -----> Compressing...
remote:        Done: 17M
remote: -----> Launching...
remote:        Released v3

@edmorley edmorley merged commit e8b8cde into master Jan 15, 2021
@edmorley edmorley deleted the edmorley/bash-strict branch January 15, 2021 18:50
@edmorley
Copy link
Member Author

$ h buildpacks:publish heroku-community/apt master
Publishing buildpack heroku-community/apt... done
Publishing buildpack was successful

$ h buildpacks:versions heroku-community/apt
Version  Released At               Status
───────  ────────────────────────  ─────────
5        2021-01-15T19:46:44.614Z  published
4        2019-10-17T17:20:09.847Z  published
3        2019-09-09T07:26:55.242Z  published
2        2019-06-11T06:47:11.497Z  published
1        2018-05-21T20:08:40.250Z  published

@edmorley
Copy link
Member Author

edmorley commented Jan 18, 2021

Note: If you see build errors after this change, when you didn't previously, check previously "successful" builds and you will see the same error message in them. The difference now is that errors are no longer silently ignored, and instead the buildpack correctly marks the build as failing, if the packages didn't install correctly. This new behaviour prevents missing dependencies from potentially causing downtime after deployment.

These errors are not an error in the APT buildpack, but due to invalid entries in your application's Aptfile. You will need to remove the invalid entries from your Aptfile to fix the build. Removing them is safe to do (ie: shouldn't cause any problems with your app), since these invalid entries weren't being installed anyway.

For example, if you see an error like:

E: Package '<some-package-name>' has no installation candidate

Then it means you need to look in your Aptfile for the line that references that package, and remove it.

Or if you see something like:

W: GPG error: http://http.us.debian.org/debian stretch Release: The following signatures couldn't be verified because the public key is not available: ...

E: The repository 'http://http.us.debian.org/debian stretch Release' is not signed.

Then remove the related :repo:deb ... line from your Aptfile.

edmorley added a commit that referenced this pull request Jan 19, 2021
In a recent support ticket, a user had copied the example `:repo:` line into their `Aptfile`,
which previously referred to the now obsolete "artful" Ubuntu LTS release.

This caused the build to fail after #79, when previously the error was ignored.

This PR makes the example more clearly just an example, and emphasises that
using a custom repository is only necessary if using packages from outside
the standard Ubuntu repository.

In addition, the bogus `pg` Gem example was removed, since `libpq-dev` is
part of the stack image, so we should not be encouraging people to install
it unnecessarily.
edmorley added a commit that referenced this pull request Jan 21, 2021
In a recent support ticket, a user had copied the example `:repo:` line into their `Aptfile`,
which previously referred to the now obsolete "artful" Ubuntu LTS release.

This caused the build to fail after #79, when previously the error was ignored.

This PR makes the example more clearly just an example, and emphasises that
using a custom repository is only necessary if using packages from outside
the standard Ubuntu repository.

In addition, the bogus `pg` Gem example was removed, since `libpq-dev` is
part of the stack image, so we should not be encouraging people to install
it unnecessarily.
EtienneM referenced this pull request in Scalingo/apt-buildpack Dec 16, 2022
Enables the bash `pipefail` mode, which ensures that a failure in a
command prior to a pipe correctly causes the script to exit 1.

Without this, failures during the `apt-get` and `curl` invocations were
ignored and the compile marked as a success. At best this leads to
confusing errors in later buildpacks (if build time dependencies are
missing), and at worst this could cause runtime failures for packages
not used during the build, but required by the app at runtime.

Enabling `pipefail` mode required a change to the custom repositories
feature, to prevent the build exiting 1 when `grep -s -e "^:repo:"`
found no matches (ie when no custom repositories are specified).

In addition, the `--show-error` and `--fail` flags have been added to
the `curl` call, otherwise non-HTTP 200 exit codes are ignored and the
compile similarly marked as successful when it should not have been.

Fixes #47.
Fixes W-8722791.
EtienneM referenced this pull request in Scalingo/apt-buildpack Dec 16, 2022
In a recent support ticket, a user had copied the example `:repo:` line into their `Aptfile`,
which previously referred to the now obsolete "artful" Ubuntu LTS release.

This caused the build to fail after #79, when previously the error was ignored.

This PR makes the example more clearly just an example, and emphasises that
using a custom repository is only necessary if using packages from outside
the standard Ubuntu repository.

In addition, the bogus `pg` Gem example was removed, since `libpq-dev` is
part of the stack image, so we should not be encouraging people to install
it unnecessarily.
Frzk referenced this pull request in Scalingo/apt-buildpack May 13, 2024
Enables the bash `pipefail` mode, which ensures that a failure in a
command prior to a pipe correctly causes the script to exit 1.

Without this, failures during the `apt-get` and `curl` invocations were
ignored and the compile marked as a success. At best this leads to
confusing errors in later buildpacks (if build time dependencies are
missing), and at worst this could cause runtime failures for packages
not used during the build, but required by the app at runtime.

Enabling `pipefail` mode required a change to the custom repositories
feature, to prevent the build exiting 1 when `grep -s -e "^:repo:"`
found no matches (ie when no custom repositories are specified).

In addition, the `--show-error` and `--fail` flags have been added to
the `curl` call, otherwise non-HTTP 200 exit codes are ignored and the
compile similarly marked as successful when it should not have been.

Fixes #47.
Fixes W-8722791.
Frzk referenced this pull request in Scalingo/apt-buildpack May 13, 2024
In a recent support ticket, a user had copied the example `:repo:` line into their `Aptfile`,
which previously referred to the now obsolete "artful" Ubuntu LTS release.

This caused the build to fail after #79, when previously the error was ignored.

This PR makes the example more clearly just an example, and emphasises that
using a custom repository is only necessary if using packages from outside
the standard Ubuntu repository.

In addition, the bogus `pg` Gem example was removed, since `libpq-dev` is
part of the stack image, so we should not be encouraging people to install
it unnecessarily.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

apt-get/curl errors don't cause the buildpack to fail early
2 participants