Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prevent APT using source lists from /etc/apt/sources.list.d/ #46

Merged
merged 2 commits into from
Jun 11, 2019

Conversation

edmorley
Copy link
Member

Previously the buildpack only passed -o dir::etc::sourcelist to APT, which meant that APT still used the default sourceparts location of /etc/apt/sources.list.d/.

This meant on cedar-14 this buildpack would use esm.ubuntu.com as an APT source (as of heroku/base-images/pull/140), which results in errors if the requested packages happened to have ESM-only updates available (since the ESM repository requires credentials since it's a paid Ubuntu offering).

Fixes #45 / W-6224944.

@edmorley edmorley added the bug label Jun 10, 2019
@edmorley edmorley self-assigned this Jun 10, 2019
@edmorley
Copy link
Member Author

edmorley commented Jun 10, 2019

The test failures on Travis are pre-existing (ie on master already), and are due to warnings being output on stderr from the deprecated use of --force-yes. Fixing these requires switching to the other --allow-* options, which aren't supported until newer APT, which isn't supported on cedar-14. (And either way is out of scope of this PR.)

I've tested this buildpack branch against a test app, and it successfully installed the packages that failed in the #45 example:

remote: -----> Apt app detected
remote: -----> Reusing cache
remote: -----> Updating apt caches
remote:        Ign http://archive.ubuntu.com trusty InRelease
remote:        Hit http://archive.ubuntu.com trusty-security InRelease
remote:        Hit http://archive.ubuntu.com trusty-updates InRelease
remote:        Hit http://archive.ubuntu.com trusty Release.gpg
remote:        Hit http://archive.ubuntu.com trusty Release
remote:        Hit http://archive.ubuntu.com trusty-security/main amd64 Packages
remote:        Hit http://archive.ubuntu.com trusty-security/universe amd64 Packages
remote:        Hit http://archive.ubuntu.com trusty-updates/main amd64 Packages
remote:        Hit http://archive.ubuntu.com trusty-updates/universe amd64 Packages
remote:        Hit http://archive.ubuntu.com trusty/main amd64 Packages
remote:        Hit http://archive.ubuntu.com trusty/universe amd64 Packages
remote:        Hit http://apt.postgresql.org trusty-pgdg InRelease
remote:        Hit http://apt.postgresql.org trusty-pgdg/main amd64 Packages
remote:        Reading package lists...
remote: -----> Fetching .debs for libchromaprint-dev
remote:        Reading package lists...
remote:        Building dependency tree...
remote:        The following extra packages will be installed:
remote:          libavcodec54 libavutil52 libchromaprint0 libgsm1 libmp3lame0 libopenjpeg2
remote:          libopus0 liborc-0.4-0 libschroedinger-1.0-0 libspeex1 libtheora0 libva1
remote:          libx264-142 libxvidcore4
remote:        Suggested packages:
remote:          libchromaprint-tools python-acoustid opus-tools speex
remote:        The following NEW packages will be installed:
remote:          libavcodec54 libavutil52 libchromaprint-dev libchromaprint0 libgsm1
remote:          libmp3lame0 libopenjpeg2 libopus0 liborc-0.4-0 libschroedinger-1.0-0
remote:          libspeex1 libtheora0 libva1 libx264-142 libxvidcore4
remote:        0 upgraded, 15 newly installed, 0 to remove and 0 not upgraded.
remote:        Need to get 2,575 kB/4,388 kB of archives.
remote:        After this operation, 13.4 MB of additional disk space will be used.
remote:        Get:1 http://archive.ubuntu.com/ubuntu/ trusty-security/universe libavutil52 amd64 6:9.20-0ubuntu0.14.04.1 [66.9 kB]
remote:        Get:2 http://archive.ubuntu.com/ubuntu/ trusty/universe libmp3lame0 amd64 3.99.5+repack1-3ubuntu1 [158 kB]
remote:        Get:3 http://archive.ubuntu.com/ubuntu/ trusty-security/universe libavcodec54 amd64 6:9.20-0ubuntu0.14.04.1 [2,350 kB]
remote:        Fetched 2,575 kB in 0s (2,784 kB/s)
remote:        Download complete and in download only mode
remote: -----> Installing libavcodec54_6%3a9.20-0ubuntu0.14.04.1_amd64.deb
remote: -----> Installing libavutil52_6%3a9.20-0ubuntu0.14.04.1_amd64.deb
remote: -----> Installing libchromaprint0_1.1-1_amd64.deb
remote: -----> Installing libchromaprint-dev_1.1-1_amd64.deb
remote: -----> Installing libgsm1_1.0.13-4_amd64.deb
remote: -----> Installing libmp3lame0_3.99.5+repack1-3ubuntu1_amd64.deb
remote: -----> Installing libopenjpeg2_1.3+dfsg-4.7ubuntu1_amd64.deb
remote: -----> Installing libopus0_1.1-0ubuntu1_amd64.deb
remote: -----> Installing liborc-0.4-0_1%3a0.4.18-1ubuntu1_amd64.deb
remote: -----> Installing libschroedinger-1.0-0_1.0.11-2ubuntu1_amd64.deb
remote: -----> Installing libspeex1_1.2~rc1.1-1ubuntu1_amd64.deb
remote: -----> Installing libtheora0_1.1.1+dfsg.1-3.2_amd64.deb
remote: -----> Installing libva1_1.3.0-2_amd64.deb
remote: -----> Installing libx264-142_2%3a0.142.2389+git956c8d8-2_amd64.deb
remote: -----> Installing libxvidcore4_2%3a1.3.2-9ubuntu1_amd64.deb
remote: -----> Writing profile script
remote: -----> Rewrite package-config files

bin/compile Outdated
@@ -54,7 +54,8 @@ else
fi

APT_OPTIONS="-o debug::nolocking=true -o dir::cache=$APT_CACHE_DIR -o dir::state=$APT_STATE_DIR"
APT_OPTIONS="$APT_OPTIONS -o dir::etc::sourcelist=$APT_SOURCES"
# Override the use of /etc/apt/sources.list (sourcelist) and /etc/apt/sources.list.d/* (sourceparts).
APT_OPTIONS="$APT_OPTIONS -o dir::etc::sourcelist=$APT_SOURCES -o dir::etc::sourceparts=-'"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why have -' as the value here? I would have expected this to be an empty or non-existent directory.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh sorry the trailing ' isn't meant to be there. The - is since that's what several SO / GitHub code references use. I tried to find a more authoritative guide as to what the supported path scheme was, however the docs are pretty lacking.

I'd be open to trying an empty string here and seeing whether that works too.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Trying with the empty string results in apt-get update returning the man page (which doesn't cause the buildpack to fail early, due to #47).

I've switched it back to - (minus the ' typo).

Copy link
Member Author

@edmorley edmorley Jun 10, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So I found a reference to /dev/null in the APT source, here:
https://salsa.debian.org/apt-team/apt/blob/f15e090301e0744471d0fbf86ea0d494f6c08215/apt-pkg/sourcelist.cc#L313

...so I've switched it one last time to use that instead now 😆
(And re-tested again to ensure that works)

Previously the buildpack only passed `-o dir::etc::sourcelist` to APT,
which meant that APT still used the default `sourceparts` location of
`/etc/apt/sources.list.d/`.

This meant on cedar-14 this buildpack would use esm.ubuntu.com as an APT
source (as of heroku/base-images/pull/140), which results in errors if
the requested packages happened to have ESM-only updates available
(since the ESM repository requires credentials since it's a paid Ubuntu
offering).

Fixes #45 / W-6224944.
@edmorley edmorley merged commit 3a59ddc into master Jun 11, 2019
@edmorley edmorley deleted the edmorley-apt-sourceparts branch June 11, 2019 06:45
@edmorley edmorley removed the request for review from joshwlewis June 11, 2019 06:48
@edmorley
Copy link
Member Author

And published:

$ heroku buildpacks:versions heroku-community/apt
Version  Released At               Status
───────  ────────────────────────  ─────────
2        2019-06-11T06:47:11.497Z  published
1        2018-05-21T20:08:40.250Z  published

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Buildpack is unexpectedly using sources lists from /etc/apt/sources.list.d/
3 participants