Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cloud-init doesn't prefer new APT config format when old and new are provided #2711

Closed
ubuntu-server-builder opened this issue May 10, 2023 · 14 comments
Closed

cloud-init doesn't prefer new APT config format when old and new are provided #2711

ubuntu-server-builder opened this issue May 10, 2023 · 14 comments
Labels
launchpad Migrated from Launchpad priority Fix soon

Comments

@ubuntu-server-builder
Copy link
Collaborator

This bug was originally filed in Launchpad as LP: #1616831

Launchpad details 
affected_projects = ['cloud-init (Ubuntu)', 'cloud-init (Ubuntu Xenial)']
assignee = paelzer
assignee_name = Christian Ehrhardt 
date_closed = 2016-09-12T20:50:29.164465+00:00
date_created = 2016-08-25T09:42:42.082471+00:00
date_fix_committed = 2016-08-30T17:27:44.183632+00:00
date_fix_released = 2016-09-12T20:50:29.164465+00:00
id = 1616831
importance = high
is_complete = True
lp_url = https://bugs.launchpad.net/cloud-init/+bug/1616831
milestone = None
owner = andreserl
owner_name = Andres Rodriguez
private = False
status = fix_released
submitter = andreserl
submitter_name = Andres Rodriguez
tags = ['4010', 'verification-done']
duplicates = []

Launchpad user Andres Rodriguez(andreserl) wrote on 2016-08-25T09:42:42.082471+00:00

==== Begin SRU Template ====
[Impact]
New apt configuration was enabled, but if the user provided both new
configuration format and old configuration format, then then cloud-init
would error.

The fixed behavior is to allow new and old and warn if they differ.

This allows a user to provide the same data for older releases and newer release.

[Test Case]

Recreate failure (this would recreate on intermediate versions of

cloud-init that were in trusty, but will not on xenial version.

testing the working path is still valid though to ensure it functions.

you will need to adjust both proxy settings to a local proxy.

$ cat >user-data <<EOF
#cloud-config
apt:
preserve_sources_list: false
primary:

apt_proxy: http://192.168.122.1:8000/
EOF

$ release=xenial
$ name=x1
$ lxc launch ubuntu-daily:$release $name "--config=user.user-data=$(cat user-data)"
$ while ! lxc exec "$name" -- [ -f /run/cloud-init/result.json ]; do sleep 1; done
$ lxc exec $name -- grep WARN /var/log/cloud-init.log

Now update container, clean and reboot to show first boot

$ lxc exec $name -- sh -c '
p=/etc/apt/sources.list.d/proposed.list
echo deb http://archive.ubuntu.com/ubuntu $(lsb_release -sc)-proposed main > "$p" &&
apt-get update -q && apt-get -qy install cloud-init'
$ lxc exec $name -- sh -c '
cd /var/lib/cloud && for d in ; do [ "$d" = "seed" ] || rm -Rf "$d"; done
rm -Rf /var/log/cloud-init'

$ lxc exec $name reboot

$ lxc exec $name -- grep WARN /var/log/cloud-init.log || echo no warn

this should show us archive used.

$ lxc exec $name -- grep us.archive /etc/apt/sources.list | grep deb | tail -n 3
$ lxc exec $name -- cat /etc/apt/apt.conf.d/95cloud-init-proxy

[Regression Potential]
The apt feature itself (added under bug 1574113) could have regressions, as seen in bug 1621180.
This fix specifically relaxed cloud-init's behavior, making it more backwards compatible. It should not be too prone to regression itself.

==== End SRU Template ====

Trying to use the new configuration format of APT configuration while still providing the OLD format, causes cloud-init fails to configure APT.

cloud-init should be ignoring the old format if the new format is provided to ensure backwards compat.

This is a problem for MAAS provided that we cannot safely differentiate / determine what cloud-init version we are using for a specific release we are deploying, and as such, we still need to send the old config while still providing the new one because:

  1. Yakkety uses newer cloud-init with new format above
  2. Xenial, Trusty, Precise use older cloud-init that doesn't support new format.

And this is a problem because:

  1. MAAS won't be able to use derived repositories in Xenial, Trusty, Precise until this gets backported into cloud-init.
  2. Commission is done in Xenial, while deployment in Yakkety, but both may require the same config, but it is only supported in Yakkety's cloud-init.
  3. Users may be using old images that may not contain new cloud-init at all, and even though the release already supports it, the image they are using doesn't and they have to continue to use the old format.
  4. MAAS cannot differentiate/identify which cloud-init version its being used, as such, needs to sends both old and new config.

Aug 25 09:44:17 node02 [CLOUDINIT] cc_apt_configure.py[ERROR]: Error in apt configuration: old and new format of apt features are mutually exclusive ('apt':'{'primary': [{'arches': ['default'], 'uri': 'http://us.archive.ubuntu.com/ubuntu'}], 'preserve_sources_list': True, 'security': [{'arches': ['default'], 'uri': 'http://us.archive.ubuntu.com/ubuntu'}], 'sources': {'launchpad_3': {'source': 'deb http://ppa.launchpad.net/maas/next/ubuntu yakkety main'}}}' vs 'apt_proxy' key)

Aug 25 09:51:58 node02 [CLOUDINIT] util.py[DEBUG]: Running module apt-configure (<module 'cloudinit.config.cc_apt_configure' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_apt_configure.py'>) failed#012Traceback (most recent call last):#12 File "/usr/lib/python3/dist-packages/cloudinit/stages.py", line 785, in _run_modules#012 freq=freq)#12 File "/usr/lib/python3/dist-packages/cloudinit/cloud.py", line 70, in run#012 return self._runners.run(name, functor, args, freq, clear_on_fail)#12 File "/usr/lib/python3/dist-packages/cloudinit/helpers.py", line 199, in run#012 results = functor(*args)#12 File "/usr/lib/python3/dist-packages/cloudinit/config/cc_apt_configure.py", line 77, in handle#012 ocfg = convert_to_v3_apt_format(ocfg)#12 File "/usr/lib/python3/dist-packages/cloudinit/config/cc_apt_configure.py", line 527, in convert_to_v3_apt_format#012 cfg = convert_v2_to_v3_apt_format(cfg)#12 File "/usr/lib/python3/dist-packages/cloudinit/config/cc_apt_configure.py", line 489, in convert_v2_to_v3_apt_format#012 raise ValueError(msg)#012ValueError: Error in apt configuration: old and new format of apt features are mutually exclusive ('apt':'{'preserve_sources_list': True, 'primary': [{'uri': 'http://us.archive.ubuntu.com/ubuntu', 'arches': ['default']}], 'security': [{'uri': 'http://us.archive.ubuntu.com/ubuntu', 'arches': ['default']}], 'sources': {'launchpad_3': {'source': 'deb http://ppa.launchpad.net/maas/next/ubuntu yakkety main'}}}' vs 'apt_proxy, apt_preserve_sources_list' key)
@ubuntu-server-builder ubuntu-server-builder added launchpad Migrated from Launchpad priority Fix soon labels May 10, 2023
@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Andres Rodriguez(andreserl) wrote on 2016-08-25T09:50:21.882627+00:00

Since not all Ubuntu releases and not all users will have the latest, the new configuration format should be preferred instead of the old configuration format if both are provided, rather than failing.

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Andres Rodriguez(andreserl) wrote on 2016-08-25T15:00:59.885033+00:00

http://pastebin.ubuntu.com/23089233/

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Scott Moser(smoser) wrote on 2016-08-25T15:16:16.311859+00:00

Here is a simple recreate
$ cat > /tmp/my.user-data <<"EOF"
#cloud-config
apt:
preserve_sources_list: false
primary:

apt_proxy: http://192.168.122.1:8000/

system_info:
package_mirrors:

$ lxc launch ubuntu-daily:yakkety "--config=user.user-data=$(cat /tmp/my.user-data)"

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Christian Ehrhardt (paelzer) wrote on 2016-08-26T12:01:14.253740+00:00

Merge request with fix and tests at https://code.launchpad.net/~paelzer/cloud-init/+git/cloud-init/+merge/304064

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Launchpad Janitor(janitor) wrote on 2016-08-26T20:55:40.887754+00:00

This bug was fixed in the package cloud-init - 0.7.7-22-g763f403-0ubuntu1

cloud-init (0.7.7-22-g763f403-0ubuntu1) yakkety; urgency=medium

  • New upstream snapshot.
    • apt-config: allow both old and new format to be present.
      [Christian Ehrhardt] (LP: #1616831)
    • python2.6: fix dict comprehension usage in _lsb_release. [Joshua Harlow]
    • Add a module that can configure spacewalk. [Joshua Harlow]
    • add install option for openrc [Matthew Thode]
    • Generate a dummy bond name for OpenStack (LP: #1605749)
  • debian/control, debian/README.source: update to reference git.
  • debian/new-upstream-snapshot: small fixes.

-- Scott Moser smoser@ubuntu.com Fri, 26 Aug 2016 15:22:13 -0400

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Andres Rodriguez(andreserl) wrote on 2016-08-30T14:11:51.465407+00:00

I'm reopening this bug and this has not really been fixed:

ubuntu@node01:~$ dpkg -l | grep cloud-init
ii cloud-init 0.7.7-22-g763f403-0ubuntu1 all Init scripts for cloud instances

Config MAAS sends cloud-init: http://pastebin.ubuntu.com/23111940/

/etc/apt/sources.list: http://pastebin.ubuntu.com/23111941/

cloud-init.log: http://paste.ubuntu.com/23111945/
cloud-init-output.log: http://paste.ubuntu.com/23111947/

Note that curtin rev 418 works just fine, while cloud-init doesn't. Note that in the cloud-init config we send "preserve_sources_list: True" while in curtin config it is False.

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Andres Rodriguez(andreserl) wrote on 2016-08-30T14:18:48.316348+00:00

http://pastebin.ubuntu.com/23111966/ -> with preserve_source_list: false, then sources.list gets updated accordingly.

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Andres Rodriguez(andreserl) wrote on 2016-08-30T14:25:53.118222+00:00

Seeting back to fix-released, as now we need to send preserve_sources_list: false

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Scott Moser(smoser) wrote on 2016-09-12T20:50:27.874440+00:00

fixed in 0.7.8.

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Chris J Arges(arges) wrote on 2016-09-13T20:18:26.798125+00:00

Hello Andres, or anyone else affected,

Accepted cloud-init into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.7-31-g65ace7b-0ubuntu1~16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Martin Pitt(pitti) wrote on 2016-09-15T14:30:50.198236+00:00

Hello Andres, or anyone else affected,

Accepted cloud-init into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/cloud-init/0.7.8-1-g3705bb5-0ubuntu1~16.04.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, and change the tag from verification-needed to verification-done. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed. In either case, details of your testing will help us make a better decision.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Scott Moser(smoser) wrote on 2016-09-21T17:56:23.394972+00:00

verified as per sru template description.

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Launchpad Janitor(janitor) wrote on 2016-09-22T17:34:08.065412+00:00

This bug was fixed in the package cloud-init - 0.7.8-1-g3705bb5-0ubuntu1~16.04.1

cloud-init (0.7.8-1-g3705bb5-0ubuntu1~16.04.1) xenial-proposed; urgency=medium

  • New upstream release 0.7.8.
  • New upstream snapshot.
    • systemd: put cloud-init.target After multi-user.target (LP: #1623868)

cloud-init (0.7.7-31-g65ace7b-0ubuntu1~16.04.2) xenial-proposed; urgency=medium

  • debian/control: add Breaks of older versions of walinuxagent (LP: #1623570)

cloud-init (0.7.7-31-g65ace7b-0ubuntu1~16.04.1) xenial-proposed; urgency=medium

  • debian/control: fix missing dependency on python3-serial,
    and make SmartOS datasource work.
  • debian/cloud-init.templates fix capitalisation in template so
    dpkg-reconfigure works to select OpenStack. (LP: #1575727)
  • d/README.source, d/control, d/new-upstream-snapshot, d/rules: sync
    with yakkety for changes due to move to git.
  • d/rules: change PYVER=python3 to PYVER=3 to adjust to upstream change.
  • debian/rules, debian/cloud-init.install: remove install file
    to ensure expected files are collected into cloud-init deb.
    (LP: #1615745)
  • debian/dirs: remove obsolete / unused file.
  • upstream move from bzr to git.
  • New upstream snapshot.
    • Allow link type of null in network_data.json [Jon Grimm] (LP: #1621968)
    • DataSourceOVF: fix user-data as base64 with python3 (LP: #1619394)
    • remove obsolete .bzrignore
    • systemd: Better support package and upgrade. (LP: #1576692, #1621336)
    • tests: cleanup tempdirs in apt_source tests
    • apt config conversion: treat empty string as not provided. (LP: #1621180)
    • Fix typo in default keys for phone_home [Roland Sommer] (LP: #1607810)
    • salt minion: update default pki directory for newer salt minion.
      (LP: #1609899)
    • bddeb: add --release flag to specify the release in changelog.
    • apt-config: allow both old and new format to be present.
      [Christian Ehrhardt] (LP: #1616831)
    • python2.6: fix dict comprehension usage in _lsb_release. [Joshua Harlow]
    • Add a module that can configure spacewalk. [Joshua Harlow]
    • add install option for openrc [Matthew Thode]
    • Generate a dummy bond name for OpenStack (LP: #1605749)
    • network: fix get_interface_mac for bond slave, read_sys_net for ENOTDIR
    • azure dhclient-hook cleanups
    • Minor cleanups to atomic_helper and add unit tests.
    • Fix Gentoo net config generation [Matthew Thode]
    • distros: fix get_primary_arch method use of os.uname [Andrew Jorgensen]
    • Apt: add new apt configuration format [Christian Ehrhardt]
    • Get Azure endpoint server from DHCP client [Brent Baude]
    • DigitalOcean: use the v1.json endpoint [Ben Howard]
    • MAAS: add vendor-data support (LP: #1612313)
    • Upgrade to a configobj package new enough to work [Joshua Harlow]
    • ConfigDrive: recognize 'tap' as a link type. (LP: #1610784)
    • NoCloud: fix bug providing network-interfaces via meta-data.
      (LP: 1577982)
    • Add distro tags on config modules that should have it [Joshua Harlow]
    • ChangeLog: update changelog for previous commit.
    • add ntp config module [Ryan Harper]
    • SmartOS: more improvements for network configuration
    • tools/read-version: update to address change in version
    • make-tarball: older versions of git with --format=tar.
    • read-version: do not attempt git-describe if no git.
    • Newer requests have strong type validation [Joshua Harlow]
    • For upstream snapshot versions do not modify git-describe output.
    • adjust signal_handler for version changes.
    • revert unintended change to ubuntu sources list
    • drop modification of version during make-tarball, tools changes.
    • adjust tools and version information.
    • Update build tools to work with git [Lars Kellogg-Stedman]
    • fix pep8 errors in mcollective unit tests
    • mcollective: add tests, cleanups and bug fix when no config in /etc.

-- Scott Moser smoser@ubuntu.com Thu, 15 Sep 2016 09:57:27 -0400

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Chris J Arges(arges) wrote on 2016-09-22T17:35:39.342746+00:00

The verification of the Stable Release Update for cloud-init has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
launchpad Migrated from Launchpad priority Fix soon
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@ubuntu-server-builder