Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cloud-init SIGTERM error #4404

Closed
kideg20 opened this issue Sep 1, 2023 · 7 comments
Closed

cloud-init SIGTERM error #4404

kideg20 opened this issue Sep 1, 2023 · 7 comments
Labels
bug Something isn't working correctly incomplete Action required by submitter

Comments

@kideg20
Copy link

kideg20 commented Sep 1, 2023 •
edited
Loading

Problem

When I run the apt-get update,dist-upgrade or install command for the first time, cloud-init fails with the error 'util.py[DEBUG]: Cloud-init 23.2.2-0ubuntu0~22.04.1 received SIGTERM, exiting...'*
after sigterm cloud-init is restarted and continues to run, apt-get works fine.

Componetns

Ubuntu 22.04.3 LTS
jammy-server-cloudimg-amd64.ova https://cloud-images.ubuntu.com/jammy/current/jammy-server-cloudimg-amd64.ova 2023-08-28 22:40

/usr/bin/cloud-init 23.2.2-0ubuntu0~22.04.1

systemd 249 (249.11-0ubuntu3.9)

apt-get update Error :

023-09-01 17:49:01,646 - helpers.py[DEBUG]: Running update-sources using lock (<FileLock using file '/var/lib/cloud/instances/423b337e-c809-8b6a-c132-b261937d0329/sem/update_sources'>)
2023-09-01 17:49:01,646 - debian.py[DEBUG]: Waiting for apt lock
2023-09-01 17:49:01,647 - debian.py[DEBUG]: apt lock available
2023-09-01 17:49:01,647 - subp.py[DEBUG]: Running command ['eatmydata', 'apt-get', '--option=Dpkg::Options::=--force-confold', '--option=Dpkg::options::=--force-unsafe-io', '--assume-yes', '--quiet', 'update'] with allowed return codes [0] (shell=False, capture=False)
2023-09-01 17:49:13,110 - util.py[DEBUG]: apt-update [eatmydata apt-get --option=Dpkg::Options::=--force-confold --option=Dpkg::options::=--force-unsafe-io --assume-yes --quiet update] took 11.463 seconds
2023-09-01 17:49:13,111 - handlers.py[DEBUG]: finish: modules-config/config-apt-configure: SUCCESS: config-apt-configure ran successfully
2023-09-01 17:49:13,111 - modules.py[DEBUG]: Running module runcmd (<module 'cloudinit.config.cc_runcmd' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_runcmd.py'>) with frequency once-per-instance
2023-09-01 17:49:13,111 - handlers.py[DEBUG]: start: modules-config/config-runcmd: running config-runcmd with frequency once-per-instance
2023-09-01 17:49:13,111 - util.py[DEBUG]: Writing to /var/lib/cloud/instances/423b337e-c809-8b6a-c132-b261937d0329/sem/config_runcmd - wb: [644] 25 bytes
2023-09-01 17:49:13,112 - helpers.py[DEBUG]: Running config-runcmd using lock (<FileLock using file '/var/lib/cloud/instances/423b337e-c809-8b6a-c132-b261937d0329/sem/config_runcmd'>)
2023-09-01 17:49:13,112 - util.py[DEBUG]: Shellified 2 commands.
2023-09-01 17:49:13,112 - util.py[DEBUG]: Writing to /var/lib/cloud/instances/423b337e-c809-8b6a-c132-b261937d0329/scripts/runcmd - wb: [700] 105 bytes
2023-09-01 17:49:13,112 - handlers.py[DEBUG]: finish: modules-config/config-runcmd: SUCCESS: config-runcmd ran successfully
2023-09-01 17:49:13,112 - modules.py[DEBUG]: Running module byobu (<module 'cloudinit.config.cc_byobu' from '/usr/lib/python3/dist-packages/cloudinit/config/cc_byobu.py'>) with frequency once-per-instance
2023-09-01 17:49:13,113 - handlers.py[DEBUG]: start: modules-config/config-byobu: running config-byobu with frequency once-per-instance
2023-09-01 17:49:13,113 - util.py[DEBUG]: Writing to /var/lib/cloud/instances/423b337e-c809-8b6a-c132-b261937d0329/sem/config_byobu - wb: [644] 25 bytes
2023-09-01 17:49:13,113 - helpers.py[DEBUG]: Running config-byobu using lock (<FileLock using file '/var/lib/cloud/instances/423b337e-c809-8b6a-c132-b261937d0329/sem/config_byobu'>)
2023-09-01 17:49:13,113 - cc_byobu.py[DEBUG]: Skipping module named byobu, no 'byobu' values found
2023-09-01 17:49:13,113 - handlers.py[DEBUG]: finish: modules-config/config-byobu: SUCCESS: config-byobu ran successfully
2023-09-01 17:49:13,113 - main.py[DEBUG]: Ran 7 modules with 0 failures
2023-09-01 17:49:13,114 - atomic_helper.py[DEBUG]: Atomically writing to file /var/lib/cloud/data/status.json (via temporary file /var/lib/cloud/data/tmpkqahxcxs) - w: [644] 561 bytes/chars
2023-09-01 17:49:13,114 - util.py[DEBUG]: Reading from /proc/uptime (quiet=False)
2023-09-01 17:49:13,114 - util.py[DEBUG]: Read 12 bytes from /proc/uptime
2023-09-01 17:49:13,114 - util.py[DEBUG]: cloud-init mode 'modules' took 12.943 seconds (12.94)
2023-09-01 17:49:13,114 - handlers.py[DEBUG]: finish: modules-config: SUCCESS: running modules for config
2023-09-01 17:49:13,492 - util.py[DEBUG]: Cloud-init v. 23.2.2-0ubuntu0~22.04.1 running 'modules:final' at Fri, 01 Sep 2023 17:49:13 +0000. Up 62.98 seconds.
2023-09-01 17:49:13,494 - util.py[DEBUG]: Reading from /usr/lib/python3/dist-packages/cloudinit/config/schemas/schema-cloud-config-v1.json (quiet=False)
2023-09-01 17:49:13,494 - util.py[DEBUG]: Read 135437 bytes from /usr/lib/python3/dist-packages/cloudinit/config/schemas/schema-cloud-config-v1.json
2023-09-01 17:49:13,500 - util.py[DEBUG]: Reading from /usr/lib/python3/dist-packages/cloudinit/config/schemas/schema-cloud-config-v1.json (quiet=False)
2023-09-01 17:49:13,500 - util.py[DEBUG]: Read 135437 bytes from /usr/lib/python3/dist-packages/cloudinit/config/schemas/schema-cloud-config-v1.json
2023-09-01 17:49:13,502 - util.py[DEBUG]: Reading from /usr/lib/python3/dist-packages/cloudinit/config/schemas/schema-cloud-config-v1.json (quiet=False)
2023-09-01 17:49:13,503 - util.py[DEBUG]: Read 135437 bytes from /usr/lib/python3/dist-packages/cloudinit/config/schemas/schema-cloud-config-v1.json
2023-09-01 17:49:13,515 - util.py[DEBUG]: Reading from /usr/lib/python3/dist-packages/cloudinit/config/schemas/schema-cloud-config-v1.json (quiet=False)
2023-09-01 17:49:13,521 - util.py[DEBUG]: Read 135437 bytes from /usr/lib/python3/dist-packages/cloudinit/config/schemas/schema-cloud-config-v1.json
2023-09-01 17:49:13,675 - util.py[DEBUG]: Cloud-init 23.2.2-0ubuntu0~22.04.1 received SIGTERM, exiting...
  Filename: <frozen importlib._bootstrap_external>
  Function: get_data
  Line number: 1074
    Filename: <frozen importlib._bootstrap_external>
    Function: get_code
    Line number: 975
      Filename: <frozen importlib._bootstrap_external>
      Function: exec_module
      Line number: 879
2023-09-01 17:49:13,676 - util.py[DEBUG]: Reading from /proc/uptime (quiet=False)
2023-09-01 17:49:13,676 - util.py[DEBUG]: Read 12 bytes from /proc/uptime
2023-09-01 17:49:13,676 - util.py[DEBUG]: cloud-init mode 'modules' took 0.276 seconds (0.27)
2023-09-01 17:49:13,676 - handlers.py[DEBUG]: finish: modules-final: FAIL: running modules for final

apt-get install error:

2023-09-01 17:59:52,821 - debian.py[DEBUG]: Waiting for apt lock
2023-09-01 17:59:52,821 - debian.py[DEBUG]: apt lock available
2023-09-01 17:59:52,821 - subp.py[DEBUG]: Running command ['eatmydata', 'apt-get', '--option=Dpkg::Options::=--force-confold', '--option=Dpkg::options::=--force-unsafe-io', '--assume-yes', '--quiet', 'install', 'itop'] with allowed return codes [0] (shell=False, capture=False)
2023-09-01 17:59:55,201 - util.py[DEBUG]: Cloud-init 23.2.2-0ubuntu0~22.04.1 received SIGTERM, exiting...
  Filename: /usr/lib/python3.10/subprocess.py
  Function: _try_wait
  Line number: 1917
    Filename: /usr/lib/python3.10/subprocess.py
    Function: _wait
    Line number: 1959
      Filename: /usr/lib/python3.10/subprocess.py
      Function: wait
      Line number: 1209
2023-09-01 17:59:55,201 - util.py[DEBUG]: apt-install [eatmydata apt-get --option=Dpkg::Options::=--force-confold --option=Dpkg::options::=--force-unsafe-io --assume-yes --quiet install itop] took 2.379 seconds
2023-09-01 17:59:55,201 - handlers.py[DEBUG]: finish: modules-final/config-package-update-upgrade-install: FAIL: running config-package-update-upgrade-install with frequency once-per-instance
2023-09-01 17:59:55,201 - util.py[DEBUG]: Reading from /proc/uptime (quiet=False)
2023-09-01 17:59:55,201 - util.py[DEBUG]: Read 12 bytes from /proc/uptime
2023-09-01 17:59:55,201 - util.py[DEBUG]: cloud-init mode 'modules' took 7.893 seconds (7.89)
2023-09-01 17:59:55,201 - handlers.py[DEBUG]: finish: modules-final: FAIL: running modules for final

cloud-config:

Content-Type: multipart/mixed; boundary="MIMEBOUNDARY"
MIME-Version: 1.0

--MIMEBOUNDARY
Content-Transfer-Encoding: 7bit
Content-Type: text/cloud-config
Mime-Version: 1.0


apt:
  sources:
    kubernetes.list:
      source: "deb http://apt.kubernetes.io/ kubernetes-xenial main"
      keyserver: "hkp://keyserver.ubuntu.com:80"
      keyid: "13EDEF05"
    docker.list:
      source: "deb https://download.docker.com/linux/ubuntu jammy stable"
      keyserver: "hkp://keyserver.ubuntu.com:80"
      keyid: "0EBFCD88"
    hashicorp.list:
      source: "deb https://apt.releases.hashicorp.com jammy main"
      keyserver: "hkp://keyserver.ubuntu.com:80"
      keyid: "A621E701"
    nginx.list:
      source: "deb http://nginx.org/packages/mainline/ubuntu jammy nginx"
      keyserver: "hkp://keyserver.ubuntu.com:80"
      keyid: "7BD9BF62"
    elastic.list:
      source: "deb https://artifacts.elastic.co/packages/8.x/apt stable main"
      keyserver: "hkp://keyserver.ubuntu.com:80"
      keyid: "D88E42B4"
    postgresql.list:
      source: "deb http://apt.postgresql.org/pub/repos/apt/ jammy-pgdg main 15"
      keyserver: "hkp://keyserver.ubuntu.com:80"
      keyid: "7FCC7D46ACCC4CF8"

packages:
  - "iftop"

package_update: true

package_upgrade: true
--MIMEBOUNDARY
Content-Transfer-Encoding: 7bit
Content-Type: text/cloud-config
Mime-Version: 1.0









--MIMEBOUNDARY
Content-Transfer-Encoding: 7bit
Content-Type: text/cloud-config
Mime-Version: 1.0



--MIMEBOUNDARY
Content-Transfer-Encoding: 7bit
Content-Type: text/cloud-config
Mime-Version: 1.0

runcmd:
  - touch /var/log/cloud-init.vanilla-to-upgraded
  - echo "echo after cloud-init.vanilla-to-upgraded"

--MIMEBOUNDARY
Content-Transfer-Encoding: 7bit
Content-Type: text/cloud-config
Mime-Version: 1.0

users:
  - name: ubuntu
    homedir: /home/ubuntu
    shell: /bin/bash
    lock_passwd: false
    sudo: 'ALL=(ALL) NOPASSWD:ALL'
    groups:
      - sudo
    ssh_authorized_keys:
      - ecdsa-sha2-nistp384 AAAAE2VjZ........................................................mi1/gNr7SjK0Tzz+cMaj1KzQZ+EEaLAox++qnRsMrYBQSNLeuY2Iow==

--MIMEBOUNDARY--
@kideg20 kideg20 added bug Something isn't working correctly new An issue that still needs triage labels Sep 1, 2023
@kideg20
Copy link
Author

kideg20 commented Sep 4, 2023 •
edited
Loading

From Journalctl, I found that after Cloud-init init-local, systemd triggers a systemd reload (which restarts the computer), but the network still starts up. And after starting the network (systemd-networkd-wait-online.service), systemd starts cloud-init.service, but the system still shuts down all services and reboots. At the end of the reboot process systemd-shutdown[1]: Sending SIGTERM to other processes...

In the February release of Ubuntu 22.04 LTS. cloud-init init-local systemd starts a reload (reboot) of the system and also tries to start the network, but does not start the systemd-networkd-wait-online.service. So cloud-init.service is not running and is not receiving any SIGTERMs.

Sep 01 17:48:19 ubuntu cloud-init[665]: Cloud-init v. 23.2.2-0ubuntu0~22.04.1 running 'init-local' at Fri, 01 Sep 2023 17:48:19 +0000. Up 8.59 seconds.
Sep 01 17:48:19 ubuntu audit[659]: AVC apparmor="STATUS" operation="profile_load" profile="unconfined" name="snap.lxd.buginfo" pid=659 comm="apparmor_parser"
Sep 01 17:48:19 ubuntu audit[675]: AVC apparmor="STATUS" operation="profile_load" profile="unconfined" name="snap.lxd.check-kernel" pid=675 comm="apparmor_parser"
Sep 01 17:48:19 ubuntu systemd[1]: Condition check resulted in OpenVSwitch configuration for cleanup being skipped.
Sep 01 17:48:20 ubuntu systemd[1]: Finished Initial cloud-init job (pre-networking).
Sep 01 17:48:20 ubuntu systemd[1]: Reached target Preparation for Network.
Sep 01 17:48:20 ubuntu systemd[1]: Starting Network Configuration...
Sep 01 17:48:20 ubuntu audit[684]: AVC apparmor="STATUS" operation="profile_load" profile="unconfined" name="snap.lxd.daemon" pid=684 comm="apparmor_parser"
Sep 01 17:48:20 ubuntu systemd-networkd[719]: lo: Link UP
Sep 01 17:48:20 ubuntu systemd-networkd[719]: lo: Gained carrier
Sep 01 17:48:20 ubuntu systemd-networkd[719]: Enumeration completed
Sep 01 17:48:20 ubuntu systemd[1]: Started Network Configuration.
Sep 01 17:48:20 ubuntu systemd[1]: Starting Wait for Network to be Configured...
Sep 01 17:48:20 ubuntu kernel: vmxnet3 0000:0b:00.0 ens192: intr type 3, mode 0, 2 vectors allocated
Sep 01 17:48:20 ubuntu kernel: vmxnet3 0000:0b:00.0 ens192: NIC Link is Down
Sep 01 17:48:20 ubuntu systemd-networkd[719]: ens192: Link UP
Sep 01 17:48:20 ubuntu systemd[1]: Starting Network Name Resolution...
Sep 01 17:48:20 ubuntu systemd-resolved[730]: Positive Trust Anchors:
Sep 01 17:48:20 ubuntu systemd-resolved[730]: . IN DS 20326 8 2 e06d44b80b8f1d39a95c0b0d7c65d08458e880409bbc683457104237c7f8ec8d
Sep 01 17:48:20 ubuntu systemd-resolved[730]: Negative trust anchors: home.arpa 10.in-addr.arpa 16.172.in-addr.arpa 17.172.in-addr.arpa 18.172.in-addr.arpa 19.172.in-addr.arpa 20.172.in-addr.arpa 21.172.in-addr.arpa 22.172.in-addr.arpa 23.172.in-addr.arpa 24.172.in-addr.arpa 25.172.in-addr.arpa 26.172.in-addr.arpa 27.172.in-addr.arpa 28.172.in-addr.arpa 29.172.in-addr.arpa 30.172.in-addr.arpa 31.172.in-addr.arpa 168.192.in-addr.arpa d.f.ip6.arpa corp home internal intranet lan local private test
Sep 01 17:48:20 ubuntu systemd-resolved[730]: Using system hostname 'ubuntu'.
Sep 01 17:48:20 ubuntu systemd[1]: Started Network Name Resolution.
Sep 01 17:48:20 ubuntu systemd[1]: Reached target Network.
Sep 01 17:48:20 ubuntu systemd[1]: Reached target Host and Network Name Lookups.
Sep 01 17:48:20 ubuntu audit[728]: AVC apparmor="STATUS" operation="profile_load" profile="unconfined" name="snap.lxd.hook.configure" pid=728 comm="apparmor_parser"
Sep 01 17:48:20 ubuntu audit[749]: AVC apparmor="STATUS" operation="profile_load" profile="unconfined" name="snap.lxd.hook.install" pid=749 comm="apparmor_parser"
Sep 01 17:48:20 ubuntu audit[750]: AVC apparmor="STATUS" operation="profile_load" profile="unconfined" name="snap.lxd.hook.remove" pid=750 comm="apparmor_parser"
Sep 01 17:48:20 ubuntu audit[751]: AVC apparmor="STATUS" operation="profile_load" profile="unconfined" name="snap.lxd.lxc" pid=751 comm="apparmor_parser"
Sep 01 17:48:21 ubuntu audit[752]: AVC apparmor="STATUS" operation="profile_load" profile="unconfined" name="snap.lxd.lxc-to-lxd" pid=752 comm="apparmor_parser"
Sep 01 17:48:21 ubuntu audit[753]: AVC apparmor="STATUS" operation="profile_load" profile="unconfined" name="snap.lxd.lxd" pid=753 comm="apparmor_parser"
Sep 01 17:48:21 ubuntu audit[754]: AVC apparmor="STATUS" operation="profile_load" profile="unconfined" name="snap.lxd.migrate" pid=754 comm="apparmor_parser"
Sep 01 17:48:21 ubuntu audit[755]: AVC apparmor="STATUS" operation="profile_load" profile="unconfined" name="snap.lxd.user-daemon" pid=755 comm="apparmor_parser"
Sep 01 17:48:21 ubuntu systemd[1]: Finished Load AppArmor profiles managed internally by snapd.
Sep 01 17:48:30 del-u2204-2309011741-upgraded systemd-resolved[730]: System hostname changed to 'del-u2204-2309011741-upgraded'.
Sep 01 17:48:30 del-u2204-2309011741-upgraded systemd[1]: Reloading.
Sep 01 17:48:31 del-u2204-2309011741-upgraded systemd-udevd[822]: Using default interface naming scheme 'v249'.
Sep 01 17:48:31 del-u2204-2309011741-upgraded systemd[1]: Condition check resulted in OpenVSwitch configuration for cleanup being skipped.
Sep 01 17:48:31 del-u2204-2309011741-upgraded systemd-networkd[719]: ens192: DHCPv6 lease lost
Sep 01 17:48:31 del-u2204-2309011741-upgraded systemd[1]: Stopping Network Configuration...
Sep 01 17:48:31 del-u2204-2309011741-upgraded systemd[1]: systemd-networkd.service: Deactivated successfully.
Sep 01 17:48:31 del-u2204-2309011741-upgraded systemd[1]: Stopped Network Configuration.
Sep 01 17:48:31 del-u2204-2309011741-upgraded systemd[1]: Starting Network Configuration...
Sep 01 17:48:31 del-u2204-2309011741-upgraded systemd-networkd[826]: ens192: Link UP
Sep 01 17:48:31 del-u2204-2309011741-upgraded systemd-networkd[826]: lo: Link UP
Sep 01 17:48:31 del-u2204-2309011741-upgraded systemd-networkd[826]: lo: Gained carrier
Sep 01 17:48:31 del-u2204-2309011741-upgraded systemd-networkd[826]: Enumeration completed
Sep 01 17:48:31 del-u2204-2309011741-upgraded systemd[1]: Started Network Configuration.
Sep 01 17:48:31 del-u2204-2309011741-upgraded systemd[1]: Condition check resulted in OpenVSwitch configuration for cleanup being skipped.
Sep 01 17:48:36 del-u2204-2309011741-upgraded kernel: vmxnet3 0000:0b:00.0 ens192: NIC Link is Up 10000 Mbps
Sep 01 17:48:36 del-u2204-2309011741-upgraded kernel: IPv6: ADDRCONF(NETDEV_CHANGE): ens192: link becomes ready
Sep 01 17:48:36 del-u2204-2309011741-upgraded systemd-networkd[826]: ens192: Gained carrier
Sep 01 17:48:36 del-u2204-2309011741-upgraded systemd-timesyncd[606]: Network configuration changed, trying to establish connection.
Sep 01 17:48:37 del-u2204-2309011741-upgraded systemd-networkd[826]: ens192: Gained IPv6LL
Sep 01 17:48:46 del-u2204-2309011741-upgraded systemd[1]: systemd-fsckd.service: Deactivated successfully.
Sep 01 17:48:46 del-u2204-2309011741-upgraded systemd-timesyncd[606]: Timed out waiting for reply from 91.189.91.157:123 (ntp.ubuntu.com).
Sep 01 17:48:49 del-u2204-2309011741-upgraded systemd-networkd-wait-online[729]: managing: ens192
Sep 01 17:48:49 del-u2204-2309011741-upgraded systemd[1]: Finished Wait for Network to be Configured.
Sep 01 17:48:50 del-u2204-2309011741-upgraded systemd[1]: Starting Initial cloud-init job (metadata service crawler)...
Sep 01 17:48:50 del-u2204-2309011741-upgraded cloud-init[847]: Cloud-init v. 23.2.2-0ubuntu0~22.04.1 running 'init' at Fri, 01 Sep 2023 17:48:50 +0000. Up 39.80 seconds.
...
...shutting down
...some cloud-init jobs
...shutting down
...
Sep 01 17:49:11 del-u2204-2309011741-upgraded systemd[1]: Finished System Reboot.
Sep 01 17:49:11 del-u2204-2309011741-upgraded systemd[1]: Reached target System Reboot.
Sep 01 17:49:11 del-u2204-2309011741-upgraded systemd[1]: Shutting down.
Sep 01 17:49:11 del-u2204-2309011741-upgraded systemd-shutdown[1]: Syncing filesystems and block devices.
Sep 01 17:49:11 del-u2204-2309011741-upgraded systemd-shutdown[1]: Sending SIGTERM to remaining processes...
Sep 01 17:49:11 del-u2204-2309011741-upgraded unknown[616]: Processing SIGTERM; service exiting
Sep 01 17:49:11 del-u2204-2309011741-upgraded unknown[616]: ServiceEndMainLoop: about to stop main loop
Sep 01 17:49:11 del-u2204-2309011741-upgraded unknown[616]: END SERVICE
Sep 01 17:49:11 del-u2204-2309011741-upgraded systemd-journald[411]: Journal stopped

@kideg20 kideg20 changed the title cloud-init apt-get update install SIGTERM error cloud-init SIGTERM error Sep 5, 2023
@blackboxsw
Copy link
Collaborator

@kideg20 thank you for filing this bug and making cloud-init better. I have a couple of concens that I don't think are represented in this bug and I am unable to reproduce this on recent Ubuntu 22.04 Jammy with your provided MIME user-data.

apt receiving a SIGTERM during early boot is something this is indicative of a separate process (possibly unattended-upgrades) generating that signal as cloud-init is trying to install/update packages. So, something outside of cloud-init is sending APT this signal, and/or forcing the system to reboot

Also, cloud-init doesn't generally trigger a reboot of your system unless either package_reboot_if_required: true or power_state:\n mode: reboot were set in cloud-config. So, the reboots triggered also points to some other service in the images.

Some questions and requests:

  1. Can you please provide the procedure or steps you are using to reproduce this issue? If I launch daily images of Jammy with the MIME userdata provided, I'm not hitting these SIGTERM condition nor reboots.
  2. Can you attach /var/log/apt/history.log and /var/log/unattended-upgrades/unattended-upgrades.log
  3. Please also attach the cloud-init.tar.gz file produced by cloud-init collect-logs as that will grab the full journal log as well as the complete cloud-init.log for debugging.

@blackboxsw
Copy link
Collaborator

marking incomplete until we get a bit more information here or a reproducible procedure. thanks again.

@blackboxsw blackboxsw added the incomplete Action required by submitter label Sep 7, 2023
@kideg20
Copy link
Author

kideg20 commented Sep 7, 2023 •
edited
Loading

I am using VMware. And I create a virtual machine through terraform. With terraform i clone ova template with linux_options options:

clone {
      template_uuid = ...
      configure {
        linux_options {
          ....
        }
      }
}

If linux_options are defined at the beginning of the first boot after running cloud-init init-local, the system triggers a reboot, but the network still starts.

The difference between the old and new Ubuntu image are:

1.) On older Ubuntu 22.04 LTS systems, April 2023, networking did not start until the reboot was complete, and Cloud-init did not start until the reboot was complete. Cloud-init started after reboot and everything worked fine after reboot.

2.) in the latest version of Ubuntu (22.04 LTS from 08/28/2023), the network was able to start during the system reboot, and Cloud-init init was able to start before the reboot was completed. At the last stage of the reboot, it sends a SIGTERM to all processes. This causes Cloud-init to receive sigterm.

@blackboxsw blackboxsw removed the new An issue that still needs triage label Sep 7, 2023
@blackboxsw
Copy link
Collaborator

@kideg20 thank you for filing this bug and the supplemental details that indicate you are running VMware.

This looks conspicuosly like #4188 which was a change in open-vm-tools basically automatically triggers a reboot during provisioning stage after a 30 second timeout which prompted @vitality411 to file an upstream issue vmware/open-vm-tools#684 to track/improve this behavior change where possible. I'm guessing that is where your SIGTERMs are coming from. Does this symptom align with that upstream bug in your case?

@blackboxsw
Copy link
Collaborator

Duplicate of #4188

@blackboxsw blackboxsw marked this as a duplicate of #4188 Sep 26, 2023
@blackboxsw
Copy link
Collaborator

//

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working correctly incomplete Action required by submitter
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@blackboxsw @kideg20