Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

dmidecode triggers system reboot on Inforce 6640 #3525

Closed
ubuntu-server-builder opened this issue May 12, 2023 · 30 comments
Closed

dmidecode triggers system reboot on Inforce 6640 #3525

ubuntu-server-builder opened this issue May 12, 2023 · 30 comments
Labels
launchpad Migrated from Launchpad

Comments

@ubuntu-server-builder
Copy link
Collaborator

This bug was originally filed in Launchpad as LP: #1858615

Launchpad details 
affected_projects = ['dmidecode (Ubuntu)', 'dmidecode (Ubuntu Xenial)', 'dmidecode (Ubuntu Bionic)', 'dmidecode (Ubuntu Eoan)', 'dmidecode (Ubuntu Focal)', 'dmidecode (Debian)']
assignee = None
assignee_name = None
date_closed = 2020-01-09T00:26:28.132896+00:00
date_created = 2020-01-07T08:29:52.548932+00:00
date_fix_committed = None
date_fix_released = None
id = 1858615
importance = undecided
is_complete = True
lp_url = https://bugs.launchpad.net/cloud-init/+bug/1858615
milestone = None
owner = ethan.hsieh
owner_name = ethan.hsieh
private = False
status = invalid
submitter = ethan.hsieh
submitter_name = ethan.hsieh
tags = ['verification-done', 'verification-done-bionic', 'verification-done-eoan', 'verification-done-xenial']
duplicates = []

Launchpad user ethan.hsieh(ethan.hsieh) wrote on 2020-01-07T08:29:52.548932+00:00

[Impact]
Running 'sudo dmidecode' on non-UEFI ARM systems can cause them to crash/reboot. cloud-init apparently runs dmidecode as root, so it breaks any cloud-init based installation.

[Test Case]
sudo dmidecode

[Fix]
Upstream has the following fix:

commit e12ec26e19e02281d3e7258c3aabb88a5cf5ec1d
Author: Jean Delvare jdelvare@suse.de
Date: Mon Aug 26 14:20:15 2019 +0200

dmidecode: Only scan /dev/mem for entry point on x86

[Regression Risk]
In Ubuntu, dmidecode only builds on amd64, arm64, armhf & i386.
The fix is to disable code on !x86, so the regression risk is restricted to ARM platforms, where we know /dev/mem trolling is bad news.
@ubuntu-server-builder ubuntu-server-builder added the launchpad Migrated from Launchpad label May 12, 2023
@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Ryan Harper(raharper) wrote on 2020-01-09T00:26:27.114717+00:00

Thanks for filing a bug. I've added a dmidecode task to track the issue with the tool. It may also affect the kernel package, and possibly firmware (though that's not something that Ubuntu provides). Cloud-init and any other tool may invoke this package and it should not reboot the system; but there's no issue with cloud-init or other callers of the tool As such, I'm marking the cloud-init task as invalid.

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user ethan.hsieh(ethan.hsieh) wrote on 2020-01-14T10:05:56.754418+00:00

@ryan
Not all aarch64 platforms can support demidecode well.
How about revert the patch? or create a black list?

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Dan Watkins(oddbloke) wrote on 2020-01-14T16:13:44.388841+00:00

Ethan, cloud-init isn't the right level to handle hardware issues with DMI (which is why we've added the dmidecode task). I've pinged kernel folks to see if we can get eyes on this issue.

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Colin Ian King(colin-king) wrote on 2020-01-14T17:32:13.042855+00:00

Does the kernel expose the DMI tables via the sysfs following sysfs file: /sys/firmware/dmi/tables/DMI ?

If so, can you do the following:

sudo cat /sys/firmware/dmi/tables/DMI > dmi.raw

and attach it to the bug report. Also a dump of the kernel dmesg log after it boots may be useful to see if it's a broken firmware DMI table or a kernel issue.

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Colin Ian King(colin-king) wrote on 2020-01-14T17:33:34.626804+00:00

Oh, stupid me, I've just read the info in comment #1

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user ethan.hsieh(ethan.hsieh) wrote on 2020-01-15T01:58:35.322325+00:00

@dan
Got it. Thanks~

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Colin Ian King(colin-king) wrote on 2020-01-27T11:31:07.891071+00:00

Hi, can you provide me instructions on how to get and install the image for this board? I'd like to reproduce this issue and get a suitable fix for this.

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Colin Ian King(colin-king) wrote on 2020-01-27T15:48:08.025090+00:00

So, dmidecode directly mmap's to /dev/mem and does some probing based on the belief that the system is a x86 architecture even on arm architectures.

openat(AT_FDCWD, "/dev/mem", O_RDONLY) = 3
fstat(3, {st_mode=S_IFCHR|0640, st_rdev=makedev(0x1, 0x1), ...}) = 0
mmap(NULL, 65536, PROT_READ, MAP_SHARED, 3, 0xf0000) = 0x7f9f6fd000

etc

So that's kind of intrusive and as root one can read any sort of physical addresses in /dev/mem that may cause breakage.

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Colin Ian King(colin-king) wrote on 2020-01-27T16:23:02.602146+00:00

dmidocode.c directly accesses memory and assumes it's an x86 without any checking that the arch is x86.. Randomly scanning arbitrary hunks of memory on non-x86 as root will lead to all sorts of woe:

memory_scan:
if (!(opt.flags & FLAG_QUIET))
printf("Scanning %s for entry point.\n", opt.devmem);
/* Fallback to memory scan (x86, x86_64) */
if ((buf = mem_chunk(0xF0000, 0x10000, opt.devmem)) == NULL)
{
ret = 1;
goto exit_free;
}

It probably needs wrapping with:

#if defined(x86_64) || defined(__x86_64) ||
defined(i386) || defined(__i386)

...

#endif

Anyhow, I don't think this is a kernel specific issue. I can trigger this with various kernels - we just don't protect users with CAP_SYS_ADMIN rights doing crazy probing on /dev/mem.

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Colin Ian King(colin-king) wrote on 2020-01-27T17:07:06.561285+00:00

I guess the next question is why dmidecode being run as root is required on a cloud init? What happens when arches don't have DMI data?

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Colin Ian King(colin-king) wrote on 2020-01-27T17:10:21.104440+00:00

Upstream has a fix like the one I was hinting at in comment #9, I'll SRU this fix.

commit e12ec26e19e02281d3e7258c3aabb88a5cf5ec1d
Author: Jean Delvare jdelvare@suse.de
Date: Mon Aug 26 14:20:15 2019 +0200

dmidecode: Only scan /dev/mem for entry point on x86

x86 is the only architecture which can have a DMI entry point scanned
from /dev/mem. Do not attempt it on other architectures, because not
only it can't work, but it can even cause the system to reboot.

This fixes support request #109697:
https://savannah.nongnu.org/support/?109697

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user dann frazier(dannf) wrote on 2020-01-27T17:12:25.707384+00:00

We carry this patch in focal - do we just need to backport it to bionic?

http://git.savannah.nongnu.org/cgit/dmidecode.git/commit/?id=e12ec26e19e02281d3e7258c3aabb88a5cf5ec1d

I uploaded a test fix to ppa:dannf/test. Could someone w/ hw access verify that?

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Colin Ian King(colin-king) wrote on 2020-01-27T17:22:17.499074+00:00

It needs backporting to eoan, disco bionic, I was just about to upload a fix to my ppa so I could get it sponsored. Do you want to take it from here Dann?

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user dann frazier(dannf) wrote on 2020-01-27T17:30:14.064347+00:00

Sure Colin, I'll take it from here - thanks for your analysis so far. As a next step, I'll wait for Ethan - or someone else w/ hw access - to verify the PPA build in Comment #12.

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user dann frazier(dannf) wrote on 2020-01-27T17:37:26.729680+00:00

Actually, I realized I can reproduce this on a dragonboard I have here. I've verified the PPA fix myself, so I'll go ahead and SRU.

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Dan Watkins(oddbloke) wrote on 2020-01-27T17:38:11+00:00

On Mon, Jan 27, 2020 at 05:07:06PM -0000, Colin Ian King wrote:

I guess the next question is why dmidecode being run as root is required
on a cloud init?

cloud-init uses DMI data to identify that it is running on certain
platforms. It will use /sys/class/dmi/... if available, but falls back
to dmidecode in its absence.

What happens when arches don't have DMI data?

If running on a platform which identifies itself only via DMI data,
cloud-init will incorrectly determine that it is not running on that
platform.

Given that other arches can provide DMI data, and this cloud-init
behaviour is known, I would be surprised if any platform that relies
on it would launch a new architecture without DMI support. So if we get
no data (instead of a reboot), then I believe the current behaviour is
correct.

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Colin Ian King(colin-king) wrote on 2020-01-27T17:48:27.598759+00:00

Dann, tested on my 6640 on an older kernel, now get:

sudo dmidecode

dmidecode 3.1

No SMBIOS nor DMI entry point found, sorry.

I guess that's expected.

I'd like to see what Ethan gets on his H/W as I'm not running a cloud installation on my dev board.

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Timo Aaltonen(tjaalton) wrote on 2020-02-07T16:43:18.718766+00:00

Hello ethan.hsieh, or anyone else affected,

Accepted dmidecode into eoan-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/dmidecode/3.2-2ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-eoan to verification-done-eoan. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-eoan. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Timo Aaltonen(tjaalton) wrote on 2020-02-07T16:48:31.862432+00:00

Hello ethan.hsieh, or anyone else affected,

Accepted dmidecode into bionic-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/dmidecode/3.1-1ubuntu0.1 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-bionic to verification-done-bionic. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-bionic. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Timo Aaltonen(tjaalton) wrote on 2020-02-07T16:51:39.972069+00:00

Hello ethan.hsieh, or anyone else affected,

Accepted dmidecode into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/dmidecode/3.0-2ubuntu0.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Ubuntu SRU Bot(ubuntu-sru-bot) wrote on 2020-02-07T18:37:15+00:00

All autopkgtests for the newly accepted dmidecode (3.2-2ubuntu0.1) for eoan have finished running.
The following regressions have been reported in tests triggered by the package:

crmsh/unknown (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/eoan/update_excuses.html#dmidecode

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Ubuntu SRU Bot(ubuntu-sru-bot) wrote on 2020-02-07T18:37:25+00:00

All autopkgtests for the newly accepted dmidecode (3.1-1ubuntu0.1) for bionic have finished running.
The following regressions have been reported in tests triggered by the package:

crmsh/3.0.1-3ubuntu1 (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/bionic/update_excuses.html#dmidecode

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user dann frazier(dannf) wrote on 2020-02-07T21:01:07.445185+00:00

e/b/x all verified on a dragonboard 410c

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user dann frazier(dannf) wrote on 2020-02-07T22:42:53.184175+00:00

Both crmsh autopkgtest failures could be explained by a flaky network connection. The eoan one is certainly that - it's unable to connect to the archive. The error from the bionic case is masked - but the test is trying to apt install vim. That could obviously also be an archive connectivity issue. I retried both tests, and they now appear to have passed.

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user ethan.hsieh(ethan.hsieh) wrote on 2020-02-08T04:35:49.615249+00:00

@colin @dann @timo

I have verified the following packages on my inforce 6640.
eoan-proposed: dmidecode_3.2-2ubuntu0.1_arm64.deb
bionic-proposed: dmidecode_3.1-1ubuntu0.1_arm64.deb
xenial-proposed: dmidecode_3.0-2ubuntu0.2_arm64.deb

They look good.
When I run dmidecode, I get the following message instead of system reboot.

dmidecode 3.X

No SMBIOS nor DMI entry point found, sorry.

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user ethan.hsieh(ethan.hsieh) wrote on 2020-02-08T06:28:27.164282+00:00

Verified dmidecode_(3.1-1ubuntu0.1_amd64.deb) on my x86 laptop (Ubuntu 18.04.3 LTS, Desktop).
It works well.

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Launchpad Janitor(janitor) wrote on 2020-02-17T09:18:49.132173+00:00

This bug was fixed in the package dmidecode - 3.2-2ubuntu0.1

dmidecode (3.2-2ubuntu0.1) eoan; urgency=medium

  • Restrict /dev/mem scanning for entry point to x86, fixing crashes
    on certain ARM platforms (LP: #1858615):
    • New debian/patches/0140-Fix_scan_entry_point.patch (Cherry picked from
      upstream Commit e12ec26e19e02281d3e7258c3aabb88a5cf5ec1d.

-- dann frazier dannf@ubuntu.com Mon, 27 Jan 2020 10:55:41 -0700

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Łukasz Zemczak(sil2100) wrote on 2020-02-17T09:18:54.421084+00:00

The verification of the Stable Release Update for dmidecode has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Launchpad Janitor(janitor) wrote on 2020-02-17T09:52:58.218543+00:00

This bug was fixed in the package dmidecode - 3.1-1ubuntu0.1

dmidecode (3.1-1ubuntu0.1) bionic; urgency=medium

  • Restrict /dev/mem scanning for entry point to x86, fixing crashes
    on certain ARM platforms (LP: #1858615):
    • New debian/patches/0140-Fix_scan_entry_point.patch (Cherry picked from
      upstream Commit e12ec26e19e02281d3e7258c3aabb88a5cf5ec1d.

-- dann frazier dannf@ubuntu.com Mon, 27 Jan 2020 11:09:10 -0700

@ubuntu-server-builder
Copy link
Collaborator Author

Launchpad user Launchpad Janitor(janitor) wrote on 2020-02-17T10:10:04.693797+00:00

This bug was fixed in the package dmidecode - 3.0-2ubuntu0.2

dmidecode (3.0-2ubuntu0.2) xenial; urgency=medium

  • Restrict /dev/mem scanning for entry point to x86, fixing crashes
    on certain ARM platforms (LP: #1858615):
    • New debian/patches/0140-Fix_scan_entry_point.patch (Cherry picked from
      upstream Commit e12ec26e19e02281d3e7258c3aabb88a5cf5ec1d.

-- dann frazier dannf@ubuntu.com Mon, 27 Jan 2020 11:29:42 -0700

@ubuntu-server-builder ubuntu-server-builder closed this as not planned Won't fix, can't repro, duplicate, stale May 12, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
launchpad Migrated from Launchpad
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@ubuntu-server-builder