Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

iotedge check crashes on 1.0.9 under Yocto Linux and Rust 1.41.0 #2747

Closed
surlemur opened this issue Mar 27, 2020 · 10 comments
Closed

iotedge check crashes on 1.0.9 under Yocto Linux and Rust 1.41.0 #2747

surlemur opened this issue Mar 27, 2020 · 10 comments
Assignees
@dylanbronson
Labels
needinfo Needs more info from issue reporter / updates from PR submitter no-issue-activity

Comments

@surlemur
Copy link

surlemur commented Mar 27, 2020 •
edited
Loading

Expected Behavior

iotedge check should output the result of some checks

Current Behavior

iotedge check crashes (output with RUST_BACKTRACE=full):

root@edge:~# iotedge check
thread 'tokio-runtime-worker-1' panicked at 'byte index 6 is out of bounds of `temp1`', src/libcore/str/mod.rs:2052:9
stack backtrace:
   0:   0x91d76e - <unknown>
   1:   0x931b4e - <unknown>
   2:   0x91c2c8 - <unknown>
   3:   0x91f61a - <unknown>
   4:   0x91f420 - <unknown>
   5:   0x91fa9e - <unknown>
   6:   0x91f7fc - <unknown>
   7:   0x9316b0 - <unknown>
   8:   0x92e6c0 - <unknown>
   9:   0x765238 - <unknown>
  10:   0x7666f8 - <unknown>
  11:   0x75c098 - <unknown>
  12:   0x5c10b0 - <unknown>
  13:   0x5c8544 - <unknown>
  14:   0x5242b4 - <unknown>
  15:   0x5202e6 - <unknown>
  16:   0x5260b6 - <unknown>
  17:   0x8b14ce - <unknown>
  18:   0x8b2adc - <unknown>
  19:   0x92115a - <unknown>
  20:   0x8b1b0e - <unknown>
  21:   0x8ad602 - <unknown>
  22:   0x8acbae - <unknown>
  23:   0x89da0c - <unknown>
  24:   0x89dbd0 - <unknown>
  25:   0x89db04 - <unknown>
  26:   0x89ed56 - <unknown>
  27:   0x8b0eb4 - <unknown>
  28:   0x8b0ffc - <unknown>
  29:   0x8b2320 - <unknown>
  30:   0x8b2a86 - <unknown>
  31:   0x92115a - <unknown>
  32:   0x8b3a30 - <unknown>
  33:   0x90f596 - <unknown>
  34:   0x91cd44 - <unknown>
  35:   0x915966 - <unknown>
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Canceled', src/libcore/result.rs:1188:5
stack backtrace:
   0:   0x91d76e - <unknown>
   1:   0x931b4e - <unknown>
   2:   0x91c2c8 - <unknown>
   3:   0x91f61a - <unknown>
   4:   0x91f420 - <unknown>
   5:   0x91fa9e - <unknown>
   6:   0x91f7fc - <unknown>
   7:   0x9316b0 - <unknown>
   8:   0x930ede - <unknown>
   9:   0x539984 - <unknown>
  10:   0x514c4e - <unknown>
  11:   0x511ff6 - <unknown>
  12:   0x515a8e - <unknown>
  13:   0x91f716 - <unknown>
  14:   0x92115a - <unknown>
  15:   0x911bf0 - <unknown>
  16:   0x5159c0 - <unknown>
  17: 0x76c624a4 - __libc_start_main

Steps to Reproduce

Provide a detailed set of steps to reproduce the bug.

  1. iotedge check

Context (Environment)

Output of iotedge check

See above...

Device Information

  • Host OS [e.g. Ubuntu 16.04, Ubuntu 18.04, Windows IoT Core]: Yocto Linux (see above)
  • Architecture [e.g. amd64, arm32, arm64]: arm32
  • Container OS [e.g. Linux containers, Windows containers]: Linux containers

Runtime Versions

  • iotedged [run iotedge version]: iotedge 1.0.9
  • Edge Agent [image tag (e.g. 1.0.0)]: 1.0
  • Edge Hub [image tag (e.g. 1.0.0)]: 1.0
  • Docker/Moby [run docker version]: 18.09.0, go1.11.1, git commit, 489b8eda6674523df8b82a210399b7d2954427d0

Note: when using Windows containers on Windows, run docker -H npipe:////./pipe/iotedge_moby_engine version instead

Logs

iotedged logs 

<Paste here between the triple backticks>
edge-agent logs 

<Paste here between the triple backticks>
edge-hub logs 

<Paste here between the triple backticks>

Additional Information

No problems in the past under Yocto and iotedge 1.0.8. However, Rust update was necessary to get 1.0.9 to work.
@ancaantochi
Copy link
Contributor

Hi @surlemur

Just to make sure the issue is only with iotedge check, is iotedged running and starting the modules?

@ancaantochi ancaantochi self-assigned this Mar 27, 2020
@surlemur
Copy link
Author

Hi Anca
Yes, just iotedge check. iotedged is starting up correctly, and the modules are running, too. (There is some re-authentication problem later between the modules and Edge Hub, apparently, but I don't think this is related.)

@surlemur surlemur mentioned this issue Mar 30, 2020
Closed
@arsing
Copy link
Member

arsing commented Mar 30, 2020

Can't really say anything without backtraces. I assume the yocto build strips debug symbols? If so, please retry with an unstripped binary.

Otherwise, just searching for temp1 or temp{ or "temp" in all our code as well as our deps doesn't find anything, so no idea where that string is coming from and who's slicing it incorrectly. Is it something in your config.yaml?

@ancaantochi ancaantochi added the needinfo Needs more info from issue reporter / updates from PR submitter label Mar 30, 2020
@surlemur surlemur mentioned this issue Mar 30, 2020
Closed
@surlemur
Copy link
Author

surlemur commented Apr 1, 2020

I will try to test on an unstripped version as soon as possible.

@github-actions
Copy link

github-actions bot commented May 2, 2020

This issue is being marked as stale because it has been open for 30 days with no activity.

@CindyXing
Copy link
Contributor

@surlemur any update on the unstripped debug dump?

@surlemur
Copy link
Author

Hi Cindy
We recently upgraded from an ARMv7 to an ARMv8 board. On the new board with the new Yocto Linux image, the issue disappeared. I assume, the Yocto build for the previous board was referencing some incompatible Rust version.

In my view, we can close this issue.

@varunpuranik
Copy link
Contributor

Great, thanks for the update @surlemur

@Lexmark-chad Lexmark-chad mentioned this issue Oct 17, 2020
Closed
@arsing
Copy link
Member

arsing commented Oct 22, 2020

This was figured out in #3746 to be due to a bug in the sysinfo dependency when it enumerates hwmon devices looking for temp sensors. If it finds a file whose name starts with "temp" but is less than six characters long it indexes it incorrectly and causes the panic. In this user's case, the old board presumably had an hwmon device that created a file named temp1, and the new board didn't.

Our 1.0.10 release updated the sysinfo dependency to a version with the fix for this bug, so this issue can be fixed by updating iotedged to 1.0.10.

@surlemur
Copy link
Author

Thank for the update!

This was referenced Feb 18, 2021
Closed
Merged
kodiakhq bot pushed a commit that referenced this issue Feb 25, 2021
@damonbarry
Upgrade sysinfo package (#4444)
b53cc33 
This change upgrades the sysinfo package in the iotedge CLI and iotedged. It fixes bugs called out in #2747 and #3746. This change was fixed in master earlier and was supposed to by ported to the 1.1 branch, but never was.
darobs pushed a commit to darobs/iotedge that referenced this issue Mar 3, 2021
@damonbarry @darobs
Upgrade sysinfo package (Azure#4444)
16730b6 
This change upgrades the sysinfo package in the iotedge CLI and iotedged. It fixes bugs called out in Azure#2747 and Azure#3746. This change was fixed in master earlier and was supposed to by ported to the 1.1 branch, but never was.
darobs added a commit that referenced this issue Mar 3, 2021
@darobs @pmzara
@damonbarry @yophilav @ms-mahuber
Merge 1.1 to 1.1-k8s-preview (#4529)
6e775c1 
* Fix GetModules call error using RFC3339 DateTime format. (#4293)

Fix issue calling GetModuleLogs using DirectMethod with a payload that contains parameters using RF3339 DateTime format.

* Prepare for release 1.1.0 (#4342)

* Update nuget config per security recommendations (#4289)

* Set IOTEDGE_HOST before 'make install' (#4396)

After commit a34e0ed, the iotedge CLI fails in Mariner because IOTEDGE_HOST was not set during compilation, so the CLI has the wrong path to iotedged's management endpoint.

To fix this, we set IOTEDGE_HOST prior to running `make install`. It was already being set prior to `make release`.

* Update Base Images for Security Vulnerability 3.1.12 (#4380)

- Update ARM32/ARM64 Bionic base images to 3.1.12
- Update AMD64 (Linux) Alpine base images to 3.1.12
- Update AMD64 (Windows) Nanoserver base images to 3.1.12

* Update release 1.1.0 (#4399)

Update the CHANGELOG to include the latest commits.

* Config yaml 1.1 (#4434)

This change updates the default agent tag in config.yaml to 1.1.

Cherry-pick a4faab5 and update 1.1-specific Windows config.yaml.

* Upgrade sysinfo package (#4444)

This change upgrades the sysinfo package in the iotedge CLI and iotedged. It fixes bugs called out in #2747 and #3746. This change was fixed in master earlier and was supposed to by ported to the 1.1 branch, but never was.

* EFLOW: Introduce environment variables for nuget operations (#4499)

Set env variables to direct nuget caches to the data partition instead of to the rootfs

Co-authored-by: Pedro Marcelo Zara <pmzara@hotmail.com>
Co-authored-by: Damon Barry <damonbarry@users.noreply.github.com>
Co-authored-by: yophilav <54859653+yophilav@users.noreply.github.com>
Co-authored-by: ms-mahuber <60939654+ms-mahuber@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dylanbronson dylanbronson

Labels
needinfo Needs more info from issue reporter / updates from PR submitter no-issue-activity
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@CindyXing @arsing @surlemur @varunpuranik @ancaantochi @dylanbronson