Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

initramfs fixes #9089

Merged
merged 1 commit into from
Aug 16, 2019
Merged

Conversation

c0d3z3r0
Copy link
Contributor

@c0d3z3r0 c0d3z3r0 commented Jul 28, 2019

Motivation and Context

Fixes #7904

Debian does not boot from zfs when using packages generated from the git repo because /etc/default/zfs and /etc/zfs/zfs-functions are missing, which both are used by the zfs initramfs script.

Description

  • Add the missing files above
  • Prevent conflict of init.d/zfs-import (sysvinit) and zfs-import.target (systemd) by precautionary masking zfs-import.service
  • fix the zfs initconfig location (debian: /etc/default) for distribs other than debian
  • fix initramfs-tools' PREREQ parameter (zdev -> udev)

How Has This Been Tested?

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Performance enhancement (non-breaking change which improves efficiency)
  • Code cleanup (non-breaking change which makes code smaller or more readable)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Documentation (a change to man pages or other documentation)

Checklist:

@c0d3z3r0 c0d3z3r0 force-pushed the for-upstream/initramfs_fixes branch 3 times, most recently from 0f4bfc5 to 61e8d9a Compare July 28, 2019 23:51
@c0d3z3r0 c0d3z3r0 changed the title initramfs fixes [WIP] initramfs fixes Jul 29, 2019
@codecov
Copy link

codecov bot commented Jul 29, 2019

Codecov Report

Merging #9089 into master will decrease coverage by 0.09%.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff            @@
##           master    #9089     +/-   ##
=========================================
- Coverage   79.04%   78.94%   -0.1%     
=========================================
  Files         401      400      -1     
  Lines      121648   121640      -8     
=========================================
- Hits        96154    96029    -125     
- Misses      25494    25611    +117
Flag Coverage Δ
#kernel 79.67% <ø> (ø) ⬆️
#user 66.54% <ø> (-0.09%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 693c1fc...61e8d9a. Read the comment docs.

@c0d3z3r0 c0d3z3r0 force-pushed the for-upstream/initramfs_fixes branch 3 times, most recently from 1051193 to 2213a72 Compare July 29, 2019 16:02
@c0d3z3r0 c0d3z3r0 changed the title [WIP] initramfs fixes initramfs fixes Jul 29, 2019
@behlendorf behlendorf added the Status: Code Review Needed Ready for review and testing label Jul 29, 2019
@ghfields
Copy link
Contributor

This would fix #7904

#8796 is another PR that aims to do multiple initramfs fixes, including this. Either we approve this and remove the zfs-functions from that PR or we just get the other one done.

Also, before a PR is committed, ZOL wants commits squashed to a single commit before being accepted. If you need assistance with this, let us know.

Thanks for looking at this. This has been a nuisance building deb packages from zfs master.

@c0d3z3r0
Copy link
Contributor Author

@ghfields my changes include some stuff from #8796, while I left out the changes to the initramfs script which are not essential for the fix but are an improvement of code quality - these should go to another PR IMHO.

Also, before a PR is committed, ZOL wants commits squashed to a single commit before being accepted. If you need assistance with this, let us know.

Thanks, I know that but generally keep PRs as multiple, logically separated commits, to make review easier. Shortly before merging I can squash them

@dreamcat4
Copy link

Sorry can I just quickly ask here: did you change any code? Or its all the same existing PR / commits from earlier? Because I already tested that one last week. And it was absolutely fine for me.

It doesnt seem like you added any further commits. I tested ontop of the 0.8.1 release git tag. Which is the latest official release at this time.

@c0d3z3r0
Copy link
Contributor Author

c0d3z3r0 commented Aug 1, 2019

huh?! what about those? https://github.com/zfsonlinux/zfs/pull/9089/commits

Edit: I don't think I really got what you were asking (may be my fault). Do you mean this PR by "earlier"? Do you mean #8796?

@dreamcat4
Copy link

dreamcat4 commented Aug 1, 2019

Ah sorry @c0d3z3r0 I was the one who had got confused with the other "[RFC] fixes" PR. Which was actually the one i had already tested earlier on. Not this one. Good good... seems like a progress. Getting closer.

Yeah I can rebuild pkgs and compare the contents with a dpkg -c file.deb then. Just to check for the presence of the file in the pkg. Since can no longer do a full reinstall from scratch at this point. So long as the resultant package contains the same missing file then it should be the same right ? So long as that is sufficient for your testing / confirmation purposes.

@c0d3z3r0
Copy link
Contributor Author

c0d3z3r0 commented Aug 1, 2019

Ah sorry @c0d3z3r0 I was the one who had got confused with the other "[RFC] fixes" PR. Which was actually the one i had already tested earlier on. Not this one. Good good... seems like a progress. Getting closer.

Aaah, at least we both were confused :D

Yeah I can rebuild pkgs and compare the contents with a dpkg -c file.deb then. Just to check for the presence of the file in the pkg. Since can no longer do a full reinstall from scratch at this point. So long as the resultant package contains the same missing file then it should be the same right ? So long as that is sufficient for your testing / confirmation purposes.

Well, my PR even does something more than #8796:

  • Precaution stuff (systemd vs. sysvinit)
  • fix the zfs config location for distribs other than debian
  • fix initramfs-tools' PREREQ parameter (zdev -> udev)

The boot fix basically consists of adding the initconf file (/etc/default/zfs) and /etc/zfs-functions to the package.

@c0d3z3r0 c0d3z3r0 force-pushed the for-upstream/initramfs_fixes branch from 1aaca6b to c36eade Compare August 5, 2019 20:01
@c0d3z3r0
Copy link
Contributor Author

c0d3z3r0 commented Aug 5, 2019

Fixed "file included twice" warning from rpmbuild, squashed everything and rebased onto master

@codecov
Copy link

codecov bot commented Aug 6, 2019

Codecov Report

Merging #9089 into master will increase coverage by 0.47%.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #9089      +/-   ##
==========================================
+ Coverage   79.18%   79.65%   +0.47%     
==========================================
  Files         400      279     -121     
  Lines      121692    80856   -40836     
==========================================
- Hits        96360    64408   -31952     
+ Misses      25332    16448    -8884
Flag Coverage Δ
#kernel 79.65% <ø> (-0.02%) ⬇️
#user ?

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c81f179...2bafa28. Read the comment docs.

* contrib/initramfs: include /etc/default/zfs and /etc/zfs/zfs-functions
At least debian needs /etc/default/zfs and /etc/zfs/zfs-functions for
its initramfs. Include both in build when initramfs is configured.

* contrib/initramfs: include 60-zvol.rules and zvol_id
Include 60-zvol.rules and zvol_id and set udev as predependency instead
of debians zdev. This makes debians additional zdev hook unneeded.

* Correct initconfdir substitution for some distros
Not every Linux distro is using @sysconfdir@/default but @initconfdir@
which is already determined by configure. Let's use it.

* systemd: prevent possible conflict between systemd and sysvinit
Systemd will not load a sysvinit service if a unit exists with the same
name. This prevents conflicts between sysvinit and systemd.
In ZFS there is one sysvinit service that does not have a systemd
service but a target counterpart, zfs-import.target.
Usually it does not make any sense to install both but it is possisble.
Let's prevent any conflict by masking zfs-import.service by default.
This does not harm even if init.d/zfs-import does not exist.

Signed-off-by: Michael Niewöhner <foss@mniewoehner.de>
@c0d3z3r0 c0d3z3r0 force-pushed the for-upstream/initramfs_fixes branch from c36eade to 2bafa28 Compare August 6, 2019 12:46
@dreamcat4
Copy link

Hey there @c0d3z3r0 I tried applying your latest code onto 0.8.1 release git tag. And it was too far ahead. So I instead just made a patch files, applied it. However this part of the patch I had to edit manually. Because it did not apply cleanly. I am guessing since the last release 0.8.1 is older. Anyhow this is the bit I had to rework slighly by myself:

https://gist.github.com/dreamcat4/56e351c4e7c64e4f97b3e7578c78a93a#file-zfs-pull-9089-patch-reworked-for-0-8-1-patch-L24-L34

Then all of the compilation steps of zfs executed OK for me. It spat out debs. However when reinstalling zfs, I then got the following error msg:

installing zfs_0.8.1-1_amd64.deb ...
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Note, selecting 'zfs' instead of './zfs_0.8.1-1_amd64.deb'
The following packages will be DOWNGRADED:
  zfs
0 to upgrade, 0 to newly install, 1 to downgrade, 0 to remove and 19 not to upgrade.
Need to get 0 B/1,194 kB of archives.
After this operation, 18.4 kB of additional disk space will be used.
Get:1 /home/id/.dev/zfs/zfs_0.8.1-1_amd64.deb zfs amd64 0.8.1-1 [1,194 kB]
dpkg: warning: downgrading zfs from 0.8.1-6 to 0.8.1-1
(Reading database ... 265906 files and directories currently installed.)
Preparing to unpack .../.dev/zfs/zfs_0.8.1-1_amd64.deb ...
Unpacking zfs (0.8.1-1) over (0.8.1-6) ...
dpkg: error processing archive /home/id/.dev/zfs/zfs_0.8.1-1_amd64.deb (--unpack):
 trying to overwrite '/etc/default/zfs', which is also in package zfs-initramfs 0.8.1-6
dpkg-deb: error: paste subprocess was killed by signal (Broken pipe)
Errors were encountered while processing:
 /home/id/.dev/zfs/zfs_0.8.1-1_amd64.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)
id@asrock-z170-ocf:~/.dev/zfs$ 

Not sure what to do now. Personally I don't want to move up to master if there is a risk that it will include other kinds of breaking changes. My generally approach is to use the last stable tagged release as per in the zfs releases page.

Please advise further. If there is anything else you want to see / need from me here.

@c0d3z3r0
Copy link
Contributor Author

c0d3z3r0 commented Aug 8, 2019

Then all of the compilation steps of zfs executed OK for me. It spat out debs. However when reinstalling zfs, I then got the following error msg:

I guess I did some mistake in the spec which includes /etc/default/zfs in zfs-initramfs*.deb and zfs*.deb. I'll fix that later and you know

@c0d3z3r0
Copy link
Contributor Author

c0d3z3r0 commented Aug 8, 2019

Then all of the compilation steps of zfs executed OK for me. It spat out debs. However when reinstalling zfs, I then got the following error msg:

I guess I did some mistake in the spec which includes /etc/default/zfs in zfs-initramfs*.deb and zfs*.deb. I'll fix that later and you know

The spec seems correct. According to the log you only installed zfs_0.8.1-1_amd64.deb. Could you try dpkg -i zfs_0.8.1-1_amd64.deb zfs-initramfs_0.8.1-1_amd64.deb, please?

@dreamcat4
Copy link

Same error msg. Then the 2nd time i re-run the cmd it's gone. So maybe that is because the version its overwriting was the other previous fix?

id@asrock-z170-ocf:~/.dev/zfs$ sudo dpkg -i zfs_0.8.1-1_amd64.deb zfs-initramfs_0.8.1-1_amd64.deb
dpkg: warning: downgrading zfs from 0.8.1-6 to 0.8.1-1
(Reading database ... 265906 files and directories currently installed.)
Preparing to unpack zfs_0.8.1-1_amd64.deb ...
Unpacking zfs (0.8.1-1) over (0.8.1-6) ...
dpkg: error processing archive zfs_0.8.1-1_amd64.deb (--install):
 trying to overwrite '/etc/default/zfs', which is also in package zfs-initramfs 0.8.1-6
dpkg-deb: error: paste subprocess was killed by signal (Broken pipe)
dpkg: warning: downgrading zfs-initramfs from 0.8.1-6 to 0.8.1-1
Preparing to unpack zfs-initramfs_0.8.1-1_amd64.deb ...
Unpacking zfs-initramfs (0.8.1-1) over (0.8.1-6) ...
Setting up zfs-initramfs (0.8.1-1) ...
Errors were encountered while processing:
 zfs_0.8.1-1_amd64.deb
id@asrock-z170-ocf:~/.dev/zfs$ 

here is the output of 2nd rerun of same cmd

$ sudo dpkg -i zfs_0.8.1-1_amd64.deb zfs-initramfs_0.8.1-1_amd64.deb
(Reading database ... 265908 files and directories currently installed.)
Preparing to unpack zfs_0.8.1-1_amd64.deb ...
Unpacking zfs (0.8.1-1) over (0.8.1-1) ...
Preparing to unpack zfs-initramfs_0.8.1-1_amd64.deb ...
Unpacking zfs-initramfs (0.8.1-1) over (0.8.1-1) ...
Setting up zfs (0.8.1-1) ...
Setting up zfs-initramfs (0.8.1-1) ...
Processing triggers for man-db (2.8.5-2) ...
id@asrock-z170-ocf:~/.dev/zfs$ 

IDK maybe I should go back and install a completely clean version of zfs-0.8.1 ontop first. Then your patch. To make sure these apt error messages dont reappear again.

@c0d3z3r0
Copy link
Contributor Author

c0d3z3r0 commented Aug 8, 2019

IDK maybe I should go back and install a completely clean version of zfs-0.8.1 ontop first. Then your patch. To make sure these apt error messages dont reappear again.

Yep, good idea.

You can check the contents (and post them here, please) with dpkg -c zfs_*.deb and dpkg -c zfs-initramfs_*.deb

@dreamcat4
Copy link

It seems that the whole business of me installing zfs on ubuntu here is a bit complicated. So you will have to bear with me on that due to my lack of experience in this area. Getting other errors now involving dkms etc. So whilst I try to tackle that, and get back to reproducing the original boot error / problem. Here is the requested dpkg -c output

https://gist.github.com/dreamcat4/0e1b8762a130853485c55a6272aaece0

@dreamcat4
Copy link

OK, got back to the kernel panic now. This reproduced the original problem. Next I will apply the version of your patch which I tweaked slightly. But only to get the patch to apply cleanly, nothing else. As had already explained in one of my earlier comments today #9089 (comment)

https://gist.github.com/dreamcat4/56e351c4e7c64e4f97b3e7578c78a93a#file-zfs-pull-9089-patch-reworked-for-0-8-1-patch-L24-L34

Then I will rebuild, reinstall, reboot. See what happens. Hopefully there won't be any other unrelated errors anymore to get in our way.

@c0d3z3r0
Copy link
Contributor Author

c0d3z3r0 commented Aug 8, 2019

So whilst I try to tackle that, and get back to reproducing the original boot error / problem. Here is the requested dpkg -c output

https://gist.github.com/dreamcat4/0e1b8762a130853485c55a6272aaece0

o.O dpkg complains about replacing /etc/default/zfs but the file is not in the package... this is getting really, really strange...

OK, got back to the kernel panic now. This reproduced the original problem. Next I will apply the version of your patch which I tweaked slightly. But only to get the patch to apply cleanly, nothing else. As had already explained in one of my earlier comments today #9089 (comment)

https://gist.github.com/dreamcat4/56e351c4e7c64e4f97b3e7578c78a93a#file-zfs-pull-9089-patch-reworked-for-0-8-1-patch-L24-L34

Your patch looks good!

@dreamcat4
Copy link

OK @c0d3z3r0 just finished testing your PR. The failed case was reproduced, as was already described above. Then I built with your codem and reinstalled installed it. In my log it says the same version of zfs is being installed over the top (0.8.1). Which is to be expected since I didn't bother to increment the version number or change anything else except for your patch.

Regenerated initramfs & updated grub.

Here is the full build log of that run:

https://gist.github.com/dreamcat4/8919af50ff5d187f6c44aa98110b8254

Then I shutdown the computer, and pulled out my backup boot disk from the system. To make sure that other one could not possibly be being booted accidentally. So then only the main boot disk which had the new initramfs on it.

Rebooted and it came back up. No kernel panic. I think this concludes my testing now. Kind Regards.

@c0d3z3r0
Copy link
Contributor Author

c0d3z3r0 commented Aug 8, 2019 via email

@dreamcat4
Copy link

@c0d3z3r0 Here is 2nd dpkg -c. Many thanks for doing this PR. Anything else please let us know.

https://gist.github.com/dreamcat4/7572fdc600a4d17c1ad603a81d9963fb

@c0d3z3r0
Copy link
Contributor Author

c0d3z3r0 commented Aug 9, 2019 via email

@c0d3z3r0
Copy link
Contributor Author

The fix has been verified successfully by @dreamcat4 and @ReimuHakurei. The failing tests are unrelated. This should be ready for final review and merging now. @behlendorf

Copy link
Contributor

@behlendorf behlendorf left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for sorting this out!

@behlendorf behlendorf added Status: Accepted Ready to integrate (reviewed, tested) and removed Status: Code Review Needed Ready for review and testing labels Aug 15, 2019
@behlendorf behlendorf merged commit 9323aad into openzfs:master Aug 16, 2019
@c0d3z3r0 c0d3z3r0 deleted the for-upstream/initramfs_fixes branch August 19, 2019 22:40
pcd1193182 added a commit to pcd1193182/zfs that referenced this pull request Aug 22, 2019
Signed-off-by: Paul Dagnelie <pcd@delphix.com>
behlendorf added a commit that referenced this pull request Aug 23, 2019
Signed-off-by: Paul Dagnelie <pcd@delphix.com>
ghost pushed a commit to zfsonfreebsd/ZoF that referenced this pull request Aug 23, 2019
Signed-off-by: Paul Dagnelie <pcd@delphix.com>
tonyhutter pushed a commit to tonyhutter/zfs that referenced this pull request Dec 24, 2019
* contrib/initramfs: include /etc/default/zfs and /etc/zfs/zfs-functions
At least debian needs /etc/default/zfs and /etc/zfs/zfs-functions for
its initramfs. Include both in build when initramfs is configured.

* contrib/initramfs: include 60-zvol.rules and zvol_id
Include 60-zvol.rules and zvol_id and set udev as predependency instead
of debians zdev. This makes debians additional zdev hook unneeded.

* Correct initconfdir substitution for some distros
Not every Linux distro is using @sysconfdir@/default but @initconfdir@
which is already determined by configure. Let's use it.

* systemd: prevent possible conflict between systemd and sysvinit
Systemd will not load a sysvinit service if a unit exists with the same
name. This prevents conflicts between sysvinit and systemd.
In ZFS there is one sysvinit service that does not have a systemd
service but a target counterpart, zfs-import.target.
Usually it does not make any sense to install both but it is possisble.
Let's prevent any conflict by masking zfs-import.service by default.
This does not harm even if init.d/zfs-import does not exist.

Reviewed-by: Chris Wedgwood <cw@f00f.org>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Tested-by: Alex Ingram <reimu@reimuhakurei.net>
Tested-by: Dreamcat4 <dreamcat4@gmail.com>
Signed-off-by: Michael Niewöhner <foss@mniewoehner.de>
Closes openzfs#7904
Closes openzfs#9089
tonyhutter pushed a commit to tonyhutter/zfs that referenced this pull request Dec 24, 2019
Signed-off-by: Paul Dagnelie <pcd@delphix.com>
tonyhutter pushed a commit to tonyhutter/zfs that referenced this pull request Dec 27, 2019
* contrib/initramfs: include /etc/default/zfs and /etc/zfs/zfs-functions
At least debian needs /etc/default/zfs and /etc/zfs/zfs-functions for
its initramfs. Include both in build when initramfs is configured.

* contrib/initramfs: include 60-zvol.rules and zvol_id
Include 60-zvol.rules and zvol_id and set udev as predependency instead
of debians zdev. This makes debians additional zdev hook unneeded.

* Correct initconfdir substitution for some distros
Not every Linux distro is using @sysconfdir@/default but @initconfdir@
which is already determined by configure. Let's use it.

* systemd: prevent possible conflict between systemd and sysvinit
Systemd will not load a sysvinit service if a unit exists with the same
name. This prevents conflicts between sysvinit and systemd.
In ZFS there is one sysvinit service that does not have a systemd
service but a target counterpart, zfs-import.target.
Usually it does not make any sense to install both but it is possisble.
Let's prevent any conflict by masking zfs-import.service by default.
This does not harm even if init.d/zfs-import does not exist.

Reviewed-by: Chris Wedgwood <cw@f00f.org>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Tested-by: Alex Ingram <reimu@reimuhakurei.net>
Tested-by: Dreamcat4 <dreamcat4@gmail.com>
Signed-off-by: Michael Niewöhner <foss@mniewoehner.de>
Closes openzfs#7904
Closes openzfs#9089
tonyhutter pushed a commit to tonyhutter/zfs that referenced this pull request Dec 27, 2019
Signed-off-by: Paul Dagnelie <pcd@delphix.com>
@NeQuissimus NeQuissimus mentioned this pull request Jan 18, 2020
10 tasks
tonyhutter pushed a commit that referenced this pull request Jan 23, 2020
* contrib/initramfs: include /etc/default/zfs and /etc/zfs/zfs-functions
At least debian needs /etc/default/zfs and /etc/zfs/zfs-functions for
its initramfs. Include both in build when initramfs is configured.

* contrib/initramfs: include 60-zvol.rules and zvol_id
Include 60-zvol.rules and zvol_id and set udev as predependency instead
of debians zdev. This makes debians additional zdev hook unneeded.

* Correct initconfdir substitution for some distros
Not every Linux distro is using @sysconfdir@/default but @initconfdir@
which is already determined by configure. Let's use it.

* systemd: prevent possible conflict between systemd and sysvinit
Systemd will not load a sysvinit service if a unit exists with the same
name. This prevents conflicts between sysvinit and systemd.
In ZFS there is one sysvinit service that does not have a systemd
service but a target counterpart, zfs-import.target.
Usually it does not make any sense to install both but it is possisble.
Let's prevent any conflict by masking zfs-import.service by default.
This does not harm even if init.d/zfs-import does not exist.

Reviewed-by: Chris Wedgwood <cw@f00f.org>
Reviewed-by: Brian Behlendorf <behlendorf1@llnl.gov>
Tested-by: Alex Ingram <reimu@reimuhakurei.net>
Tested-by: Dreamcat4 <dreamcat4@gmail.com>
Signed-off-by: Michael Niewöhner <foss@mniewoehner.de>
Closes #7904
Closes #9089
tonyhutter pushed a commit that referenced this pull request Jan 23, 2020
Signed-off-by: Paul Dagnelie <pcd@delphix.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Status: Accepted Ready to integrate (reviewed, tested)
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Script zfs-functions missing from generated .deb packages, causes kernel panic
5 participants