Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

mock: allow pre-creating users in chroot #1103

Merged
merged 3 commits into from
Jul 24, 2023

Conversation

praiskup
Copy link
Member

@praiskup praiskup commented Jun 9, 2023

No description provided.

@praiskup praiskup marked this pull request as draft June 9, 2023 08:33
@@ -466,6 +468,9 @@ def _fixup_build_user(self):
@traceLog()
@noop_in_bootstrap
def _make_build_user(self):
self.shadow_utils.create_user("pesign", copy_from_host=True)
self.shadow_utils.create_user("random-user")
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This part likely needs to be done elsewhere and needs to be configurable! This is just a dummy preview for experiments.

@belegdol
Copy link
Contributor

belegdol commented Jun 9, 2023

This fixes #1091 without having to set up ACLs. /run/pesign on the host remains owned by pesign:pesign even after installing pesign in the chroot.

@xsuchy
Copy link
Member

xsuchy commented Jun 12, 2023

@praiskup please check buildroot._make_build_user() for historical knowledge. Like deleting the user first, because it may exists from previous run. And fixing the uid.

@praiskup
Copy link
Member Author

Yes, I've seen the history - in this case, I think we can afford to keep it simple, at least at the beginning; if this logic fails, it fails for misconfiguration (which is not very likely because we do this only when the chroot is first initialized).

praiskup added a commit to praiskup/mock that referenced this pull request Jun 21, 2023
The trick is that the {user,group}{add,del} commands can be executed
on host with `--prefix <chroot>` nowadays.

This commit removes one unnecessary dependency from the minimal
buildroot, so the 'shadow-utils' are removed from all the configs.
The good thing is also that we don't need a hacks around the potentially
obsolescent '/sbin/useradd' utility (the `config_opts['useradd']`
option) because since Mock v3 we support only RHEL 8 hosts and newer.

The `_make_build_user()` was renamed to `_make_users()`, but the
unnecessary side-effect of homedir cleanup was cut-out as a separate
method `_cleanup_homedir` that needs to be called only before build, not
for normal `mock --init` or `mock --shell`.  But we can otherwise
re-create the users in-chroot for every single mock run to assure the
UID/GID match the requirements.

The `chown_home_dir` used to be called in some unclear conditions, but
that call can not cause any harm and gives us guarantees that homedir is
readable by `mockbuilder`.  We newly call it everytime.

The `_setup_build_dirs()` is called a bit later for the `prebuild ==
True` case to align with `not self.chroot_was_initialized`.  But this
shouldn't have any real consequence (we call `chown_home_dir()` anyway).

There's one different thing.  We we don't do `userdel -r` anymore for
mockbuilder.  This shouldn't be needed, because when user is really
re-created (prebuild == True), we already did `_cleanup_homedir()`.
The other thing is that before we `useradd`, we already have `/builddir`
pre-created -> so useradd doesn't create `.bashrc`, etc.  We can iterate
on this, but it doesn't seem to be Mock's responsibility.

Fixes: rpm-software-management#1102
Closes: rpm-software-management#1103
@praiskup praiskup marked this pull request as ready for review June 21, 2023 09:24
@praiskup
Copy link
Member Author

@praiskup please check buildroot._make_build_user() for historical knowledge. Like deleting the user first, because it may exists from previous run. And fixing the uid.

I changed my opinion about this, and I agree that removing the user first is better (makes me more confident that UID/GIDs are aligned).

While doing this quite complex feature, I also propose changing the default way of handling mockbuilder (user) and mock (group) in-chroot -> so we use the same principle in both cases.

Ready for the review.

praiskup added a commit to praiskup/mock that referenced this pull request Jun 21, 2023
The trick is that the {user,group}{add,del} commands can be executed
on host with `--prefix <chroot>` nowadays.

This commit removes one unnecessary dependency from the minimal
buildroot, so the 'shadow-utils' are removed from all the configs.
The good thing is also that we don't need a hacks around the potentially
obsolescent '/sbin/useradd' utility (the `config_opts['useradd']`
option) because since Mock v3 we support only RHEL 8 hosts and newer.

The `_make_build_user()` was renamed to `_make_users()`, but the
unnecessary side-effect of homedir cleanup was cut-out as a separate
method `_cleanup_homedir` that needs to be called only before build, not
for normal `mock --init` or `mock --shell`.  But we can otherwise
re-create the users in-chroot for every single mock run to assure the
UID/GID match the requirements.

The `chown_home_dir` used to be called in some unclear conditions, but
that call can not cause any harm and gives us guarantees that homedir is
readable by `mockbuilder`.  We newly call it everytime.

The `_setup_build_dirs()` is called a bit later for the `prebuild ==
True` case to align with `not self.chroot_was_initialized`.  But this
shouldn't have any real consequence (we call `chown_home_dir()` anyway).

There's one different thing.  We we don't do `userdel -r` anymore for
mockbuilder.  This shouldn't be needed, because when user is really
re-created (prebuild == True), we already did `_cleanup_homedir()`.
The other thing is that before we `useradd`, we already have `/builddir`
pre-created -> so useradd doesn't create `.bashrc`, etc.  We can iterate
on this, but it doesn't seem to be Mock's responsibility.

Fixes: rpm-software-management#1102
Closes: rpm-software-management#1103
@belegdol
Copy link
Contributor

I tested the updated code and it works as expected, pesign works correctly.

praiskup added a commit to praiskup/mock that referenced this pull request Jun 26, 2023
The trick is that the {user,group}{add,del} commands can be executed
on host with `--prefix <chroot>` nowadays.

This commit removes one unnecessary dependency from the minimal
buildroot, so the 'shadow-utils' are removed from all the configs.
The good thing is also that we don't need a hacks around the potentially
obsolescent '/sbin/useradd' utility (the `config_opts['useradd']`
option) because since Mock v3 we support only RHEL 8 hosts and newer.

The `_make_build_user()` was renamed to `_make_users()`, but the
unnecessary side-effect of homedir cleanup was cut-out as a separate
method `_cleanup_homedir` that needs to be called only before build, not
for normal `mock --init` or `mock --shell`.  But we can otherwise
re-create the users in-chroot for every single mock run to assure the
UID/GID match the requirements.

The `chown_home_dir` used to be called in some unclear conditions, but
that call can not cause any harm and gives us guarantees that homedir is
readable by `mockbuilder`.  We newly call it everytime.

The `_setup_build_dirs()` is called a bit later for the `prebuild ==
True` case to align with `not self.chroot_was_initialized`.  But this
shouldn't have any real consequence (we call `chown_home_dir()` anyway).

There's one different thing.  We we don't do `userdel -r` anymore for
mockbuilder.  This shouldn't be needed, because when user is really
re-created (prebuild == True), we already did `_cleanup_homedir()`.
The other thing is that before we `useradd`, we already have `/builddir`
pre-created -> so useradd doesn't create `.bashrc`, etc.  We can iterate
on this, but it doesn't seem to be Mock's responsibility.

Fixes: rpm-software-management#1102
Closes: rpm-software-management#1103
praiskup added a commit to praiskup/mock that referenced this pull request Jun 26, 2023
The trick is that the {user,group}{add,del} commands can be executed
on host with `--prefix <chroot>` nowadays.

This commit removes one unnecessary dependency from the minimal
buildroot, so the 'shadow-utils' are removed from all the configs.
The good thing is also that we don't need a hacks around the potentially
obsolescent '/sbin/useradd' utility (the `config_opts['useradd']`
option) because since Mock v3 we support only RHEL 8 hosts and newer.

The `_make_build_user()` was renamed to `_make_users()`, but the
unnecessary side-effect of homedir cleanup was cut-out as a separate
method `_cleanup_homedir` that needs to be called only before build, not
for normal `mock --init` or `mock --shell`.  But we can otherwise
re-create the users in-chroot for every single mock run to assure the
UID/GID match the requirements.

The `chown_home_dir` used to be called in some unclear conditions, but
that call can not cause any harm and gives us guarantees that homedir is
readable by `mockbuilder`.  We newly call it everytime.

The `_setup_build_dirs()` is called a bit later for the `prebuild ==
True` case to align with `not self.chroot_was_initialized`.  But this
shouldn't have any real consequence (we call `chown_home_dir()` anyway).

There's one different thing.  We we don't do `userdel -r` anymore for
mockbuilder.  This shouldn't be needed, because when user is really
re-created (prebuild == True), we already did `_cleanup_homedir()`.
The other thing is that before we `useradd`, we already have `/builddir`
pre-created -> so useradd doesn't create `.bashrc`, etc.  We can iterate
on this, but it doesn't seem to be Mock's responsibility.

Fixes: rpm-software-management#1102
Closes: rpm-software-management#1103
praiskup added a commit to praiskup/mock that referenced this pull request Jun 26, 2023
The trick is that the {user,group}{add,del} commands can be executed
on host with `--prefix <chroot>` nowadays.

This commit removes one unnecessary dependency from the minimal
buildroot, so the 'shadow-utils' are removed from all the configs.
The good thing is also that we don't need a hacks around the potentially
obsolescent '/sbin/useradd' utility (the `config_opts['useradd']`
option) because since Mock v3 we support only RHEL 8 hosts and newer.

The `_make_build_user()` was renamed to `_make_users()`, but the
unnecessary side-effect of homedir cleanup was cut-out as a separate
method `_cleanup_homedir` that needs to be called only before build, not
for normal `mock --init` or `mock --shell`.  But we can otherwise
re-create the users in-chroot for every single mock run to assure the
UID/GID match the requirements.

The `chown_home_dir` used to be called in some unclear conditions, but
that call can not cause any harm and gives us guarantees that homedir is
readable by `mockbuilder`.  We newly call it everytime.

The `_setup_build_dirs()` is called a bit later for the `prebuild ==
True` case to align with `not self.chroot_was_initialized`.  But this
shouldn't have any real consequence (we call `chown_home_dir()` anyway).

There's one different thing.  We we don't do `userdel -r` anymore for
mockbuilder.  This shouldn't be needed, because when user is really
re-created (prebuild == True), we already did `_cleanup_homedir()`.
The other thing is that before we `useradd`, we already have `/builddir`
pre-created -> so useradd doesn't create `.bashrc`, etc.  We can iterate
on this, but it doesn't seem to be Mock's responsibility.

Fixes: rpm-software-management#1102
Closes: rpm-software-management#1103
mock/mock.spec Outdated
@@ -9,7 +9,7 @@

Summary: Builds packages inside chroots
Name: mock
Version: 4.1
Version: 4.1.post
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

???

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is an experiment (likely I should use post1) that I don't insist on,
but seems really attractive.

Simply, this way I can signalize that the Version is higher than the one
released in Fedora, always beating the NVR there (even if Release goes up due
to rebuilds) and in mock-core-configs I still can require the version newer
then the one from this PR.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I moved this experiment into #1138, it was rather disturbing change in this PR.

@xsuchy
Copy link
Member

xsuchy commented Jun 28, 2023

There's one different thing. We we don't do userdel -r anymore for
mockbuilder. This shouldn't be needed, because when user is really
re-created (prebuild == True), we already did _cleanup_homedir().
The other thing is that before we useradd, we already have /builddir
pre-created -> so useradd doesn't create .bashrc, etc. We can iterate
on this, but it doesn't seem to be Mock's responsibility.

I do not get this. When you call mock with --clean you want to clean /builddir and with -noclean you want to preserve the content. Not just on the first run, but on consecutive runs.

@praiskup
Copy link
Member Author

I do not get this. When you call mock with --clean you want to clean /builddir
and with -noclean you want to preserve the content. Not just on the first run,
but on consecutive runs.

I don't think --no-clean works this way. That's executed unconditionally (note that _make_build_user() is always called for prebuild == True). Which IMO implies that playing with -r option is just redundant (those files are already removed).

@praiskup
Copy link
Member Author

praiskup commented Jun 29, 2023

But honestly, I'm a bit confused, don't ask me what the effect of --no-clean actually is.

praiskup added a commit to praiskup/mock that referenced this pull request Jun 29, 2023
The trick is that the {user,group}{add,del} commands can be executed
on host with `--prefix <chroot>` nowadays.

This commit removes one unnecessary dependency from the minimal
buildroot, so the 'shadow-utils' are removed from all the configs.
The good thing is also that we don't need a hacks around the potentially
obsolescent '/sbin/useradd' utility (the `config_opts['useradd']`
option) because since Mock v3 we support only RHEL 8 hosts and newer.

The `_make_build_user()` was renamed to `_make_users()`, but the
unnecessary side-effect of homedir cleanup was cut-out as a separate
method `_cleanup_homedir` that needs to be called only before build, not
for normal `mock --init` or `mock --shell`.  But we can otherwise
re-create the users in-chroot for every single mock run to assure the
UID/GID match the requirements.

The `chown_home_dir` used to be called in some unclear conditions, but
that call can not cause any harm and gives us guarantees that homedir is
readable by `mockbuilder`.  We newly call it everytime.

The `_setup_build_dirs()` is called a bit later for the `prebuild ==
True` case to align with `not self.chroot_was_initialized`.  But this
shouldn't have any real consequence (we call `chown_home_dir()` anyway).

There's one different thing.  We we don't do `userdel -r` anymore for
mockbuilder.  This shouldn't be needed, because when user is really
re-created (prebuild == True), we already did `_cleanup_homedir()`.
The other thing is that before we `useradd`, we already have `/builddir`
pre-created -> so useradd doesn't create `.bashrc`, etc.  We can iterate
on this, but it doesn't seem to be Mock's responsibility.

Fixes: rpm-software-management#1102
Closes: rpm-software-management#1103
praiskup added a commit to praiskup/mock that referenced this pull request Jun 29, 2023
The trick is that the {user,group}{add,del} commands can be executed
on host with `--prefix <chroot>` nowadays.

This commit removes one unnecessary dependency from the minimal
buildroot, so the 'shadow-utils' are removed from all the configs.
The good thing is also that we don't need a hacks around the potentially
obsolescent '/sbin/useradd' utility (the `config_opts['useradd']`
option) because since Mock v3 we support only RHEL 8 hosts and newer.

The `_make_build_user()` was renamed to `_make_users()`, but the
unnecessary side-effect of homedir cleanup was cut-out as a separate
method `_cleanup_homedir` that needs to be called only before build, not
for normal `mock --init` or `mock --shell`.  But we can otherwise
re-create the users in-chroot for every single mock run to assure the
UID/GID match the requirements.

The `chown_home_dir` used to be called in some unclear conditions, but
that call can not cause any harm and gives us guarantees that homedir is
readable by `mockbuilder`.  We newly call it everytime.

The `_setup_build_dirs()` is called a bit later for the `prebuild ==
True` case to align with `not self.chroot_was_initialized`.  But this
shouldn't have any real consequence (we call `chown_home_dir()` anyway).

There's one different thing.  We we don't do `userdel -r` anymore for
mockbuilder.  This shouldn't be needed, because when user is really
re-created (prebuild == True), we already did `_cleanup_homedir()`.
The other thing is that before we `useradd`, we already have `/builddir`
pre-created -> so useradd doesn't create `.bashrc`, etc.  We can iterate
on this, but it doesn't seem to be Mock's responsibility.

Fixes: rpm-software-management#1102
Closes: rpm-software-management#1103
@praiskup praiskup force-pushed the praiskup-shadow-utils branch 2 times, most recently from 83ce4b6 to c576130 Compare June 29, 2023 07:28
praiskup added a commit to praiskup/mock that referenced this pull request Jun 29, 2023
The trick is that the {user,group}{add,del} commands can be executed
on host with `--prefix <chroot>` nowadays.

This commit removes one unnecessary dependency from the minimal
buildroot, so the 'shadow-utils' are removed from all the configs.
The good thing is also that we don't need a hacks around the potentially
obsolescent '/sbin/useradd' utility (the `config_opts['useradd']`
option) because since Mock v3 we support only RHEL 8 hosts and newer.

The `_make_build_user()` was renamed to `_make_users()`, but the
unnecessary side-effect of homedir cleanup was cut-out as a separate
method `_cleanup_homedir` that needs to be called only before build, not
for normal `mock --init` or `mock --shell`.  But we can otherwise
re-create the users in-chroot for every single mock run to assure the
UID/GID match the requirements.

The `chown_home_dir` used to be called in some unclear conditions, but
that call can not cause any harm and gives us guarantees that homedir is
readable by `mockbuilder`.  We newly call it everytime.

The `_setup_build_dirs()` is called a bit later for the `prebuild ==
True` case to align with `not self.chroot_was_initialized`.  But this
shouldn't have any real consequence (we call `chown_home_dir()` anyway).

There's one different thing.  We we don't do `userdel -r` anymore for
mockbuilder.  This shouldn't be needed, because when user is really
re-created (prebuild == True), we already did `_cleanup_homedir()`.
The other thing is that before we `useradd`, we already have `/builddir`
pre-created -> so useradd doesn't create `.bashrc`, etc.  We can iterate
on this, but it doesn't seem to be Mock's responsibility.

Fixes: rpm-software-management#1102
Closes: rpm-software-management#1103
The trick is that the {user,group}{add,del} commands can be executed
on host with `--prefix <chroot>` nowadays.

This commit removes one unnecessary dependency from the minimal
buildroot, so the 'shadow-utils' are removed from all the configs.
The good thing is also that we don't need a hacks around the potentially
obsolescent '/sbin/useradd' utility (the `config_opts['useradd']`
option) because since Mock v3 we support only RHEL 8 hosts and newer.

The `_make_build_user()` was renamed to `_make_users()`, but the
unnecessary side-effect of homedir cleanup was cut-out as a separate
method `_cleanup_homedir` that needs to be called only before build, not
for normal `mock --init` or `mock --shell`.  But we can otherwise
re-create the users in-chroot for every single mock run to assure the
UID/GID match the requirements.

The `chown_home_dir` used to be called in some unclear conditions, but
that call can not cause any harm and gives us guarantees that homedir is
readable by `mockbuilder`.  We newly call it everytime.

The `_setup_build_dirs()` is called a bit later for the `prebuild ==
True` case to align with `not self.chroot_was_initialized`.  But this
shouldn't have any real consequence (we call `chown_home_dir()` anyway).

There's one different thing.  We we don't do `userdel -r` anymore for
mockbuilder.  This shouldn't be needed, because when user is really
re-created (prebuild == True), we already did `_cleanup_homedir()`.
The other thing is that before we `useradd`, we already have `/builddir`
pre-created -> so useradd doesn't create `.bashrc`, etc.  We can iterate
on this, but it doesn't seem to be Mock's responsibility.

Fixes: rpm-software-management#1102
Closes: rpm-software-management#1103
@praiskup
Copy link
Member Author

So, the consequence here is that e.g. manual /etc/passwd modifications
are going to be dropped between mock --shell calls, because instead of
using usermod calls (old behavior), we are now newly re-creating the
from scratch (new behavior). The new behavior can drop potentially
user-changed /etc/{passwd,group} entries.

Per documentation

    -n, --no-clean
           Do not clean chroot before building a package.
    ...
    --buildsrpm
           Build the specified SRPM either from  a  spec  file  and  source
           file/directory  or SCM. The chroot (including the results direc‐
           tory) is cleaned first, unless --no-clean is specified.

So keeping the old behavior for the "non build" use case seems like we
might be overcautious. That said, the --no-clean option of course has
a meaning, but the thing users really want is to not clean up
the installed packages from the previous build (to potentially speed up
the subsequent build(s).

@praiskup
Copy link
Member Author

We know that @voxik and @mkoncek are heavy --no-clean users. Folks, would you mind testing this not-yet-merged variant of mock? Steps to install this:

sudo dnf copr enable @mock/mock-pull-requests:pr:1103
sudo dnf copr update mock mock-core-configs
mock --scrub=all -r <your chroot>
... test the use-case you typically have ...

@mkoncek
Copy link

mkoncek commented Jul 19, 2023

Sure, can start tomorrow.
Just to be clear, my whole usage of --no-clean is:

  • Build newer version of a dependency
  • Install newer version of a dependency in mock
  • Build newer version of a dependent package with --no-clean

I hope this is how it is intended to be used.

@hobbes1069
Copy link

Sure, can start tomorrow. Just to be clear, my whole usage of --no-clean is:

  • Build newer version of a dependency
  • Install newer version of a dependency in mock
  • Build newer version of a dependent package with --no-clean

I use --no-clean via fedpkg mockbuild when I'm doing test builds of new versions of packages and the build fails for some reason. No point in reinstalling the whole buildroot.

@fweimer-rh
Copy link

I use --no-clean occasionally. The change isn't that radical in principle. The --shell option with the implied --no-clean still overwrites /etc/dnf/dnf.conf even before this change, which I found terribly confusing at one point.

But I think rewriting /etc/passwd is different because it reverts the the actions of postinst scripts of already-installed packages. I think this means that just re-running make in the otherwise unchanged buildroot re-entered via --shell can produce unexpected results. That can't be good?

@praiskup
Copy link
Member Author

Thank you for responding here!

@hobbes1069 wrote

I use --no-clean via fedpkg mockbuild when I'm doing test builds of new versions of packages and the build fails for some reason. No point in reinstalling the whole buildroot.

Indeed, this behavior of --no-clean shouldn't be affected by this PR.

@fweimer-rh wrote

The --shell option with the implied --no-clean still overwrites /etc/dnf/dnf.conf even before this change, which I found terribly confusing at one point.

Good point. There already are other things that Mock sets up for every run no matter --{,no-}clean. So this isn't precedent at least.

But I think rewriting /etc/passwd is different because it reverts the
the actions of postinst scripts of already-installed packages. I think
this means that just re-running make in the otherwise unchanged
buildroot re-entered via --shell can produce unexpected results. That
can't be good?

Well, in theory at least (we may e.g. do mock --shell => mock --install missinb-builddep => mock --shell again). But the original motivation was different dcad166 (it's not obvious, but the modified else: branch calling _fixup_build_user basically relates to --shell, not --rebuild, which btw did re-create the user even before, since 7814ebd). Note mock always does other things like "chown -R " so changing UID actually has some real effect.

@xsuchy xsuchy merged commit 01e8cc4 into rpm-software-management:main Jul 24, 2023
10 of 16 checks passed
xsuchy pushed a commit that referenced this pull request Jul 24, 2023
The trick is that the {user,group}{add,del} commands can be executed
on host with `--prefix <chroot>` nowadays.

This commit removes one unnecessary dependency from the minimal
buildroot, so the 'shadow-utils' are removed from all the configs.
The good thing is also that we don't need a hacks around the potentially
obsolescent '/sbin/useradd' utility (the `config_opts['useradd']`
option) because since Mock v3 we support only RHEL 8 hosts and newer.

The `_make_build_user()` was renamed to `_make_users()`, but the
unnecessary side-effect of homedir cleanup was cut-out as a separate
method `_cleanup_homedir` that needs to be called only before build, not
for normal `mock --init` or `mock --shell`.  But we can otherwise
re-create the users in-chroot for every single mock run to assure the
UID/GID match the requirements.

The `chown_home_dir` used to be called in some unclear conditions, but
that call can not cause any harm and gives us guarantees that homedir is
readable by `mockbuilder`.  We newly call it everytime.

The `_setup_build_dirs()` is called a bit later for the `prebuild ==
True` case to align with `not self.chroot_was_initialized`.  But this
shouldn't have any real consequence (we call `chown_home_dir()` anyway).

There's one different thing.  We we don't do `userdel -r` anymore for
mockbuilder.  This shouldn't be needed, because when user is really
re-created (prebuild == True), we already did `_cleanup_homedir()`.
The other thing is that before we `useradd`, we already have `/builddir`
pre-created -> so useradd doesn't create `.bashrc`, etc.  We can iterate
on this, but it doesn't seem to be Mock's responsibility.

Fixes: #1102
Closes: #1103
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants