Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Generate all Agama json profiles using jsonnet profiles #21108

Open
wants to merge 11 commits into
base: master
Choose a base branch
from

Conversation

okynos
Copy link
Contributor

@okynos okynos commented Feb 4, 2025

We have added one method to auto-generate JSON profiles that can be later imported into Agama.

Copy link

github-actions bot commented Feb 4, 2025

Great PR! Please pay attention to the following items before merging:

Files matching lib/**.pm:

  • Consider adding or extending unit tests in t/

This is an automatically generated QA checklist based on modified files.

Squashed commit of the following:

commit 1abca87
Author: José Luis Fernández Aguilera <jose.fernandez1@suse.com>
Date:   Mon Feb 3 16:24:54 2025 +0100

    Improved the way to install jsonnet

commit b014cdb
Author: José Luis Fernández Aguilera <jose.fernandez1@suse.com>
Date:   Mon Feb 3 12:26:27 2025 +0100

    Renamed encrypted variable

commit 0578b9c
Author: José Luis Fernández Aguilera <jose.fernandez1@suse.com>
Date:   Mon Feb 3 12:06:29 2025 +0100

    Added wipefs script

commit 5c9961b
Author: José Luis Fernández Aguilera <jose.fernandez1@suse.com>
Date:   Mon Feb 3 11:19:46 2025 +0100

    Fixed encryption definition

commit a4d3c6f
Author: José Luis Fernández Aguilera <jose.fernandez1@suse.com>
Date:   Mon Feb 3 11:01:35 2025 +0100

    Squashed commit of the following:

    commit 59b65d8
    Author: José Luis Fernández Aguilera <jose.fernandez1@suse.com>
    Date:   Mon Feb 3 10:40:56 2025 +0100

        Added encryption to template

    commit e79bc06
    Author: José Luis Fernández Aguilera <jose.fernandez1@suse.com>
    Date:   Fri Jan 31 17:05:42 2025 +0100

        Applied template modifications

    commit 815a68b
    Author: José Luis Fernández Aguilera <jose.fernandez1@suse.com>
    Date:   Fri Jan 31 13:21:51 2025 +0100

        Removed profile_generator reference

    commit d2f121c
    Author: José Luis Fernández Aguilera <jose.fernandez1@suse.com>
    Date:   Fri Jan 31 13:16:29 2025 +0100

        Fix export function autoyast

    commit 224f792
    Author: José Luis Fernández Aguilera <jose.fernandez1@suse.com>
    Date:   Fri Jan 31 13:14:02 2025 +0100

        Applied design changes

    commit 8a09472
    Author: José Luis Fernández Aguilera <jose.fernandez1@suse.com>
    Date:   Fri Jan 31 11:16:36 2025 +0100

        Clean the code

    commit 2257c12
    Author: José Luis Fernández Aguilera <jose.fernandez1@suse.com>
    Date:   Fri Jan 31 09:59:48 2025 +0100

        Added parameters

    commit dae3ca3
    Author: José Luis Fernández Aguilera <jose.fernandez1@suse.com>
    Date:   Thu Jan 30 17:47:49 2025 +0100

        Fix product definition

    commit 451ca41
    Author: José Luis Fernández Aguilera <jose.fernandez1@suse.com>
    Date:   Thu Jan 30 17:29:31 2025 +0100

        Added agama.auto parameter

    commit 0213b41
    Author: José Luis Fernández Aguilera <jose.fernandez1@suse.com>
    Date:   Thu Jan 30 17:00:23 2025 +0100

        Moved profile generation to boot Agama step

    commit 860343f
    Author: José Luis Fernández Aguilera <jose.fernandez1@suse.com>
    Date:   Thu Jan 30 16:38:28 2025 +0100

        Added import

    commit 319eca2
    Author: José Luis Fernández Aguilera <jose.fernandez1@suse.com>
    Date:   Thu Jan 30 16:20:57 2025 +0100

        Fix typo

    commit 58b2bbd
    Author: José Luis Fernández Aguilera <jose.fernandez1@suse.com>
    Date:   Thu Jan 30 16:18:06 2025 +0100

        Added casedir variable

    commit 9d010e8
    Author: José Luis Fernández Aguilera <jose.fernandez1@suse.com>
    Date:   Thu Jan 30 09:54:38 2025 +0100

        Minor fixes

    commit b92df71
    Author: José Luis Fernández Aguilera <jose.fernandez1@suse.com>
    Date:   Wed Jan 29 17:59:29 2025 +0100

        Added profile generation and upload

    commit 8d82a93
    Author: José Luis Fernández Aguilera <jose.fernandez1@suse.com>
    Date:   Wed Jan 29 17:50:11 2025 +0100

        Moved flags

    commit 9f0b376
    Author: José Luis Fernández Aguilera <jose.fernandez1@suse.com>
    Date:   Wed Jan 29 16:51:13 2025 +0100

        Changed GPG check

    commit 342698c
    Author: José Luis Fernández Aguilera <jose.fernandez1@suse.com>
    Date:   Wed Jan 29 16:39:47 2025 +0100

        Remove previously added repo

    commit dae109f
    Author: José Luis Fernández Aguilera <jose.fernandez1@suse.com>
    Date:   Wed Jan 29 16:27:53 2025 +0100

        Fixed repository URL

    commit 699a246
    Author: José Luis Fernández Aguilera <jose.fernandez1@suse.com>
    Date:   Wed Jan 29 16:19:31 2025 +0100

        Fixed parameters

    commit ae431a4
    Author: José Luis Fernández Aguilera <jose.fernandez1@suse.com>
    Date:   Wed Jan 29 16:09:58 2025 +0100

        Replace options order

    commit f766a07
    Author: José Luis Fernández Aguilera <jose.fernandez1@suse.com>
    Date:   Wed Jan 29 16:01:44 2025 +0100

        Added non-interactive in zypper

    commit 5ff245e
    Author: José Luis Fernández Aguilera <jose.fernandez1@suse.com>
    Date:   Wed Jan 29 15:54:18 2025 +0100

        Added sudo to the commands

    commit 4264a4e
    Author: José Luis Fernández Aguilera <jose.fernandez1@suse.com>
    Date:   Wed Jan 29 13:22:31 2025 +0100

        Moved to system calls

    commit bd708c5
    Author: José Luis Fernández Aguilera <jose.fernandez1@suse.com>
    Date:   Wed Jan 29 11:34:38 2025 +0100

        Select console before test

    commit 307c6f0
    Author: José Luis Fernández Aguilera <jose.fernandez1@suse.com>
    Date:   Wed Jan 29 11:05:40 2025 +0100

        Moved generate agama profile and test jsonnet package in worker

    commit 157f993
    Author: José Luis Fernández Aguilera <jose.fernandez1@suse.com>
    Date:   Tue Jan 28 17:48:43 2025 +0100

        Fix variable definition

    commit 99610a3
    Author: José Luis Fernández Aguilera <jose.fernandez1@suse.com>
    Date:   Tue Jan 28 17:44:57 2025 +0100

        Fix default profile import

    commit c36f92d
    Author: José Luis Fernández Aguilera <jose.fernandez1@suse.com>
    Date:   Tue Jan 28 17:40:54 2025 +0100

        First approach to generate Agama profile

Removed install jsonnet function

Fix syntax error

Restore install_jsonnet function

Restore deleted templates

Removed Pom model file
@okynos okynos force-pushed the 175728-profile-generator branch from 0efea3d to 7269a65 Compare February 4, 2025 09:36
@okynos okynos added qe-yam WIP Work in progress labels Feb 4, 2025
Comment on lines 66 to 69
unless (get_var('AGAMA_AUTO')) {
my $profile_url = generate_json_profile();
push(@params, "agama.auto=\"$profile_url\"");
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this block should go under if agama auto above, so if it is x86_64 we will go with this and if not, the previous mechanism for the other architectures. It is preparation, doesn't need to to be in the middle of the ui code, anyway it runs in the workers, you don't need openqa api.

save_tmp_file($profile_name, $profile_content);

my $profile_url = autoinst_url . "/files/$profile_name";
upload_profile(path => $profile_name, profile => $profile_content);
Copy link
Contributor

@jknphy jknphy Feb 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in case of failure we might not have the uploaded json file, so I think we should also diag for example the generated json file to display it in the screen, otherwise could be hard to dissect.

lib/autoyast.pm Outdated
sub generate_json_profile {
my $profile_name = "generated_profile.json";

install_jsonnet();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's have all the initialization together, so you can move this block before jsonnet execution.

my $profile_content = `jsonnet $profile_parameters $casedir/data/yam/agama/auto/template.jsonnet`;

save_tmp_file($profile_name, $profile_content);

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this extra line is not needed, the whole thing could be logic block, kind of blocks (1) preparation (2) execution (3) post-stuff and no need for comment in each block because it is obvious that is the structure.

lib/autoyast.pm Outdated
my $profile_name = "generated_profile.json";

install_jsonnet();
my $profile_parameters = get_var('AGAMA_PROFILE_PARAMETERS');
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AGAMA_PROFILE_OPTIONS so it goes with the same type of naming than we have for Puppeteer.

Another important point here that would make less error-prone these openQA variable, as we can guess with string 'true' of 'false' that they are string or code, we should avoid to pass --tla-str and --tla-code. Could you please implement a simple parse for that.

lib/autoyast.pm Outdated
@@ -783,6 +784,48 @@ sub expand_agama_profile {
return $profile_url;
}

=head2 install_jsonnet
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

call it workaround_* and document it before some other contributors starts to scream :) because it is not good practice and we are being promised that will be packaged for all architectures but it wouldn't be the case we would need to rise the point or salt this other repo.

@@ -0,0 +1,12 @@
local lib = import 'lib/base.libsonnet';

function(storage='default', product='SLES', scripts='default', encrypted=false) {
Copy link
Contributor

@jknphy jknphy Feb 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as I mentioned to you, the 'default' word is too tricky and it is already used by jsonnet schema, so it difficult its understanding, we could go just with storage='' if that is allow?

local lib = import 'lib/base.libsonnet';

function(storage='default', product='SLES', scripts='default', encrypted=false) {
product: lib.getProduct(product),
Copy link
Contributor

@jknphy jknphy Feb 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no need for a function here, let's use the external variable directly, otherwise we cannot reuse for openSUSE, each product needs to pass it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually, it works like that, if you run jsonnet --tla-str product="Tumbleweed" template.jsonnet the output will contains Tumbleweed in the product id.
Example:

> jsonnet --tla-str product="Tumbleweed" data/yam/agama/auto/template.jsonnet 
{
   "product": {
      "id": "Tumbleweed"
   },
   "root": {
      "hashedPassword": true,
      "password": "$6$vYbbuJ9WMriFxGHY$gQ7shLw9ZBsRcPgo6/8KmfDvQ/lCqxW8/WnMoLCoWGdHO6Touush1nhegYfdBbXRpsQuy/FTZZeg7gQL50IbA/"
   },
.
.
.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The function allow us to set a default variable, avoid errors on empty of parameters run.

Copy link
Contributor Author

@okynos okynos Feb 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok I removed the function, the product id definition will be in the template.

|||
}
],
post: [ enable_root_login ],
Copy link
Contributor

@jknphy jknphy Feb 4, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should not make wipefs (let's use full names, wipe_filesystem) dependent on enable_root_login, we should compose them, add them seprately. Because this logic for these kind of workarounds tends to disappear in future builds (we don't know but happens a lot) so no logic is required, all dummy building blocks as much as possible.

Comment on lines 2 to 3
storage: import 'storage.libsonnet',
scripts: import 'scripts.libsonnet',
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why to add this here if it is not used?

}]
},
wipefs: {
pre: [
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should be able to reference the array item, so we can use more than one pre.

partitions: [
{ search: "*", delete: true },
{ generate: 'default' },
{ filesystem: { path: '/', type: 'ext4' } },
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this could be a way to reuse it to avoid repetition below, passing the value from the openQA variable with the list of options.

@@ -0,0 +1,12 @@
local lib = import 'lib/base.libsonnet';

function(storage='default', product='SLES', scripts='default', encrypted=false) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we have right now a case for "import profile" that add only scripts, so in a way user and root are also optional.

@@ -0,0 +1,12 @@
local lib = import 'lib/base.libsonnet';
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess we should import here all the libs and not have import libraries in libraries for simplicity.

Comment on lines 61 to 63
} else {
my $profile_url = generate_json_profile();
set_var('EXTRABOOTPARAMS', get_var('EXTRABOOTPARAMS', '') . " agama.auto=\"$profile_url\"");
Copy link
Contributor

@jknphy jknphy Feb 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand when we use it, I would expect something like:

if (agama_auto && x86)
   create json profile
else
   existing code (expand profile)

otherwise existing change would run for interactive installation.
we need to point in AGAMA_AUTO to the template yam/agama/auto/template.jsonnet

lib/autoyast.pm Outdated
Comment on lines 817 to 827
my ($options) = @_;

my @params = split(' ', trim($options));
for my $element (@params) {
if ($element =~ /true/ || $element =~/false/) {
$element = " --tla-code " . $element
} else {
$element = " --tla-str " . $element
}
}
return @params
Copy link
Contributor

@jknphy jknphy Feb 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

try to avoid loops at all cost and use higher-order functions, Perl provides grep, map, the code is much more concise, something like this:
return map { " --tla-" . (/true|false/ ? "code" : "str") . " $_" } split ' ', trim($options);
You could even avoid to create a function and just include it below.

lib/autoyast.pm Outdated

sub generate_json_profile {
my $profile_name = "generated_profile.json";
my $profile_options = parse_profile_options(get_var('AGAMA_PROFILE_OPTIONS'));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the lines which contains some logic should go below, the casedir is just an assignation and should go here instead.

local scripts = import 'lib/scripts.libsonnet';
local storage = import 'lib/storage.libsonnet';

function(storage_schema='', product='SLES', pre_scripts='',
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

let's not invent words here, the only schema that we have is the one in the Agama repo against which the profile is validated, this should just be storage, we should make all the libraries looks like so, so above local storage_lib same for any other library.

local scripts = import 'lib/scripts.libsonnet';
local storage = import 'lib/storage.libsonnet';

function(storage_schema='', product='SLES', pre_scripts='',
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for the same reason to make it flexible for openSUSE, product=''

local storage = import 'lib/storage.libsonnet';

function(storage_schema='', product='SLES', pre_scripts='',
post_scripts='', encrypted=false) {
Copy link
Contributor

@jknphy jknphy Feb 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

perhaps this could be scripts_pre so we don't need to invent names in the future, but to follow from outer to inner the structure and use that always as parameter names.

echo 'PermitRootLogin yes' > /etc/ssh/sshd_config.d/root.conf
|||
},
multipath: {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in case we do some more multipath actions, better "activate_multipath".

@@ -0,0 +1,32 @@
{
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I find really unlikely that we use a pre script also as post script, wouldn't make more sent to separate them in different files, so the user of this feature can see clearly the catalog for each.

@@ -0,0 +1,45 @@
local get_root_filesystem(filesystem='ext4') = {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shouldn't be any default for this, looks to me like some external variable.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
qe-yam WIP Work in progress
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants