-
Notifications
You must be signed in to change notification settings - Fork 282
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Generate all Agama json profiles using jsonnet profiles #21108
base: master
Are you sure you want to change the base?
Conversation
Great PR! Please pay attention to the following items before merging: Files matching
This is an automatically generated QA checklist based on modified files. |
Squashed commit of the following: commit 1abca87 Author: José Luis Fernández Aguilera <jose.fernandez1@suse.com> Date: Mon Feb 3 16:24:54 2025 +0100 Improved the way to install jsonnet commit b014cdb Author: José Luis Fernández Aguilera <jose.fernandez1@suse.com> Date: Mon Feb 3 12:26:27 2025 +0100 Renamed encrypted variable commit 0578b9c Author: José Luis Fernández Aguilera <jose.fernandez1@suse.com> Date: Mon Feb 3 12:06:29 2025 +0100 Added wipefs script commit 5c9961b Author: José Luis Fernández Aguilera <jose.fernandez1@suse.com> Date: Mon Feb 3 11:19:46 2025 +0100 Fixed encryption definition commit a4d3c6f Author: José Luis Fernández Aguilera <jose.fernandez1@suse.com> Date: Mon Feb 3 11:01:35 2025 +0100 Squashed commit of the following: commit 59b65d8 Author: José Luis Fernández Aguilera <jose.fernandez1@suse.com> Date: Mon Feb 3 10:40:56 2025 +0100 Added encryption to template commit e79bc06 Author: José Luis Fernández Aguilera <jose.fernandez1@suse.com> Date: Fri Jan 31 17:05:42 2025 +0100 Applied template modifications commit 815a68b Author: José Luis Fernández Aguilera <jose.fernandez1@suse.com> Date: Fri Jan 31 13:21:51 2025 +0100 Removed profile_generator reference commit d2f121c Author: José Luis Fernández Aguilera <jose.fernandez1@suse.com> Date: Fri Jan 31 13:16:29 2025 +0100 Fix export function autoyast commit 224f792 Author: José Luis Fernández Aguilera <jose.fernandez1@suse.com> Date: Fri Jan 31 13:14:02 2025 +0100 Applied design changes commit 8a09472 Author: José Luis Fernández Aguilera <jose.fernandez1@suse.com> Date: Fri Jan 31 11:16:36 2025 +0100 Clean the code commit 2257c12 Author: José Luis Fernández Aguilera <jose.fernandez1@suse.com> Date: Fri Jan 31 09:59:48 2025 +0100 Added parameters commit dae3ca3 Author: José Luis Fernández Aguilera <jose.fernandez1@suse.com> Date: Thu Jan 30 17:47:49 2025 +0100 Fix product definition commit 451ca41 Author: José Luis Fernández Aguilera <jose.fernandez1@suse.com> Date: Thu Jan 30 17:29:31 2025 +0100 Added agama.auto parameter commit 0213b41 Author: José Luis Fernández Aguilera <jose.fernandez1@suse.com> Date: Thu Jan 30 17:00:23 2025 +0100 Moved profile generation to boot Agama step commit 860343f Author: José Luis Fernández Aguilera <jose.fernandez1@suse.com> Date: Thu Jan 30 16:38:28 2025 +0100 Added import commit 319eca2 Author: José Luis Fernández Aguilera <jose.fernandez1@suse.com> Date: Thu Jan 30 16:20:57 2025 +0100 Fix typo commit 58b2bbd Author: José Luis Fernández Aguilera <jose.fernandez1@suse.com> Date: Thu Jan 30 16:18:06 2025 +0100 Added casedir variable commit 9d010e8 Author: José Luis Fernández Aguilera <jose.fernandez1@suse.com> Date: Thu Jan 30 09:54:38 2025 +0100 Minor fixes commit b92df71 Author: José Luis Fernández Aguilera <jose.fernandez1@suse.com> Date: Wed Jan 29 17:59:29 2025 +0100 Added profile generation and upload commit 8d82a93 Author: José Luis Fernández Aguilera <jose.fernandez1@suse.com> Date: Wed Jan 29 17:50:11 2025 +0100 Moved flags commit 9f0b376 Author: José Luis Fernández Aguilera <jose.fernandez1@suse.com> Date: Wed Jan 29 16:51:13 2025 +0100 Changed GPG check commit 342698c Author: José Luis Fernández Aguilera <jose.fernandez1@suse.com> Date: Wed Jan 29 16:39:47 2025 +0100 Remove previously added repo commit dae109f Author: José Luis Fernández Aguilera <jose.fernandez1@suse.com> Date: Wed Jan 29 16:27:53 2025 +0100 Fixed repository URL commit 699a246 Author: José Luis Fernández Aguilera <jose.fernandez1@suse.com> Date: Wed Jan 29 16:19:31 2025 +0100 Fixed parameters commit ae431a4 Author: José Luis Fernández Aguilera <jose.fernandez1@suse.com> Date: Wed Jan 29 16:09:58 2025 +0100 Replace options order commit f766a07 Author: José Luis Fernández Aguilera <jose.fernandez1@suse.com> Date: Wed Jan 29 16:01:44 2025 +0100 Added non-interactive in zypper commit 5ff245e Author: José Luis Fernández Aguilera <jose.fernandez1@suse.com> Date: Wed Jan 29 15:54:18 2025 +0100 Added sudo to the commands commit 4264a4e Author: José Luis Fernández Aguilera <jose.fernandez1@suse.com> Date: Wed Jan 29 13:22:31 2025 +0100 Moved to system calls commit bd708c5 Author: José Luis Fernández Aguilera <jose.fernandez1@suse.com> Date: Wed Jan 29 11:34:38 2025 +0100 Select console before test commit 307c6f0 Author: José Luis Fernández Aguilera <jose.fernandez1@suse.com> Date: Wed Jan 29 11:05:40 2025 +0100 Moved generate agama profile and test jsonnet package in worker commit 157f993 Author: José Luis Fernández Aguilera <jose.fernandez1@suse.com> Date: Tue Jan 28 17:48:43 2025 +0100 Fix variable definition commit 99610a3 Author: José Luis Fernández Aguilera <jose.fernandez1@suse.com> Date: Tue Jan 28 17:44:57 2025 +0100 Fix default profile import commit c36f92d Author: José Luis Fernández Aguilera <jose.fernandez1@suse.com> Date: Tue Jan 28 17:40:54 2025 +0100 First approach to generate Agama profile Removed install jsonnet function Fix syntax error Restore install_jsonnet function Restore deleted templates Removed Pom model file
0efea3d
to
7269a65
Compare
tests/yam/agama/boot_agama.pm
Outdated
unless (get_var('AGAMA_AUTO')) { | ||
my $profile_url = generate_json_profile(); | ||
push(@params, "agama.auto=\"$profile_url\""); | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this block should go under if agama auto
above, so if it is x86_64 we will go with this and if not, the previous mechanism for the other architectures. It is preparation, doesn't need to to be in the middle of the ui code, anyway it runs in the workers, you don't need openqa api.
save_tmp_file($profile_name, $profile_content); | ||
|
||
my $profile_url = autoinst_url . "/files/$profile_name"; | ||
upload_profile(path => $profile_name, profile => $profile_content); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
in case of failure we might not have the uploaded json file, so I think we should also diag
for example the generated json file to display it in the screen, otherwise could be hard to dissect.
lib/autoyast.pm
Outdated
sub generate_json_profile { | ||
my $profile_name = "generated_profile.json"; | ||
|
||
install_jsonnet(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's have all the initialization together, so you can move this block before jsonnet execution.
my $profile_content = `jsonnet $profile_parameters $casedir/data/yam/agama/auto/template.jsonnet`; | ||
|
||
save_tmp_file($profile_name, $profile_content); | ||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this extra line is not needed, the whole thing could be logic block, kind of blocks (1) preparation (2) execution (3) post-stuff and no need for comment in each block because it is obvious that is the structure.
lib/autoyast.pm
Outdated
my $profile_name = "generated_profile.json"; | ||
|
||
install_jsonnet(); | ||
my $profile_parameters = get_var('AGAMA_PROFILE_PARAMETERS'); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
AGAMA_PROFILE_OPTIONS
so it goes with the same type of naming than we have for Puppeteer.
Another important point here that would make less error-prone these openQA variable, as we can guess with string 'true' of 'false' that they are string or code, we should avoid to pass --tla-str
and --tla-code
. Could you please implement a simple parse for that.
lib/autoyast.pm
Outdated
@@ -783,6 +784,48 @@ sub expand_agama_profile { | |||
return $profile_url; | |||
} | |||
|
|||
=head2 install_jsonnet |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
call it workaround_*
and document it before some other contributors starts to scream :) because it is not good practice and we are being promised that will be packaged for all architectures but it wouldn't be the case we would need to rise the point or salt this other repo.
data/yam/agama/auto/template.jsonnet
Outdated
@@ -0,0 +1,12 @@ | |||
local lib = import 'lib/base.libsonnet'; | |||
|
|||
function(storage='default', product='SLES', scripts='default', encrypted=false) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
as I mentioned to you, the 'default' word is too tricky and it is already used by jsonnet schema, so it difficult its understanding, we could go just with storage=''
if that is allow?
data/yam/agama/auto/template.jsonnet
Outdated
local lib = import 'lib/base.libsonnet'; | ||
|
||
function(storage='default', product='SLES', scripts='default', encrypted=false) { | ||
product: lib.getProduct(product), |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
no need for a function here, let's use the external variable directly, otherwise we cannot reuse for openSUSE, each product needs to pass it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actually, it works like that, if you run jsonnet --tla-str product="Tumbleweed" template.jsonnet
the output will contains Tumbleweed in the product id.
Example:
> jsonnet --tla-str product="Tumbleweed" data/yam/agama/auto/template.jsonnet
{
"product": {
"id": "Tumbleweed"
},
"root": {
"hashedPassword": true,
"password": "$6$vYbbuJ9WMriFxGHY$gQ7shLw9ZBsRcPgo6/8KmfDvQ/lCqxW8/WnMoLCoWGdHO6Touush1nhegYfdBbXRpsQuy/FTZZeg7gQL50IbA/"
},
.
.
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The function allow us to set a default variable, avoid errors on empty of parameters run.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ok I removed the function, the product id definition will be in the template.
||| | ||
} | ||
], | ||
post: [ enable_root_login ], |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we should not make wipefs (let's use full names, wipe_filesystem) dependent on enable_root_login, we should compose them, add them seprately. Because this logic for these kind of workarounds tends to disappear in future builds (we don't know but happens a lot) so no logic is required, all dummy building blocks as much as possible.
storage: import 'storage.libsonnet', | ||
scripts: import 'scripts.libsonnet', |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
why to add this here if it is not used?
}] | ||
}, | ||
wipefs: { | ||
pre: [ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we should be able to reference the array item, so we can use more than one pre
.
partitions: [ | ||
{ search: "*", delete: true }, | ||
{ generate: 'default' }, | ||
{ filesystem: { path: '/', type: 'ext4' } }, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
this could be a way to reuse it to avoid repetition below, passing the value from the openQA variable with the list of options.
data/yam/agama/auto/template.jsonnet
Outdated
@@ -0,0 +1,12 @@ | |||
local lib = import 'lib/base.libsonnet'; | |||
|
|||
function(storage='default', product='SLES', scripts='default', encrypted=false) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
we have right now a case for "import profile" that add only scripts, so in a way user
and root
are also optional.
data/yam/agama/auto/template.jsonnet
Outdated
@@ -0,0 +1,12 @@ | |||
local lib = import 'lib/base.libsonnet'; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I guess we should import here all the libs and not have import libraries in libraries for simplicity.
tests/yam/agama/boot_agama.pm
Outdated
} else { | ||
my $profile_url = generate_json_profile(); | ||
set_var('EXTRABOOTPARAMS', get_var('EXTRABOOTPARAMS', '') . " agama.auto=\"$profile_url\""); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't understand when we use it, I would expect something like:
if (agama_auto && x86)
create json profile
else
existing code (expand profile)
otherwise existing change would run for interactive installation.
we need to point in AGAMA_AUTO to the template yam/agama/auto/template.jsonnet
lib/autoyast.pm
Outdated
my ($options) = @_; | ||
|
||
my @params = split(' ', trim($options)); | ||
for my $element (@params) { | ||
if ($element =~ /true/ || $element =~/false/) { | ||
$element = " --tla-code " . $element | ||
} else { | ||
$element = " --tla-str " . $element | ||
} | ||
} | ||
return @params |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
try to avoid loops at all cost and use higher-order functions, Perl provides grep, map, the code is much more concise, something like this:
return map { " --tla-" . (/true|false/ ? "code" : "str") . " $_" } split ' ', trim($options);
You could even avoid to create a function and just include it below.
lib/autoyast.pm
Outdated
|
||
sub generate_json_profile { | ||
my $profile_name = "generated_profile.json"; | ||
my $profile_options = parse_profile_options(get_var('AGAMA_PROFILE_OPTIONS')); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the lines which contains some logic should go below, the casedir is just an assignation and should go here instead.
data/yam/agama/auto/template.jsonnet
Outdated
local scripts = import 'lib/scripts.libsonnet'; | ||
local storage = import 'lib/storage.libsonnet'; | ||
|
||
function(storage_schema='', product='SLES', pre_scripts='', |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
let's not invent words here, the only schema that we have is the one in the Agama repo against which the profile is validated, this should just be storage
, we should make all the libraries looks like so, so above local storage_lib
same for any other library.
data/yam/agama/auto/template.jsonnet
Outdated
local scripts = import 'lib/scripts.libsonnet'; | ||
local storage = import 'lib/storage.libsonnet'; | ||
|
||
function(storage_schema='', product='SLES', pre_scripts='', |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
for the same reason to make it flexible for openSUSE, product=''
data/yam/agama/auto/template.jsonnet
Outdated
local storage = import 'lib/storage.libsonnet'; | ||
|
||
function(storage_schema='', product='SLES', pre_scripts='', | ||
post_scripts='', encrypted=false) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
perhaps this could be scripts_pre
so we don't need to invent names in the future, but to follow from outer to inner the structure and use that always as parameter names.
echo 'PermitRootLogin yes' > /etc/ssh/sshd_config.d/root.conf | ||
||| | ||
}, | ||
multipath: { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
in case we do some more multipath actions, better "activate_multipath".
@@ -0,0 +1,32 @@ | |||
{ |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I find really unlikely that we use a pre script also as post script, wouldn't make more sent to separate them in different files, so the user of this feature can see clearly the catalog for each.
@@ -0,0 +1,45 @@ | |||
local get_root_filesystem(filesystem='ext4') = { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
shouldn't be any default for this, looks to me like some external variable.
We have added one method to auto-generate JSON profiles that can be later imported into Agama.