inf-terraform-[aws|azure], jenkins-agent-terraform-2306 with tooling update (ruby 3.2.2, python 3.11)

[tbugfinder]

This PR add a new jenkins-agent-terraform-2305 with updated tools used within the could quickstarter components.
Especially upgrade to Ruby 3.2.2 as Ruby 2.7.5 is EOL and switching the default python version to 3.11.
Dependent updates are also incorporated as well as overall version updates.

Closes #914

Tasks:

• Updated documentation in docs/modules/... directory
• Ran tests in <quickstarter>/testdata directory

[BraisVQ]

@tbugfinder , This update in the agent will make older versions of the AWS and Azure quickstarter still functional ? is it a breaking change ?

[tbugfinder]

@tbugfinder , This update in the agent will make older versions of the AWS and Azure quickstarter still functional ? is it a breaking change ?

@braisvq1996 , that's correct.
Previous quickstarter codebases will fail (pipeline - make init stage), as the current agent tagging approach does not map quickstarter versions to a defined image tag but a generic :4.x tag - actually only because of the Ruby upgrade which has to be applied, rather now.
So it has to be well documented and communicated that previous versions have to be "touched" and which steps should be applied.

There are two options:
Option A):
Preserve an older build of jenkins-terraform-agent and use e.g. it's sha tag within the Jenkinsfile. So those codebases would not need any update but would use a previous version of the agent.
At some point those codebaes would have to be upgraded, though.

Option B):
Update files .ruby-version, Gemfile and Gemfile.lock (mandatory) and some other files optionally (.pre-commit-config.yaml, Pipfile, .terraform-version, versions.tf)
We could provide patch files or use a file based git merge.

If there's any other option, happy to discuss.

[BraisVQ]

The older image will always be preserved, just that the tag 4.x will pount to the most recent builded agent.

Another option would be to make a new mayor release of ODS (5.x) even though I would not really want to go with this option...

@metmajer @tbugfinder I would go with the option to prepare some documentation on how to use older image, how to update and notify/inform users of this breacking change.

[metmajer]

@tbugfinder @braisvq1996 I would recommend leaving existing use cases intact and not introducing breaking changes automatically. We can use our existing communication channels including Release Notes for the upcoming release to explain the breaking change and how to use the new image. Brais, what's your final take on the tag name?

[BraisVQ]

@metmajer @tbugfinder The tag for the agents are the same for everyone, 4.x because we are using ODS 4.x
The provisioing of all quickstarters also uses this configuration and cannot change for only one of them.

The only way to get this update and not have the breaking change is to either have a different name for the agent (as we did for Java) or release ODS 5.x

Personally, I would preffer to either re-name the agent or take the breaking change into the existeing one and communicate/document

[tbugfinder]

@metmajer @tbugfinder The tag for the agents are the same for everyone, 4.x because we are using ODS 4.x The provisioing of all quickstarters also uses this configuration and cannot change for only one of them.

The only way to get this update and not have the breaking change is to either have a different name for the agent (as we did for Java) or release ODS 5.x

Personally, I would preffer to either re-name the agent or take the breaking change into the existeing one and communicate/document

Good point, that's option C.

I rather prefer that existing teams manage and update their code. This should be a normal maintenance activity.

[tbugfinder]

What does renaming the agent mean?
We'll still maintain a single agent in this repo but would not actively maintain existing built agents?

Is there any side effect if built agent images might get deleted mistakenly?

[tbugfinder]

Option D)
Add EOL ruby 2.7.5 into the image preserving backward compatibility.

This requires some additional testing, increases image size and the image itself might be marked having security issues.

[BraisVQ]

What does renaming the agent mean? We'll still maintain a single agent in this repo but would not actively maintain existing built agents?

Is there any side effect if built agent images might get deleted mistakenly?

In Java agent we renamed the agent to jdk because maven was removed from it (wrappers approach is recomended instead).
This means that maven agent is removed from the code but not frmo the ODS installation. It will be just an image that will not receive updates.

[BraisVQ]

Option D) Add EOL ruby 2.7.5 into the image preserving backward compatibility.

This requires some additional testing, increases image size and the image itself might be marked having security issues.

This would be a prefered option if posible. BUT we would still need to communicate to users that Ruby 2.x will be removed soon and they need to update their code then remove Ruby 2.x at a later date.

[nichtraunzer]

Sorry for jumping in late ... considering our previous discussion my understanding was that we simply introduce a new agent and keep the old agent in parallel for compatibility

[BraisVQ]

Always welcome @nichtraunzer, we were just checking all possibilities and its impact to the users.
Using a new agent will make 0 impact to users but might not be the best approach if we start to do so each time a breaking change comes. We can think of it as a last option.
Adding both Ruby versions would be a better option (If it is even possible)

[gerardcl]

Hi! Just to add my two cents: I think having a generic agent image for all terraform quickstarters is the way to go, because both azure and aws quickstarters use the same, right?

[tbugfinder]

Hi! Just to add my two cents: I think having a generic agent image for all terraform quickstarters is the way to go, because both azure and aws quickstarters use the same, right?

Still aws&azure QS rely on the same image.

The question is how to deal with existing environments which depend on a previous image build.

@gerardcl Are you saying there should also be one image for existing and new environments? Is it ok that admins of existing environments have to update the code or should the image be versatile?

[tbugfinder]

Sorry for jumping in late ... considering our previous discussion my understanding was that we simply introduce a new agent and keep the old agent in parallel for compatibility

How often can these be updated or how many versions should be maintained? Is it terraform-agent and terraform-agent-2?

[gerardcl]

Hi!
first, sorry that I exchanged two topics in one (agent naming and quickstarters naming), my last comment on the naming goes for the quickstarters, which to me should also be just one and then offer switch of "providers" when making use of the component library, but that is a thought.
second, why not make then the agent naming also dependant on the ruby version, something like jenkins-agent-terraform-ruby-<2,3,4,...>?

[BraisVQ]

@gerardcl I do think it is a good idea as it seems ruby version is a breaking change and it will follow the same approach we have with node agents.

@tbugfinder @nichtraunzer we can follow the last suggested naming convention that way we keep last agent and we have new one available. New naming convention makes sense in the following versions of ruby.

[nichtraunzer]

Sorry for jumping in late ... considering our previous discussion my understanding was that we simply introduce a new agent and keep the old agent in parallel for compatibility

How often can these be updated or how many versions should be maintained? Is it terraform-agent and terraform-agent-2?

My understanding is that we are more flexible and do not depend on ODS releases - @braisvq1996 correct me if I am wrong ...

My preferred approach to this topic would be:

• having now a second agent with updated ruby available to "new customers"
• "old customers" shall stick to the current version with maintenance limited to bugs / security fixes only

Going this approach allows us to inform "old customers" about the availability of the new agent version in a follow-up "sunset" phase (3-6 months). So we would have enough time to

• stabilize the new agent
• provide proper migration documentation / paths for both the new agent and(!) the state file migration

[BraisVQ]

Sorry for jumping in late ... considering our previous discussion my understanding was that we simply introduce a new agent and keep the old agent in parallel for compatibility

How often can these be updated or how many versions should be maintained? Is it terraform-agent and terraform-agent-2?

My understanding is that we are more flexible and do not depend on ODS releases - @braisvq1996 correct me if I am wrong ...

My preferred approach to this topic would be:

• having now a second agent with updated ruby available to "new customers"
• "old customers" shall stick to the current version with maintenance limited to bugs / security fixes only

Going this approach allows us to inform "old customers" about the availability of the new agent version in a follow-up "sunset" phase (3-6 months). So we would have enough time to

• stabilize the new agent
• provide proper migration documentation / paths for both the new agent and(!) the state file migration

We are not necessarily dependent of ODS releases but we can take advantage that one is near in order to update some quickstarters.

Lets go with the approach of having a new agent and as you mention work on the migration path

[tbugfinder]

As a side note, I've done a build with two ruby versions which could work too, but might increase over time.

Anyway the approach is one active/maintained agent and an deprecated agent.

The naming convention for the agent (from now on) is:
jenkins-agent-terraform-yymm

[BraisVQ]

As a side note, I've done a build with two ruby versions which could work too, but might increase over time.

Anyway the approach is one active/maintained agent and an deprecated agent.

The naming convention for the agent (from now on) is:
jenkins-agent-terraform-yymm

I would be more inclined to use package versions instead of dates like Gerard example: jenkins-agent-terraform-ruby3

[tbugfinder]

Today the agent uses ruby3.2.2, tomorrow ruby 3.4.5.

So it might not work well with ruby3 only or even a breaking change is raised by another component, meaning the name could be jenkins-agent-terraform-ruby3-azure3-aws2? Therefore yymm is simple and makes the version transparent (jenkins-agent-terraform-2305:4.x).

[BraisVQ]

@tbugfinder we should also update the makefile to build this new agent