Updated pdk version

[16c7x]

This addresses issues #146 and #145.

The REDME has been updated to remove the line "The latest version of the PDK is also pulled in at build time." which is now incorrect.

The version of the PDK has been updated to version 3.0.1.

The pdk_1_18_1_dependencies test module has been removed as it was out of date and contained a Gemfile that was trying to download non-existent Gems such as puppet-module-posix-default-r3.2 which are no longer needed as the PDK manages its own Gem dependencies. It has been replaced with module create using pdk new module using PDK 3.

[Magisus]

Are you aware of any changes in PDK 3 that might break existing users? You mention people wanting to test Puppet 8 code, but we don't want to accidentally drop support for older code versions either, that might impact tests people have currently.

[16c7x]

@Magisus Looking at https://github.com/puppetlabs/pdk/blob/main/CHANGELOG.md there doesn't look like anything that would drop support, if users are using deprecated commands such as --pe-version they may have to update their jobs to use --puppet-version instead -> puppetlabs/pdk#1250

The biggest change is going to be the addition of ruby 3.2 but ruby 2.7 is still included in the PDK.

Do you think releasing it as puppet-dev-tools:5.x instead of puppet-dev-tools:4.x might be a safer option?

I've tagged @davidsandilands into this for any additional comments on the PDK.

[davidsandilands]

Tagging team to help comment @puppetlabs/devx @pmcmaw

[16c7x]

I've updated the skipfiles list in the build test yaml files, it originally had /root/.pdk/cache/ruby/*/gems/aws-sdk-core-*/lib/aws-sdk-ssooidc/client.rb but needed this /opt/puppetlabs/pdk/share/cache/ruby/*/gems/aws-sdk-core-*/lib/aws-sdk-ssooidc/client.rb adding to pass the trivy scan test. It covers some tokens that are contained in this gem https://github.com/aws/aws-sdk-ruby, for example https://github.com/aws/aws-sdk-ruby/blob/509b182371d88f60919e3035450014552b66aaad/gems/aws-sdk-core/lib/aws-sdk-ssooidc/client.rb#L470

[Magisus]

What we'll probably do after the 5.4.0 release is over, is choose a new fixed tag (maybe it will be 5.x, maybe not), and build a version of this image with PDK 3 but configured to test with Puppet 7 by default (there's an environment variable you can set to accomplish this), and use that as the default in CD4PE 5, probably with a DB migration to fix up the built-in jobs in any existing install. That should have minimal impact on most existing users, but it will make it easy to test against Puppet 8 with a custom job (still using the new default image).

If that proves successful/useful, we can think about if we want to also make changes to CD4PE 4. However, we're trying to get 5.x to a place where it's got feature parity with 4, plus new stuff, and encourage people to move to it, so we may use this as another incentive to upgrade.

[Magisus]

Thanks for this! I'll publish this version of the image early next week, once I've had a chance to share this with the rest of the team and make sure there are no objections. Then customers on both CD4PE 4 and 5 can start using it right away in custom jobs, no upgrade required.

[Magisus]

I'm curious, if we're going to go the route of having them create custom jobs based on a new image for this release, could they just use the official PDK image instead? https://hub.docker.com/r/puppet/pdk

If we're going to update the default for all jobs, it makes sense to make a new puppet-dev-tools, since we also bundle other tools that are used in other jobs. But if all they need is a newer PDK for specific modules, I think they could probably just as easily use the official PDK image as a different puppet-dev-tools image.

If that doesn't work, let me know, and I can still go ahead an publish this.

[16c7x]

https://hub.docker.com/r/puppet/pdk doesn't include Onceover.
I think Onceover uses the Puppet gem listed in the Gemfile to compile it's catalogs and that Gemfile currently states 7.5 (I've only just thought of this) so it can't switch between versions of Puppet the way the PDK can. I'll need to check Onceover with a Puppet 8 gem.
We don't need environmental variables to control the Puppet version for PDK, it will read the metadata.json file and use the correct version of Puppet/Ruby based on that, it's probably a safer way to go as it keeps all the dependencies/config in once place.
If we were to continue with the Puppet-dev-tools container then maybe we should use a Puppet8 tag, as that seems to be the real difference.
Or even switch to https://hub.docker.com/r/puppet/pdk for PDK validate and create a separate container for Onceover so we can have a Puppet 7 and Puppet 8 version of that?

[Magisus]

Ah I see, yeah I think if we were going to release a puppet8 tag of this image, it should include Puppet 8. I can look into making that change, for this opt-in version we've been discussing.

However, since we can only include one, either 7 or 8, I'm not sure what to do about this in the case where we want to make a new default image... 🤔 Will need to think more about that. I don't think we have any default jobs that use onceover at least...

[16c7x]

@Magisus I think it would be safest to continue to ship the current Docker image with cd4pe, customers can carry on as normal and only have to do something when they want to move to Puppet 8, I can imaging they wouldn't be happy if they did an upgrade and found things didn't work exactly as they did before. It might be useful if in future if cd4pe could ship with more than one Docker container, we'd be ready for Puppet 9 then 😆. It might be a good idea anyway as we'd be able to split Onceover into a seperate container and then use the official https://hub.docker.com/r/puppet/pdk and avoid having to upgrade both at once.
I've updated Onceover and the Puppet gem and tested it and Onceover now try and do $testvar = '1' + 1 we get �[31mEvaluation Error: The string '1' was automatically coerced to the numerical value 1 which is what we'd expect for Puppet 8.
If you think pupept-dev-tools:puppet8 is an OK name/tag then I can make the changes to the build scripts and move this into a puppet8 branch.

[Magisus]

Thanks for doing this! I hadn't had time to circle back to this yet, with the big release. I'll get the jobs fixed up today and get an image published. If it needs more changes we can always publish again.

[Magisus]

I've retargeted this at the newly-created puppet8 branch, which I will keep around for now. I'm going to merge it, and make the job updates on that branch.

[Magisus]

@16c7x I've published the puppet8 images to dockerhub: https://hub.docker.com/layers/puppet/puppet-dev-tools/puppet8/images/sha256-6da1c6cdde55a3b174a6042b0e724ce0340d146f616e4ea1853aee78d4e87677 Feel free to give them a try, including with customers! If there are any issues, feel free to let us know, or open a PR based on the puppet8 branch of this repo.