Migrate ci.jenkins.io to the sponsored subscription

[dduportal]

Service(s)

ci.jenkins.io

Summary

This issue tracks the migration of ci.jenkins.io's controller VM to the secondary sponsored Azure subscription.

Ref. #3908 (comment)

The goals are:

• Move ~450$ of monthly billing out of the main subscription
• Improve network performances by keeping all of ci.jenkins.io controller and VMs in the same vnet

Prerequisites are fulfilled:

• Azure Kubernetes publick8s suffers from SNAT port exhaustion: network slowness #3908 led to adding a NAT gateway in the sponsored network (for the controller AND agent subnets)

Plan

• Create a new (and distinct) ci.jenkins.io VM using our module in the new subscription
  • Terraform creation
  • Puppet setup
  • Allow the new VM in all required locations:
    • LDAP (for logging in)
    • AWS cloud (for agents)
    • DigitalOcean (for agents)
• Initial copy of disk data from existing VM to the new VM
  • Snapshot
  • New disk from snapshot
  • Mount manually new disk
  • Rsync from new disk to VM disk
  • Cleanup
• Effective migration to the new VM
  • Announce operation
  • Stop current ci.jenkins.io
  • Sync last changes
  • Move DNS records
  • Restart VM
  • Check everything works (starting, accessing with HTTP and SSH, run Infra acceptance test job to check for allocation of all kind of agents
  • Announce end of operation

[dduportal]

Update on creation with Terraform:

• New subnet created in feat(vnets) add a subnet for ci.jenkins.io controller in the sponsored subscription azure-net#195
• Module update for controller to support migration of VM but not DNS:
  • shared-tools: feat(terraform: azure-jenkinsinfra-controller) allow DNS to use distinct provider or to be undefined shared-tools#135
  • Apply on existing: chore(ci.jenkins.io, trusted.ci.jenkins.io, cert.ci.jenkins.io) update azure VM controller module to support distinct AzureRM provider for DNS azure#585

TODO:

• Effective creation by feat(ci.jenkins.io) add a secondary (empty) VM to prepare moving controller to secondary sponsored subscription azure#583

[dduportal]

Update: new VM created along with its resources in terraform:

• Preparatory work: feat(terraform: azure-jenkinsinfra-controller) allow DNS to use distinct provider or to be undefined shared-tools#135 and chore(ci.jenkins.io, trusted.ci.jenkins.io, cert.ci.jenkins.io) update azure VM controller module to support distinct AzureRM provider for DNS azure#585
• feat(ci.jenkins.io) add a secondary (empty) VM to prepare moving controller to secondary sponsored subscription azure#583
• hotfix(ci.jenkins.io) migrate controller RG to the module azure#593

Next step: Puppet for this VM!

[dduportal]

Update:

• New VM created and available (was able to reach it through the VPN)

  • Routing: feat: add the new (sponsored) vnet routes to CCDs docker-openvpn#326, feat: add the new (sponsored) vnet routes to routing table of the VPN VM jenkins-infra#3268, fix(vnets) required peering for VPN to reach the new controller VM azure-net#203
  • Re-created the VM with a complete DNS/hostname controller.sponsorship.ci.jenkins.io: fix(ci.jenkins.io) re-create the new VM (sponsored) with proper DNS azure#601

• Started the data copy (snapshot of current ci.jenkins.io data, mounted and copy in progress to the datadisk)

WiP:

• Puppet setup (to finish tomorrow with proper data disk UUID)

[dduportal]

Update:

• Initial data copy done
  • snapshot -> disk -> attached to new VM
  • Format and mount the data disk in /var/lib/jenkins
  • Mount the temp disk
  • Ran rsync
  • After rsync finished: unmount all disks, VM reboot, delete temp disk and temp snapshot
• WiP Puppet initialization
  • Declared the node and its settings in Puppet - add a new host: sponsorship.ci.jenkins.io jenkins-infra#3272 (note UUID of the datadisk was retrieved after the initial format and mount during the data initial copy)
  • Host requested a certificate on the puppet master: gotta validate it an run the initial puppet apply

[dduportal]

Update:

• Maintenance announced in https://groups.google.com/g/jenkinsci-dev/c/XPVvL3devzI and status.jenkins.io
• Puppet ran with success.
• Controller did not start though as the content /var/lib/jenkins had to be owned by the user jenkins instead of jenkins-infra-team (the latter was used for the rsync copy) => important to note and fix during the final "diff data sync"

[dduportal]

Update:

• New VM is now using a NAT gateway (ref. feat(gateways) setup ci.jenkins.io sponsorship controller subnet to use NAT gateway azure-net#204) which IP is in the LDAP allow list: https://github.com/jenkins-infra/kubernetes-management/blob/2b1376c452b39a6be70e2596cabe1e8c2ea4ee2f/config/ldap.yaml#L20
• No IP restriction (yes it could be better) for jenkins-infra/aws (https://github.com/jenkins-infra/aws/blob/main/cik8s-cluster.tf) and jenkins-infra/digitalocean (https://github.com/jenkins-infra/digitalocean/blob/main/doks-cluster.tf).

=> we are ready to roll

[dduportal]

Update:

• We had to fix the default user UIDs as the container requires jenkins to have UID 1000

groupmod -g 998 jenkins-infra-team
groupmod -g jenkins 1000
groupmod -g 1000 jenkins
usermod -u 1000 jenkins

• We had to migrate the /etc/letsencrypt directory from the original VM to the new one
• Final sync ran successfully (including a chown -R 1000:1000 /var/lib/jenkins`)
• Migrated CNAMES: feat(ci.jenkins.io): change cname target to new subscription vm azure#604
• Puppet run was successful and started the services (including letsencrypt renewal)
• We had to cleanup old resources and fix the permissions in cleanup(ci.jenkins.io) remove all non sponsored resources azure#605
• Credential was updated manually after this ran

[dduportal]

• Operation closed in close operation for ci.j status#464
• Unmounted and deleted the "old disk". Let's keep the snapshot though

[dduportal]

A word on costs:

• ci.jenkins.io is no longer costing money on our "normal" Azure subscription:

• This cost did not disappeared: it is used by the (credits) sponsored subscription. The current consumption is currentl ~ 1700$ for January (non spot agents and ci.jenkins.io controller): we have to be carefull on these costs (but assuming 2000$ per month with current usage we can expect 18 month)

• The efforts of this issue, [get.jenkins.io/mirrors/mirrorbit - Azure] High costs due to usage of Azure File Storage #3917 and infra.ci.jenkins.io on arm64 (controller and agents) #3823 ensures that we should not cross the 7000$ threshold for January 2024 (check the forecast vs. reality on the last days), and we clearly can expect around 6000$ (we planned 6500$ for January, Feb and March, and then 6000$ for months after that):

[dduportal]

Reopening as there are missing elements:

• Cleanup puppet from controller.ci.jenkins.io former VM references

  • Puppet Master

    $ puppetserver ca revoke --certname controller.ci.jenkins.io
    Certificate for controller.ci.jenkins.io has been revoked
    

  • Puppet code - jenkins-infra/jenkins-infra@185a120 and jenkins-infra/jenkins-infra@4bafec5

• Update runbook - https://github.com/jenkins-infra/runbooks/pull/80

[dduportal]

Runbook fixups: https://github.com/jenkins-infra/runbooks/pull/82 and https://github.com/jenkins-infra/runbooks/pull/87