Implement process for upgrading old service instances on cloud.gov

[mogul]

User Story

In order to ensure existing instances get updates relating to compliance or security, the SSB should include a mechanism for updating existing instances, or helping clients migrate between plans.

Acceptance Criteria

[ACs should be clearly demoable/verifiable whenever possible. Try specifying them using BDD.]

• GIVEN I have provisioned an EKS instance INSTANCENAME
  AND a the SSB is updated to include a new brokerpak version implementing a new feature
  WHEN I run cf update-service INSTANCENAME
  THEN I can see the new feature is available in INSTANCENAME

Background

[Any helpful contextual notes or links to artifacts/evidence, if needed]

Security Considerations (required)

[Any security concerns that might be implicated in the change. "None" is OK, just be explicit here!]

Sketch

The CSB team is implementing this functionality upstream. See this epic in Pivotal Tracker and the set of stories attached to it.

The feature is now available, but is behind a feature flag we need to enable.

[mogul]

The CSB is implementing this capability upstream, and should be shipping it any day now.

[mogul]

Surprise!

[jbrown-xentity]

@nickumia-reisys can you add link to upstream documentation

[nickumia-reisys]

There actually isn't a good central location where this is documented. Various links,

• Slack Discussion (referenced above) from cloud foundry support.
• Upstream Github Issue
• Upstream Pivotal Issue
• Upstream 0.10.0 Release Notes

[hkdctol]

This is relevant only if/when we move to EKS broker so moving to Icebox for now.

[nickumia-reisys]

As a summary of how this works:

• Before:
  • When a Service was created, it gets a copy of the terraform code from the Broker.
  • The Service instance code can be different from the Broker code.
  • When the Broker is updated, the Service instance code remains unchanged.
  • Running an update-service on an existing Service just re-activated the existing terraform code with new configuration parameters.
  • If the code in the Broker was updated, then new Service instances would get the code changes, but existing instances would not.
• After:
  • When a Service is created, it gets a copy of the terraform code from the Broker.
  • The Service instance code can be different from the Broker code.
  • When the Broker is updated, the Service instance code remains unchanged.
  • Running an update-service on an existing Service installs the most recent code from the Broker into the Service instance and then activates the new code.
  • If the code in the Broker was updated, then new Service instances would get the code changes. Existing instances will not get updated unless the update-service is run.

Because we made this the default behavior, to achieve the old behavior of leaving the terraform code unchanged on update-service, the BROKERPAK_UPDATES_ENABLED variable would need to be disabled on the Broker and restarted before the update-service occurs.

This feature was critical in pushing updates to existing production instances without downtime.

• Before:
  • There was a service instance in production with production data.
  • To get the newest code into the service, a new service would need to be created, loaded with the data and then swapped for no downtime.
  • For our CKAN application, the only way to load the data was to connect to production and rebuild the data from scratch. For this reason, having no downtime was impractical.
• After:
  • There was a service instance in production with production data.
  • To get the newest code into the service, a simple update-service would update the existing architecture with the new changes. As long as the terraform code was non-destructive in the persistent storage resources, minutes-level downtime or less was possible.
  • Example of minutes-level downtime:
    • Solr on ECS - Change Solr locktype to none datagov-ssb#154
  • Example of days-level downtime:
    • Solr on ECS - EFS - Max IO vs. General Purpose datagov-ssb#147
    • Solr on ECS - Setup Solr Replication datagov-ssb#149

Very much related to

• [Tiny EPIC] Support SOLR standalone on ECS #3826
• Use brokered, persistent Solr instead of an ephemeral app #2778
• Configure solr at app startup inventory-app#386
• Inventory uses SOLR 8 #3697
• Solr 8 instance credentials are not bound on cloud.gov #3523
• Lots of EKS Brokerpak issues that are not centrally linked to an epic or main issue. Anytime a new feature was added the the EKS Brokerpak (prior to this ticket), a new service needed to be created. Example: Limit brokered EKS service endpoints to cloud.gov egress IP ranges by default #3744

[nickumia-reisys]

🖖 Big shoutout to @mogul for keeping on top of the Brokerpak community and mentoring me through this!