[New Process Improvement Need]: Continuous Deployment

[riverma]

Checked for duplicates

Yes - I've already checked

Category

Software Lifecycle - the creation, change, and release of software

Describe the need

We have a need for:

• Standardized deployment tools project wide (+1'd by @jjacob7734 @LucaCinquini @nttoole @mcduffie)
• Guidance and procedures for blue/green deployment and rolling updates (+1'd by @stirlingalgermissen @kgrimes2 @LucaCinquini)
• Automated aggregation / packaging of build dependencies with delivered software
• Automated packaging and delivery of documentation

Tasks

Preview Give feedback

1. Recommendations on deployment tools (terraform, ansible, etc.)
2. (Infusion artifact) template terraform deployment for cloud project
3. SLIM best practice guide

[riverma]

+1'd by @ramesh-maddegoda, @drewm-jpl, @nttoole, @MJJoyce, @galenatjpl, @sharimayer, @Scotchester, @kgrimes2, @sujen1412

[hookhua]

+1 @pymonger @mcayan @stirlingalgermissen NISAR and SWOT are looking into further improvements to blue/green deployment. e.g. blue/green deployments with lower downtime. Would be useful to hear of any early findings from NISAR and SWOT SDS PCM teams on this.

[pymonger]

@hookhua: I've got Alex starting work on that in the next sprint. Will keep you informed on our findings and development.

[yunks128]

Summary of current status:

Recommendations on Deployment Tools

• Mentioned in Continuous Integration: https://github.com/NASA-AMMOS/slim/tree/main/docs/guides/software-lifecycle/continuous-integration
• Tools are organized here: https://github.com/NASA-AMMOS/slim/blob/main/docs/guides/software-lifecycle/continuous-integration/continuous-integration-frameworks.md?plain=1

1. Terraform:

   • Use Case: Infrastructure as Code (IaC) for provisioning and managing cloud resources.
   • Benefits:
     • Platform-agnostic (AWS, Azure, GCP).
     • Reproducible deployments with version control.
     • Easy collaboration and state tracking.
   • Usage: Create reusable modules for standardizing deployment pipelines.

2. Ansible:

   • Use Case: Configuration management and application deployment.
   • Benefits:
     • Agentless, simple YAML syntax.
     • Supports blue/green deployment strategies.
     • Integrates seamlessly with CI/CD tools like Jenkins, GitHub Actions.
   • Usage: Automate post-deployment configurations and ensure systems are consistent.

3. Containerization:

   • Use Docker for creating isolated build environments and bundling applications with dependencies.
   • Leverage Kubernetes for managing rolling updates and blue/green deployments.

4. CI/CD Pipelines:

   • Tools like Jenkins, GitHub Actions, or GitLab CI/CD can automate testing, building, and deployment pipelines.
   • Focus on integrating blue/green deployment or rolling updates through automation.

---

Guidance for Blue/Green Deployment and Rolling Updates

1. Blue/Green Deployment:

   • Deploy new updates to a separate environment (blue) while the live environment (green) remains untouched.
   • Tools: AWS Elastic Load Balancer (ELB), NGINX, Kubernetes services with traffic splitting.

2. Rolling Updates:

   • Gradually update instances in a fleet to the new version without downtime.
   • Use orchestration tools like Kubernetes or AWS Auto Scaling Groups with Launch Configurations.
   • Incorporate health checks and monitoring during the update process.

3. Best Practices:

   • Define rollback procedures for both strategies.
   • Implement canary testing (deploy to a small subset of users initially).
   • Use version tags and environment-specific variables to manage deployments.

---

Automated Aggregation/Packaging of Build Dependencies

1. Dependency Management:

   • Use package managers (e.g., pipenv, npm, maven) to lock dependencies at specific versions.

2. Delivery Automation:

   • Bundle dependencies as part of Docker images.
   • Create pipelines for dependency scanning (e.g., Dependabot) to ensure security and compatibility.

---

Automated Packaging and Delivery of Documentation

1. Generate Documentation:

   • Use Slim-cli for automated document generation: https://github.com/NASA-AMMOS/slim-cli/tree/main?tab=readme-ov-file#generate-docusaurus-documentation
   • Integrate automated documentation generation into CI/CD pipelines.

2. Delivery:

   • Host documentation with tools like GitHub Pages, Docusaurus, or internal wiki systems.
   • Automate packaging and deployment as part of the pipeline using build tools.

---

SLIM Best Practice Guide

1. Structure:

   • Include step-by-step instructions for setting up a deployment pipeline.
   • Document tools, templates, and configurations for different deployment scenarios (cloud vs. on-prem).

2. Templates:

   • Provide a Terraform template for a typical deployment.
   • Include configuration files for Ansible to set up and manage environments.

---

Next Steps:

1. Develop a Terraform deployment template for cloud projects.
2. Draft a SLIM Deployment Best Practices Guide with community input.
3. Implement blue/green and rolling update procedures in a test environment and document results.

[yunks128]

We need to refine our deployment strategies and tools, focusing on continuous deployment from day one. Below are key areas for improvement:

1. Reorganize Deployment Categories

   • Currently, categories are jumbled. They should be structured more clearly:
     • Tools: Terraform, Ansible
     • Containerization & CI/CD Pipelines: Needs reorganization and cleanup

2. Follow-Up Action

   • Review M20 lessons learned regarding blue/green deployment.

3. Blue-Green Deployment Complexity

   • Due to increasing complexity, we need a structured Continuous Deployment Guideline based on project sizes and complexities.
   • Reference SRL blue-green deployment example: Zero downtime deployment with Kubernetes + ArgoCD

4. Tooling Comparison

   • Evaluate Terraform vs. Kubernetes, their pros/cons, and suitability for different deployment needs.
   • Terraform: Crude but widely adopted.
   • Kubernetes: More robust, but requires deep expertise.

5. Starter Kits & Templates

   • Provide starter kits, sample projects, and templates to ensure deployments are streamlined from day one.

6. Tooling & Best Practices

   • How to decouple project-agnostic vs. project-specific parts in deployment templates.
   • Goal: Shorten deployment cycles and introduce automated deployments effectively.
   • Address environment isolation:
     • Terraform separate environment settings
     • M20 & Unity faced similar issues.
     • LocalStack for local settings—what configurations should be maintained
     • AWS Systems Manager (SSM) for managing environment variables (consider M20 & Unity use cases).
     • SSM challenges: Ensuring correct settings, handling throttling/caching issues (M20).

7. Application Lifecycle Considerations

   • Define deployment strategies based on project lifecycle.
   • Explore technology-agnostic aspects of continuous deployment vs. project-dependent aspects.
   • GDS section (Needs further discussion).

8. Automate Multi-Environment Testing

   • To ensure environment-agnostic Infrastructure as Code (IaC), early automation of both deployment and testing is critical.

Next Steps:

• Organize categories and tools properly.
• Draft a Continuous Deployment strategy document based on project complexities.
• Conduct Terraform vs. Kubernetes comparison.
• Define best practices for environment isolation and deployment lifecycle strategies.
• Provide starter kits and templates for immediate adoption.

[hookhua]

From our recent meeting https://github.com/NASA-AMMOS/slim/wiki/SLIM-Open-Source-Community-Meeting#agenda
it appears that there are several key best practices insights that we could fold into a continuous deployment guide. Perhaps these could go into a section called "Best Practices" or "General Principles" ?

Automate Deployment and Testing Early:
Integrate deployment to I&T pipeline early on.
Ensure multiple environment testing to validate environment-agnostic Infrastructure as Code (IaC).

Environment Variables Management
Consider utilizing tooling for managing environment variables separate from deployed software to ease transition during and across deployments. e.g. AWS SSM.

Blue/Green Deployment:
Develop strategies for decoupling more of the ball of state from key microservices to enable smoother blue/green deployments.
Examples include considerations for ElasticSearch/OpenSearch cluster blue/green deployments of indices across nodes to help with transition across blue/green cluster nodes.

Deployment Duration:
Aim to reduce deployment times. e.g. from days to hours to minutes.

Avoid Human-in-the-Loop Approaches:
Minimize manual intervention to streamline processes and reduce potential errors.

Zero-Downtime Deployment:
Build upon deployment and automation strategies above to target zero-downtime deployment.

[yunks128]

Draft in progress:
https://github.com/NASA-AMMOS/slim/blob/continuous-deployment/docs/guides/software-lifecycle/continuous-deployment/readme.md

Continuous Deployment Starterkit ("Hello World") draft in progress:
https://github.com/NASA-AMMOS/slim-cd-starterkit