release: record release phases #334

hiddeco · 2020-03-19T08:59:28Z

This PR adds a phase field to the status object of the HelmRelease resource, this field records the current or latest action phase performed for the resource.

It also introduces new Prometheus metrics for this field and restores the failure Kubernetes Event, addressing #281 and #62 (comment).

hiddeco · 2020-03-19T18:08:48Z

@sa-spag can you take a look and see if the added histogram (flux_helm_operator_release_phase_duration_seconds) provides all the information necessary for you to do your dashboard and alerting?

sa-spag · 2020-03-20T07:27:43Z

@sa-spag can you take a look and see if the added histogram (flux_helm_operator_release_phase_duration_seconds) provides all the information necessary for you to do your dashboard and alerting?

Yes it does, and...

helm-operator/pkg/release/release.go

Lines 165 to 173 in 9a0512f

    
           const ( 
        
           	InstallAction       action = "install" 
        
           	UpgradeAction       action = "upgrade" 
        
           	SkipAction          action = "skip" 
        
           	RollbackAction      action = "rollback" 
        
           	UninstallAction     action = "uninstall" 
        
           	DryRunCompareAction action = "dry-run-compare" 
        
           	AnnotateAction      action = "annotate" 
        
           )

... seems to cover all interesting cases. Great work, thanks!

sa-spag · 2020-03-27T12:53:54Z

I successfully deployed an image built from your work and applied the new CRD as you suggested on a staging cluster. I'll keep you posted.

sa-spag · 2020-03-27T13:49:00Z

Seems like there's a regression throughput-wise, perhaps workers no longer parallelize the releases:

The green curve corresponds to flux_helm_operator_release_queue_length_count, the vertical blue line is a Grafana annotation which marks the new version deployment.

Turns out it's not related to this pull-request, as fluxcd/helm-operator-prerelease:master-997e8f23 also shows the same behavior.

pkg/status/conditions.go

pkg/operator/operator.go

pkg/apis/helm.fluxcd.io/v1/types_helmrelease.go

pkg/release/release.go

sa-spag · 2020-03-31T09:58:15Z

pkg/operator/operator.go

-	// is synced.
-	MessageChartSynced = "Chart managed by HelmRelease processed"
+	ReleaseSynced       = "ReleaseSynced"
+	FailedReleaseSync   = "FailedReleaseSync"


Might be out of scope but it would be awesome if we could send a separate event for rollbacks (e.g. ReleaseRollBack).

stefanprodan

LGTM

Great improvements! thanks @hiddeco 🥇

This commit introduces the recording of release phases in the `HelmRelease` and as a Prometheus histogram metrics. It also re-introduces the recording of release failures as a Kubernetes Event with a proper failure reason. Limitizes the amount of (obsolete) error wraps and restructures returned error messages.

hiddeco changed the title ~~Enhancement/status~~ release: record release phases Mar 19, 2020

hiddeco changed the title ~~release: record release phases~~ release: record release phases in status Mar 19, 2020

hiddeco added the enhancement New feature or request label Mar 19, 2020

hiddeco force-pushed the enhancement/status branch 2 times, most recently from 4a89d12 to 3aa6b4d Compare March 19, 2020 17:04

hiddeco force-pushed the enhancement/status branch from ebe545a to 9a0512f Compare March 19, 2020 19:30

This was referenced Mar 23, 2020

HelmRelease status seems incorrect #293

Closed

kubectl show wrong crd status #67

Closed

docs: complete overhaul and switch to mkdocs #321

Merged

hiddeco force-pushed the enhancement/status branch from 9a0512f to 98d4eb2 Compare March 27, 2020 05:59

sa-spag reviewed Mar 27, 2020

View reviewed changes

pkg/status/conditions.go Outdated Show resolved Hide resolved

pkg/operator/operator.go Outdated Show resolved Hide resolved

sa-spag reviewed Mar 27, 2020

View reviewed changes

hiddeco force-pushed the enhancement/status branch from d07e554 to 99ce4ce Compare March 30, 2020 09:25

stefanprodan mentioned this pull request Mar 30, 2020

1.0 Release #346

Closed

hiddeco force-pushed the enhancement/status branch 2 times, most recently from 47f0880 to ca01cf0 Compare March 30, 2020 20:29

sa-spag reviewed Mar 31, 2020

View reviewed changes

hiddeco force-pushed the enhancement/status branch 3 times, most recently from 792f511 to aaa51db Compare March 31, 2020 12:30

hiddeco changed the title ~~release: record release phases in status~~ release: record release phases Mar 31, 2020

sa-spag approved these changes Mar 31, 2020

View reviewed changes

hiddeco force-pushed the enhancement/status branch from aaa51db to a0541b6 Compare March 31, 2020 13:24

hiddeco marked this pull request as ready for review March 31, 2020 13:24

hiddeco force-pushed the enhancement/status branch 2 times, most recently from 90612df to e125d83 Compare March 31, 2020 14:36

hiddeco force-pushed the enhancement/status branch from e125d83 to 1813965 Compare March 31, 2020 14:58

hiddeco requested a review from stefanprodan March 31, 2020 15:24

stefanprodan approved these changes Mar 31, 2020

View reviewed changes

hiddeco force-pushed the enhancement/status branch from 5c56508 to 68793ac Compare March 31, 2020 16:35

hiddeco added 3 commits March 31, 2020 19:28

docs: document FailedReleaseSync and new metric

8def91c

docs: fix README.md

7556bd6

hiddeco force-pushed the enhancement/status branch from 68793ac to 7556bd6 Compare March 31, 2020 17:29

hiddeco merged commit 706bcb3 into master Mar 31, 2020

alex-shpak mentioned this pull request Apr 8, 2020

Update helm-operator image tag in values file #366

Closed

sa-spag mentioned this pull request Apr 30, 2020

Release condition gauge (instead of phase gauge) #395

Closed

seaneagan mentioned this pull request May 4, 2020

Deprecate status "phases" #404

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

release: record release phases #334

release: record release phases #334

hiddeco commented Mar 19, 2020 •

edited

Loading

hiddeco commented Mar 19, 2020

sa-spag commented Mar 20, 2020

sa-spag commented Mar 27, 2020

sa-spag commented Mar 27, 2020 •

edited

Loading

sa-spag Mar 31, 2020

stefanprodan left a comment

release: record release phases #334

release: record release phases #334

Conversation

hiddeco commented Mar 19, 2020 • edited Loading

hiddeco commented Mar 19, 2020

sa-spag commented Mar 20, 2020

sa-spag commented Mar 27, 2020

sa-spag commented Mar 27, 2020 • edited Loading

sa-spag Mar 31, 2020

Choose a reason for hiding this comment

stefanprodan left a comment

Choose a reason for hiding this comment

hiddeco commented Mar 19, 2020 •

edited

Loading

sa-spag commented Mar 27, 2020 •

edited

Loading