re-adjust finalizer/status update call order

[HumairAK]

this should fix the errors that occur when dspa is deleted, which seems to happen because finalizers are removed (and thus dspa clean up happens before status update is called)

2024-08-19T18:23:53Z INFO DataSciencePipelinesApplication.datasciencepipelinesapplications.opendatahub.io "sample" not found {"namespace": "dspa", "dspa_name": "sample"}
2024-08-19T18:23:53Z INFO Updating components endpoints {"namespace": "dspa", "dspa_name": "sample"}
2024-08-19T18:23:53Z ERROR	Encountered error when updating the DSPA status {"namespace": "dspa", "dspa_name": "sample", "error": "Operation cannot be fulfilled on datasciencepipelinesapplications.datasciencepipelinesapplications.opendatahub.io \"sample\": StorageError: invalid object, Code: 4, Key: /kubernetes.io/datasciencepipelinesapplications.opendatahub.io/datasciencepipelinesapplications/dspa/sample, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 976fd0ef-3cdc-4435-9254-f77f1206f0d1, UID in object meta: "}
github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).updateStatus
/workspace/controllers/dspipeline_controller.go:369
github.com/opendatahub-io/data-science-pipelines-operator/controllers.(*DSPAReconciler).Reconcile
/workspace/controllers/dspipeline_controller.go:224
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:118
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:314
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/opt/app-root/src/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226
2024-08-19T18:23:53Z INFO DSPA resource was not found, assuming it was recently deleted, nothing to do here {"namespace": "dspa", "dspa_name": "sample"}
2024-08-19T18:23:53Z INFO DSPA resource was not found, assuming it was recently deleted, nothing to do here {"namespace": "dspa", "dspa_name": "sample"}

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from humairak. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[dsp-developers]

A new image has been built to help with testing out this PR: quay.io/opendatahub/data-science-pipelines-operator:pr-692
An OCP cluster where you are logged in as cluster admin is required.

To use this image run the following:

cd $(mktemp -d)
git clone git@github.com:opendatahub-io/data-science-pipelines-operator.git
cd data-science-pipelines-operator/
git fetch origin pull/692/head
git checkout -b pullrequest aa064649e73047e955701fca258af86966d6eb8a
oc new-project opendatahub
make deploy IMG="quay.io/opendatahub/data-science-pipelines-operator:pr-692"

More instructions here on how to deploy and test a Data Science Pipelines Application.

[HumairAK]

/hold

I think we need to still investigate this in more detail on the root cause, it's not yet clear it's related to the finalizer, and there's a lot of flakiness involved in reproducing it

[hbelmiro]

/lgtm