[BUG] OwnerReferences block disappearing after few hours

[thomasschmitt73]

Components and versions
[x] Controller, version: 1.6.0
[ ] Env-Injector (webhook), version: 1.6.0
[ ] Other

Describe the bug
Link to Secret loses ownerReferences and fails to sync #69

Owner reference on secret vanished :
following block disappears after few hours

  ownerReferences:
    - apiVersion: spv.no/v2beta1
      kind: AzureKeyVaultSecret
      name: admin-awx
      uid: %uid%
      controller: false
      blockOwnerDeletion: true

I am getting this warning for AWX application seen it's being managed by awx-operator
A temporary workaround would be to delete the secret, which is been creating back with akv2k8s ownership but then getting automatically destroyed after few hours.

Any useful workaround ? force the secret to be handle by akv2k8s?
Thanks a lot :)

Expected behavior
No warning event messages

Logs
On controller :

I0326 15:21:56.353663       1 worker.go:97] Error syncing key automation-awx/admin-awx: Resource 'awx-admin-password' already exists and is not managed by AzureKeyVaultSecret
E0326 15:21:56.378315       1 worker.go:93] Failed to process key automation-awx/awx-postgres-configuration. Reason: Resource 'awx-postgres-configuration' already exists and is not managed by AzureKeyVaultSecret
E0326 15:21:56.378345       1 worker.go:108] Resource 'awx-postgres-configuration' already exists and is not managed by AzureKeyVaultSecret
I0326 15:21:56.378354       1 worker.go:110] Dropping key "automation-awx/awx-postgres-configuration" out of the queue: Resource 'awx-postgres-configuration' already exists and is not managed by AzureKeyVaultSecret
E0326 15:21:56.452316       1 worker.go:93] Failed to process key automation-awx/admin-awx. Reason: Resource 'awx-admin-password' already exists and is not managed by AzureKeyVaultSecret
E0326 15:21:56.452346       1 worker.go:108] Resource 'awx-admin-password' already exists and is not managed by AzureKeyVaultSecret
I0326 15:21:56.452355       1 worker.go:110] Dropping key "automation-awx/admin-awx" out of the queue: Resource 'awx-admin-password' already exists and is not managed by AzureKeyVaultSecret

Additional context
AKS on Azure 1.27.7 (kubernetes version)
AWX Version 23.6.0