Volume mount prevents refreshing configuration properties

[everflux]

I have a simple spring boot application and provide two config maps using a volume mount (paths) and the config API access (config sources).
As soon as a paths is configured, the application will not refresh after a change of the config map. The config map values are updated in the volume, but the refresh does not take place. Using the refresh endpoint solves this, but needs to be done manually.
I fact, when a paths configuration is present the refreshing of config-sources is prevented.

Version: 1.0.2.RELEASE

bootstrap.yaml:

  spring:
  cloud:
    kubernetes:
      config:
        sources:
        - name: commons
        paths:
        - /spring/config/application.yml

Used deployment:

---
apiVersion: apps/v1
kind: Deployment
...
    spec:
      serviceAccountName: spring-boot
      containers:
      - name: spring
        image: trion/spring-demo-k8s
        ports:
        - name: http
          containerPort: 8080
        volumeMounts:
        - name: "data"
          mountPath: "/spring/config/"
      volumes:
      - name: "data"
        configMap:
          name: mountconfig

[markfisher]

can you list your spring.cloud.kubernetes.* properties?

[everflux]

See content of 'bootstrap.yaml' above (sorry, indentation is wrong in the ticket)

[markfisher]

spring.cloud.kubernetes.reload.enabled is false by default

[everflux]

I am terribly sorry, indeed my bootstrap.yml was incomplete (due to experiments to triage the issue)
This is the correct bootstrap.yaml:

spring:
  cloud:
    kubernetes:
      config:
        sources:
        - name: commons
#        paths:
#        - /spring/config/application.yml
      reload:
        enabled: true
#        mode: polling
#        period: 5000
        strategy: refresh

and my application.yaml (since I read in another issue that the refresh endpoint must be enabled)

spring:
  application:
    name: spring-demo

management:
  endpoint:
    refresh:
      enabled: true
    restart:
      enabled: true
    health:
      enabled: true
      show-details: always
    info:
      enabled: true
  endpoints:
    web:
      exposure:
        include: "*"
        #restart must be enabled currently
        #https://github.com/spring-cloud/spring-cloud-kubernetes/issues/402

As soon as the commented "paths" gets enabled the refresh no longer works, no matter if I use the kubernetes watch API or polling mode. Disabling the volume mount immediately resolves the issue. Manually refreshing (through the http endpoint) updates the values from the volume.

[Haybu]

Current implementation triggers a reload event once a configmap is updated. However, this event occurs soon before the new configmap content is projected into the mounted volume.

The implementation currently compares config sources in the current state of the environment against the content located from the mounted volume path (while not updated yet).

Refer to the update mounted configmap delay at https://kubernetes.io/docs/tasks/configure-pod-container/configure-pod-configmap/#mounted-configmaps-are-updated-automatically.

It may need to add a feature to watch files system for any update to trigger a reload from mounted volume paths. @geoand @ryanjbaxter thought?

[geoand]

Doesn't the current implementation only apply to reading the configmap for the API and not the file system?

[everflux]

Even if there is some problem with the config map volume update, it should not prevent updating the API based configuration imho.

[markfisher]

even though it's technically separate from this issue, once spring-cloud/spring-cloud-commons#605 and/or spring-cloud/spring-cloud-config#1439 is available, that would allow application config to be updated in an eventually consistent manner while relying on standard Kubernetes ConfigMap volume mounting (no api access from the app)

[Haybu]

yes @geoand, the current implementation uses API to apply reading configmaps directly but not their projected contents from mounted volumes.

@markfisher, that would be an awesome enhancement, it would help with this case to watch the projected files and eventually refresh configs.

[wind57]

The issue reported here, might very likely be a cause of this bug, which seems that will be fixed in both 2.1.x and 3.x.x releases. There are already some PRs pending approval and review.

The real enhancement here would be to actually watch the file mounted in the container, without POLLING and/or EVENT based reloading.

We could have another way to watch, called MOUNT_WATCH or something, where we implement a real directory/file watcher, without any polling on the existing property sources (POLLING) nor any API calls (EVENT).

So you would configure:

spring:
  cloud:
    kubernetes:
      config:
        paths:
        - /spring/config/application.yml

We would watch this file and reload the app when it changes. You can even mount that path as a ConfigMap/Secret and k8s will ultimately update the contents when those change.

This is just an idea, but as usual, before I even start thinking about this, I need to debate it properly to understand the scope with @ryanjbaxter.

[ryanjbaxter]

I am not in favor of adding another mechanism for watching mounted file changes at the moment since the configuration watcher if already meant to solve this.

[wind57]

How would config watcher solve watching for a mounted file though? It still uses the API and expects to catch a ConfigMap change via informers. What am I missing?

[ryanjbaxter]

It doesn't watch the mounted files it watches the configmaps and secrets backing those files. When one of those are changed it tells the app using those config maps and secrets after the k8s maximum period of time to refresh those files so it can refresh its configuration.

Yes the config watcher uses the k8s api to do this, but it alleviates every app having to do this and instead we just have 1 app using the api.

[wind57]

That's what I thought... Thank you, it makes sense.

On a different note, do you think this issue can be closed after we close the one I linked?

[ryanjbaxter]

I think we can close this issue because the config watcher solves the OP request