'Unrecognized apiVersion: apps/v1beta1' using (local-)openfaas on Kubernetes 1.16

[Tails]

Bug

Current Behavior

With garden.yml Project left as is: ('openfaas' provider)

With the 'openfaas' provider in garden.yml Project changed to 'local-openfaas':

openfaas$ garden scan
 providers                 → Getting status...

Failed resolving provider local-kubernetes. Here is the output:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Command failed with exit code 70 (Unknown system error -70): minikube addons enable dashboard
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Failed resolving one or more provider configurations:
- local-kubernetes: Command failed with exit code 70 (Unknown system error -70): minikube addons enable dashboard

See error.log for detailed error message
openfaas$ garden scan
 providers                 → Getting status...

Failed resolving provider local-kubernetes. Here is the output:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Command failed with exit code 70 (Unknown system error -70): minikube config set WantUpdateNotification false
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Failed resolving one or more provider configurations:
- local-kubernetes: Command failed with exit code 70 (Unknown system error -70): minikube config set WantUpdateNotification false

See error.log for detailed error message
openfaas$ garden scan
✔ providers                 → Preparing environment... → Done
   ✔ local-kubernetes          → Configuring... → Ready
   ✔ tiller                    → Deploying Tiller... → Done (took 19.8 sec)
      ℹ tiller                    → Resources ready
      ℹ tiller                    → Resources ready
modules:
  - allowPublish: false
    apiVersion: garden.io/v0
    build:
      dependencies: []
    description: OpenFaaS runtime
    kind: Module
    name: local-openfaas--system
    outputs:
      release-name: openfaas--openfaas
    path: >-
      /usr/local/Cellar/garden-cli/0.10.13/libexec/static/openfaas/system/openfaas-system
    plugin: local-openfaas
    serviceConfigs:
      - dependencies: []
        hotReloadable: false
        name: local-openfaas--system
    taskConfigs: []
    testConfigs: []
    type: helm
  - allowPublish: false
    apiVersion: garden.io/v0
    build:
      dependencies: []
    description: OpenFaaS templates for building functions
    kind: Module
    name: local-openfaas--templates
    outputs: {}
    path: >-
      /Users/username/Desktop/garden-test/examples/openfaas/.garden/sources/module/templates--3d9439e2ea
    plugin: local-openfaas
    repositoryUrl: 'https://github.com/openfaas/templates.git#master'
    serviceConfigs: []
    taskConfigs: []
    testConfigs: []
    type: exec
  - allowPublish: true
    apiVersion: garden.io/v0
    build:
      dependencies:
        - copy:
            - source: template
              target: .
          name: templates
          plugin: local-openfaas
    configPath: >-
      /Users/username/Desktop/garden-test/examples/openfaas/services/hello-function/garden.yml
    description: Hello world serverless function
    kind: Module
    name: hello-function
    outputs:
      endpoint: 'http://gateway.openfaas.svc.cluster.local:8080/function/hello-function'
    path: >-
      /Users/Desktop/garden-test/examples/openfaas/services/hello-function
    serviceConfigs:
      - dependencies:
          - local-openfaas--system
        hotReloadable: false
        name: hello-function
    taskConfigs: []
    testConfigs: []
    type: openfaas
  - allowPublish: true
    apiVersion: garden.io/v0
    build:
      dependencies: []
    configPath: >-
      /Users/Desktop/garden-test/examples/openfaas/libraries/hello-npm-package/garden.yml
    description: Hello world npm package
    kind: Module
    name: hello-npm-package
    outputs: {}
    path: >-
      /Users/Desktop/garden-test/examples/openfaas/libraries/hello-npm-package
    serviceConfigs: []
    taskConfigs: []
    testConfigs: []
    type: exec
  - allowPublish: true
    apiVersion: garden.io/v0
    build:
      dependencies:
        - copy:
            - source: ./
              target: libraries/hello-npm-package/
          name: hello-npm-package
    configPath: >-
      /Users/Desktop/garden-test/examples/openfaas/services/hello-container/garden.yml
    description: Hello world container service
    kind: Module
    name: hello-container
    outputs:
      deployment-image-name: hello-container
      local-image-name: hello-container
    path: >-
      /Users/Desktop/garden-test/examples/openfaas/services/hello-container
    serviceConfigs:
      - dependencies:
          - hello-function
        hotReloadable: false
        name: hello-container
    taskConfigs: []
    testConfigs:
      - dependencies: []
        name: unit
        spec:
          args:
            - npm
            - test
          dependencies: []
          env: {}
          name: unit
          timeout: null
        timeout: null
      - dependencies:
          - hello-function
        name: integ
        spec:
          args:
            - npm
            - run
            - integ
          dependencies:
            - hello-function
          env:
            FUNCTION_ENDPOINT: >-
              http://gateway.openfaas.svc.cluster.local:8080/function/hello-function
          name: integ
          timeout: null
        timeout: null
    type: container

openfaas$ garden deploy
Deploy 🚀 

 providers                 → Getting status...

Failed resolving provider local-kubernetes. Here is the output:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Command failed with exit code 70 (Unknown system error -70): minikube addons enable dashboard
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

Failed resolving one or more provider configurations:
- local-kubernetes: Command failed with exit code 70 (Unknown system error -70): minikube addons enable dashboard

See error.log for detailed error message
openfaas$ garden deploy
Deploy 🚀 

✔ providers                 → Getting status... → Done
✔ local-openfaas--system    → Building version v-01f7a13ab5... → Done (took 2.6 sec)
✔ hello-npm-package         → Getting build status for v-e801a13beb... → Done (took 0.2 sec)
✔ local-openfaas--templates → Getting build status for v-a4f73398a2... → Done (took 0.2 sec)
✔ hello-container           → Building hello-container:v-607922e1b7... → Done (took 33.7 sec)
✔ hello-function            → Building hello-function:v-9680e39137... → Done (took 35.3 sec)

Failed getting status for service 'local-openfaas--system' (from module 'local-openfaas--system'). Here is the output:
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Unrecognized apiVersion: apps/v1beta1
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

1 deploy task(s) failed!

See error.log for detailed error message

Expected behavior

Having a running OpenFaas instance.

Reproducible example

# initialize minikube kubernetes cluster
$ minikube start --kubernetes-version=1.16.0

# get garden example repo
$ git clone https://github.com/garden-io/garden.git . && cd ./examples/openfaas

# let garden scan state
$ garden scan # -> local-kubernetes: Command failed with exit code 70 (Unknown system error -70): minikube addons enable dashboard

$ garden scan # only works second or third time

$ garden deploy # missing provider dependency 'kubernetes' in provider 'openfaas'

# change '- name: openfaas' to '- name: local-openfaas' in garden.yml
$ garden scan

$ garden deploy # Failed getting status for service 'local-openfaas--system' (from module 'local-openfaas--system'). Unrecognized apiVersion: apps/v1beta1

Workaround

I'm trying to find a Kubernetes version where the OpenFaas example works by downgrading through every minor version, but no luck yet.

Additional context

With default Openfaas example:

[2019-11-02T20:12:44.307Z] Error: Missing provider dependency 'kubernetes' in configuration for provider 'openfaas'. Are you missing a provider configuration?
    at ResolveProviderTask.<anonymous> (/snapshot/dist/build/src/tasks/resolve-provider.js:0)
    at Generator.next (<anonymous>)
    at __awaiter (/snapshot/dist/build/src/tasks/resolve-provider.js:0)
    at new Promise (<anonymous>)
    at __awaiter (/snapshot/dist/build/src/tasks/resolve-provider.js:0)
    at getDependencies.bluebird_1.default.map (/snapshot/dist/build/src/tasks/resolve-provider.js:0)
    at tryCatcher (/snapshot/dist/node_modules/bluebird/js/release/util.js:16:23)
    at MappingPromiseArray.module.exports.MappingPromiseArray._promiseFulfilled (/snapshot/dist/node_modules/bluebird/js/release/map.js:61:38)
    at MappingPromiseArray.module.exports.PromiseArray._iterate (/snapshot/dist/node_modules/bluebird/js/release/promise_array.js:114:31)
    at MappingPromiseArray.init (/snapshot/dist/node_modules/bluebird/js/release/promise_array.js:78:10)
    at MappingPromiseArray.module.exports.MappingPromiseArray._asyncInit (/snapshot/dist/node_modules/bluebird/js/release/map.js:30:10)
    at _drainQueueStep (/snapshot/dist/node_modules/bluebird/js/release/async.js:142:12)
    at _drainQueue (/snapshot/dist/node_modules/bluebird/js/release/async.js:131:9)
    at Async._drainQueues (/snapshot/dist/node_modules/bluebird/js/release/async.js:147:5)
    at Immediate.Async.drainQueues (/snapshot/dist/node_modules/bluebird/js/release/async.js:17:14)
    at runCallback (timers.js:705:18)
    at tryOnImmediate (timers.js:676:5)
    at processImmediate (timers.js:658:5)
Error Details:
config:
  name: openfaas
missingProviderName: kubernetes

Note that I had to run garden scan a few times consecutively (without changing anything else) before it worked.

With 'local-openfaas':

[2019-11-02T20:42:14.738Z] Error: Unrecognized apiVersion: apps/v1beta1
    at KubeApi.<anonymous> (/snapshot/dist/build/src/plugins/kubernetes/api.js:0)
    at Generator.next (<anonymous>)
    at fulfilled (/snapshot/dist/build/src/plugins/kubernetes/api.js:0)
Error Details:
apiVersion: apps/v1beta1
resource:
  apiVersion: apps/v1beta1
  kind: Deployment
  metadata:
    labels:
      app: openfaas
      chart: openfaas-4.4.0
      component: alertmanager
      heritage: Tiller
      release: openfaas--openfaas
    name: alertmanager
    namespace: openfaas
    annotations: {}
  spec:
    replicas: 1
    template:
      metadata:
        labels:
          app: alertmanager
        annotations:
          sidecar.istio.io/inject: 'false'
          checksum/alertmanager-config: ad8df4c40683190fc1cf01c20a8b288d009aaa7663ab18e7e324c241a8f1341b
      spec:
        containers:
          - name: alertmanager
            image: 'prom/alertmanager:v0.16.1'
            imagePullPolicy: Always
            command:
              - alertmanager
              - '--config.file=/alertmanager.yml'
              - '--storage.path=/alertmanager'
            livenessProbe:
              httpGet:
                path: /-/ready
                port: 9093
              timeoutSeconds: 30
            readinessProbe:
              httpGet:
                path: /-/ready
                port: 9093
              timeoutSeconds: 30
            ports:
              - containerPort: 9093
                protocol: TCP
            resources:
              limits:
                memory: 50Mi
              requests:
                memory: 25Mi
            volumeMounts:
              - mountPath: /alertmanager.yml
                name: alertmanager-config
                subPath: alertmanager.yml
        volumes:
          - name: alertmanager-config
            configMap:
              name: alertmanager-config
              items:
                - key: alertmanager.yml
                  path: alertmanager.yml
                  mode: 420


[2019-11-02T20:42:14.792Z] Error: 1 deploy task(s) failed!
    at Object.<anonymous> (/snapshot/dist/build/src/commands/base.js:0)
    at Generator.next (<anonymous>)
    at __awaiter (/snapshot/dist/build/src/commands/base.js:0)
    at new Promise (<anonymous>)
    at __awaiter (/snapshot/dist/build/src/commands/base.js:0)
    at Object.handleTaskResults (/snapshot/dist/build/src/commands/base.js:0)
    at DeployCommand.<anonymous> (/snapshot/dist/build/src/commands/deploy.js:0)
    at Generator.next (<anonymous>)
    at fulfilled (/snapshot/dist/build/src/commands/deploy.js:0)
Error Details:
results:
  taskResults:
    build.hello-npm-package:
      type: build
      key: build.hello-npm-package
      name: hello-npm-package
      description: building hello-npm-package
      completedAt: '2019-11-02T20:42:07.583Z'
      output:
        fresh: false
      dependencyResults: {}
    build.local-openfaas--templates:
      type: build
      key: build.local-openfaas--templates
      name: local-openfaas--templates
      description: building local-openfaas--templates
      completedAt: '2019-11-02T20:42:07.626Z'
      output:
        fresh: false
      dependencyResults: {}
    build.hello-container:
      type: build
      key: build.hello-container
      name: hello-container
      description: building hello-container
      completedAt: '2019-11-02T20:42:07.999Z'
      output:
        fresh: false
      dependencyResults:
        build.hello-npm-package:
          type: build
          key: build.hello-npm-package
          name: hello-npm-package
          description: building hello-npm-package
          completedAt: '2019-11-02T20:42:07.583Z'
          output:
            fresh: false
          dependencyResults: {}
    build.hello-function:
      type: build
      key: build.hello-function
      name: hello-function
      description: building hello-function
      completedAt: '2019-11-02T20:42:08.031Z'
      output:
        fresh: false
      dependencyResults:
        build.local-openfaas--templates:
          type: build
          key: build.local-openfaas--templates
          name: local-openfaas--templates
          description: building local-openfaas--templates
          completedAt: '2019-11-02T20:42:07.626Z'
          output:
            fresh: false
          dependencyResults: {}
    build.local-openfaas--system:
      type: build
      key: build.local-openfaas--system
      name: local-openfaas--system
      description: building local-openfaas--system
      completedAt: '2019-11-02T20:42:14.594Z'
      output:
        fresh: true
      dependencyResults: {}
    get-service-status.local-openfaas--system:
      type: get-service-status
      description: >-
        getting status for service 'local-openfaas--system' (from module
        'local-openfaas--system')
      key: get-service-status.local-openfaas--system
      name: local-openfaas--system
      error:
        detail:
          apiVersion: apps/v1beta1
          resource:
            apiVersion: apps/v1beta1
            kind: Deployment
            metadata:
              labels:
                app: openfaas
                chart: openfaas-4.4.0
                component: alertmanager
                heritage: Tiller
                release: openfaas--openfaas
              name: alertmanager
              namespace: openfaas
              annotations: {}
            spec:
              replicas: 1
              template:
                metadata:
                  labels:
                    app: alertmanager
                  annotations:
                    sidecar.istio.io/inject: 'false'
                    checksum/alertmanager-config: >-
                      ad8df4c40683190fc1cf01c20a8b288d009aaa7663ab18e7e324c241a8f1341b
                spec:
                  containers:
                    - name: alertmanager
                      image: 'prom/alertmanager:v0.16.1'
                      imagePullPolicy: Always
                      command:
                        - alertmanager
                        - '--config.file=/alertmanager.yml'
                        - '--storage.path=/alertmanager'
                      livenessProbe:
                        httpGet:
                          path: /-/ready
                          port: 9093
                        timeoutSeconds: 30
                      readinessProbe:
                        httpGet:
                          path: /-/ready
                          port: 9093
                        timeoutSeconds: 30
                      ports:
                        - containerPort: 9093
                          protocol: TCP
                      resources:
                        limits:
                          memory: 50Mi
                        requests:
                          memory: 25Mi
                      volumeMounts:
                        - mountPath: /alertmanager.yml
                          name: alertmanager-config
                          subPath: alertmanager.yml
                  volumes:
                    - name: alertmanager-config
                      configMap:
                        name: alertmanager-config
                        items:
                          - key: alertmanager.yml
                            path: alertmanager.yml
                            mode: 420
        type: kubernetes
      completedAt: '2019-11-02T20:42:14.730Z'
    deploy.local-openfaas--system: null
    deploy.hello-container: null
    deploy.hello-function: null
  restartRequired: false

Your environment

garden version
0.10.13

minikube version
1.5.2

kubectl version
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.2", GitCommit:"c97fe5036ef3df2967d086711e6c0c405941e14b", GitTreeState:"clean", BuildDate:"2019-10-15T19:18:23Z", GoVersion:"go1.12.10", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.0", GitCommit:"2bd9643cee5b3b3a5ecbd3af49d09018f0773c77", GitTreeState:"clean", BuildDate:"2019-09-18T14:27:17Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}

docker version

Client: Docker Engine - Community
 Version:           18.09.2
 API version:       1.39
 Go version:        go1.10.8
 Git commit:        6247962
 Built:             Sun Feb 10 04:12:39 2019
 OS/Arch:           darwin/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.2
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.6
  Git commit:       6247962
  Built:            Sun Feb 10 04:13:06 2019
  OS/Arch:          linux/amd64
  Experimental:     false

[Tails]

It seems it works on Kubernetes 1.15.2! Then this should be mentioned in the docs, that Garden is not compatible with Kubernetes 1.16.

[edvald]

I think it may just be this particular provider, I'll take a look. We may just need to update it and fix it. We aim to be compatible with new major releases as they come out.

[Tails]

Perhaps related: #1273