Feature Request - Support for podman builder

[vikas027]

It would be great to have podman in the builders list.

I have also tried using podman-docker to fake docker commands to use podman in vain (as expected).

❯ skaffold build 
Generating tags...
 - azure.azurecr.io/myimage -> azure.azurecr.io/myimage:20230211_1707_3829a31
Checking cache...
 - azure.azurecr.io/myimage: Not found. Building
Starting build...
Building [azure.azurecr.io/myimage]...
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post "http://%2Fvar%2Frun%2Fdocker.sock/v1.24/build?buildargs=%7B%7D&cachefrom=null&cgroupparent=&cpuperiod=0&cpuquota=0&cpusetcpus=&cpusetmems=&cpushares=0&dockerfile=Dockerfile&labels=null&memory=0&memswap=0&networkmode=&rm=0&shmsize=0&t=azure.azurecr.io%2Fmyimage%3A20230211_1707_3829a31&target=&ulimits=null&version=": dial unix /var/run/docker.sock: connect: permission denied. Docker build ran into internal error. Please retry.
If this keeps happening, please open an issue..

[ericzzzzzzz]

@vikas027 Thank you for filing the request, agreed this is a nice to have. Mark this as p3 at the moment, we can bump this later if we see more people ask this feature.

[AndiDog]

I have a working Podman setup on macOS, and pulling/pushing images works fine with the podman CLI (to GitLab image registry). But with Skaffold, it fails:

build [...] failed: could not push image "registry.gitlab.com/...:...": trying to reuse blob sha256:... at destination: Requesting bearer token: invalid status code from registry 403 (Forbidden)

Even with useDockerCLI: true (with a dummy docker binary that calls podman instead) doesn't help – it may be ignored?!

[userbradley]

I too would like to see this feature!

---

@AndiDog can you share your steps on how you got yours working?

[AndiDog]

I meant above that Podman doesn't work well for me with Skaffold. By now, I've dropped Podman on macOS in favor of colima due to all the problems and because it's hard to contribute to the open source podman project using a Mac...

If you meant how to set up Podman, then Podman Desktop works out of the box.

[userbradley]

I was referring to this section

Even with useDockerCLI: true (with a dummy docker binary that calls podman instead) doesn't help – it may be ignored?!

How have you created a dummy binary? I though symlink but docker uses a daemon (annoyingly) so that wouldnt work

I've just moved from docker to podman, and everything other than skaffold works!

[Larswa]

I second the request for Podman support. Running everything from windows with a podman machine and k3s/containerd on another wsl instance is a really good workflow, and I only miss Skaffold to make it a complete solution.

[sumkincpp]

That's something that would be really important to have!

[userbradley]

So I'm not sure if this is a mixture of me missing something, or something starting to work, but I will post it here.

I seem to have gotten skaffold and podman to work together?

When you start a podman machine, it gives you a command saying (paraphrased) Use this for stuff that uses docker blah blah

I tried it, and it seems to now work

export DOCKER_HOST='unix:///Users/<username>/.local/share/containers/podman/machine/qemu/podman.sock'

Note, this is on a mac

Proof of no docker installed

Skaffold file

apiVersion: skaffold/v4beta5
kind: Config
metadata:
  name: gcs-web-server
build:
  artifacts:
    - image: ghcr.io/userbradley/gcs-web-server
      context: .
      docker:
        dockerfile: Dockerfile

I am more than happy to be proved wrong on this one, as this does feel a little too good to be true

[ericzzzzzzz]

#7078 (comment)

[foxydevloper]

#7078 (comment)

That workaround stopped working a year ago. One of skaffold's releases made it so docker doesn't use the docker remote api and always uses the docker cli #7883 (comment)

[ericzzzzzzz]

hi @foxydevloper the workaround is supposed to work, #8430 (comment) talked about the same approach 3 month ago,
changing the DOCKER_HOST env variable is also suggested here https://podman-desktop.io/docs/migrating-from-docker/using-the-docker_host-environment-variable

[ericzzzzzzz]

Just to clarify, even the workaround works, we're still working on this feature.

[foxydevloper]

@ericzzzzzzz The workaround doesn't appear to work for me on windows, it says the docker executable is required.

Setting the DOCKER_HOST in podman isn't necessary since it uses the default "docker_engine" named pipe on windows anyways.
Also, I don't see the need for a seperate "podman" builder since it still supports docker engine's api anyways. It'd be nice for switching from docker to podman to be able to use existing skaffold configuration that uses docker. But skaffold doesn't appear to be using the docker engine api and instead using the docker executable.

[ericzzzzzzz]

Hi @foxydevloper , Skaffold does use golang docker rest client to talk to docker host by default, unless user explicitly sets Docker useCli to true or skaffold detects that user needs cross-platform build to make the run successful.
It is possible for you to share your skaffold project setup, so we can have a better understanding of what's happening.

[foxydevloper]

@ericzzzzzzz The configuration is as follows:

apiVersion: skaffold/v4beta6
kind: Config
metadata:
  name: dummyproj
build:
  local:
    useDockerCLI: false # default value so shouldn't matter
  artifacts:
    - image: hello
      docker:
        dockerfile: Dockerfile
manifests:
  rawYaml:
    - hello.yaml

As you can see, I explicitly have useDockerCLI set to false.
However when I run skaffold run, I get this, saying exec: "docker": executable file not found in %PATH%. Docker build ran into internal error. Please retry.:

I've also tried setting "useBuildkit" to false and get the same thing.
My project is simply a dockerfile, a hello.py, and a simple deployment

[foxydevloper]

I sent this issue earlier: #7883, which appears to be the same issue, where it's using the docker CLI instead of docker's api. It recommends downgrading to a very old version of skaffold.

[ericzzzzzzz]

ahhh..

skaffold/cmd/skaffold/app/cmd/flags.go

Lines 719 to 729 in 5943bd5

	Name: "check-cluster-node-platforms",
	Usage: "When set to true, images are built for the target platforms matching the active kubernetes cluster node platforms. Enabled by default for `dev`, `debug` and `run`",
	Value: &opts.CheckClusterNodePlatforms,
	DefValue: false,
	DefValuePerCommand: map[string]interface{}{
	"dev": true,
	"debug": true,
	"run": true,
	},
	FlagAddMethod: "BoolVar",
	DefinedOn: []string{"build", "dev", "run", "debug"},

for dev, run, debug "check-cluster-node-platforms" is set to true , that will lead skaffold to use cli for build..

Could you try to set this flag to false to see if it works? skaffold run --check-cluster-node-platforms=false

[foxydevloper]

@ericzzzzzzz Yes, that does indeed fix it and it works perfectly.

[foxydevloper]

@ericzzzzzzz Is this something that will always require this flag to work properly? it's nice there's a workaround but the expectation is that skaffold would work with anything that supports docker's api out of the box without a flag. Skaffold should at least be more clear as to why it's looking for the docker executable and recommend using the flag to opt out. I'd love for this to be improved somewhat

[ericzzzzzzz]

@foxydevloper agreed! I think the reason skaffold is doing that is mainly for corss-platform builds, by default skaffold tries to figure out what platform a image should be build for, that's why it uses target k8s cluster arch as platform, and when platform is specified, the implementation will use docker cli, probably building cross-platform image through docker rest client was not an option during that time. It should be something doable now, thank you for bring this up.
I'll create an issue for it.

[foxydevloper]

I have a working Podman setup on macOS, and pulling/pushing images works fine with the podman CLI (to GitLab image registry). But with Skaffold, it fails:

build [...] failed: could not push image "registry.gitlab.com/...:...": trying to reuse blob sha256:... at destination: Requesting bearer token: invalid status code from registry 403 (Forbidden)

Ran into this as well on Windows, had to install the docker cli, then do docker login, which is seperate from podman's login, and skaffold push started to work. docker login says that it stored the login credentials in ~/.docker\config.json., which podman doesn't save to when logging in. Podman's equivalent directory seems to be ~/.config/containers/auth.json. It has the exact same format, so it would be nice if skaffold also used that path, instead of us having to use the docker cli to login as well.

[AlexGluck]

Fedora 38, podman 4.7.2, skaffold 2.9.0
Build correctly, but deploy helm error becouse;
WARN[0010] unable to extract values for IMAGE_REPO, IMAGE_TAG and IMAGE_DIGEST from image due to error:

Anyone know workaround for this?

[ericzzzzzzz]

Hi @AlexGluck , could you provide minimal reproducible project for us to get a better understanding of the problem you encountered, also you may use this https://github.com/GoogleContainerTools/skaffold/tree/639694dcfead2a825450cc9f3b4696f39d8cf961/examples/helm-deployment helm example project to verifiy if podman works on your end.

[AlexGluck]

@ericzzzzzzz
helm version: version.BuildInfo{Version:"v3.8.2", GitCommit:"6e3701edea09e5d55a8ca2aae03a68917630e91b", GitTreeState:"clean", GoVersion:"go1.17.5"}

Example of my skaffold file:

---
apiVersion: skaffold/v3
kind: Config
metadata:
  name: local
build:
  tagPolicy:
    sha256: {}
  local:
    useBuildkit: true
  artifacts:
  - image: skaffold-helm
    docker:
      dockerfile: ./Dockerfile
deploy:
  helm:
    releases:
    - name: skaffold-helm
      chartPath: charts
      namespace: rooster
      createNamespace: true
      setValueTemplates:
        image: "{{.IMAGE_REPO_skaffold_helm}}:{{.IMAGE_TAG_skaffold_helm}}@{{.IMAGE_DIGEST_skaffold_helm}}"
...

[ericzzzzzzz]

Hi, @AlexGluck I cannot reproduce this on my Mac machine and I commented out useBuildkit as I don't have docker-cli installed, using that will direct it to use docker-cli but I don't think that is the issue. Everything works fine when I use this config with skaffold example project, is it possible for you to share a complete reproducible project or could you use -vdebug flag to get some debug log and paste it here?

[AlexGluck]

Full log here

 1048  git clone https://github.com/GoogleContainerTools/skaffold.git
 1049  cd skaffold/
 1050  git checkout 639694dcfead2a825450cc9f3b4696f39d8cf961
 1051  cd examples/helm-deployment
 1052  cat <<EOF > skaffold.yaml
---
apiVersion: skaffold/v3
kind: Config
metadata:
  name: local
build:
  tagPolicy:
    sha256: {}
  local:
    useBuildkit: true
  artifacts:
  - image: skaffold-helm
    docker:
      dockerfile: ./Dockerfile
deploy:
  helm:
    releases:
    - name: skaffold-helm
      chartPath: charts
      namespace: rooster
      createNamespace: true
      setValueTemplates:
        image: "{{.IMAGE_REPO_skaffold_helm}}:{{.IMAGE_TAG_skaffold_helm}}@{{.IMAGE_DIGEST_skaffold_helm}}"
...
EOF
 1053  skaffold dev -v DEBUG

[VINAYJAGANNATH]

I would like this feature to be made available as soon as possible.

[bkuzmic]

Maybe you'll find this useful. Podman is used for building (with cache enabled) and pushing image to local registry. This speeds up the development cycle because 'kind load' is slow.
https://github.com/bkuzmic/skaffold-podman-kind