Podman 2.2.1 client not connecting to Podman 2.0.5 (nighty rhcos 4.7) due to API version mismatch? #9098

gbraad · 2021-01-26T06:13:25Z

When I execute the following command

PS> .\podman.exe system connection add --identity C:\Users\gbraad\.crc\machines\crc\id_ecdsa --port 2222 crc core@127.0.0.1

an error is returned:

 Error: failed to parse 'podman info' results: json: cannot unmarshal number into Go struct field Version.version.APIVersion of type string

The command

PS> ssh -i C:\Users\gbraad\.crc\machines\crc\id_ecdsa -p 2222 core@127.0.0.1 podman info

provides the following output:

host:
  arch: amd64
  buildahVersion: 1.15.1
  cgroupVersion: v1
  conmon:
    package: conmon-2.0.21-2.rhaos4.6.el8.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.21, commit: e5c6f4123734b4e16cbd22b8f9475a6d02a52e8e'
  cpus: 4
  distribution:
    distribution: '"rhcos"'
    version: "4.7"
  eventLogger: file
  hostname: crc-ctkkw-master-0
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1000
      size: 1
    - container_id: 1
      host_id: 100000
      size: 65536
  kernel: 4.18.0-240.10.1.el8_3.x86_64
  linkmode: dynamic
  memFree: 213295104
  memTotal: 9403904000
  ociRuntime:
    name: runc
    package: runc-1.0.0-81.rhaos4.6.git5b757d4.el8.x86_64
    path: /usr/bin/runc
    version: |-
      runc version spec: 1.0.2-dev
      go: go1.15.0
      libseccomp: 2.4.3
  os: linux
  remoteSocket:
    exists: true
    path: /run/user/1000/podman/podman.sock
  rootless: true
  slirp4netns:
    executable: /bin/slirp4netns
    package: slirp4netns-1.1.4-2.module+el8.3.0+8221+97165c3f.x86_64
    version: |-
      slirp4netns version 1.1.4
      commit: b66ffa8e262507e37fca689822d23430f3357fe8
      libslirp: 4.3.1
      SLIRP_CONFIG_VERSION_MAX: 3
  swapFree: 0
  swapTotal: 0
  uptime: 28m 54.87s
registries:
  search:
  - registry.access.redhat.com
  - docker.io
store:
  configFile: /var/home/core/.config/containers/storage.conf
  containerStore:
    number: 0
    paused: 0
    running: 0
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /bin/fuse-overlayfs
      Package: fuse-overlayfs-1.1.2-3.module+el8.3.0+8221+97165c3f.x86_64
      Version: |-
        fuse-overlayfs: version 1.1.0
        FUSE library version 3.2.1
        using FUSE kernel interface version 7.26
  graphRoot: /var/home/core/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 0
  runRoot: /run/user/1000/containers
  volumePath: /var/home/core/.local/share/containers/storage/volumes
version:
  APIVersion: 1
  Built: 1600877882
  BuiltTime: Wed Sep 23 16:18:02 2020
  GitCommit: ""
  GoVersion: go1.14.7
  OsArch: linux/amd64
  Version: 2.0.5

The client used was: https://github.com/containers/podman/releases/tag/v2.2.1 podman-remote-release-windows.zip

The text was updated successfully, but these errors were encountered:

gbraad · 2021-01-26T06:26:09Z

Using https://github.com/containers/podman/releases/tag/v2.0.6 seems to work on the Command Prompt:

C:\Users\gbraad\Downloads\podman>.\podman.exe system connection add --identity C:\Users\gbraad\.crc\machines\crc\id_ecdsa --port 2222 crc core@127.0.0.1

C:\Users\gbraad\Downloads\podman>podman ps
CONTAINER ID  IMAGE   COMMAND  CREATED  STATUS  PORTS   NAMES

C:\Users\gbraad\Downloads\podman>podman version
Client:
Version:      2.0.6
API Version:  1
Go Version:   go1.14.6
Git Commit:   27362ba1ad8879ea71610fa68a651a1651e0180f
Built:        Wed Sep  2 01:08:00 2020
OS/Arch:      windows/amd64

Server:
Version:      2.0.5
API Version:  0
Go Version:   go1.14.7
Built:        Thu Sep 24 00:18:02 2020
OS/Arch:      linux/amd64

C:\Users\gbraad\Downloads\podman>

Although it works from PowerShell, it will always complain about the console mode (Filed as issue: #9100)

PS> ./podman ps
time="2021-01-26T14:28:58+08:00" level=error msg="Failed to set console mode for cli" error="The parameter is incorrect."
CONTAINER ID  IMAGE   COMMAND  CREATED  STATUS  PORTS   NAMES
PS> ./podman version
time="2021-01-26T14:29:15+08:00" level=error msg="Failed to set console mode for cli" error="The parameter is incorrect."
Client:
Version:      2.0.6
API Version:  1
Go Version:   go1.14.6
Git Commit:   27362ba1ad8879ea71610fa68a651a1651e0180f
Built:        Wed Sep  2 01:08:00 2020
OS/Arch:      windows/amd64

Server:
Version:      2.0.5
API Version:  0
Go Version:   go1.14.7
Built:        Thu Sep 24 00:18:02 2020
OS/Arch:      linux/amd64

In other words; there is no backwards compatibility for the client when using a higher version with an older server component.

baude · 2021-01-27T18:27:19Z

@gbraad in your first comment, after you added the connection ... did it get set to the default? podman system connection ls should indicate the default connection with a *. If it isn't set, try setting the default and seeing if your command works.

gbraad · 2021-01-28T04:37:08Z

It was the only connection (and default)

crc*    C:\Users\gbraad\.crc\machines\crc\id_ecdsa  ssh://core@127.0.0.1:2222/run/user/1000/podman/podman.sock

you noticed it failed with:

Error: failed to parse 'podman info' results: json: cannot unmarshal number into Go struct field Version.version.APIVersion of type string

In the info this says for RHCOS (podman 2.0.5):

version:
  APIVersion: 1
  Built: 1600877882
  BuiltTime: Wed Sep 23 16:18:02 2020
  GitCommit: ""
  GoVersion: go1.14.7
  OsArch: linux/amd64
  Version: 2.0.5

While on Fedora 33 (podman 2.1.1) this is:

version:
  APIVersion: 2.1.0
  Built: 1607438270
  BuiltTime: Tue Dec  8 22:37:50 2020
  GitCommit: ""
  GoVersion: go1.15.5
  OsArch: linux/amd64
  Version: 2.2.1

notet the APIVersion.

afbjorklund · 2021-01-30T16:12:28Z

It changed in c4b49af - I don't think there is compatibility between 2.0 and 2.1, so you need to use a 2.0 client ?

Same thing with 1.9, due to using a different protocol and socket. So far, 2.2 server seems to work with 3.0 client...

But the API bumped from 2.0.0 to 3.0.0 so there are no guarantees.

Version	API	Commit
1.9	varlink
2.0	`1`	`5c8e7ed`
2.1	`2.0.0`	`c4b49af`
2.2	`2.1.0`	`02843f8`
3.0	`3.0.0`	`12f835d`

gbraad · 2021-01-31T14:26:41Z

For now I am stuck on 2.0.6 as client which seems to work. So, yeah. Can confirm this behaviour, but not as expected.

afbjorklund · 2021-01-31T14:56:44Z

From what I can tell, the podman3 server is still using podman2 protocol. Or I guess that should read "Podman 2.1+"

Client:
Version:      3.0.0-rc1
API Version:  3.0.0
Go Version:   go1.15.5
Git Commit:   1cbe815ee74e835f72ee7c63857126b5aedbee39
Built:        Fri Jan 22 17:28:01 2021
OS/Arch:      linux/amd64

Server:
Version:      3.0.0-rc1
API Version:  2.0.0
Go Version:   go1.15.5
Git Commit:   1cbe815ee74e835f72ee7c63857126b5aedbee39
Built:        Fri Jan 22 17:27:52 2021
OS/Arch:      linux/amd64

I'm just going to write "Podman 2.1 or later" for my requirements, and leave support for 1.6 and 2.0 as RHEL exclusive...

mheon · 2021-01-31T16:27:11Z

Eh?

Alright, that's a bug. Supposed to be 3.0 on both ends.

afbjorklund · 2021-01-31T18:07:35Z

Alright, that's a bug. Supposed to be 3.0 on both ends.

Should I open an issue, seems to be the same with RC2 ?

Client:
Version:      3.0.0-rc2
API Version:  3.0.0
Go Version:   go1.15.5
Git Commit:   510a983a752a136a5df0bdbff9d14db0490956b2
Built:        Sun Jan 31 18:56:25 2021
OS/Arch:      linux/amd64

Server:
Version:      3.0.0-rc2
API Version:  2.0.0
Go Version:   go1.15.5
Git Commit:   510a983a752a136a5df0bdbff9d14db0490956b2
Built:        Sun Jan 31 18:56:08 2021
OS/Arch:      linux/amd64

Should this really be "Libpod" twice ?

        // LibpodTree supports Libpod endpoints
        LibpodTree = VersionTree(iota)
        // CompatTree supports Libpod endpoints
        CompatTree

Not sure difference between "CurrentAPIVersion" and "APIVersion" ?

	// APIVersion provides the current and minimal API versions for compat and libpod endpoint trees
	// Note: GET|HEAD /_ping is never versioned and provides the API-Version and Libpod-API-Version headers to allow
	//       clients to shop for the Version they wish to support
	APIVersion = map[VersionTree]map[VersionLevel]semver.Version{
		LibpodTree: {
			CurrentAPIVersion: semver.MustParse("2.0.0"),
			MinimalAPIVersion: semver.MustParse("2.0.0"),
		},
		CompatTree: {
			CurrentAPIVersion: semver.MustParse("1.40.0"),
			MinimalAPIVersion: semver.MustParse("1.24.0"),
		},
	}

// APIVersion is the version for the remote
// client API.  It is used to determine compatibility
// between a remote podman client and its backend
var APIVersion = semver.MustParse("3.0.0")

afbjorklund · 2021-01-31T19:12:22Z

Found a third place, in the bindings (podman-remote)

        // APIVersion - podman will fail to run if this value is wrong
        APIVersion = semver.MustParse("2.0.0")

If only that is changed, it's compatible-like-python:

Error: Cannot connect to the Podman socket, make sure there is a Podman REST API service running.: server API version is too old. Client "3.0.0" server "2.0.0"

Fail to see where the version negotiation is taking place ?

It seems to assume that server version is always newer.

mheon · 2021-01-31T20:02:06Z

Yep, please open a bug. I thought I hit everywhere when I bumped us to 3.0.0-dev, but clearly I missed one.

rhatdan · 2021-02-05T12:01:05Z

Since this is now fixed in podman 3.0, I am going to close this issue. Reopen if I am mistaken.

afbjorklund · 2021-02-05T12:30:47Z

Since this is now fixed in podman 3.0, I am going to close this issue. Reopen if I am mistaken.

It probably won't fix it for podman 2.0, so there one has to keep using the podman 2.0 client...

Might want to document those shortcomings, when running older servers ? (before version 2.1)

But seems to be working for podman 2.2, so as long as client is 2.1+ (not 1.9, not 2.0) it's OK.

Client:
Version:      2.1.1
API Version:  2.0.0
Go Version:   go1.15.5
Built:        Thu Jan  1 01:00:00 1970
OS/Arch:      linux/amd64

Server:
Version:      3.0.0-dev
API Version:  3.0.0
Go Version:   go1.15.5
Git Commit:   c421127dd7f700829a8e5265d8ddad102061bebc
Built:        Fri Feb  5 13:19:30 2021
OS/Arch:      linux/amd64

Client:
Version:      2.2.1
API Version:  2.1.0
Go Version:   go1.15.2
Built:        Thu Jan  1 01:00:00 1970
OS/Arch:      linux/amd64

Server:
Version:      3.0.0-dev
API Version:  3.0.0
Go Version:   go1.15.5
Git Commit:   c421127dd7f700829a8e5265d8ddad102061bebc
Built:        Fri Feb  5 13:19:30 2021
OS/Arch:      linux/amd64

If I understand the original bug report (above), this might be something to document on the RHCOS 4.7 side ?

gbraad changed the title ~~Podman v2 not adding connection when using Windows client and nighty rhcos~~ Podman 2.2.1 not adding connection when using Windows client and nighty rhcos (2.0.5) Jan 26, 2021

gbraad changed the title ~~Podman 2.2.1 not adding connection when using Windows client and nighty rhcos (2.0.5)~~ Podman 2.2.1 client not connecting to Podman 2.0.5 (nighty rhcos 4.7) due to API version mismatch? Jan 29, 2021

afbjorklund mentioned this issue Feb 1, 2021

podman 3.0.0 server seems to still be using the 2.0.0 API #9175

Closed

rhatdan closed this as completed Feb 5, 2021

gbraad mentioned this issue Feb 8, 2021

CentOS fails on iptables when starting a container gbraad/pind-images#2

Closed

gbraad mentioned this issue Mar 29, 2021

[Epic] Basic Podman 2 support crc-org/crc#1611

Closed

2 tasks

github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 22, 2023

github-actions bot locked as resolved and limited conversation to collaborators Sep 22, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Podman 2.2.1 client not connecting to Podman 2.0.5 (nighty rhcos 4.7) due to API version mismatch? #9098

Podman 2.2.1 client not connecting to Podman 2.0.5 (nighty rhcos 4.7) due to API version mismatch? #9098

gbraad commented Jan 26, 2021

gbraad commented Jan 26, 2021 •

edited

Loading

baude commented Jan 27, 2021

gbraad commented Jan 28, 2021 •

edited

Loading

afbjorklund commented Jan 30, 2021 •

edited

Loading

gbraad commented Jan 31, 2021

afbjorklund commented Jan 31, 2021

mheon commented Jan 31, 2021

afbjorklund commented Jan 31, 2021 •

edited

Loading

afbjorklund commented Jan 31, 2021

mheon commented Jan 31, 2021

rhatdan commented Feb 5, 2021

afbjorklund commented Feb 5, 2021

Podman 2.2.1 client not connecting to Podman 2.0.5 (nighty rhcos 4.7) due to API version mismatch? #9098

Podman 2.2.1 client not connecting to Podman 2.0.5 (nighty rhcos 4.7) due to API version mismatch? #9098

Comments

gbraad commented Jan 26, 2021

gbraad commented Jan 26, 2021 • edited Loading

baude commented Jan 27, 2021

gbraad commented Jan 28, 2021 • edited Loading

afbjorklund commented Jan 30, 2021 • edited Loading

gbraad commented Jan 31, 2021

afbjorklund commented Jan 31, 2021

mheon commented Jan 31, 2021

afbjorklund commented Jan 31, 2021 • edited Loading

afbjorklund commented Jan 31, 2021

mheon commented Jan 31, 2021

rhatdan commented Feb 5, 2021

afbjorklund commented Feb 5, 2021

gbraad commented Jan 26, 2021 •

edited

Loading

gbraad commented Jan 28, 2021 •

edited

Loading

afbjorklund commented Jan 30, 2021 •

edited

Loading

afbjorklund commented Jan 31, 2021 •

edited

Loading