Copy Files to a pod

[ltamaster]

Is there a way to copy files to a pod?, I am trying with connect_get_namespaced_pod_exec but I think is very a limited workaround.

Thanks

[aogier]

Hi, unfortunately it doesn't seem there is another way to cope with that as even the kubectl implementation or this ticket seems to suggest this is the only way.
Have you managed to copy files via exec() ?

[ltamaster]

Hi @aogier,
Yes, I used the exec to copy files. My user case is coping scripts to pods (a rundeck plugin), so using exec is enough. However, my workaround won't work with binary files.

[aogier]

Hi @ltamaster, AFAICT current examples/implementation use tar on a standard stream pipe, like:

source -> tar -> stdout -> kubernetes -> stdin -> tar -> target

so I'd expect it should work for any file type, as long as tar can cope with them. If you don't mind share your current implementation I could try to help you make things work and maybe promote it to an example script to give to others too :)

[ltamaster]

Hi @aogier ,

I am doing something like this:

    # Calling exec interactively.
    exec_command = [shell]
    resp = stream(api.connect_get_namespaced_pod_exec, name, namespace,
                  command=exec_command,
                  stderr=True, stdin=True,
                  stdout=True, tty=False,
                  _preload_content=False)

    file = open(source_file, "r")

    commands = []
    commands.append("cat <<'EOF' >" + destination_file + "\n")
    commands.append(file.read())
    commands.append("EOF\n")

    while resp.is_open():
        resp.update(timeout=1)
        if resp.peek_stdout():
            print("STDOUT: %s" % resp.read_stdout())
        if resp.peek_stderr():
            print("STDERR: %s" % resp.read_stderr())

        if commands:
            c = commands.pop(0)
            resp.write_stdin(c)
        else:
            break

    resp.close()

[aogier]

Ah, I've seen that now. There are some problems with strings send over the websocket, this will in turn also become a python 2/3 issue. I think the problem reside in kubernetes.stream.WSClient implementation, I've trying patching his write_channel() method that way:

def write_channel(self, channel, data):
    """Write data to a channel."""
    self.sock.send(bytes(chr(channel), 'utf-8') + data)

now your modified example does not give errors:

# Copying file
exec_command = ['/bin/sh']
resp = stream(api.connect_get_namespaced_pod_exec, name, 'default',
              command=exec_command,
              stderr=True, stdin=True,
              stdout=True, tty=False,
              _preload_content=False)

source_file = '/bin/sh'
destination_file = '/tmp/sh'

file = open(source_file, "rb")

buffer = b''
with open(source_file, "rb") as file:
    buffer += file.read()

commands = []
commands.append(bytes("cat <<'EOF' >" + destination_file + "\n", 'utf-8'))
commands.append(buffer)
commands.append(bytes("EOF\n", 'utf-8'))
commands.append(bytes("date\n", 'utf-8'))

while resp.is_open():
    resp.update(timeout=1)
    if resp.peek_stdout():
        print("STDOUT: %s" % resp.read_stdout())
    if resp.peek_stderr():
        print("STDERR: %s" % resp.read_stderr())
    if commands:
        c = commands.pop(0)
        #print("Running command... %s\n" % c)
        resp.write_stdin(c)
    else:
        break

resp.write_stdin(bytes("date\n", 'utf-8'))
sdate = resp.readline_stdout(timeout=3)
print("Server date command returns: %s" % sdate)
resp.write_stdin(bytes("whoami\n", 'utf-8'))
user = resp.readline_stdout(timeout=3)
print("Server user is: %s" % user)
resp.close()

however, output is unexpected:

Server date command returns: None
Server user is: None

and copied file is truncated. I'd like to use tar rather than shell heredocs just for.

What do you think ? Ciao

[aogier]

Ok this PoC works well:

# Copying file
exec_command = ['tar', 'xvf', '-', '-C', '/']
resp = stream(api.connect_get_namespaced_pod_exec, name, 'default',
              command=exec_command,
              stderr=True, stdin=True,
              stdout=True, tty=False,
              _preload_content=False)

source_file = '/tmp/dash.tar'
destination_file = '/tmp/sh'

file = open(source_file, "rb")

buffer = b''
with open(source_file, "rb") as file:
    buffer += file.read()

commands = []
commands.append(buffer)

while resp.is_open():
    resp.update(timeout=1)
    if resp.peek_stdout():
        print("STDOUT: %s" % resp.read_stdout())
    if resp.peek_stderr():
        print("STDERR: %s" % resp.read_stderr())
    if commands:
        c = commands.pop(0)
        #print("Running command... %s\n" % c)
        resp.write_stdin(c)
    else:
        break
resp.close()

Things to do:

• programmatically produce tar stream
• send a PR against python-base repo

Some preliminary tests will be needed, though, as well as six'ing websocket/example code

HTH, ciao !

[aogier]

real tar file PoC implemented here, let's wait for kubernetes-client/python-base#52 needed for this to properly work.

ATM only client -> pod binary traffic works on py3, either direction seems to work in py2, ymmv

ciao

[ltamaster]

Hi @aogier ,

I tested your code and It worked great. I was trying to figure out how I can set a particular destination path on the tar command, is that possible?

Thanks
Luis

[aogier]

Hi @ltamaster, first of all I'm very glad this is working yeah :)

Defining destination path is indeed possibile and it depends on how you want to do it. I'll suggest you couple of option starting from what i prefer the most:

Using proposed tar -C /

From manual, tar's -C flag:

Change to DIR before performing any operations.
This option is order-sensitive, i.e. it affects all options that follow.

so if you want to put under /usr/local/foo/bar/:

oggei@cane:/tmp$ cd $(mktemp -d)
oggei@cane:/tmp/tmp.u9nN0XILDP$ mkdir -p usr/local/foo/bar
oggei@cane:/tmp/tmp.u9nN0XILDP$ echo hello > usr/local/foo/bar/my_file
oggei@cane:/tmp/tmp.u9nN0XILDP$ tar cv . > ../package.tar
./
./usr/
./usr/local/
./usr/local/foo/
./usr/local/foo/bar/
./usr/local/foo/bar/my_file
oggei@cane:/tmp/tmp.u9nN0XILDP$

That way, you have a /tmp/package.tar that when decompressed place a usr/local/foo/bar/my_file under your root filesystem. Tar being tar, this is not limited to a single file BTW (in fact even the example above ships 4 dir and 1 file).

A similar concept is used eg. in Debian packaging system, where a filesystem root is produced under debian/$pkg_name and then compressed in .deb package.

Runtime -C flag definition

You could write your code so you define operations' root via -C flag, and then you tar a hierarchy relative to this root. It's more or less like the above, except you start from a custom root and not / (so you can tar c file > archive.tar and then place it where you want via -C). Seems less straightforward to me, but it works.

OK that's all, happy hacking on that :)

[ltamaster]

Thanks @aogier

Adding this on your code made the trick:
tar.add(name=source_file,arcname=destination_path+"/"+file)

[aogier]

ah brilliant, this is a lot simpler ... I admit it, didn't studied well the tar module 🐐

thanks for sharing !

[ltamaster]

BTW, I got an error message on the line resp.close():

ERROR: websocket: close status: 891

The file is copied OK; the message is produced when that line is called.

[aogier]

I've pushed a new feature/476-copy-example branch with an updated submodule revision pointer could you please try this one ? No problem happens to me with either python 2 and 3, double check you're using 7b133cf for repo and kubernetes-client/python-base@7d67d12 for base submodule ! :)

[ltamaster]

Ok, I will try that

[aogier]

@ltamaster FYI I've just deleted old tags and squashed/rebased all the stuff now everything reside at 5cb61bb and two PR submitted and I'm still curious if everything works well for you :)

hope this will get accepted soon my work is done on that ciao ! 👍

[ltamaster]

@aogier I used 5cb61bb and I still got the message ERROR:websocket:close status: 891 when the method resp.close() is called, weird. Maybe is a problem with my env (I am using python 2.7.14).

[aogier]

@ltamaster cannot recreate this on examples/exec.py with either py2 and py3 ... are you using different code ? If yes could you please share it with me ?

[ltamaster]

Yes, I tried your code using python 2.7.14.
I debugged it and I noticed that I got the 891 response from this code:
https://github.com/websocket-client/websocket-client/blob/master/websocket/_core.py#L403
This line returns the 891 response code: recv_status = struct.unpack("!H", frame.data[0:2])[0]

I will try with a fresh env.

[ltamaster]

The fresh env (ubuntu 17.10 with python 2.7) gives me the same result

[aogier]

ok thank you for your commitment I'll try to recreate it on travis

[aogier]

Hi @ltamaster , I'm unfortunately unable to reproduce your issue on travis, relevant build here. Which exec.py stage is returning this error ? Could you paste python -Wall examples/exec.py output ? Which k8s version are you using ?

[ltamaster]

Hi @aogier

I am using the new OIDC authentication (my Kubernetes environment is tectonic ).
That was merged into the master: kubernetes-client/python-base#48

I clone your repo but I cannot use because it doesn't have that support.
What I tried before was merge your changes on my repo, which has the OIDC support.

Maybe the problem is there.

[ltamaster]

Hi @aogier ,

I didn't get the error with your code (which create a pod directly). I noticed that when I run the same code to a pod created with a development I got the error.

I created an example here, that is based on example/exec.py, but it create a deployment and connect to a pod from that deployment: example

[ltamaster]

Hi @aogier

Did you have the change to reproduce the warning message with the script that I sent?

Thanks
Luis

[aogier]

ehi ciao @ltamaster just got some vacations :) I'll look at the script by the end of the week ! Thanks, ciao

[plaurin84]

This really useful, thank you sirs ;-)

[plaurin84]

I am trying to use this PoC https://github.com/aogier/k8s-client-python/blob/12f1443895e80ee24d689c419b5642de96c58cc8/examples/exec.py#L101

I'm not sure how I should specify the file I want to pull from the pod using this solution. I am probably missing something here...

[revollat]

Hi @aogier I tryed your code (with python 3.6) and I get this error
can you help please ...

Traceback (most recent call last):
  File "cp_test.py", line 136, in <module>
    resp.write_stdin(c)
  File "/home/ubuntu/kubengine/kublog/venv/lib/python3.5/site-packages/kubernetes/stream/ws_client.py", line 160, in write_stdin
    self.write_channel(STDIN_CHANNEL, data)
  File "/home/ubuntu/kubengine/kublog/venv/lib/python3.5/site-packages/kubernetes/stream/ws_client.py", line 114, in write_channel
    self.sock.send(chr(channel) + data)
TypeError: Can't convert 'bytes' object to str implicitly

[micw523]

@revollat This might help:
https://stackoverflow.com/questions/16699362/python3-error-typeerror-cant-convert-bytes-object-to-str-implicitly
As a side note, your path says python 3.5.

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[Ark-kun]

/reopen

[k8s-ci-robot]

@Ark-kun: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/reopen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[sona201]

Hi @ltamaster，

i want to copy foo.tar.gz from pod, when i doing like this

    # Calling exec interactively

    src_path = pod/to/foo.tar.gz
    target_path = local/to/foo.tar.gz
    _exec_copy_command = ['/bin/sh', '-c', 'cat {src_path}'.format(src_path=src_path)]
    with open(target_path, 'a') as tar_file:
        resp = stream(self.api_instance.connect_get_namespaced_pod_exec,
                      pod_name,
                      namespace,
                      command=_exec_copy_command,
                      container=container,
                      stderr=True, stdin=True,
                      stdout=True, tty=False,
                      _preload_content=False)

        while resp.is_open():
            resp.update(timeout=1)
            if resp.peek_stdout():
                out = resp.read_stdout()
                # print("STDOUT: %s" % len(out))
                tar_file.write(out)
            if resp.peek_stderr():
                # print('STDERR: {0}'.format(resp.read_stderr()))
                logger.error(f'STDERR: {resp.read_stderr()}')
        resp.close()

however，Compare the MD5 of the two files, inconsistent

is there anything wrong？

[QIvan]

Hey, @aogier, thanks for sharing your code snippets earlier. I know it's been a while, but could you help me compile the final example of copying a file with this library?
Something like that fits in this signature:

def copy_file(kube_conn, namespace: str, pod_name: str, source_file, dest_path: str):
    pass 

Unfortunately, most of the links you shared earlier don't work nowadays:

real tar file PoC implemented here

Thanks!

[QIvan]

Okay, after several hours of experiments, I finally made it.
For future explores:

def copy_file(kube_conn, namespace: str, pod_name: str, source_file: pathlib.Path, dest_path: str):
    buf = io.BytesIO()
    with tarfile.open(fileobj=buf, mode='w:tar') as tar:
        tar.add(source_file, arcname=pathlib.Path(dest_path).joinpath(source_file.name))
    commands = [buf.getvalue()]

    # Copying file
    exec_command = ['tar', 'xvf', '-', '-C', '/']
    resp = stream.stream(kube_conn.connect_get_namespaced_pod_exec, pod_name, namespace,
                         command=exec_command,
                         stderr=True, stdin=True,
                         stdout=True, tty=False,
                         _preload_content=False)

    while resp.is_open():
        resp.update(timeout=1)
        if resp.peek_stdout():
            print(f"STDOUT: {resp.read_stdout()}")
        if resp.peek_stderr():
            print(f"STDERR: {resp.read_stderr()}")
        if commands:
            c = commands.pop(0)
            resp.write_stdin(c)
        else:
            break
    resp.close()

[iTaybb]

This is my version, which passes gzipped data for better performance, based on @QIvan 's solution:

def k8s_copy_file_from_pod(namespace: str, pod_name: str, source_file: str, dest_path: str):
    exec_command = ['/bin/sh', '-c', f'cd {Path(source_file).parent}; tar czf - {Path(source_file).name}']
    with tempfile.TemporaryFile() as buff:
        resp = k.stream.stream(k.client.CoreV1Api().connect_get_namespaced_pod_exec, pod_name, namespace,
                      command=exec_command,
                      binary=True,
                      stderr=True, stdin=True,
                      stdout=True, tty=False,
                      _preload_content=False)
        while resp.is_open():
            resp.update(timeout=1)
            if resp.peek_stdout():
                out = resp.read_stdout()
                logging.info(f"got {len(out)} bytes")
                buff.write(out)
            if resp.peek_stderr():
                logging.warning(f"STDERR: {resp.read_stderr().decode('utf-8')}")
        resp.close()
        buff.flush()
        buff.seek(0)
        with tarfile.open(fileobj=buff, mode='r:gz') as tar:
            subdir_and_files = [
                tarinfo for tarinfo in tar.getmembers()
            ]
            tar.extractall(path=dest_path, members=subdir_and_files)