`Container.put_archive` fails to raise proper exception copying big enough file into non-existent path inside container

[pavdmyt]

Container.put_archive(path, data) raises docker.errors.NotFound if path argument not found inside container. But if data argument is a sufficiently big file (in my tests 780K is enough), it is fails with requests.exceptions.ConnectionError.

Environment details:

$ pip freeze | grep docker && python --version && docker version
docker==2.6.1
docker-pycreds==0.2.1
Python 3.6.2
Client:
 Version:      17.07.0-ce
 API version:  1.31
 Go version:   go1.9
 Git commit:   8784753
 Built:        unknown-buildtime
 OS/Arch:      darwin/amd64

Server:
 Version:      17.09.0-ce
 API version:  1.32 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:45:38 2017
 OS/Arch:      linux/amd64
 Experimental: false

OS details

$ sw_vers
ProductName:	Mac OS X
ProductVersion:	10.10.5
BuildVersion:	14F2511

Steps to reproduce

Run container:

$ docker run -d --name cp_test ubuntu:16.04 /bin/sh -c "while true; do echo hello world; sleep 1; done"
$ docker ps
CONTAINER ID        NAMES               IMAGE               PORTS               STATUS
574e03b4aca8        cp_test             ubuntu:16.04                            Up About an hour

Consider dkr_cp_bug.py script that puts file into cp_test container:

# -*- coding: utf-8 -*-

import os
import tarfile

import docker


FNAME = "testdata.txt"
TAR_NAME = "testdata.tar"
CONTAINER_NAME = "cp_test"
LINES_QTY = 10 ** 4

# Destination to put file inside container
# (non-existent)
DST_PATH = "/foo"


def compose_data(qty=LINES_QTY):
    line = "-" * 79
    line += "\n"
    return line * qty


def write_data(FNAME, data):
    with open(FNAME, "w") as fd:
        fd.write(data)


def make_tar(TAR_NAME, path):
    with tarfile.open(TAR_NAME, 'w') as tar:
        tar.add(path)


def main():
    try:
        data = compose_data()
        write_data(FNAME, data)
        make_tar(TAR_NAME, "./" + FNAME)

        client = docker.from_env(version="auto")
        container = client.containers.get(CONTAINER_NAME)

        with open(TAR_NAME, 'rb') as fd:
            ok = container.put_archive(path=DST_PATH, data=fd)

    # Cleanup
    finally:
        os.remove(FNAME)
        os.remove(TAR_NAME)

    if ok:
        print("Success")
    else:
        print("Fail")


if __name__ == '__main__':
    main()

Traceback (same happens under Python 2.7.14 and 3.5.2):

$ python dkr_cp_bug.py
Traceback (most recent call last):
  File "/usr/local/var/pyenv/versions/dkr-py36/lib/python3.6/site-packages/urllib3/connectionpool.py", line 601, in urlopen
    chunked=chunked)
  File "/usr/local/var/pyenv/versions/dkr-py36/lib/python3.6/site-packages/urllib3/connectionpool.py", line 357, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/var/pyenv/versions/3.6.2/lib/python3.6/http/client.py", line 1239, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/var/pyenv/versions/3.6.2/lib/python3.6/http/client.py", line 1285, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/var/pyenv/versions/3.6.2/lib/python3.6/http/client.py", line 1234, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/var/pyenv/versions/3.6.2/lib/python3.6/http/client.py", line 1065, in _send_output
    self.send(chunk)
  File "/usr/local/var/pyenv/versions/3.6.2/lib/python3.6/http/client.py", line 986, in send
    self.sock.sendall(data)
  File "/usr/local/var/pyenv/versions/3.6.2/lib/python3.6/ssl.py", line 965, in sendall
    v = self.send(data[count:])
  File "/usr/local/var/pyenv/versions/3.6.2/lib/python3.6/ssl.py", line 935, in send
    return self._sslobj.write(data)
  File "/usr/local/var/pyenv/versions/3.6.2/lib/python3.6/ssl.py", line 636, in write
    return self._sslobj.write(data)
BrokenPipeError: [Errno 32] Broken pipe

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/var/pyenv/versions/dkr-py36/lib/python3.6/site-packages/requests/adapters.py", line 440, in send
    timeout=timeout
  File "/usr/local/var/pyenv/versions/dkr-py36/lib/python3.6/site-packages/urllib3/connectionpool.py", line 639, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/local/var/pyenv/versions/dkr-py36/lib/python3.6/site-packages/urllib3/util/retry.py", line 357, in increment
    raise six.reraise(type(error), error, _stacktrace)
  File "/usr/local/var/pyenv/versions/dkr-py36/lib/python3.6/site-packages/urllib3/packages/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/usr/local/var/pyenv/versions/dkr-py36/lib/python3.6/site-packages/urllib3/connectionpool.py", line 601, in urlopen
    chunked=chunked)
  File "/usr/local/var/pyenv/versions/dkr-py36/lib/python3.6/site-packages/urllib3/connectionpool.py", line 357, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/local/var/pyenv/versions/3.6.2/lib/python3.6/http/client.py", line 1239, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/var/pyenv/versions/3.6.2/lib/python3.6/http/client.py", line 1285, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/var/pyenv/versions/3.6.2/lib/python3.6/http/client.py", line 1234, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/var/pyenv/versions/3.6.2/lib/python3.6/http/client.py", line 1065, in _send_output
    self.send(chunk)
  File "/usr/local/var/pyenv/versions/3.6.2/lib/python3.6/http/client.py", line 986, in send
    self.sock.sendall(data)
  File "/usr/local/var/pyenv/versions/3.6.2/lib/python3.6/ssl.py", line 965, in sendall
    v = self.send(data[count:])
  File "/usr/local/var/pyenv/versions/3.6.2/lib/python3.6/ssl.py", line 935, in send
    return self._sslobj.write(data)
  File "/usr/local/var/pyenv/versions/3.6.2/lib/python3.6/ssl.py", line 636, in write
    return self._sslobj.write(data)
urllib3.exceptions.ProtocolError: ('Connection aborted.', BrokenPipeError(32, 'Broken pipe'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "dkr_cp_bug.py", line 59, in <module>
    main()
  File "dkr_cp_bug.py", line 45, in main
    ok = container.put_archive(path=DST_PATH, data=fd)
  File "/Users/me/git_cloned/docker-py/docker/models/containers.py", line 267, in put_archive
    return self.client.api.put_archive(self.id, path, data)
  File "/Users/me/git_cloned/docker-py/docker/utils/decorators.py", line 19, in wrapped
    return f(self, resource_id, *args, **kwargs)
  File "/Users/me/git_cloned/docker-py/docker/utils/decorators.py", line 34, in wrapper
    return f(self, *args, **kwargs)
  File "/Users/me/git_cloned/docker-py/docker/api/container.py", line 941, in put_archive
    res = self._put(url, params=params, data=data)
  File "/Users/me/git_cloned/docker-py/docker/utils/decorators.py", line 46, in inner
    return f(self, *args, **kwargs)
  File "/Users/me/git_cloned/docker-py/docker/api/client.py", line 195, in _put
    return self.put(url, **self._set_request_timeout(kwargs))
  File "/usr/local/var/pyenv/versions/dkr-py36/lib/python3.6/site-packages/requests/sessions.py", line 566, in put
    return self.request('PUT', url, data=data, **kwargs)
  File "/usr/local/var/pyenv/versions/dkr-py36/lib/python3.6/site-packages/requests/sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/var/pyenv/versions/dkr-py36/lib/python3.6/site-packages/requests/sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/var/pyenv/versions/dkr-py36/lib/python3.6/site-packages/requests/adapters.py", line 490, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', BrokenPipeError(32, 'Broken pipe'))

However, if file to copy is small, e.g. LINES_QTY = 10, it raises docker.errors.NotFound as expected:

# LINES_QTY =  10
$ python dkr_cp_bug.py
Traceback (most recent call last):
  File "/Users/me/git_cloned/docker-py/docker/api/client.py", line 222, in _raise_for_status
    response.raise_for_status()
  File "/usr/local/var/pyenv/versions/dkr-py36/lib/python3.6/site-packages/requests/models.py", line 935, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://192.168.99.100:2376/v1.32/containers/574e03b4aca807939fed04298d877b927fd8b604e476e703cede5a73c4b84b11/archive?path=%2Ffoo

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "dkr_cp_bug.py", line 60, in <module>
    main()
  File "dkr_cp_bug.py", line 46, in main
    ok = container.put_archive(path=DST_PATH, data=fd)
  File "/Users/me/git_cloned/docker-py/docker/models/containers.py", line 267, in put_archive
    return self.client.api.put_archive(self.id, path, data)
  File "/Users/me/git_cloned/docker-py/docker/utils/decorators.py", line 19, in wrapped
    return f(self, resource_id, *args, **kwargs)
  File "/Users/me/git_cloned/docker-py/docker/utils/decorators.py", line 34, in wrapper
    return f(self, *args, **kwargs)
  File "/Users/me/git_cloned/docker-py/docker/api/container.py", line 942, in put_archive
    self._raise_for_status(res)
  File "/Users/me/git_cloned/docker-py/docker/api/client.py", line 224, in _raise_for_status
    raise create_api_error_from_http_exception(e)
  File "/Users/me/git_cloned/docker-py/docker/errors.py", line 31, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
docker.errors.NotFound: 404 Client Error: Not Found ("Could not find the file /foo in container 574e03b4aca807939fed04298d877b927fd8b604e476e703cede5a73c4b84b11")

Some thoughts

Digging into this I've found that it's actually a requests known issue, raised several times already. From this discussion it seems that it will not get fixed any time soon.

Meantime as a workaround it's possible to catch requests.exceptions.ConnnectionError in res = self._put(url, params=params, data=data) call here:

    @utils.check_resource('container')
    @utils.minimum_version('1.20')
    def put_archive(self, container, path, data):
        """
        Insert a file or folder in an existing container using a tar archive as
        source.

        Args:
            container (str): The container where the file(s) will be extracted
            path (str): Path inside the container where the file(s) will be
                extracted. Must exist.
            data (bytes): tar data to be extracted

        Returns:
            (bool): True if the call succeeds.

        Raises:
            :py:class:`docker.errors.APIError`
                If the server returns an error.
        """
        params = {'path': path}
        url = self._url('/containers/{0}/archive', container)
        res = self._put(url, params=params, data=data)
        self._raise_for_status(res)
        return res.status_code == 200

and try to compose docker.errors.NotFound similarly as it is done in docker.errors.create_api_error_from_http_exception.

I can try to compose a PR for this.

[Blizzke]

Any chance this will get fixed in the official repo? @pavdmyt can perhaps create a PR as it looks like a clean fix?
I've just ran into this and wasted half an hour debugging things to figure out what went wrong.

[macuyler]

I recently ran into this same issue, and I was able to figure out a solution by reading these examples:

• https://gist.github.com/michaelconnor00/b3c332a2d6b70f6443d33459d3a731aa
• https://gist.github.com/zbyte64/6800eae10ce082bb78f0b7a2cca5cbc2

It appears that the issue is put_archive needs a byte stream as input instead of just the raw bytes you get from opening the file. Both of the linked examples used BytesIO to accomplish this.

Here is a brief summary of the linked examples:

import tarfile
import time
from io import BytesIO

example_data = 'example'
data = example_data.encode('utf-8')

tarstream = BytesIO()
tar = tarfile.TarFile(fileobj=tarstream, mode='w')
tarinfo = tarfile.TarInfo(name='example.txt')
tarinfo.size = len(data)
tarinfo.mtime = time.time()
tar.addfile(tarinfo, BytesIO(data))
tar.close()

tarstream.seek(0)
r = container.put_archive('/root/', tarstream)

[lobotmcj]

@macuyler thank you -- this saved me from spending even longer debugging this very cryptic behavior