If building a python image from a folder, which is also a git-folder build silently fails #452

NegatioN · 2019-02-19T15:08:17Z

There seems to be a bug where if you call s2i build $PATH seldonio/seldon-core-s2i-python3 $NAME in a folder which is also a git-directory, it defaults to treating it as a git-directory, and is somehow unable to load in the files/model in it.

It does output a Build completed successfully at the end, but starting the model will crash with:

docker run -p 5000:5000 minimalmodel:0.1
starting microservice
2019-02-19 14:49:41,565 - seldon_core.microservice:main:277 - INFO:  Starting microservice.py:main
2019-02-19 14:49:41,570 - seldon_core.microservice:main:312 - INFO:  Annotations: {}
Traceback (most recent call last):
  File "/usr/local/bin/seldon-core-microservice", line 11, in <module>
    load_entry_point('seldon-core', 'console_scripts', 'seldon-core-microservice')()
  File "/microservice/python/seldon_core/microservice.py", line 314, in main
    interface_file = importlib.import_module(args.interface_name)
  File "/usr/local/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'Model'

as far as I can see, this bug occurs for all model-complexities on my macbook.

Example outputs with and without .git-folder inside the target folder:

Running s2i build $(pwd) -e MODEL_NAME=Model -e API_TYPE=REST -e SERVICE_TYPE=MODEL -e PERSISTENCE=0 seldonio/seldon-core-s2i-python3:0.5-SNAPSHOT minimalmodel:0.1 --loglevel 5 in a folder without .git in it we get this log-output:

s2i build $(pwd) -e MODEL_NAME=Model -e API_TYPE=REST -e SERVICE_TYPE=MODEL -e PERSISTENCE=0 seldonio/seldon-core-s2i-python3:0.5-SNAPSHOT minimalmodel:0.1 --loglevel 5
I0219 15:52:00.842313    7490 build.go:50] Running S2I version "v1.1.13-2-gbb50d6fa"
I0219 15:52:00.843076    7490 util.go:58] Getting docker credentials for seldonio/seldon-core-s2i-python3:0.5-SNAPSHOT
I0219 15:52:00.850923    7490 docker.go:488] Using locally available image "seldonio/seldon-core-s2i-python3:0.5-SNAPSHOT"
I0219 15:52:00.854582    7490 build.go:163]
Builder Image:			seldonio/seldon-core-s2i-python3:0.5-SNAPSHOT
Source:				/Users/my_user/test/Seldon
Output Image Tag:		minimalmodel:0.1
Environment:			MODEL_NAME=Model,API_TYPE=REST,SERVICE_TYPE=MODEL,PERSISTENCE=0
Labels:
Incremental Build:		disabled
Remove Old Build:		disabled
Builder Pull Policy:		if-not-present
Previous Image Pull Policy:	if-not-present
Quiet:				disabled
Layered Build:			disabled
Docker Endpoint:		unix:///var/run/docker.sock
Docker Pull Config:		/Users/my_user/.docker/config.json
Docker Pull User:

I0219 15:52:00.860118    7490 docker.go:488] Using locally available image "seldonio/seldon-core-s2i-python3:0.5-SNAPSHOT"
I0219 15:52:00.875772    7490 docker.go:488] Using locally available image "seldonio/seldon-core-s2i-python3:0.5-SNAPSHOT"
I0219 15:52:00.875806    7490 docker.go:728] Image sha256:b046a65dcfef796079eb73e9387f5762184e5c92e75b9d930bbab73cd1a2e90f contains io.openshift.s2i.scripts-url set to "image:///s2i/bin"
I0219 15:52:00.876118    7490 scm.go:20] DownloadForSource /Users/my_user/test/Seldon
I0219 15:52:00.876360    7490 sti.go:204] Preparing to build minimalmodel:0.1
I0219 15:52:00.876967    7490 download.go:37] Copying sources from "/Users/my_user/test/Seldon" to "/var/folders/wr/wp4vh7794wn2jbbts6p1t0qxl3krlf/T/s2i861549457/upload/src"
I0219 15:52:00.877277    7490 fs.go:260] F "/Users/my_user/test/Seldon/minimalmodel.onnx" -> "/var/folders/wr/wp4vh7794wn2jbbts6p1t0qxl3krlf/T/s2i861549457/upload/src/minimalmodel.onnx"
I0219 15:52:00.877641    7490 fs.go:260] F "/Users/my_user/test/Seldon/requirements.txt" -> "/var/folders/wr/wp4vh7794wn2jbbts6p1t0qxl3krlf/T/s2i861549457/upload/src/requirements.txt"
I0219 15:52:00.877732    7490 fs.go:247] D "/Users/my_user/test/Seldon/.s2i" -> "/var/folders/wr/wp4vh7794wn2jbbts6p1t0qxl3krlf/T/s2i861549457/upload/src/.s2i"
I0219 15:52:00.877967    7490 fs.go:260] F "/Users/my_user/test/Seldon/.s2i/environment" -> "/var/folders/wr/wp4vh7794wn2jbbts6p1t0qxl3krlf/T/s2i861549457/upload/src/.s2i/environment"
I0219 15:52:00.878297    7490 fs.go:260] F "/Users/my_user/test/Seldon/Model.py" -> "/var/folders/wr/wp4vh7794wn2jbbts6p1t0qxl3krlf/T/s2i861549457/upload/src/Model.py"
I0219 15:52:00.881019    7490 fs.go:260] F "/Users/my_user/test/Seldon/build.sh" -> "/var/folders/wr/wp4vh7794wn2jbbts6p1t0qxl3krlf/T/s2i861549457/upload/src/build.sh"
I0219 15:52:00.881244    7490 fs.go:260] F "/Users/my_user/test/Seldon/.gitignore" -> "/var/folders/wr/wp4vh7794wn2jbbts6p1t0qxl3krlf/T/s2i861549457/upload/src/.gitignore"
I0219 15:52:00.881450    7490 fs.go:260] F "/Users/my_user/test/Seldon/minimalmodel.pt" -> "/var/folders/wr/wp4vh7794wn2jbbts6p1t0qxl3krlf/T/s2i861549457/upload/src/minimalmodel.pt"
I0219 15:52:00.881597    7490 fs.go:247] D "/Users/my_user/test/Seldon/.idea" -> "/var/folders/wr/wp4vh7794wn2jbbts6p1t0qxl3krlf/T/s2i861549457/upload/src/.idea"
I0219 15:52:00.881849    7490 fs.go:260] F "/Users/my_user/test/Seldon/.idea/Seldon.iml" -> "/var/folders/wr/wp4vh7794wn2jbbts6p1t0qxl3krlf/T/s2i861549457/upload/src/.idea/Seldon.iml"
I0219 15:52:00.882087    7490 fs.go:260] F "/Users/my_user/test/Seldon/.idea/encodings.xml" -> "/var/folders/wr/wp4vh7794wn2jbbts6p1t0qxl3krlf/T/s2i861549457/upload/src/.idea/encodings.xml"
I0219 15:52:00.882256    7490 fs.go:247] D "/Users/my_user/test/Seldon/.idea/codeStyles" -> "/var/folders/wr/wp4vh7794wn2jbbts6p1t0qxl3krlf/T/s2i861549457/upload/src/.idea/codeStyles"
I0219 15:52:00.882477    7490 fs.go:260] F "/Users/my_user/test/Seldon/.idea/codeStyles/Project.xml" -> "/var/folders/wr/wp4vh7794wn2jbbts6p1t0qxl3krlf/T/s2i861549457/upload/src/.idea/codeStyles/Project.xml"
I0219 15:52:00.882741    7490 fs.go:260] F "/Users/my_user/test/Seldon/.idea/codeStyles/codeStyleConfig.xml" -> "/var/folders/wr/wp4vh7794wn2jbbts6p1t0qxl3krlf/T/s2i861549457/upload/src/.idea/codeStyles/codeStyleConfig.xml"
I0219 15:52:00.882904    7490 fs.go:247] D "/Users/my_user/test/Seldon/.idea/libraries" -> "/var/folders/wr/wp4vh7794wn2jbbts6p1t0qxl3krlf/T/s2i861549457/upload/src/.idea/libraries"
I0219 15:52:00.883095    7490 fs.go:260] F "/Users/my_user/test/Seldon/.idea/libraries/R_User_Library.xml" -> "/var/folders/wr/wp4vh7794wn2jbbts6p1t0qxl3krlf/T/s2i861549457/upload/src/.idea/libraries/R_User_Library.xml"
I0219 15:52:00.883302    7490 fs.go:260] F "/Users/my_user/test/Seldon/.idea/dbnavigator.xml" -> "/var/folders/wr/wp4vh7794wn2jbbts6p1t0qxl3krlf/T/s2i861549457/upload/src/.idea/dbnavigator.xml"
I0219 15:52:00.883613    7490 fs.go:260] F "/Users/my_user/test/Seldon/.idea/vcs.xml" -> "/var/folders/wr/wp4vh7794wn2jbbts6p1t0qxl3krlf/T/s2i861549457/upload/src/.idea/vcs.xml"
I0219 15:52:00.883943    7490 fs.go:260] F "/Users/my_user/test/Seldon/.idea/workspace.xml" -> "/var/folders/wr/wp4vh7794wn2jbbts6p1t0qxl3krlf/T/s2i861549457/upload/src/.idea/workspace.xml"
I0219 15:52:00.884202    7490 fs.go:260] F "/Users/my_user/test/Seldon/.idea/modules.xml" -> "/var/folders/wr/wp4vh7794wn2jbbts6p1t0qxl3krlf/T/s2i861549457/upload/src/.idea/modules.xml"
I0219 15:52:00.884430    7490 fs.go:260] F "/Users/my_user/test/Seldon/.idea/misc.xml" -> "/var/folders/wr/wp4vh7794wn2jbbts6p1t0qxl3krlf/T/s2i861549457/upload/src/.idea/misc.xml"
I0219 15:52:00.884601    7490 install.go:261] Using "assemble" installed from "image:///s2i/bin/assemble"

But running s2i build $(pwd) -e MODEL_NAME=Model -e API_TYPE=REST -e SERVICE_TYPE=MODEL -e PERSISTENCE=0 seldonio/seldon-core-s2i-python3:0.5-SNAPSHOT minimalmodel:0.1 --loglevel 5 in a folder with .git in it, we get this:

I0219 15:50:27.752463    7287 build.go:50] Running S2I version "v1.1.13-2-gbb50d6fa"
I0219 15:50:27.753375    7287 util.go:58] Getting docker credentials for seldonio/seldon-core-s2i-python3:0.5-SNAPSHOT
I0219 15:50:27.765996    7287 docker.go:488] Using locally available image "seldonio/seldon-core-s2i-python3:0.5-SNAPSHOT"
I0219 15:50:27.768786    7287 build.go:163]
Builder Image:			seldonio/seldon-core-s2i-python3:0.5-SNAPSHOT
Source:				/Users/my_user/gherepos/Seldon
Output Image Tag:		minimalmodel:0.1
Environment:			MODEL_NAME=Model,API_TYPE=REST,SERVICE_TYPE=MODEL,PERSISTENCE=0
Labels:
Incremental Build:		disabled
Remove Old Build:		disabled
Builder Pull Policy:		if-not-present
Previous Image Pull Policy:	if-not-present
Quiet:				disabled
Layered Build:			disabled
Docker Endpoint:		unix:///var/run/docker.sock
Docker Pull Config:		/Users/my_user/.docker/config.json
Docker Pull User:

I0219 15:50:27.771522    7287 docker.go:488] Using locally available image "seldonio/seldon-core-s2i-python3:0.5-SNAPSHOT"
I0219 15:50:27.786031    7287 docker.go:488] Using locally available image "seldonio/seldon-core-s2i-python3:0.5-SNAPSHOT"
I0219 15:50:27.786051    7287 docker.go:728] Image sha256:b046a65dcfef796079eb73e9387f5762184e5c92e75b9d930bbab73cd1a2e90f contains io.openshift.s2i.scripts-url set to "image:///s2i/bin"
I0219 15:50:27.786271    7287 scm.go:20] DownloadForSource /Users/my_user/gherepos/Seldon
I0219 15:50:27.787027    7287 sti.go:204] Preparing to build minimalmodel:0.1
I0219 15:50:27.787471    7287 clone.go:36] Downloading "/Users/my_user/gherepos/Seldon" ...
I0219 15:50:27.787484    7287 clone.go:40] Cloning sources into "/var/folders/wr/wp4vh7794wn2jbbts6p1t0qxl3krlf/T/s2i498479362/upload/src"
Votre branche est à jour avec 'origin/master'.
I0219 15:50:27.863207    7287 clone.go:56] Checked out "HEAD"
I0219 15:50:28.045069    7287 clone.go:62] Updated submodules for "HEAD"
I0219 15:50:28.150102    7287 install.go:261] Using "assemble" installed from "image:///s2i/bin/assemble"
I0219 15:50:28.150142    7287 install.go:261] Using "run" installed from "image:///s2i/bin/run"
I0219 15:50:28.150163    7287 install.go:261] Using "save-artifacts" installed from "image:///s2i/bin/save-artifacts"
I0219 15:50:28.150199    7287 ignore.go:64] .s2iignore file does not exist
I0219 15:50:28.150210    7287 sti.go:213] Clean build will be performed
I0219 15:50:28.150216    7287 sti.go:216] Performing source build from /Users/my_user/gherepos/Seldon
I0219 15:50:28.150222    7287 sti.go:227] Running "assemble" in "minimalmodel:0.1"

I haven't tested what's supposed to happen in the case of a real git repo being targeted externally/locally, but right now it seems to struggle a bit at least.

If there's anything more I can help with, just holler. :)

My test-model, Model.py:

import onnx
import caffe2.python.onnx.backend
import numpy as np

class Model:
    def __init__(self):
        self.model = onnx.load('minimalmodel.onnx')

    def predict(self, X ,features_names):
        return caffe2.python.onnx.backend.run_model(self.model, [X.astype(np.float32)])

The text was updated successfully, but these errors were encountered:

ukclivecox · 2019-02-19T15:18:52Z

Interesting. Can you start the created image in Docker and run with "bash" and see what files are available in the /microservice folder? I assume the Model.py is not there for some reason.

NegatioN · 2019-02-19T15:29:25Z

Correct.
Under all versions of the builder i tried (0.4, 0.5-SNAPSHOT, latest), there are no files copied. so ls just gives an empty directory.

example of running 0.4 below:

I0219 16:30:26.251209    8324 build.go:175] Build completed successfully
➜  Seldon git:(master) ✗ docker run -it --rm minimalmodel:0.1 /bin/bash
root@f9337048df6c:/microservice# ls
root@f9337048df6c:/microservice#

While removing the .git folder copies over all the files in the given directory as one would expect.

I0219 16:31:26.797914    8416 build.go:175] Build completed successfully
➜  Seldon docker run -it --rm minimalmodel:0.1 /bin/bash
root@0dc94d2c8e7c:/microservice# ls
Model.py  build.sh  minimalmodel.onnx  minimalmodel.pt	requirements.txt

jklaise · 2019-02-19T16:14:29Z

@NegatioN I can reproduce this, if you have files in a git repository that are not commited yet, they will not be copied to the image, this is because s2i internally uses git clone on the target. To fix this you can pass the --copy flag to the s2i build command which will use the local file system copy and ignore the git repo status (reference here: https://github.com/openshift/source-to-image/blob/master/docs/cli.md).

NegatioN · 2019-02-19T16:29:25Z

Interesting! Thanks for the clarification. A bit too magical for my taste, but as you say it is because of s2i.
Then unless you want to keep this open to put in some documentation somewhere, the ticket can be closed imo :)

NegatioN changed the title ~~If building from a folder, which is also a git-folder build silently fails~~ If building a python image from a folder, which is also a git-folder build silently fails Feb 19, 2019

ukclivecox closed this as completed Feb 20, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

If building a python image from a folder, which is also a git-folder build silently fails #452

If building a python image from a folder, which is also a git-folder build silently fails #452

NegatioN commented Feb 19, 2019 •

edited

Loading

ukclivecox commented Feb 19, 2019

NegatioN commented Feb 19, 2019 •

edited

Loading

jklaise commented Feb 19, 2019 •

edited

Loading

NegatioN commented Feb 19, 2019 •

edited

Loading

If building a python image from a folder, which is also a git-folder build silently fails #452

If building a python image from a folder, which is also a git-folder build silently fails #452

Comments

NegatioN commented Feb 19, 2019 • edited Loading

ukclivecox commented Feb 19, 2019

NegatioN commented Feb 19, 2019 • edited Loading

jklaise commented Feb 19, 2019 • edited Loading

NegatioN commented Feb 19, 2019 • edited Loading

NegatioN commented Feb 19, 2019 •

edited

Loading

NegatioN commented Feb 19, 2019 •

edited

Loading

jklaise commented Feb 19, 2019 •

edited

Loading

NegatioN commented Feb 19, 2019 •

edited

Loading