Fixed kompose build failure #837

surajnarwade · 2017-10-05T16:47:20Z

While local build, kompose was not recognizing dockerfile key
Hence it was breaking the build as mentioned in issue #832.
This PR will fix the issue.

surajnarwade · 2017-10-10T07:48:22Z

@cdrage review needed

cdrage

please fix comments, otherwise LGTM

cdrage · 2017-10-10T16:23:31Z

pkg/transformer/utils.go

+	if service.Dockerfile != "" {
+		//imagePath = path.Join(relativePath, service.Build)
+		dockerfile_dir, _ := path.Split(service.Dockerfile)
+		//imagePath = path.Join(imagePath, dockerfile_dir)


remove commented out lines

cdrage · 2017-10-10T16:23:42Z

pkg/transformer/utils.go

+	var imagePath string
+	if service.Dockerfile != "" {
+		//imagePath = path.Join(relativePath, service.Build)
+		dockerfile_dir, _ := path.Split(service.Dockerfile)


not correct variable name, use dockerfileDir

surajnarwade · 2017-10-11T06:40:56Z

@cdrage changed variable name, good to go

cdrage · 2017-10-11T13:38:02Z

Hey @surajd can you do a quick review to see if this closes your issue?

Ideally... one day we'll have some integration tests for build...

surajssd · 2017-10-19T03:25:15Z

sorry for the late reply, it didn't work for me:

$ kompose version
1.3.0 (7dac78e)

$ docker version
Client:
 Version:      17.09.0-ce
 API version:  1.32
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:42:30 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.09.0-ce
 API version:  1.32 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   afdb6d4
 Built:        Tue Sep 26 22:43:51 2017
 OS/Arch:      linux/amd64
 Experimental: false

trying to build using kompose

$ kompose up -v
DEBU Docker Compose version: 2                    
DEBU Opening compose files: docker-compose.yml    
DEBU [0/1] [gen]: Adding                          
DEBU [0/1] [default]: EventType: 32               
DEBU Default network found                        
INFO Build key detected. Attempting to build and push image 'surajd/kedgeschema' 
DEBU Compose file dir: /home/fedora/go/src/github.com/kedgeproject/json-schema-generator 
INFO Building image 'surajd/kedgeschema' from directory 'scripts' 
DEBU Created temporary file /tmp/kompose-image-build-681917583 for Docker image tarballing 
FATA Error while deploying application: k.Transform failed: Unable to build Docker image for service gen: Unable to build image. For more output, use -v or --verbose when converting.: The command '/bin/sh -c cd /go/src/github.com/kedgeproject/kedge-jsonschema && go build -o kedge-jsonschema main.go parsego.go' returned a non-zero code: 1

works with docker cli

$ docker build -t surajd/kedgeschema -f ./scripts/Dockerfile .
Sending build context to Docker daemon  40.92MB
Step 1/11 : FROM golang:1.9.0 AS build-env
 ---> 1cdc81f11b10
Step 2/11 : ADD . /go/src/github.com/kedgeproject/kedge-jsonschema
 ---> 08dc2baeca36
Step 3/11 : RUN cd /go/src/github.com/kedgeproject/kedge-jsonschema && go build -o kedge-jsonschema main.go parsego.go
 ---> Running in 0c0a2eb954bc
 ---> 4159167cb457
Removing intermediate container 0c0a2eb954bc
Step 4/11 : FROM garethr/openapi2jsonschema
 ---> a889e74d0c7b
Step 5/11 : RUN apk add --update curl &&     rm -rf /var/cache/apk/*
 ---> Running in e0b9df79e429
fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/community/x86_64/APKINDEX.tar.gz
(1/3) Installing libssh2 (1.7.0-r0)
(2/3) Installing libcurl (7.55.0-r0)
(3/3) Installing curl (7.55.0-r0)
Executing busybox-1.24.2-r13.trigger
OK: 32 MiB in 35 packages
 ---> e1c4f68be0f3
Removing intermediate container e0b9df79e429
Step 6/11 : RUN mkdir /lib64 && ln -s /lib/libc.musl-x86_64.so.1 /lib64/ld-linux-x86-64.so.2
 ---> Running in a687d92837ba
 ---> c87a0b8a94ee
Removing intermediate container a687d92837ba
Step 7/11 : COPY --from=build-env /go/src/github.com/kedgeproject/kedge-jsonschema/kedge-jsonschema /usr/local/bin/
 ---> ebeb9c27eca1
Step 8/11 : COPY ./scripts/entrypoint.sh /usr/local/bin/
 ---> 879a1dbeeb42
Step 9/11 : WORKDIR /data
 ---> 01e4d1e6444e
Removing intermediate container e7f7c96fe212
Step 10/11 : ENTRYPOINT
 ---> Running in d822bde63994
 ---> 0ee419b4a177
Removing intermediate container d822bde63994
Step 11/11 : CMD /bin/sh /usr/local/bin/entrypoint.sh
 ---> Running in fbf29bf08841
 ---> 74e63cef9742
Removing intermediate container fbf29bf08841
Successfully built 74e63cef9742
Successfully tagged surajd/kedgeschema:latest

surajnarwade · 2017-10-19T07:45:31Z

@surajssd , problem with context and build has been solved with this PR, as you can see logs problem is with line in Dockerfile

The command '/bin/sh -c cd /go/src/github.com/kedgeproject/kedge-jsonschema && go build -o kedge-jsonschema main.go parsego.go

cdrage · 2017-10-23T17:29:23Z

We really need to add tests for this, I believe we should add some integration tests with this too...

surajnarwade · 2017-11-08T08:32:22Z

@cdrage @surajssd since we need docker login, as of now we don't have integration tests for such cases
cc @ashetty1

cdrage · 2017-11-08T13:07:02Z

We don't need docker login... We can build these locally.

…

On Nov 8, 2017 03:32, "Suraj Narwade" ***@***.***> wrote: @cdrage <https://github.com/cdrage> @surajssd <https://github.com/surajssd> since we need docker login, as of now we don't have integration tests for such cases cc @ashetty1 <https://github.com/ashetty1> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#837 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AGH-oBeff12WgGZxPslTuNprtQ0MuFzUks5s0WcXgaJpZM4PvYa6> .

surajnarwade · 2017-11-27T08:04:41Z

@surajssd @cdrage it seems, error is coming from fsouza/go-dockerclient , I have filed an issue there which can be tracked here fsouza/go-dockerclient#698

fsouza · 2017-11-29T16:01:40Z

pkg/transformer/utils.go

+	var imagePath string
+	if service.Dockerfile != "" {
+		dockerfileDir, _ := path.Split(service.Dockerfile)
+		imagePath = path.Join(relativePath, service.Build, dockerfileDir)


This is not the right fix. imagePath is context. You need to modify your BuildImage function to support specifying the Dockerfile location, because the docker daemon needs the whole thing in a tar package along with the parameter that specifies where to find the Dockerfile inside the package.

One possible fix: https://github.com/fsouza/kompose/pull/1 (this is probably too hacky to be merged as-is and I didn't really look into the tests).

If you want to test this locally (assuming you have kompose cloned inside your $GOPATH):

% cd $GOPATH/src/github.com/kubernetes/kompose % curl https://patch-diff.githubusercontent.com/raw/fsouza/kompose/pull/1.diff | patch -p1 % go install

This will give you the fixed kompose, and you can use it to build the images locally:

% cd $GOPATH/src/github.com/kedgeproject/json-schema-generator % cat docker-compose.yml version: "2" services: my-service: build: context: . dockerfile: scripts/Dockerfile image: docker.io/fsouza/kompose-example % kompose up -v DEBU Docker Compose version: 2 DEBU Opening compose files: docker-compose.yml DEBU [0/1] [my-service]: Adding DEBU [0/1] [default]: EventType: 32 DEBU Default network found INFO Build key detected. Attempting to build and push image 'docker.io/fsouza/kompose-example' DEBU Compose file dir: /Users/fsouza/src/github.com/kedgeproject/json-schema-generator INFO Building image 'docker.io/fsouza/kompose-example' from directory 'json-schema-generator' DEBU Created temporary file /tmp/kompose-image-build-068813155 for Docker image tarballing INFO Image 'docker.io/fsouza/kompose-example' from directory 'json-schema-generator' built successfully DEBU Image docker.io/fsouza/kompose-example build output: [ omitting pull output] Status: Downloaded newer image for garethr/openapi2jsonschema:latest ---> a889e74d0c7b Step 5/11 : RUN apk add --update curl && rm -rf /var/cache/apk/* ---> Running in 9550de763469 fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/main/x86_64/APKINDEX.tar.gz fetch http://dl-cdn.alpinelinux.org/alpine/v3.4/community/x86_64/APKINDEX.tar.gz (1/3) Installing libssh2 (1.7.0-r0) (2/3) Installing libcurl (7.55.0-r2) (3/3) Installing curl (7.55.0-r2) Executing busybox-1.24.2-r13.trigger OK: 32 MiB in 35 packages Removing intermediate container 9550de763469 ---> 0581d5f0334b Step 6/11 : RUN mkdir /lib64 && ln -s /lib/libc.musl-x86_64.so.1 /lib64/ld-linux-x86-64.so.2 ---> Running in 614e1a2f354d Removing intermediate container 614e1a2f354d ---> ce00b0eca38b Step 7/11 : COPY --from=build-env /go/bin/schemagen /usr/local/bin/ ---> 8a10cbb79e51 Step 8/11 : COPY ./scripts/entrypoint.sh /usr/local/bin/ ---> f724b972bbd9 Step 9/11 : WORKDIR /data Removing intermediate container 4641e6f13a49 ---> 76ef64fb903b Step 10/11 : ENTRYPOINT [] ---> Running in 7a646cbf785c Removing intermediate container 7a646cbf785c ---> 52de1720118a Step 11/11 : CMD ["/bin/sh", "/usr/local/bin/entrypoint.sh"] ---> Running in 12da818ac4f0 Removing intermediate container 12da818ac4f0 ---> 761f958cfea5 Successfully built 761f958cfea5 Successfully tagged fsouza/kompose-example:latest [ omit push ]

(The built image is actually on Docker Hub under fsouza/kompose-example)

surajnarwade · 2017-11-29T16:19:22Z

Thanks a lot @fsouza for clearing out the confusion, I will update PR accordingly

surajnarwade · 2017-11-30T14:45:56Z

@surajssd @cdrage can you please test the PR ?

While `local` build, kompose was not recognizing `dockerfile` key Hence it was breaking the build as mentioned in issue kubernetes#832. This PR will fix the issue.

surajssd · 2017-12-04T09:30:28Z

I am using this file in https://github.com/kedgeproject/json-schema-generator

$ cat docker-compose.yml 
version: "2"
services:
  gen:
    build:
      context: .
      dockerfile: scripts/Dockerfile
    image: surajd/testschema

When I run kompose I get error:

$ kompose up -v
DEBU Docker Compose version: 2                    
DEBU Opening compose files: docker-compose.yml    
DEBU [0/1] [gen]: Adding                          
DEBU [0/1] [default]: EventType: 32               
DEBU Default network found                        
INFO Build key detected. Attempting to build and push image 'surajd/testschema' 
DEBU Compose file dir: /home/hummer/go/src/github.com/kedgeproject/json-schema-generator 
INFO Building image 'surajd/testschema' from directory 'json-schema-generator' 
DEBU Created temporary file /tmp/kompose-image-build-720868890 for Docker image tarballing 
FATA Error while deploying application: k.Transform failed: Unable to build Docker image for service gen: Unable to build image.
For more output, use -v or --verbose when converting.: API error (500): {"message":"Cannot locate specified Dockerfile: Dockerfile"}

I am not sure if it is taking the Dockerfile which is somewhere else and the context is somewhere else.

I think you might want to run this test to test if the code works.

surajnarwade · 2017-12-05T11:31:02Z

@surajssd , can you please check one more time,

$ cat docker-compose.yml 
version: "2"
services:
  gen:
    build:
      context: .
      dockerfile: scripts/Dockerfile
    image: docker.io/surajnarwade/kedgeschema

and output is:

$ kompose up -v
DEBU Docker Compose version: 2                    
DEBU Opening compose files: docker-compose.yml    
DEBU [0/1] [gen]: Adding                          
DEBU [0/1] [default]: EventType: 32               
DEBU Default network found                        
INFO Build key detected. Attempting to build and push image 'docker.io/surajnarwade/kedgeschema' 
DEBU Compose file dir: /home/snarwade/go/src/github.com/kedgeproject/json-schema-generator 
INFO Building image 'docker.io/surajnarwade/kedgeschema' from directory 'json-schema-generator' 
DEBU Created temporary file /tmp/kompose-image-build-521645935 for Docker image tarballing 
INFO Image 'docker.io/surajnarwade/kedgeschema' from directory 'json-schema-generator' built successfully 
DEBU Image docker.io/surajnarwade/kedgeschema build output:
Step 1/11 : FROM golang:1.9.0 AS build-env
 ---> 1cdc81f11b10
Step 2/11 : ADD . /go/src/github.com/kedgeproject/kedge-jsonschema
 ---> Using cache
 ---> 35f4234b278f
Step 3/11 : RUN cd /go/src/github.com/kedgeproject/kedge-jsonschema && go build -o kedge-jsonschema main.go parsego.go
 ---> Using cache
 ---> d7d43a0e297b
Step 4/11 : FROM garethr/openapi2jsonschema
 ---> a889e74d0c7b
Step 5/11 : RUN apk add --update curl &&     rm -rf /var/cache/apk/*
 ---> Using cache
 ---> 7710d0d40f49
Step 6/11 : RUN mkdir /lib64 && ln -s /lib/libc.musl-x86_64.so.1 /lib64/ld-linux-x86-64.so.2
 ---> Using cache
 ---> a3e8a01c47d7
Step 7/11 : COPY --from=build-env /go/src/github.com/kedgeproject/kedge-jsonschema/kedge-jsonschema /usr/local/bin/
 ---> Using cache
 ---> c2f9b8443f30
Step 8/11 : COPY ./scripts/entrypoint.sh /usr/local/bin/
 ---> Using cache
 ---> 5b3610c2f411
Step 9/11 : WORKDIR /data
 ---> Using cache
 ---> f9e63f6aff5b
Step 10/11 : ENTRYPOINT
 ---> Using cache
 ---> 46aba1cefd60
Step 11/11 : CMD /bin/sh /usr/local/bin/entrypoint.sh
 ---> Using cache
 ---> 348c0ebcc764
Successfully built 348c0ebcc764
Successfully tagged surajnarwade/kedgeschema:latest
 
DEBU Pushing Docker image 'docker.io/surajnarwade/kedgeschema' 
INFO Pushing image 'surajnarwade/kedgeschema:latest' to registry 'docker.io' 
INFO Attempting authentication credentials 'https://index.docker.io/v1/ 
DEBU Image 'surajnarwade/kedgeschema:latest' push output:
The push refers to a repository [docker.io/surajnarwade/kedgeschema]
Preparing
Preparing
Preparing
Preparing
Preparing
Preparing
Preparing
Preparing
Preparing
Preparing
Preparing
Preparing
Waiting
Waiting
Waiting
Waiting
Waiting
Waiting
Waiting
Layer already exists
Layer already exists
Layer already exists
Layer already exists
Layer already exists
Layer already exists
Layer already exists
Layer already exists
Layer already exists
Layer already exists
Layer already exists
Layer already exists
latest: digest: sha256:4fe015cfbc12ff21e2c5da135a124447fdf0e423e1c69da38e02c8535927b104 size: 2828
 
INFO Successfully pushed image 'surajnarwade/kedgeschema:latest' to registry 'docker.io' 
DEBU [gen] No ports defined. Headless service will be created. 
INFO We are going to create Kubernetes Deployments, Services and PersistentVolumeClaims for your Dockerized application. If you need different kind of resources, use the 'kompose convert' and 'kubectl create -f' commands instead. 
 
INFO Deploying application in "default" namespace 
INFO Successfully created Service: gen            
INFO Successfully created Deployment: gen         

Your application has been deployed to Kubernetes. You can run 'kubectl get deployment,svc,pods,pvc' for details.

surajssd

This LGTM!

cdrage · 2017-12-11T14:33:00Z

LGTM 👍

CzarMich · 2020-08-07T09:20:15Z

Hello, anybody still here for some help. IO ma stuck as i ma getting similar error.
Will surely be grateful for any help

mikik8s-admin@k8s-master:$ kompose up WARN Unsupported depends_on key - ignoring INFO Build key detected. Attempting to build and push image 'openempi-app' INFO Building image 'openempi-app' from directory 'openempi' INFO Image 'openempi-app' from directory 'openempi' built successfully INFO Pushing image 'library/openempi-app:latest' to registry 'docker.io' WARN Unable to retrieve .docker/config.json authentication details. Check that 'docker login' works successfully on the command line.: open /home/mikik8s-admin/.dockercfg: no such file or directory INFO Authentication credentials are not detected. Will try push without authentication. INFO Attempting authentication credentials 'docker.io ERRO Unable to push image 'library/openempi-app:latest' to registry 'docker.io'. Error: denied: requested access to the resource is denied FATA Error while deploying application: k.Transform failed: Unable to push Docker image for service app: unable to push docker image(s). Check that docker login` works successfully on the command line
$ kompose version
1.19.0 (f63a961)
$ docker version
Client:
Version: 19.03.6
API version: 1.40
Go version: go1.12.17
Git commit: 369ce74a3c
Built: Fri Feb 28 23:45:43 2020
OS/Arch: linux/amd64
Experimental: false

Server:
Engine:
Version: 19.03.6
API version: 1.40 (minimum version 1.12)
Go version: go1.12.17
Git commit: 369ce74a3c
Built: Wed Feb 19 01:06:16 2020
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.3.3-0ubuntu1~18.04.2
GitCommit:
runc:
Version: spec: 1.0.1-dev
GitCommit:
docker-init:
Version: 0.18.0
GitCommit:
`

hangyan · 2020-08-07T09:22:06Z

@CzarMich kompose up command is removed for now, you can use kubectl apply to create your resources

CzarMich · 2020-08-07T09:55:57Z

@CzarMich kompose up command is removed for now, you can use kubectl apply to create your resources

Hello @hangyan. thanks for the quick reply. Does that mean that provided I have Kompose installed, i can use Kubectl to create resources that were packaged for Docker with docker compose; Like for my case I have 3 files (docker-compose; xd1, xx1)
ie: $ kubectl apply -f docker-compose.yml

Thanks

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Oct 5, 2017

surajnarwade changed the title ~~Fixed kompose build failure~~ [WIP]Fixed kompose build failure Oct 5, 2017

k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 5, 2017

surajssd mentioned this pull request Oct 5, 2017

(feat): build docker images using, kedge build kedgeproject/kedge#295

Merged

surajnarwade force-pushed the fix-docker-build branch 2 times, most recently from 4c4e783 to cb82a2c Compare October 9, 2017 08:18

surajnarwade changed the title ~~[WIP]Fixed kompose build failure~~ Fixed kompose build failure Oct 9, 2017

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 9, 2017

cdrage requested changes Oct 10, 2017

View reviewed changes

surajnarwade force-pushed the fix-docker-build branch from cb82a2c to 7dac78e Compare October 11, 2017 06:39

surajnarwade requested a review from surajssd October 16, 2017 06:00

surajnarwade force-pushed the fix-docker-build branch from 7dac78e to b4830e6 Compare November 8, 2017 08:31

surajnarwade force-pushed the fix-docker-build branch from b4830e6 to cb5ed97 Compare November 20, 2017 08:44

fsouza reviewed Nov 29, 2017

View reviewed changes

surajnarwade force-pushed the fix-docker-build branch 2 times, most recently from 01000c9 to ec4e92c Compare November 30, 2017 05:24

surajnarwade force-pushed the fix-docker-build branch from ec4e92c to 693877e Compare December 1, 2017 04:26

Fixed kompose build failure

286fbd1

While `local` build, kompose was not recognizing `dockerfile` key Hence it was breaking the build as mentioned in issue kubernetes#832. This PR will fix the issue.

surajnarwade force-pushed the fix-docker-build branch from 693877e to 286fbd1 Compare December 1, 2017 05:45

surajssd approved these changes Dec 11, 2017

View reviewed changes

cdrage approved these changes Dec 11, 2017

View reviewed changes

cdrage merged commit 5498cf5 into kubernetes:master Dec 11, 2017

fsouza mentioned this pull request Mar 3, 2018

Docker image failed to build fsouza/go-dockerclient#698

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fixed kompose build failure #837

Fixed kompose build failure #837

surajnarwade commented Oct 5, 2017

surajnarwade commented Oct 10, 2017

cdrage left a comment

cdrage Oct 10, 2017

cdrage Oct 10, 2017

surajnarwade commented Oct 11, 2017

cdrage commented Oct 11, 2017

surajssd commented Oct 19, 2017

surajnarwade commented Oct 19, 2017

cdrage commented Oct 23, 2017

surajnarwade commented Nov 8, 2017

cdrage commented Nov 8, 2017 via email

surajnarwade commented Nov 27, 2017

fsouza Nov 29, 2017

surajnarwade commented Nov 29, 2017

surajnarwade commented Nov 30, 2017

surajssd commented Dec 4, 2017

surajnarwade commented Dec 5, 2017

surajssd left a comment

cdrage commented Dec 11, 2017

CzarMich commented Aug 7, 2020

hangyan commented Aug 7, 2020

CzarMich commented Aug 7, 2020 •

edited

Loading

Fixed kompose build failure #837

Fixed kompose build failure #837

Conversation

surajnarwade commented Oct 5, 2017

surajnarwade commented Oct 10, 2017

cdrage left a comment

Choose a reason for hiding this comment

cdrage Oct 10, 2017

Choose a reason for hiding this comment

cdrage Oct 10, 2017

Choose a reason for hiding this comment

surajnarwade commented Oct 11, 2017

cdrage commented Oct 11, 2017

surajssd commented Oct 19, 2017

surajnarwade commented Oct 19, 2017

cdrage commented Oct 23, 2017

surajnarwade commented Nov 8, 2017

cdrage commented Nov 8, 2017 via email

surajnarwade commented Nov 27, 2017

fsouza Nov 29, 2017

Choose a reason for hiding this comment

surajnarwade commented Nov 29, 2017

surajnarwade commented Nov 30, 2017

surajssd commented Dec 4, 2017

surajnarwade commented Dec 5, 2017

surajssd left a comment

Choose a reason for hiding this comment

cdrage commented Dec 11, 2017

CzarMich commented Aug 7, 2020

hangyan commented Aug 7, 2020

CzarMich commented Aug 7, 2020 • edited Loading

CzarMich commented Aug 7, 2020 •

edited

Loading