Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

receiving http error: 400 #113

Open
dawaj71 opened this issue Mar 9, 2018 · 14 comments
Open

receiving http error: 400 #113

dawaj71 opened this issue Mar 9, 2018 · 14 comments

Comments

@dawaj71
Copy link

dawaj71 commented Mar 9, 2018

Hello,

As you can see with the title i got this error "receiving http error: 400" with some images that i want to scan, but with others images it scans well and without errors.

Both kind of images are local that i pull from my registry before i do the scan, the command that i use for the scan is : clairctl analyze -l animage

I used this command to see what are the logs :
clairctl analyze -l --log-level debug animage

2018-03-09 15:01:32.913560 D | config: No config file used
2018-03-09 15:01:32.915134 D | dockercli: docker image to save: animage
2018-03-09 15:01:32.915202 D | dockercli: saving in: /tmp/clairctl/animage
2018-03-09 15:01:47.371101 I | config: retrieving interface for local IP
2018-03-09 15:01:47.371213 D | config: no interface provided, looking for docker0
2018-03-09 15:01:47.372097 D | server: Update local server port from "0" to "53700"
2018-03-09 15:01:47.372147 I | server: Starting Server on X.X.X.X
2018-03-09 15:01:47.377015 I | config: retrieving interface for local IP
2018-03-09 15:01:47.377065 D | config: no interface provided, looking for docker0
2018-03-09 15:01:47.377479 I | clair: using http://X.X.X.X/local as local url
2018-03-09 15:01:47.377524 I | clair: Pushing Layer 1/11 [17efd5fdbef8]
2018-03-09 15:01:47.377741 D | clair: Saving 17efd5fdbef8f0b0dcedd31beb409982edfcae8a6a9b9dbc45f7517e129e6497[https://registry-1.docker.io/v2]
2018-03-09 15:01:47.423263 I | clair: Pushing Layer 2/11 [002fafa83f3b]
2018-03-09 15:01:47.423502 D | clair: Saving 002fafa83f3b6f94b3f44b3df6b1828bb8bd8fb8ded6b68fc2565dbaaef28fa6[https://registry-1.docker.io/v2]
2018-03-09 15:01:47.431985 I | clair: Pushing Layer 3/11 [a43065cd7262]
2018-03-09 15:01:47.432204 D | clair: Saving a43065cd72626b5a3495cb4c53c391a9beb5add727b1ce824eae4394e3b28776[https://registry-1.docker.io/v2]
2018-03-09 15:01:47.439857 I | clair: Pushing Layer 4/11 [35e854e2bf81]
2018-03-09 15:01:47.440034 D | clair: Saving 35e854e2bf8171c89ec6e99d849846137651e28c6ec0f5f39a5de1cfec724232[https://registry-1.docker.io/v2]
2018-03-09 15:01:47.444071 I | clair: Pushing Layer 5/11 [723b01ba3a25]
2018-03-09 15:01:47.444224 D | clair: Saving 723b01ba3a25ca4df42da84356ec6c776e64e0c5967acc08aadae84ae3a05e1b[https://registry-1.docker.io/v2]
2018-03-09 15:01:47.448404 I | clair: Pushing Layer 6/11 [ad2ae30a3f16]
2018-03-09 15:01:47.448534 D | clair: Saving ad2ae30a3f1625a5f5688d0e6ccdaff56abbba45c296c96741ab6379a4b032c7[https://registry-1.docker.io/v2]
2018-03-09 15:01:47.457143 I | clair: Pushing Layer 7/11 [169b8319d348]
2018-03-09 15:01:47.457295 D | clair: Saving 169b8319d3481f95e77e13436276b0e27d0cc2a4c5b8ae8171f5ff3efb75aadc[https://registry-1.docker.io/v2]
2018-03-09 15:01:47.461696 I | clair: Pushing Layer 8/11 [702a418e95b6]
2018-03-09 15:01:47.461837 D | clair: Saving 702a418e95b6793f8e37ba6ac09ed110df4cd83fd2b61d6ae5ff18ed72ba7990[https://registry-1.docker.io/v2]
2018-03-09 15:01:47.466871 I | clair: Pushing Layer 9/11 [6d5e09df55e7]
2018-03-09 15:01:47.467038 D | clair: Saving 6d5e09df55e7ce71e3811be0a43cc8a8b4269caf01802dfc0404f6bc6fcb53b0[https://registry-1.docker.io/v2]
2018-03-09 15:01:47.476673 I | clair: Pushing Layer 10/11 [66d2113695b9]
2018-03-09 15:01:47.476837 D | clair: Saving 66d2113695b97bc90b91abd4645d81cbf25258b3c7391727366d0aa1986a9b88[https://registry-1.docker.io/v2]
2018-03-09 15:01:47.481223 I | clair: Pushing Layer 11/11 [bb4732f083a2]
2018-03-09 15:01:47.481363 D | clair: Saving bb4732f083a2aa5cd1ad07f3427fbdf201ff0f496ed0f04a5f1914a49f675f4c[https://registry-1.docker.io/v2]
2018-03-09 15:01:47.488110 I | config: retrieving interface for local IP
2018-03-09 15:01:47.488146 D | config: no interface provided, looking for docker0
2018-03-09 15:01:47.488585 I | clair: using http://X.X.X.X/local as local url
2018-03-09 15:01:47.652981 I | clair: analysing layer [bb4732f083a2] 1/11
2018-03-09 15:01:47.680441 I | clair: analysing layer [66d2113695b9] 2/11
2018-03-09 15:01:47.701403 I | clair: analysing layer [6d5e09df55e7] 3/11
2018-03-09 15:01:47.720298 I | clair: analysing layer [702a418e95b6] 4/11
2018-03-09 15:01:47.737839 I | clair: analysing layer [169b8319d348] 5/11
2018-03-09 15:01:47.756311 I | clair: analysing layer [ad2ae30a3f16] 6/11
2018-03-09 15:01:47.773774 I | clair: analysing layer [723b01ba3a25] 7/11
2018-03-09 15:01:47.792441 I | clair: analysing layer [35e854e2bf81] 8/11
2018-03-09 15:01:47.809847 I | clair: analysing layer [a43065cd7262] 9/11
2018-03-09 15:01:47.834309 I | clair: analysing layer [002fafa83f3b] 10/11
2018-03-09 15:01:47.851754 I | clair: analysing layer [17efd5fdbef8] 11/11
2018-03-09 15:01:47.852069 D | cmd: Using priority filters:

and here the logs from a scan without errors

2018-03-09 14:41:39.643721 D | config: No config file used
2018-03-09 14:41:39.647539 D | dockercli: docker image to save: animage
2018-03-09 14:41:39.647612 D | dockercli: saving in: /tmp/clairctl/animage
2018-03-09 14:42:19.298488 I | config: retrieving interface for local IP
2018-03-09 14:42:19.301301 D | config: no interface provided, looking for docker0
2018-03-09 14:42:19.312669 I | config: retrieving interface for local IP
2018-03-09 14:42:19.312767 D | config: no interface provided, looking for docker0
2018-03-09 14:42:19.316913 D | server: Update local server port from "0" to "53364"
2018-03-09 14:42:19.316970 I | server: Starting Server on X.X.X.X
2018-03-09 14:42:19.318944 I | clair: using http://X.X.X.X/local as local url
2018-03-09 14:42:19.319343 I | clair: Pushing Layer 1/11 [894c1784ca81]
2018-03-09 14:42:19.321202 D | clair: Saving 894c1784ca81caa3fd379b360c3194cef4043aeb8122d1cd2fc4d3736d43f825[https://registry-1.docker.io/v2]
2018-03-09 14:42:19.419638 I | clair: adding layer 1/11 [894c1784ca81]: receiving http error: 400
client quit unexpectedly
2018-03-09 14:42:19.419861 C | cmd: pushing image "canimage": receiving http error: 400

here my docker compose file :

version: '2.1'
services:
postgres:
image: postgres:9.6
restart: unless-stopped
volumes:
- ./docker-compose-data/postgres-data/:/var/lib/postgresql/data:rw
environment:
- POSTGRES_PASSWORD=ChangeMe
- POSTGRES_USER=clair
- POSTGRES_DB=clair
- http_proxy=myproxy
- https_proxy=myproxy
- no_proxy=/var/run/docker.sock,localhost,127.0.0.1
clair:
group_add:
- 244
image: quay.io/coreos/clair:latest
restart: unless-stopped
environment:
- http_proxy=myproxy
- https_proxy=myproxy
- no_proxy=/var/run/docker.sock,localhost,127.0.0.1
ports:
- "6060:6060"
- "6061:6061"
privileged: true
volumes:
- ./docker-compose-data/clair-config/:/config/:ro
- ./docker-compose-data/clair-tmp/:/tmp/:rw
depends_on:
postgres:
condition: service_started
command: [--log-level=debug, --config, /config/config.yml]
clairctl:
group_add:
- 244
image: jgsqware/clairctl:latest
restart: unless-stopped
environment:
- http_proxy=myproxy
- https_proxy=myproxy
- no_proxy=/var/run/docker.sock,localhost,127.0.0.1
environment:
- DOCKER_API_VERSION=1.3
volumes:
- ./docker-compose-data/clairctl-reports/:/reports/:rw
- /var/run/docker.sock:/var/run/docker.sock:ro
depends_on:
clair:
condition: service_started

I have no errors from the logs of the three containers (postgres clair and clairctl)

@lukeb2e
Copy link

lukeb2e commented Mar 11, 2018

This pull request fixes the issue you are encountering: #112

It is basically the same issue already documented in #110

@lukeb2e lukeb2e mentioned this issue Mar 11, 2018
@dawaj71
Copy link
Author

dawaj71 commented Mar 20, 2018

Hello,
Thanks for your answer, i pulled with this command

git clone -b fix-temp-path https://github.com/glookie1/clairctl.git

and i have the same error :

2018-03-20 09:46:52.472923 D | config: No config file used
2018-03-20 09:46:52.473545 D | dockercli: docker image to save: serverweb:latest
2018-03-20 09:46:52.473584 D | dockercli: saving in: /tmp/clairctl/serverweb/latest/blobs
2018-03-20 09:47:02.196738 I | config: retrieving interface for local IP
2018-03-20 09:47:02.196848 D | config: no interface provided, looking for docker0
2018-03-20 09:47:02.197882 D | server: Update local server port from "0" to "50153"
2018-03-20 09:47:02.197936 I | server: Starting Server on X.X.X.X
2018-03-20 09:47:02.202660 I | config: retrieving interface for local IP
2018-03-20 09:47:02.202709 D | config: no interface provided, looking for docker0
2018-03-20 09:47:02.203060 I | clair: using http://X.X.X.X/local as local url
2018-03-20 09:47:02.203123 I | clair: Pushing Layer 1/2 [313a85ae8f22]
2018-03-20 09:47:02.203412 D | clair: Saving 313a85ae8f22ccda740b507b12894c4695ae3f75d819ad8cc31e33f413c47809[https:///v2]
2018-03-20 09:47:02.212524 I | clair: adding layer 1/2 [313a85ae8f22]: receiving http error: 400
client quit unexpectedly
2018-03-20 09:47:02.213402 C | cmd: pushing image "serverweb:latest": receiving http error: 400

here my dockercli.go files

import (
"compress/bzip2"
"compress/gzip"
"context"
"encoding/json"
"fmt"
"io"
"os"
"strings"
"syscall"
"github.com/artyom/untar"
"github.com/coreos/pkg/capnslog"
"github.com/docker/distribution"
"github.com/docker/distribution/manifest/schema1"
"github.com/docker/docker/client"
"github.com/docker/docker/image"
"github.com/docker/docker/layer"
"github.com/docker/docker/reference"
"github.com/jgsqware/clairctl/config"
"github.com/opencontainers/go-digest"
)
var log = capnslog.NewPackageLogger("github.com/jgsqware/clairctl", "dockercli")
func parseImage(imageName string) (reference.NamedTagged, error) {
n, err := reference.ParseNamed(imageName)
if err != nil {
return nil, err
}
var image reference.NamedTagged
if reference.IsNameOnly(n) {
image = reference.WithDefaultTag(n).(reference.NamedTagged)
} else {
image = n.(reference.NamedTagged)
}
return image, nil
}
func tempImagePath(image reference.NamedTagged) string {
return fmt.Sprintf("%s", image.FullName())
}
//GetLocalManifest retrieve manifest for local image
func GetLocalManifest(imageName string, withExport bool) (reference.NamedTagged, distribution.Manifest, error) {
image, err := parseImage(imageName)
if err != nil {
return nil, nil, err
}
var manifest distribution.Manifest
if withExport {
manifest, err = save(image)
} else {
manifest, err = historyFromCommand(image)
}
if err != nil {
return nil, schema1.SignedManifest{}, err
}
+more lignes

and dockercli_test.go

package dockercli
import (
"testing"
"github.com/docker/docker/reference"
)
func TestImageParsing(t *testing.T) {
images := map[string]string{
"ubuntu:14.04": "docker_io/library/ubuntu/14_04",
"ubuntu/ubuntu:14.04": "docker_io/ubuntu/ubuntu/14_04",
"registry.com/ubuntu:14.04": "registry_com/ubuntu/14_04",
"registry.com/ubuntu/ubuntu:14.04": "registry_com/ubuntu/ubuntu/14_04",
"registry.com:5000/ubuntu:14.04": "registry_com_5000/ubuntu/14_04",
"registry.com:5000/ubuntu/ubuntu:14.04": "registry_com_5000/ubuntu/ubuntu/14_04",
}
for value, expected := range images {
n, err := reference.ParseNamed(value)
if err != nil {
t.Error("Error:", err, expected)
}
var image reference.NamedTagged
if reference.IsNameOnly(n) {
image = reference.WithDefaultTag(n).(reference.NamedTagged)
} else {
image = n.(reference.NamedTagged)
}
result := tempImagePath(image)
if result != expected {
t.Errorf("Expecting %s, got %s", expected, result)
}
}
}

@lukeb2e
Copy link

lukeb2e commented Mar 20, 2018 via email

@dawaj71
Copy link
Author

dawaj71 commented Mar 21, 2018

Thanks for your time.

I don't understand the

you have to copy my source in the original directory because some paths are hardcoded somewhere

You mean that I have to download it and paste it in my directory instead of using git clone ?

PS : I'm not in this directory btw '$GOPATH/src/github.com/jgsqware/clairctl"

PS1; I tried to gitclone the master branch from jsqawe and just replace the 2 files changed in /docker/dockercli and build with dockerfile and it does not work

@lukeb2e
Copy link

lukeb2e commented Mar 21, 2018 via email

@dawaj71
Copy link
Author

dawaj71 commented Mar 21, 2018

Ok so what I did is :

copy the master branch from the jqsware projet and change only the 2 files that are affected by your pull request. Same error ^^

I have time no worries if you don't it is ok :-)

PS: in the docker file i changed ARG CLAIRCTL_VERSION=${CLAIRCTL_VERSION:-master} by ARG CLAIRCTL_VERSION=${CLAIRCTL_VERSION:-fix-temp-master}

PS2:
clair logs :

{"Event":"could not download layer: expected 2XX","Level":"warning","Location":"driver.go:135","Time":"2018-03-22 07:28:02.991983","status code":404}
{"Event":"failed to extract data from path","Level":"error","Location":"worker.go:122","Time":"2018-03-22 07:28:02.992164","error":"could not find layer","layer":"17efd5fdbef8f0b0dcedd31beb409982edfcae8a6a9b9dbc45f7517e129e6497","path":"http://X.X.X.X/local//planningdb/blobs/17efd5fdbef8f0b0dcedd31beb409982edfcae8a6a9b9dbc45f7517e129e6497/layer.tar"}
{"Event":"Handled HTTP request","Level":"info","Location":"router.go:57","Time":"2018-03-22 07:28:02.993259","elapsed time":17228471,"method":"POST","remote addr":"172.19.0.1:54468","request uri":"/v1/layers","status":"400"}

PS3: now it doesnt work for every images not just for some images like before

@NigelWan
Copy link

NigelWan commented Mar 31, 2018

@dawaj7 my issue is the same as you. let me know if you had found out the way to fix. thanks

Update: use v1.2.6 in https://github.com/jgsqware/clairctl/releases temporary fix my problem

@lukeb2e
Copy link

lukeb2e commented Apr 5, 2018

I always have my images tagged with the full name (registry/namespace/name:tag) - it might not work if you only have set a name for your image...

I also always have to push the image first before I can generate a report.

clairctl push -l ${container}
clairctl report --format html -l ${container}

Can you try to tag your image with a domain & namespace as well?

@pengyaoxu
Copy link

pengyaoxu commented Apr 17, 2018

I also meet this problem.My config file below:

version: '2.1'
services:
  postgres:
    container_name: clair_postgres
    image: postgres:9.6
    restart: unless-stopped
    environment:
      POSTGRES_PASSWORD: password

  clair:
    container_name: clair_clair
    image: quay.io/coreos/clair:v2.0.0
    restart: unless-stopped
    depends_on:
      - postgres
    ports:
      - "6060-6061:6060-6061"
    links:
      - postgres
    volumes:
      - /tmp:/tmp
      - ./clair_config:/config
    command: [-config, /config/config.yaml]

When i run clairctl analyze -l --log-level debug centos:

2018-04-17 02:53:43.098375 D | config: No config file used
2018-04-17 02:53:43.098702 D | dockercli: docker image to save: centos:latest
2018-04-17 02:53:43.098721 D | dockercli: saving in: /tmp/clairctl/centos/latest/blobs
2018-04-17 02:53:46.727978 I | config: retrieving interface for local IP
2018-04-17 02:53:46.728022 D | config: no interface provided, looking for docker0
2018-04-17 02:53:46.728987 D | server: Update local server port from "0" to "40347"
2018-04-17 02:53:46.729003 I | server: Starting Server on 172.17.0.1:40347
2018-04-17 02:53:46.734048 I | config: retrieving interface for local IP
2018-04-17 02:53:46.734090 D | config: no interface provided, looking for docker0
2018-04-17 02:53:46.734512 I | clair: using http://172.17.0.1:40347/local as local url
2018-04-17 02:53:46.734552 I | clair: Pushing Layer 1/1 [6ce355201742]
2018-04-17 02:53:46.734756 D | clair: Saving 6ce35520174293337cf4fa0039f04e5cecfcaceb5b968d89fcc9c6fd7b952d97[https:///v2]
2018-04-17 02:53:46.742209 I | clair: adding layer 1/1 [6ce355201742]: receiving http error: 400
client quit unexpectedly
2018-04-17 02:53:46.742254 C | cmd: pushing image "centos:latest": receiving http error: 400

Is that anything wrong with my config file?

@lukeb2e
Copy link

lukeb2e commented Apr 21, 2018

Did you push your image before you triggered the analyze?

clairctl push -l ${container}

@djl197
Copy link

djl197 commented May 15, 2018

I too am having similar issues.
I am analysing Centos based containers and pushing 'local' images into clair works for some layers but for others I get the same messages as above.
I have tried 1.2.8 and the 'fix-temp-path' branch but with no difference in behaviour

@pengyaoxu
Copy link

Now is ok with my clairctl.I pull the latest clairctl resource code and compile.It works.

@animale66
Copy link

animale66 commented Jun 14, 2018

I am also getting HTTP 400s while doing clairctl push or clairctl analyze with images that are publically available on dockerhub. The issue appears to me that clairctl is not honoring http_proxy or https_proxy variables (and likely, no_proxy too).

Clairctl version 1.2.8

None of the machines (either the clair server, which I run as a containerized instance), nor the clairctl client have direct non-proxy access to the real-world. I am able to run clairctl pull without any problems, just push and analyze don't work.

My ~/clairctl.yaml contains the following:

clair:
port: 6060
healthPort: 6061
uri: http://127.0.0.1
report:
path: /opt/data/httpd/htdocs/clair_reports/
format: html

luser@bigbox ~ $ http_proxy=http://myproxyserver.domain.net:8080 https_proxy=http://myproxyserver.domain.net:8080 clairctl push rancher/dns:v0.15.3

2018-06-14 18:38:20.612975 E | clair: response error: Head https://registry-1.docker.io/v2/rancher/dns/blobs/sha256:b3e1c725a85f0953e81815b7c7aabfad9ebfd90af53f99248981282b8045d787: dial tcp 34.200.28.105:443: getsockopt: connection refused

client quit unexpectedly

2018-06-14 18:38:20.614321 C | cmd: pushing image "rancher/dns:v0.15.3": receiving http error: 400

Clair has it both as environment variables passed via launch, as well as in it's config.yaml, so if I'm missing it somewhere I have no idea where I'm supposed to set it.

/usr/bin/docker run --net bridge -m 0b --net=host --userns=host -e http_proxy=http://myproxyserver.domain.net:8080 -e https_proxy=http://myproxyserver.domain.net:8080 -e no_proxy=localhost,*.domain.net -p 6060-6061:6060-6061 -v /opt/data/clair_config:/config \ --name clair \ clair

And the tail part of my clair-config lists the proxy server:

{stock stuff at the top}

notifier:
# Number of attempts before the notification is marked as failed to be sent
attempts: 3

# Duration before a failed notification is retried
renotifyinterval: 2h

http:
  # Optional endpoint that will receive notifications via POST requests
  endpoint:

  # Optional PKI configuration
  # If you want to easily generate client certificates and CAs, try the following projects:
  # https://github.com/cloudflare/cfssl
  # https://github.com/coreos/etcd-ca
  servername:
  cafile:
  keyfile:
  certfile:

  # Optional HTTP Proxy: must be a valid URL (including the scheme).
  proxy: http://myproxyserver.domain.net:8080

I feel this error is with the clairctl client, since I'm able to pull vulnerability data from the Internet via the proxy server.

@chirangaalwis
Copy link

I am also facing a similar issue. Also explained in #74.

Is this fixed yet? @jgsqware is Clairctl deprecated as I find no development or support for existing issues in recent times?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

7 participants