Allow localhost repositories

[coryrc]

crane ls localhost:5000/something
worked for me after this change. Also a crane cp.

Fixes #840

[coryrc]

@hasheddan was pinged in 840 so pinging him here

[jonjohnsonjr]

This scares me a little bit but hopefully our tests are sufficient to make this change safely @dekkagaijin

[jonjohnsonjr]

Okay excellent, the tests did fail.

[coryrc]

-- FAIL: TestRawManifestDigests (0.02s)
    --- FAIL: TestRawManifestDigests/normal_pull,_by_digest (0.00s)
        image_test.go:166: Unexpected path: /v2/foo/bar:sha256/manifests/b1f8e8e133091bc49ac82bc634314b6ea0883f9d21da0a2bf1d80b31cc6753e4
        image_test.go:189: RawManifest() wrong error: true, want false: Get "http://127.0.0.1:32919/v2/foo/bar:sha256/manifests/b1f8e8e133091bc49ac82bc634314b6ea0883f9d21da0a2bf1d80b31cc6753e4": EOF

So just taking the first error above, when I run it w/o my change:
The "Unexpected Path" is /v2/foo/bar/manifests/sha256:082e0f844d18b00750c38f2d1203dfbdb435eb91a2d92d43ee56f83fa 6ccb739

so "sha256:" is inserted in the wrong place.

[hasheddan]

We want the registry to allow : not the repository right?

[coryrc]

Registry delegates the decision making on validity of the "repository" to Repository

[jonjohnsonjr]

crane ls localhost:5000/something

This works already, but:

crane ls localhost:5000

does not, right?

[coryrc]

crane ls localhost:5000/something

This works already, but:

crane ls localhost:5000

does not, right?

Correct, but neither does crane ls gcr.io

[coryrc]

There's zero authoritative way to decide if crane ls ubuntu:5000 means access port 5000 of host ubuntu or list tag 5000 of the ubuntu image on docker.

I could make a special case for localhost?

[jonjohnsonjr]

I don't think that's actually valid, per the distribution spec. Do other tools work with this?

[coryrc]

I don't think that's actually valid, per the distribution spec. Do other tools work with this?

I'm confused what you mean? I think I should be able to say gcr.io:5000/blah just as acceptably as gcr.io/blah but I'm just making that up on my own.

[coryrc]

The spec only seems to care about the HTTP request and not what we do on the command line, but I'm probably misunderstanding your statement.

[jonjohnsonjr]

So we need to translate:

crane ls localhost:5000

Into some http request to list tags (or anything). That http request looks like this:

GET /v2/<name>/tags/list

And <name> is defined as this regex:

[a-z0-9]+([._-][a-z0-9]+)*(/[a-z0-9]+([._-][a-z0-9]+)*)*

I guess I'm wondering if you can just do something like:

docker push localhost:5000:v3

And have that work? That would be somewhat equivalent to crane ls localhost:5000.

In theory, you may want to pull the tags that come out of that command, but how?

docker run localhost:5000:v3

Does this work for anything?

[jonjohnsonjr]

Looks like not:

$ docker tag ubuntu localhost:1338:latest
Error parsing reference: "localhost:1338:latest" is not a valid repository/tag: invalid reference format

$ docker push localhost:1338
The push refers to repository [docker.io/library/localhost]
346be19f13b0: Preparing 

[coryrc]

docker run localhost:5000:v3 doesn't make any sense to me; it'd be equivalent to gcr.io:v3.

crane catalog localhost:5000 works though

[coryrc]

(btw I'm still tracking down the bug and adding in some unit tests, I think I've isolated the problem to name.ParseReference())

[jonjohnsonjr]

I think I understand the confusion -- this isn't about listing tags, you're expecting crane ls to also list repositories, similarly to how gcrane ls works (or even normal ls).

We could print a warning like:

fmt.Printf("did you mean: crane catalog %s", src)

Or even just default to running crane catalog if we can't parse the input as a repository but can parse the input as a registry?

[jonjohnsonjr]

I've also run into a similar situation where I do something like:

crane ls example.com/repo:tag

Where I obviously mean to do:

crane ls example.com/repo

But I'm not sure how many of these "do what I mean" niceties we could add before things get unwieldy...

[coryrc]

I've also run into a similar situation where I do something like:

crane ls example.com/repo:tag

Where I obviously mean to do:

crane ls example.com/repo

But I'm not sure how many of these "do what I mean" niceties we could add before things get unwieldy...

I assumed crane ls example.com/repo:tag works because you could use it as a "exists" check; I wouldn't want to change it.

[coryrc]

I think I understand the confusion -- this isn't about listing tags, you're expecting crane ls to also list repositories, similarly to how gcrane ls works (or even normal ls).

We could print a warning like:

fmt.Printf("did you mean: crane catalog %s", src)

Or even just default to running crane catalog if we can't parse the input as a repository but can parse the input as a registry?

I'm really just trying to use crane as a library and copy from a registry server on localhost:5000, I was just demonstrating I did at least some basic manual tests before submitting :). I don't have feelings one way or the other on how the CLI works. EDIT: I mean, I always got opinions, I just don't care much which way the CLI ends up :)

[jonjohnsonjr]

I assumed crane ls example.com/repo:tag works because you could use it as a "exists" check; I wouldn't want to change it.

It actually fails to parse:

$ crane ls example.com/repo:tag
2021/04/08 12:52:22 reading tags for example.com/repo:tag: parsing repo "example.com/repo:tag": repository can only contain the runes `abcdefghijklmnopqrstuvwxyz0123456789_-./`: repo:tag

You'd want to use crane manifest or crane digest for an existence check.

I'm really just trying to use crane as a library and copy from a registry server on localhost:5000

So unfortunately there's not really a standard way to enumerate repositories and images in a registry. Catalog + tag listing is one way that works for some registries, but only works for tags. GCR does its own thing, which gcrane exposes.

I've tried to push on a standard for this, but it seems like a lost cause: opencontainers/distribution-spec#222 (comment)

I think what you want to do, for copying from some registry that supports catalog (don't do this for GCR), is roughly this:

srcRegistry := "localhost:5000"
dstRegistry := "localhost:5001"

catalog, err := crane.Catalog(srcRegistry)
if err != nil {
  return err
}

for _, repo := range catalog {
  tags, err := crane.ListTags(repo)
  if err != nil {
     return err
  }
  for _, tag := range tags {
    src := fmt.Sprintf("%s/%s:%s", srcRegistry, repo, tag)
    dst := fmt.Sprintf("%s/%s:%s", dstRegistry, repo, tag)
    if err := crane.Copy(src, dst); err != nil {
      return err
    }
  }
}

The crane.Catalog response may or may not include the hostname, so you might have to munge some strings a bit...

This is the best you can do in a "generic" way, unfortunately, and a lot of registries don't support catalog.

[coryrc]

Oh, I did get confused and the copy worked and when I went to verify with the CLI it broke and I wandered off without realizing the copy was successful. D'oh! I can still finish this though, I think I've found the bug and am building up the test cases to prove it.

[jonjohnsonjr]

D'oh! I can still finish this though, I think I've found the bug and am building up the test cases to prove it.

I'm not entirely sure if it makes sense. In my mind, a registry is just a hostname, and a repository includes a path component. This is an important distinction to make because listing tags requires a path component, and calling catalog forbids one.

I think this already works, as intended, and the linked bug doesn't reproduce?

[coryrc]

...oh

Well that's what I get for not updating my crane installation for a year or more. Yes, it's fine :).

I think there's missing test cases in tag_test.go:

Shouldn't we be checking gcr.io/g-convoy/hello-world:sha256:a428de44a9059f31a59237a5881c2d2cffa93757d99026156e4ea544577ab7f3 as a valid reference?

EDIT: d'oh, then it's handled by the digest code

[coryrc]

(I still think there's a different subtle bug I'm working on fixing re: that test case, unless I misunderstood the format) EDIT: I did misunderstand tag vs digest handling code.

[jonjohnsonjr]

I still think there's a different subtle bug I'm working on fixing re: that test case, unless I misunderstood the format

FWIW you are probably right, and I'd like to rework this package entirely, but haven't had the time... though as long as you're not blocked, I'm happy.