Unable to parse Devfile with parent on GitHub: url path to directory or file should contain 'tree' or 'blob' if parent is a GitHub release artifact, or failed to switch repo to revision if parent is a blob or raw URL

[rm3l]

Which area this feature is related to?

/kind bug

Which area this bug is related to?

/area library

What versions of software are you using?

Go project

Operating System and version:
Fedora 38

Go Pkg Version:
Latest commit on the main branch: devfile/library@f041d79

Bug Summary

Describe the bug:

In redhat-developer/odo#6768, we have updated the Devfile library to the latest commit. And now, some of our tests are failing due to Devfile parsing issues.

To Reproduce:
I've narrowed down this issue to the following reproduction steps:

1. Sample Devfile

$ cat << EOF > /tmp/devfile-issue-with-parent-uri.yaml
schemaVersion: 2.1.0
metadata:
  name: new-ol-version
  description: Child devfile to test OL image updating
parent:
  uri: https://github.com/OpenLiberty/application-stack/releases/download/maven-0.7.0/devfile.yaml
  components:
    - name: dev
      container:
        image: maven:3.6-adoptopenjdk-11-openj9
EOF

2. Parser is now failing (I'm running this from the library repo: https://github.com/devfile/library/blob/main/main.go)

$ go run .  /tmp/devfile-issue-with-parent-uri.yaml        

parsing devfile from /tmp/devfile-issue-with-parent-uri.yaml
failed to populateAndParseDevfile: url path to directory or file should contain 'tree' or 'blob'

NOTES:

• I've tried to replace the parent URL with the blob one (https://github.com/OpenLiberty/devfile-stack/blob/maven-0.7.0/templates/devfiles/maven/devfile.yaml), but the parser still fails:

$ go run .  /tmp/devfile-issue-with-parent-uri.yaml
parsing devfile from /tmp/devfile-issue-with-parent-uri.yaml
failed to populateAndParseDevfile: 1 error occurred:
        * failed to switch repo to revision. repo dir: /tmp/git-resources2839940370, revision: maven-0.7.0

• I've also tried to replace the parent URL with the raw one (https://raw.githubusercontent.com/OpenLiberty/devfile-stack/maven-0.7.0/templates/devfiles/maven/devfile.yaml), but the parser still fails:

$ go run .  /tmp/devfile-issue-with-parent-uri.yaml
parsing devfile from /tmp/devfile-issue-with-parent-uri.yaml
failed to populateAndParseDevfile: 1 error occurred:
        * failed to switch repo to revision. repo dir: /tmp/git-resources3340282075, revision: maven-0.7.0

Expected behavior

• Is it intentional that we cannot reference a parent from a GitHub release artifact?
• I guess however that using the raw or blob URLs should have worked correctly.

Suggestion on how to fix the bug

[mike-hoang]

It seems like using git switch is a little too limiting in that it only works well when switching branches. Will open a PR to change to using git checkout instead.

devfile/library#173

[rm3l]

It seems like using git switch is a little too limiting in that it only works well when switching branches. Will open a PR to change to using git checkout instead.

devfile/library#173

Thanks, @mike-hoang !!
Indeed, git switch expects a branch, but I think it can also work by detaching the working tree at the tag/branch commit; so a command like git switch --detach <branch_or_tag> should also work.
From your diff in https://github.com/devfile/library/pull/173/files, I think the issue is because of the origin/ prefix. There is only one remote after the repo was cloned, so Git should find the right remote branch/tag.

Another point about error handling: would it be possible to wrap the original error of the git commands executed by the parser, to help debug such errors in the future?

[rm3l]

Is it intentional that we cannot reference a parent from a GitHub release artifact?

Back to the original issue, the Devfile in our tests contains a parent referenced via a GitHub release artifact URL.
Is this not supposed to work anymore?

[mike-hoang]

so a command like git switch --detach <branch_or_tag> should also work.

When testing, I ran into some issues where checkout would still work:

➜  devfile-stack git:(fcee161) git switch --detach master
fatal: '--detach' cannot be used with '-b/-B/--orphan'

➜  devfile-stack git:(fcee161) git checkout master
Previous HEAD position was fcee161 Merge pull request #166 from mezarin/portDevfileRegDevfileUpdatesToStack
branch 'master' set up to track 'origin/master'.

So I think checkout would be a better option here.

Another point about error handling: would it be possible to wrap the original error of the git commands executed by the parser, to help debug such errors in the future?

Yup I can wrap the git error

Back to the original issue, the Devfile in our tests contains a parent referenced via a GitHub release artifact URL.
Is this not supposed to work anymore?

Yes, with the recent changes this will not work anymore. I wasn't aware that release artifact urls were being used, maybe this can be raised in the community call?

[rm3l]

Back to the original issue, the Devfile in our tests contains a parent referenced via a GitHub release artifact URL.
Is this not supposed to work anymore?

Yes, with the recent changes this will not work anymore. I wasn't aware that release artifact urls were being used, maybe this can be raised in the community call?

Yes, why not! I'll add it to the agenda.

I wasn't aware of that fact either, until I stumbled upon this in our tests :-)

At first, I thought we were not using the uri field correctly, but the spec and doc say:

So, unless I'm missing something from the docs/spec, it looks like it should be possible to use any hosted devfile.yaml as parent; and if the URI points to a supported Git repo, the library will clone it.

I tried to workaround this issue by serving the Devfile from a test HTTP server, but the parser was expecting a GitHub, Gitlab or BitBucket host.

# With this URI: http://localhost:5000/devfile.yaml

$ go run .  /tmp/devfile-issue-with-parent-uri.yaml 
parsing devfile from /tmp/devfile-issue-with-parent-uri.yaml
failed to populateAndParseDevfile: url host should be a valid GitHub, GitLab, or Bitbucket host; received: localhost:5000

[mike-hoang]

So, unless I'm missing something from the docs/spec, it looks like it should be possible to use any hosted devfile.yaml as parent; and if the URI points to a supported Git repo, the library will clone it.

Yes, this should be the correct behaviour 😄
Updated the pr to clone only when it is a git repo.

[mike-hoang]

@rm3l pr has been merged now which should allow the blob and raw urls to work. Not too sure what the consensus was about artifact links, but we can continue that discussion in the community calls

Let me know if you run into more issues!

[rm3l]

@rm3l pr has been merged now which should allow the blob and raw urls to work. Not too sure what the consensus was about artifact links, but we can continue that discussion in the community calls

Let me know if you run into more issues!

Thanks @mike-hoang !! I've just tested that your changes work with blob and raw URLs.
For artifact links, we thought it would make sense to also handle them as they are downloadable too. But for now, I think it is okay for us (until we hear someone complain about this ^^).
I'm updating odo to use the latest changes in the library.
Thanks again.