Fix gradle file fetcher when included build is in submodule

[patari]

If gradle included build is in submodule, dependabot gives error dependency_file_not_found from request to https://api.github.com:443/repos/<owner>/<repo>/contents/<submodule_path>?ref=<ref>.

This fixes it by ignoring 404 errors from included build files (gradle subprojects already has similar ignore).

[jeffwidman]

👋 Thanks for putting this up.

This fixes it by ignoring 404 errors from included build files (gradle subprojects already has similar ignore).

Can you link to the code containing this ignore or explain this more? Do you mean upstream native Gradle java code already has this, or that Dependabot already has a similar ignore in a different section of our codebase?

[patari]

wave Thanks for putting this up.

This fixes it by ignoring 404 errors from included build files (gradle subprojects already has similar ignore).

Can you link to the code containing this ignore or explain this more? Do you mean upstream native Gradle java code already has this, or that Dependabot already has a similar ignore in a different section of our codebase?

dependabot-core/gradle/lib/dependabot/gradle/file_fetcher.rb

Lines 83 to 85 in d8840c0

	rescue Dependabot::DependencyFileNotFound
	# Gradle itself doesn't worry about missing subprojects, so we don't
	nil

[jeffwidman]

Thanks!

I am not familiar enough with Gradle to even know the difference between Gradle subprojects vs submodules so I'm not able to evaluate this properly w/o some further digging. So gonna "phone-a-friend(ly-contributor)"... 😄

@soulus13 @yeikel I don't recall if you both are primarily Maven users or also familiar with Gradle... your thoughts on this?

[patari]

Yeah, gradle seems to have many ways to define dependencies. There is also something similar in Dependabot for gradle dependency script plugins:

dependabot-core/gradle/lib/dependabot/gradle/file_fetcher.rb

Lines 113 to 117 in d8840c0

	rescue Dependabot::DependencyFileNotFound
	next nil if file_exists_in_submodule?(path)
	next nil if path.include?("${")
	raise

This is just a quick fix to make Dependabot work with gradle included builds in git submodules. This might require some refactoring later but maybe wait for #1164?

[jeffwidman]

Just realized this is going to catch all failures to find a file, which seems overly broad.

Is there a way to definitively identify that it's in a git submodule? (since it sounds like Gradle Submodules refer to files in git submodules?)

You pointed out the file_exists_in_submodule? method, could we just use that here too?

dependabot-core/gradle/lib/dependabot/gradle/file_fetcher.rb

Lines 130 to 135 in 4cab0ee

	def file_exists_in_submodule?(path)
	fetch_file_from_host(path, fetch_submodules: true)
	true
	rescue Dependabot::DependencyFileNotFound
	false
	end

Ie,

next nil if file_exists_in_submodule?(path)

[soulus13]

I'm more of a Maven guy, but indeed, this solution sounds more logical and aligned with the existing implementation, if we only ignore the File Not Found from submodule

[patari]

Fixed!

[jakecoffman]

For a submodule, GitHub doesn't return a 404, but instead a 200 with "type": "submodule". Are you seeing a 404 in the logs? That might be more of an indicator that the included build directory doesn't exist at all.

I think for submodules the reason it's failing is we don't pass fetch_sumodules: true into fetch_file_if_present call in the FileFetcher. I assume that is because we won't be updating the dependencies in those build files, so they shouldn't affect the dependencies of the local files? Not sure if that is a correct assumption.

So I think I'm with @jeffwidman, it would be good to check if it's caused by a submodule vs a completely missing directory.

[yeikel]

Another reason why a 404 might be returned can be due to permission denied(401) error. For some reason, the Github API was designed to behave that way

How do dependabot token work when they need to cross the boundaries of a repository?

[patari]

For a submodule, GitHub doesn't return a 404, but instead a 200 with "type": "submodule". Are you seeing a 404 in the logs? That might be more of an indicator that the included build directory doesn't exist at all.

I think for submodules the reason it's failing is we don't pass fetch_sumodules: true into fetch_file_if_present call in the FileFetcher. I assume that is because we won't be updating the dependencies in those build files, so they shouldn't affect the dependencies of the local files? Not sure if that is a correct assumption.

So I think I'm with @jeffwidman, it would be good to check if it's caused by a submodule vs a completely missing directory.

The 404 comes from request to https://api.github.com:443/repos/<owner>/<repo>/contents/<submodule>?ref=<ref> (file_fetchers/base#repo_contents). If we pass fetch_submodules: true Dependabot finds all files from the git submodule and starts to find updates for those files also.

I added a commit to ignore included build only if its in a git submodule.

[jakecoffman]

I've setup what I think the situation is here: https://github.com/dsp-testing/gradle-simple/tree/f43d406d67a96305fe5f0adce7a104d541ff5acd

If I dry-run this with the CLI dependabot update gradle dsp-testing/gradle-simple I get:

  proxy | 2023/04/21 12:49:59 [016] GET https://api.github.com:443/repos/dsp-testing/gradle-simple/contents/include?ref=f43d406d67a96305fe5f0adce7a104d541ff5acd
  proxy | 2023/04/21 12:49:59 [016] * authenticating github api request
  proxy | 2023/04/21 12:49:59 [016] 200 https://api.github.com:443/repos/dsp-testing/gradle-simple/contents/include?ref=f43d406d67a96305fe5f0adce7a104d541ff5acd
updater | 2023/04/21 12:49:59 ERROR Error during file fetching; aborting
  proxy | 2023/04/21 12:49:59 [017] POST http://host.docker.internal:58592/update_jobs/cli/record_update_job_error
  proxy | 2023/04/21 12:49:59 [017] 200 http://host.docker.internal:58592/update_jobs/cli/record_update_job_error
  proxy | 2023/04/21 12:49:59 [018] PATCH http://host.docker.internal:58592/update_jobs/cli/mark_as_processed
  proxy | 2023/04/21 12:49:59 [018] 200 http://host.docker.internal:58592/update_jobs/cli/mark_as_processed
updater | 2023/04/21 12:49:59 INFO Finished job processing
updater | 2023/04/21 12:49:59 INFO Results:
updater | Dependabot encountered '1' error(s) during execution, please check the logs for more details.
updater | +---------------------------+
updater | |          Errors           |
updater | +---------------------------+
updater | | dependency_file_not_found |
updater | +---------------------------+

Hitting https://api.github.com/repos/dsp-testing/gradle-simple/contents/include?ref=f43d406d67a96305fe5f0adce7a104d541ff5acd I get a 200 with "type": "submodule".

404 due to not authorized isn't possible because it's getting the submodule directory from the current project, not trying to get the contents of the submodule from another repo.

If you're getting a 404 still from this I'm interested in what your setup is to reproduce that.

[patari]

Oh yeah, my included build is in subfolder of a git submodule so requests are little different. The 404 comes from request to /repos/<owner>/<repo>/contents/<submodule>/<path>, e.g. https://api.github.com/repos/dsp-testing/gradle-simple/contents/include/path?ref=f43d406d67a96305fe5f0adce7a104d541ff5acd.

I updated spec to match original case (=included build in root of a submodule), the fix works for both cases.

[Thomas-Vos]

Thanks for creating this PR, I am running into the same issue. There is an issue #7201 with some details. I checked the logs and for me it's a 404 error. My included build is also in subfolder of a git submodule, and private as well.

[vitorhugods]

For more people that may find this issue through the PR, there is a work-around suggested by @patari here:

• Dependabot fails when including a git submodule Gradle project using Gradle includeBuild feature #7201 (comment)

[jakecoffman]

This should be fixed with #9278, since we're cloning and fetching submodules up front the file should be present. Let me know if it's working for you without this PR.