Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Exclude .git directory content when calculating package file count #525

Merged
merged 4 commits into from
Mar 18, 2024

Conversation

qtomlinson
Copy link
Collaborator

@qtomlinson qtomlinson commented Sep 28, 2023

This was discovered during sanity test for upgrading crawler image to use node:18-bullseye. The file count for pod/cocoapods/-/SoftButton/0.1.0 was different:

  • 78 in the node:16 based image
  • 80 in the node:18-bullseye based image.
  • The difference are due to the files in .git directory: .git/hooks/pre-merge-commit.sample, and .git/hooks/push-to-checkout.sample.

The content of .git directory should be excluded for the package file count.

Task: #529

@qtomlinson qtomlinson marked this pull request as ready for review September 28, 2023 23:08
@qtomlinson
Copy link
Collaborator Author

@mpcen ready for review

@qtomlinson qtomlinson marked this pull request as draft October 31, 2023 17:55
@qtomlinson qtomlinson marked this pull request as ready for review November 1, 2023 00:26
@qtomlinson
Copy link
Collaborator Author

qtomlinson commented Nov 1, 2023

@jeffwilcox @mpcen rebased and ready for review.

@qtomlinson
Copy link
Collaborator Author

@elrayle ready for review

@qtomlinson
Copy link
Collaborator Author

The existing code only exclude .git directory itself, but not its containing files. This fix intends to exclude .git directory and its contents.

@qtomlinson qtomlinson requested a review from elrayle January 3, 2024 23:25
Copy link
Collaborator

@elrayle elrayle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like where you are going with this, especially pulling out the validation into a separate function. I have a question about the expectation of the validation and whether or not the use of intersection could have false negatives.

providers/process/abstractProcessor.js Outdated Show resolved Hide resolved
@qtomlinson qtomlinson requested a review from elrayle January 31, 2024 20:36
@qtomlinson
Copy link
Collaborator Author

@lumaxis @elrayle ready for review

lib/utils.js Outdated Show resolved Hide resolved
lib/utils.js Outdated Show resolved Hide resolved
Copy link
Collaborator

@elrayle elrayle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good to see this bug fixed. Really nice to see a function created that can be called from multiple places resulting in the same behavior.

@@ -150,7 +150,7 @@ class AbstractProcessor extends BaseHandler {
*/
async filterFiles(location) {
const fullList = await this.getFiles(location)
const filteredList = fullList.filter(file => isValidExcludingGit(file))
const filteredList = fullList.filter(file => file && !isGitFile(file))
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't necessarily need to check if file is undefined here since isGitFile already does: if (!file) return false

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm... I am afraid that the check for undefined file is still necessary. In case of falsy file, isGitFile(file) returns false, and !isGitFile returns true. Using !isGitFile alone would not exclude falsy file values.

@lumaxis lumaxis merged commit 8a04999 into clearlydefined:master Mar 18, 2024
1 check passed
@qtomlinson qtomlinson deleted the qt/exclude-git-dir branch April 10, 2024 17:35
qtomlinson added a commit to qtomlinson/crawler that referenced this pull request Apr 10, 2024
This is for fix to exclude .git directory content in recent PR (clearlydefined#525).
Bump up the version to allow reharvest of pod components.
qtomlinson added a commit to qtomlinson/crawler that referenced this pull request Apr 17, 2024
The recent fix to exclude content in the .git directory (clearlydefined#525) from pod packages will cause the file count to be different from the previous version. Update the toolVersion for PodExtract to 2.0.0 to reflect this.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants