Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add options to fast-filedeps to bring it close to parity with V1 filedeps #8184

Merged
merged 5 commits into from
Nov 8, 2019

Conversation

Eric-Arellano
Copy link
Contributor

@Eric-Arellano Eric-Arellano commented Aug 18, 2019

Problem

We want to finish replacing V1 filedeps with V2 fast-filedeps. Keeping both goals around gives us added complexity, e.g. more time in CI for tests.

To finish the replacement, fast-filedeps must first get support for --absolute and --globs.

Solution

Add support for --absolute and --globs.

Also, fix and refactor the unit tests, which were skipped in #6880 and never restored. Update the integration test to mirror the V1 test.

We can't yet actually drop the V1 goal yet due to two features that need to be implemented for full feature parity: 1) handling scala_library's java_sources and 2) supporting jvm_apps. This PR adds failing unit tests for both those cases. Once those tests can be un-skipped, then we can drop V1.

@Eric-Arellano Eric-Arellano changed the title Add --absolute to V2 filedeps goal WIP: Finish replacing filedeps with V2 rule Aug 20, 2019
Copy link
Contributor Author

@Eric-Arellano Eric-Arellano left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was able to finish the port thanks to @stuhood's suggestion!

However, I'm not confident merging this until we can confirm that both of these special JVM related cases are handled:

# TODO(John Sirois): This hacks around ScalaLibraries' psuedo-deps on JavaLibraries. We've
# already tried to tuck away this hack by subclassing closure() in ScalaLibrary - but in this
# case that's not enough when a ScalaLibrary with java_sources is an interior node of the
# active context graph. This awkwardness should be eliminated when ScalaLibrary can point
# to a java source set as part of its 1st class sources.
if isinstance(concrete_target, ScalaLibrary):
concrete_targets.update(concrete_target.java_sources)

# TODO(John Sirois): BundlePayload should expose its sources in a way uniform to
# SourcesPayload to allow this special-casing to go away.
if isinstance(target, JvmApp) and not output_globs:
for bundle in target.bundles:
files.update(
bundle.filemap.keys() if self.get_options().absolute else bundle.relative_filemap.keys()
)

I need to figure out how to add tests for these situations.

examples/src/resources/org/pantsbuild/example/hello/world.txt
''')
)
def test_filedeps_globs(self):
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be nice to update examples to have at least two files of the same extension (like .scala) in one directory to test that --globe option works for more than one file.

@cattibrie
Copy link
Contributor

@Eric-Arellano Thanks for this PR! Really like type annotations and how tests look now!

@Eric-Arellano Eric-Arellano force-pushed the filedeps branch 2 times, most recently from 44f38b2 to aa4fcaa Compare November 7, 2019 16:52
Copy link
Member

@stuhood stuhood left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!



@console_rule
def file_deps(
console: Console,
filedeps_options: Filedeps.Options,
build_root: BuildRoot,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So... one development since this was opened is the Workspace type.

I think that the semantics of the BuildRoot type (exposes some information) are sufficiently different from the Workspace type (implies mutable access) that having them separate could be reasonable. I could also see merging them, and exposing Workspace.root or something. "Relatively small single purpose things to DI" is a thing... but the BuildRoot is almost too specific a thing, and would always have a subset of the fields of Workspace.

Fine either way though.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@stuhood stuhood requested a review from gshuflin November 7, 2019 20:45
@Eric-Arellano Eric-Arellano changed the title WIP: Finish replacing filedeps with V2 rule Add options to fast-filedeps to bring it close to parity with V1 filedeps Nov 7, 2019
@Eric-Arellano
Copy link
Contributor Author

@stuhood could you please re-review? We can't drop V1 yet because we don't properly handle two JVM related use cases :/ I updated the PR description and added skipped failing unit tests.

This does at least bring us much closer to dropping the V1 goal!

@Eric-Arellano Eric-Arellano merged commit a3b4b34 into pantsbuild:master Nov 8, 2019
@Eric-Arellano Eric-Arellano deleted the filedeps branch November 8, 2019 13:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants