Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Skip native composite ReadyToRun images as inputs to Crossgen2 #62537

Merged
merged 3 commits into from
Dec 11, 2021

Conversation

trylek
Copy link
Member

@trylek trylek commented Dec 8, 2021

According to the issue #49247
a known pre-existing Crossgen2 bug is that it fails when presented
with the components of a previous Crossgen2 compilation in the
composite mode. This is because Crossgen2 lacks proper logic to
recognize the composite images and mistakes them for currently
unsupported managed C++ MSIL assemblies. This change adds the extra
check; I have also unified it between Crossgen2 and R2RDump.

Thanks

Tomas

Fixes: #49247
Fixes: #60381

Copy link
Member

@jkoritzinsky jkoritzinsky left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This test uses crossgen2 directly in its custom scripting. I think we should mark this with CrossgenIncompatible in the project file.

Copy link
Member

@jkoritzinsky jkoritzinsky left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ignore my last comment. We don't have a CrossgenIncompatible property similar to GCStressIncompatible. Using issues.targets is the right thing here.

@trylek
Copy link
Member Author

trylek commented Dec 8, 2021

@jkoritzinsky - well, we do have the property CrossGenTest, e.g. here:

I do see that the test uses explicit CG2 invocation and I believe it should be trivial to convert it to use the standard test scripting logic akin to what we do e.g. in the test

https://github.com/dotnet/runtime/blob/main/src/tests/readytorun/crossgen2/crossgen2smoke.csproj

I still however find it weird how come the test only seems to fail in CG2 runs - or are you aware of instances where it fails in JITted outerloop runs?

Thanks

Tomas

@jkoritzinsky
Copy link
Member

That's the name... I didn't realize it didn't follow the pattern of the rest. Either using that property or following a similar mechanism as the smoke test would be something I'd agree with.

@trylek
Copy link
Member Author

trylek commented Dec 8, 2021

Thanks Jeremy for your feedback. I have converted the test to our standard infra logic for running tests with forced CG2 execution. According to my local testing the test seems to be passing so for now I removed the issues.targets entry as I suspect the apparent failures may have been unintended side effects of the custom CG2 execution logic. For now I plan to close the issue assuming the PR test run finishes fine; we will reopen the issue if the test turns out to still fail in some situations.

Copy link
Member

@jkoritzinsky jkoritzinsky left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think your change has the same semantics as the old scripts. Can you take another look?

@trylek
Copy link
Member Author

trylek commented Dec 8, 2021

Thanks Jeremy, I have fixed the execution argument --runCustomTest, are you aware of any additional differences between the current test behavior and the simplified version using the stock CG2 compilation scripting?

@jkoritzinsky
Copy link
Member

jkoritzinsky commented Dec 8, 2021

Yes, it looks like the test wants both the non-R2R and the R2R versions of the DLL. It wants the R2R one in a separate R2R subfolder.

Additionally, it uses custom scripts to actually do the test run: https://github.com/dotnet/runtime/blob/main/src/tests/baseservices/TieredCompilation/RunBasicTestWithMcj.cmd
https://github.com/dotnet/runtime/blob/main/src/tests/baseservices/TieredCompilation/RunBasicTestWithMcj.sh

@trylek
Copy link
Member Author

trylek commented Dec 9, 2021

Thanks Jeremy for pointing that out, I was originally confused by the fact that the custom scripts are named the same as the default scripts generated by CLRTest.Execute.targets. In such case neither of the options I proposed is the right fix for the issue, which is in fact

#49247

When the test runs in a Crossgen2 composite run, the custom tooling for running Crossgen2 actually causes the compiler to be run twice and a known deficiency makes the second compilation fail because it cannot make sense of the composite image produced by the prior compilation. I'll update this PR by removing changes to the scripting and by fixing the Crossgen2 issue. The scripting can be also cleaned up but that's orthogonal to the root issue.

Thanks

Tomas

According to the issue dotnet#49247
a known pre-existing Crossgen2 bug is that it fails when presented
with the components of a previous Crossgen2 compilation in the
composite mode. This is because Crossgen2 lacks proper logic to
recognize the composite images and mistakes them for currently
unsupported managed C++ MSIL assemblies. This change adds the extra
check; I have also unified it between Crossgen2 and R2RDump.

Thanks

Tomas
@trylek trylek requested review from mangod9, cshung and janvorli December 9, 2021 23:59
@trylek trylek changed the title Disable test BasicTestWithMcj for CG2 with #60381 Skip native composite ReadyToRun images as inputs to Crossgen2 Dec 10, 2021
@trylek
Copy link
Member Author

trylek commented Dec 10, 2021

/cc @dotnet/crossgen-contrib

Copy link
Member

@janvorli janvorli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thank you!

@trylek trylek requested a review from jkoritzinsky December 10, 2021 00:50
I have found out that my previous version of the change wasn't
completely correct - the check was getting performed "too late"
and so the potential composite image remained in the list of
component assemblies and ended up getting rewritten to the output.
I have moved it to an earlier point in command line processing.

Thanks

Tomas
@trylek
Copy link
Member Author

trylek commented Dec 11, 2021

The assertion failure op2->TypeIs(TYP_LONG) is apparently known and tracked under #60624, merging in.

@trylek trylek merged commit 4fb7f77 into dotnet:main Dec 11, 2021
@trylek trylek deleted the BasicTestWithMcj branch December 11, 2021 01:15
@ghost ghost locked as resolved and limited conversation to collaborators Jan 10, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
4 participants