Varargs support for system ABI

[beepster4096]

This PR allows functions with the system ABI to be variadic (under the extended_varargs_abi_support feature tracked in #100189). On x86 windows, the system ABI is equivalent to C for variadic functions. On other platforms, system is already equivalent to C.

Fixes #110505

[rustbot]

r? @petrochenkov

(rustbot has picked a reviewer for you, use r? to override)

[rustbot]

These commits modify compiler targets.
(See the Target Tier Policy.)

[Soveu]

That makes me think, maybe we should first adjust the abi, then check if it is possible to use varargs?
This has an advantage, that the compiler will be more "flexible", at the cost of code being probably less portable - we could add some warnings for using system with varargs, but that will be it, just a warning.

[bjorn3]

Does windows have any function that both uses the system abi and is a vararg? How is the abi handled for that function on x86?

[beepster4096]

@Soveu

maybe we should first adjust the abi, then check if it is possible to use varargs?

There's some edge cases where this would make the varargs error dependent on the target. For example, a variadic stdcall function on i686-pc-windows-msvc would fail to compile. The same function on x86_64-pc-windows-msvc would pass because stdcall would be adjusted into C. Right now, the check is independent of the target, it just relies on the ABI the function is declared with. I don't think changing that is a good idea.

@bjorn3

Does windows have any function that both uses the system abi and is a vararg?

The example from the issue, DbgPrint, actually appears to be defined with __cdecl explicitly, but I found a function RtlInitializeSidEx that is both variadic and uses __stdcall.

How is the abi handled for that function on x86?

According to https://learn.microsoft.com/en-us/cpp/cpp/stdcall, variadic functions marked with __stdcall just use __cdecl. I am going to add that link in a comment to satisfy this review comment.

[petrochenkov]

r=me after adding cross-link comments to different places that "normalize" calling conventions (#119587 (comment)) + squashing commits.
@rustbot author

[beepster4096]

I'm a little unsure of what exactly I am being asked to do here. It seems like the review (#119587 (comment)) says to add a comment in build_dll_import's abi logic linking to adjust_abi. Meanwhile, the plurals in the r=me prerequisite lead me to believe it means to have both link to each other. To be clear, adjust_abi and build_dll_import are the only places that normalize calling conventions I could find.

I've done only the former change (and squished) for now.
@rustbot ready

[petrochenkov]

I've done only the former change

Okay, that's good enough.
@bors r+

[bors]

📌 Commit 41e224b has been approved by petrochenkov

It is now in the queue for this repository.