tracking issue for ? macro repetition

[nikomatsakis]

RFC: rust-lang/rfcs#2298

Status

• RFC accepted
• Preliminary implementation started (Implement ? macro repetition #47752)
• Preliminary implementation available
• Edition-dependent behavior proposed (see Change behavior of ? as a macro separator and Kleene op in 2018 edition #51934)
• Stabilization proposed
• Stabilized

Known bugs

None.

Unresolved questions to be answered before stabilization

• Should the ? Kleene operator accept a separator? Adding a separator is completely meaningless (since we don't accept trailing separators, and ? can accept "at most one" repetition), but allowing it is consistent with + and *. Currently, we allow a separator. We could also make it an error or lint.

[nikomatsakis]

cc @mark-i-m

[mark-i-m]

@nikomatsakis Could you update the checklist? The preliminary impl is available on nightly.

[mark-i-m]

@Centril I think you need to update the tracking issue in the RFC, too 😛

[Centril]

@mark-i-m Already done 🤣

[clarfonthey]

Is there any plan to stabilise this soon?

[mark-i-m]

I think it depends on experience if this is the right way to solve the problems involved... Frankly, I haven't really written any macros since I implemented this 😛 ...

I think there is still the one problem that was pointed out in the RFC thread: this doesn't provide a clean solution for trailing commas in * macros, since you would end up accepting foo!{,} which is a bit weird.

We could conceivably stabilize this anyway if people find it really useful. Any thoughts?

[clarfonthey]

I personally see it more ergonomic to have rules for () and $(...),+ $(,)? than $(...),* and $(...,)*, as the former usually requires way less duplication than the latter. The former it also natural for recursive macros, and it basically solves the trailing comma problem nicely.

Considering how it also allows optional arguments as well, I think it's pretty much a win here.

[clarfonthey]

As far as the one unresolved question (following separators) goes, I'm in favour of allowing it for the sake of a simpler grammar, but linting it in rustc (not clippy). That seems pretty ergonomic to me, and the compiler internally would have to accept it with a separator anyways to provide useful error messages.

[alexreg]

I don’t see the point of allowing the trailing separator even with a lint. It’s totally meaningless as you point out, so its existence in code would probably just perplex readers!

[mark-i-m]

the compiler internally would have to accept it with a separator anyways to provide useful error messages.

Hmm... I guess this is a good point. It means that probably the code complexity (in the compiler) of keeping or not keeping the separator is about the same. Still, having a simpler grammar is nice, especially if libsyntax2.0 ever actually intends have something auto-generated.

What about a deny-by-default lint?

[durka]

When would anyone ever allow the lint, though? This kinda sounds like "we can't decide, just make it configurable".

[mark-i-m]

When would anyone ever allow the lint, though?

My guess: probably never. On the other hand, when is anyone likely to actually use a separator for a single item? Will anyone ever run into the lint? The tradeoff is between a mildly simpler grammar and mildly simpler UX.

[mark-i-m]

The changes if we want disallow it are pretty minimal:

diff --git a/src/libsyntax/ext/tt/quoted.rs b/src/libsyntax/ext/tt/quoted.rs
index f324ede..a6fb782 100644
--- a/src/libsyntax/ext/tt/quoted.rs
+++ b/src/libsyntax/ext/tt/quoted.rs
@@ -470,8 +470,11 @@ where
                         GateIssue::Language,
                         explain,
                     );
+                } else {
+                    sess.span_diagnostic
+                        .span_err(span, "`?` macro repetition does not allow a separator");
                 }
-                return (Some(tok), op);
+                return (None, op);
             }
             Ok(Ok(op)) => return (Some(tok), op),

[alexreg]

My guess: probably never. On the other hand, when is anyone likely to actually use a separator for a single item? Will anyone ever run into the lint? The tradeoff is between a mildly simpler grammar and mildly simpler UX.

I think this is really "designer's/developer's motivation" vs. "user's motivation". And making the user happy should always win, I think. After all, there's a fair bit of ugly stuff in most compilers (including the Rust one), but most people don't care about it, because they don't touch it, and if they do they expect a significant learning curve. A compiler's end goal is to support a language rather than have an elegant internal structure (which is more of a bonus). The fact is, if you want to go messing with the compiler or libsyntax, taking the time to learn the slight difference between ? and */+ is no big deal. If you're a user, especially one new to Rust, that could be a real gotcha.

Now, since from the Rust user's perspective this feature adds literally nothing, and adding an error (as @mark-i-m's diff shows) is very straightforward, I maintain we disallow it. @durka's point and the article he links to is also a good one. Configuration is only a good thing when it's absolutely needed. Convention over it any other time.

[mark-i-m]

I went back and re-read part of the RFC thread, and this comment stuck out to me: rust-lang/rfcs#2298 (comment)

There is the question of what this matches: $(a)?+. It could be:

• a+ (hypothetical)
• a?a?a (current implementation)

Allowing the separator actually allows us to disambiguate:

• $(a),?+ matches a+
• $(a)?+ matches a?a?a

But at the same time, this seems really dubious, since removing the separator seems like it should not change the pattern matched. If we disallow separator on ?, then things are a bit cleaner:

• $(a),?+ is parsing error
• $(a)?+ matches a?a?a

Admittedly, this is all a bit contrived, but it does seem to throw the balance in favor of disallowing a separator on ?...

EDIT: corrected typo

[durka]

Agreed that if removing the separator changes things that seems weird and bad. But it's a shame that in the latter scenario you can't match aa+...

[mark-i-m]

One option would be to change the disambiguation strategy. That is, we stop allowing ? as a separator for any of the Kleene operators. This would be a breaking change, but perhaps it might not be that bad?

[durka]

We could allow precisely one separator, namely `?`, for this edge case: `$(a)?? +` matches `a+` or `aa+`. That eliminates the trouble where an unrelated separator has effects. Getting a little hacky though.

…

On Thu, Apr 5, 2018 at 4:20 PM, Who? Me?! ***@***.***> wrote: One option would be to change the disambiguation strategy. That is, we stop allowing ? as a separator for any of the Kleene operators. This would be a breaking change, but perhaps it might not be that bad? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#48075 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAC3n2Z5dotnKZCdGgLKn_SW42RzRPboks5tlnyqgaJpZM4R-yf0> .

[alexreg]

Now that you mention that, maybe a two-character ?? operator makes the most sense. No ambiguity, no allowance for specification of meaningless separators, and no prevention of using ? as a separator for the other Kleene operators.

[mark-i-m]

Stabilization Report 🎉

(Let me know if I missed anything)

Description of behavior

• Edition 2015 ? is a macro separator, not a kleene op, but using it as a separator triggers a migration lint.

  • This represents no change from previous behavior except the migration lint.

• Edition 2018 ? is not a valid separator. ? is a valid Kleene op.

  • Add a Kleene operator to macros to repeat a pattern at most once: $(pat)?. Here, ? behaves like + or * but represents at most one repetition of pat.
  • The ? Kleene operator does not accept any separator token, unlike + and *. For example, $(a),? is an error. This is because we don't allow trailing separators, so any separator used with the ? Kleene operator would be impossible to use.
  • For example, under this proposal, $(a)?+ matches + and a+ unambiguously.

• For a longer description of alternatives and behavior, see this FCP.

• Note, that the stabilization only impacts Edition 2018.

Differences from RFC

Link to RFC

• The RFC proposed a disambiguation strategy that used lookahead to discern whether a ? token was a separator for + or * OR was a ? Kleene op. We opted instead to remove ? as a separator in Edition 2018 as a breaking change, removing the ambiguity altogether.

• The RFC proposed making a minor breaking change in Edition 2015 to change the disambiguation as mentioned above. Instead, we opted to only enable ? Kleene operator in Edition 2018. That is, Edition 2015 is unchanged (except for a migration lint), whereas Edition 2018 has the changes described above.

• The RFC had an unresolved question: "Should the ? Kleene operator accept a separator token?". We decided the answer is "no".

Documentation and Testing

• Reference: Add some notes on repetition reference#424

• Edition Guide: Edition notes about ? edition-guide#111

• Test: ? is not a Kleene op on 2015
  https://github.com/rust-lang/rust/blob/f498e4ec1b57c3245a2fb8e0d5d836ed56760d2d/src/test/ui/macros/macro-at-most-once-rep-2015-ques-rep.rs

• Test: ? is a separator on 2015
  https://github.com/rust-lang/rust/blob/f498e4ec1b57c3245a2fb8e0d5d836ed56760d2d/src/test/ui/macros/macro-at-most-once-rep-2015-ques-sep.rs

• Test: ? is a Kleene op on 2018, and not a separator
  https://github.com/rust-lang/rust/blob/f498e4ec1b57c3245a2fb8e0d5d836ed56760d2d/src/test/ui/macros/macro-at-most-once-rep-2018.rs

[mark-i-m]

@nikomatsakis @Centril done :) Let me know if I missed anything

[mark-i-m]

Also, it looks like the checklist in the OP should be updated.

[Centril]

@rfcbot resolve report-and-tests

@mark-i-m that's a most excellent report. :)

[alexreg]

Well done on the super-meticulous work @mark-i-m. Glad we can finally get this feature stabilised!

[rfcbot]

🔔 This is now entering its final comment period, as per the review above. 🔔

[Centril]

@mark-i-m Since you did the implementation work, could you prepare a PR for the stabilization as well? :)

[mark-i-m]

Sure, but it might be a few days before I can get to it

[Centril]

@mark-i-m Ah; there's no great rush; 1.32 goes into beta on the 7th. Do you think you can get it landed by then?

[mark-i-m]

Yeah, I think I could do it by the end of the week.

[mark-i-m]

#56245 has merged