nom 7.0 roadmap

[Geal]

no real plan for now, what do we do?

• update bitvec
• update lexical-core (move to minimal-lexical?)
• update memchr
  => upgrading those dependencies will raise the MSRV. Maybe separate bit parsing in another crate to keep MSRV low?

removing macros?

[djc]

I think bitvec should definitely not be part of the default features, and I think moving it into a separate crate would make a lot of sense. See also xd009642/tarpaulin#756 (many of my coverage CI jobs are currently failing because of this issue) and https://github.com/djc/askama/issues/446. I would be happy to help out with this.

Removing macros makes sense to me -- the function-based approach is just so much better.

[Geal]

other idea: more support for text parsing. nom has been usable for that for a long time, but the experience could be improved:

• better error types. VerboseError is a step in that direction, but there are better solutions now, like nom-supreme::error or https://docs.rs/dangerous/0.2.0/dangerous/error/index.html
• good input types like nom_locate
• error recovery: https://eyalkalderon.com/blog/nom-error-recovery/
• using grammar descriptions: https://github.com/rust-bakery/nom-peg
• handling precedence in expressions (something like https://hackage.haskell.org/package/parsec-3.1.14.0/docs/Text-Parsec-Expr.html would be nice)
• easy ways to work with a token stream or even an AST as input type
• reintroduce whitespace parsers

This could live in nom, or in a separate crate with a faster release cycle (that could be a crate that aggregates the other solution in one nice package).

a related goal I'm interested in would be to help in the interaction with configuration file formats, especially being able to parse a config file, modify it, serialize it (with cookie-factory) and the file's content would stay stable.

[meritozh]

+1 for nom_locate and better error types.

[eaglgenes101]

I'd also add to the list a type implementing nom's traits such that parser output can outlive the original input string. I don't know if it's within scope, but having to deal with the parser's output lifetime being constrained to the input's lifetime was the main pain point I experienced when I was trying to use nom for my parsing.

[djc]

@Geal I'm curious how you plan to proceed with this? cargo-tarpaulin is still broken (and using the alpha isn't a good option for them). I'd like to do a new Askama release and the bitvec setup is also causing issues there. Maybe rather than waiting for a bunch of feature development to get done, it would be good to release nom 7 with dependency updates and potentially the removal of the macro system only? (I'd be happy to do the legwork to work on a release like that.)

[Geal]

I already extracted bitvec and regex combinators in their own crates, so the missing part now is removing macros, which can take some time because most of the unit tests still use macros. I'm not planning any large feature dev for this release so once that's done (and a small pass over current issues and PRs) I can release nom 7

[djc]

Sounds good, thanks! I'll see if I can help with converting the tests.

[nickelc]

@djc have you started converting the tests? I could help you with that task

[Geal]

@djc @nickelc let's talk about macro removal in issue #1343 to synchronize work

[Geal]

I published a 7.0.0-alpha2 with yesterday's changes. There's still a bit of cleanup and docs to do, and maybe a few more combinator.
I'll combine the release with announcing https://github.com/rust-bakery/nom-bufreader

[eaglgenes101]

Oh, and: EoF handling. This was another pain point for me, as many of the combinators don't have corresponding versions that return Incomplete upon hitting EoF, or such combinators aren't discoverable in the documentation.

An issue of this kind is already reported and remains open: #1222

EDIT: My bad, my issue is actually somewhat different, but still involves EoF.

[Geal]

@eaglgenes101 which combinators did not have both versions? Now with macros removed, there should be both versions of everything

[eaglgenes101]

Whoops, my bad. My issue was that, like the poster of that issue, I wanted to make a parser that could repeatedly take in input of a particular kind, and if EoF happened right after the end of a successful iteration, return what was gathered rather than an incomplete error, while still returning the usual error (which might possibly an incomplete input error) that would be returned in all other cases. Maybe this would be better discussed in that other issue?

[Geal]

That would probably be out of scope for what combinators like many0 can do. And this is something you can do manually by parsing in a loop

[Geal]

there's a 7.0.0-alpha3 crate: https://crates.io/crates/nom/7.0.0-alpha3
everything is there for the release, it still needs a bit of testing but everything looks fine for now

[djc]

Thanks for the release! Should this issue be closed/unpinned/changed to be about nom 8?

[xpe]

Since 7.0.0 was released on 2021-08-21, it seems like a good idea to:

• close this issue, and
• create a follow-up roadmap (or tracking) issue for anything that remains