Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Command parser mangles unicode inputs #1133

Closed
kevsmith opened this issue Nov 6, 2016 · 2 comments
Closed

Command parser mangles unicode inputs #1133

kevsmith opened this issue Nov 6, 2016 · 2 comments
Assignees
Labels
Milestone

Comments

@kevsmith
Copy link
Member

kevsmith commented Nov 6, 2016

The new command parser added to Piper for the 0.16 Cog release mangles Unicode inputs. The mangling has two sources:

  • Use of Erlang binary and iolist conversion functions (:erlang.binary_to_list/1 and :erlang.iolist_to_binary/1) in the leex generated lexers and yecc generated parsers.
  • Omission of the unicode option in calls to re:run/3 and re:replace/3 also in the generated lexers and parsers.

This bug is likely present in the old parser as well.

@kevsmith kevsmith added the piper label Nov 6, 2016
@kevsmith kevsmith added this to the Cog 0.17 milestone Nov 6, 2016
@kevsmith kevsmith self-assigned this Nov 6, 2016
@kevsmith kevsmith added the review label Nov 6, 2016
@kevsmith
Copy link
Member Author

kevsmith commented Nov 6, 2016

The fix pending in operable/piper#35 ensures Unicode passes thru the parsing pipeline unmangled. It does not add support for Unicode bundle or command names as doing so would require deeper changes to the parsers and test suite.

That said, making the parsing pipeline Unicode-safe is a necessary first step towards supporting Unicode bundle and command names.

@kevsmith
Copy link
Member Author

kevsmith commented Nov 6, 2016

Duplicates #1125

@kevsmith kevsmith removed the review label Nov 7, 2016
@kevsmith kevsmith modified the milestones: Cog 0.17, Cog 0.16.1 Nov 7, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant