tryparse can throw InvalidCharError

[c42f]

According to the tryparse docs, tryparse is meant to return nothing if the string doesn't contain a valid number. I just got the following surprise:

julia> tryparse(Int, "\xb5s")
ERROR: Base.InvalidCharError{Char}('\xb5')
Stacktrace:
 [1] invalid_char(::Char) at ./char.jl:85
 [2] Type at ./char.jl:130 [inlined]
 [3] tryparse_internal(::Type{Int64}, ::String, ::Int64, ::Int64, ::Int64, ::Bool) at ./parse.jl:126
 [4] #tryparse#347 at ./parse.jl:234 [inlined]
 [5] tryparse(::Type{Int64}, ::String) at ./parse.jl:234
 [6] top-level scope at none:0

I'm not quite sure whether this is a bug, or whether it's up to the user to validate the Strings they're passing around.

[StefanKarpinski]

Definitely a bug. The whole point of tryparse is that you don't get an error no matter what the data you encounter.

[c42f]

I looked into a fix for this and I also came across the following possible bug:

julia> badchar = reinterpret(Char, UInt32(0x64000BAD))
'\x64\x00\x0b\xad': Malformed UTF-8 (category Ma: Malformed, bad data)

julia> UInt32(badchar)
0x00000064

julia> Char(UInt32(badchar))
'd': ASCII/Unicode U+0064 (category Ll: Letter, lowercase)

This is due to the code at

julia/base/char.jl

Lines 124 to 127 in 0af2f74

	function UInt32(c::Char)
	# TODO: use optimized inline LLVM
	u = reinterpret(UInt32, c)
	u < 0x80000000 && return u >> 24

The Char documentation claims:

julia/base/char.jl

Lines 39 to 46 in 0af2f74

	In order to losslessly represent arbitrary byte streams stored in a `String`,
	a `Char` value may store information that cannot be converted to a Unicode
	codepoint — converting such a `Char` to `UInt32` will throw an error.
	The [`isvalid(c::Char)`](@ref) function can be used to query whether `c`
	represents a valid Unicode character.
	"""
	Char

Which one is wrong; the documentation or the implementation? It seems that changing the check u < 0x80000000 to something involving isvalid could have quite a performance hit?

[StefanKarpinski]

I’ll take a look next week. Can you bump the issue so I don’t forget? I really wish GitHub allowed one to snooze an issue or PR.

[c42f]

Sure thing.

See the branch https://github.com/JuliaLang/julia/tree/cjf/tryparse-fix-invalid-char-error for one fast way to fix this, mirroring what's done in UInt32(::Char). I haven't dug far enough into the Char encoding stuff to know whether this makes sense but it seems like it might be slightly dubious.

[StefanKarpinski]

That seems pretty good to me.