Scrub past history input before split

[hogelog]

This PR fixes #474.

As described in this #474, if there is a .irb_history from a startup with a different LANG setting, irb crashes when exiting and the .irb_history history is lost.

This problem is caused by the fact that past lines loaded from .irb_history (by irb launched with different encoding settings) become invalid strings from irb during launch.
String#split does not allow invalid strings.
https://github.com/ruby/ruby/blob/59f31a66180b8b118c93cdb3dad20e467b18f7fc/string.c#L8865

So I ran scrub before the split to remove the invalid string.
The contents of the original .irb_history are not preserved, but I believe it is a better solution than losing every history.

Solution that did not

I also considered the following methods, but they are all responses that would result in mixed strings of different encodings in .irb_history.
I did not choose such a response because I thought that creating a .irb_history that simply cannot be handled in such a way could produce unexpected bugs in irb's behavior.
What do you think?

Force split

How to forcibly split the past input. This may work if the "\n" is between character codes represented by the same byte sequence.

"😀\n😀".encode("utf-8").force_encoding("binary").split("\n".encode("us-ascii").force_encoding("binary")).join("\\\n").force_encoding("utf-8")
=> "😀\\\n😀"

But it does not work correctly between encodings where "\n" is represented by different byte sequences like below.

"＊＊＊".encode("utf-16be").force_encoding("binary").split("\n".force_encoding("binary")).join("\\\n")
=> "\xFF\\\n\xFF\\\n\xFF"

Always open with append mode

I thought that if I always append with append mode, I can avoid mentioning past lines.
However, I have not thought about it specifically because I think it would not be a good solution since the same problem occurs in the history truncation process after all.

How to reproduce the problem

You can reproduce the problem with the irb code in the repository by following the steps below.
Note that the contents of .irb_history will be lost if you perform this procedure.

$ LANG=en_US.UTF-8 ./bin/console 
irb(main):001> "😀"
=> "😀"
irb(main):002> 
$ LANG=en_US.UTF-8 ./bin/console 
irb(main):001> "😀"
=> "😀"
irb(main):002> exit
$ LANG=C ./bin/console 
irb(main):001> exit
/Users/hogelog/repos/oss/ruby/irb/lib/irb/history.rb:63:in `split': invalid byte sequence in US-ASCII (ArgumentError)

          hist = history.map{ |l| l.split("\n").join("\\\n") }
                                          ^^^^
        from /Users/hogelog/repos/oss/ruby/irb/lib/irb/history.rb:63:in `block (2 levels) in save_history'
        from /Users/hogelog/repos/oss/ruby/irb/lib/irb/history.rb:63:in `map'
        from /Users/hogelog/repos/oss/ruby/irb/lib/irb/history.rb:63:in `block in save_history'
        from /Users/hogelog/repos/oss/ruby/irb/lib/irb/history.rb:62:in `open'
        from /Users/hogelog/repos/oss/ruby/irb/lib/irb/history.rb:62:in `save_history'
        from /Users/hogelog/repos/oss/ruby/irb/lib/irb.rb:500:in `run'
        from /Users/hogelog/repos/oss/ruby/irb/lib/irb.rb:396:in `start'
        from ./bin/console:6:in `<main>'

[hogelog]

The problem with different encoding history is also on reline, so once the policy here is finalized, I would like to fix the reline as well.

[hogelog]

Based on @ima1zumi's advice that "rewriting ENV["LANG"] may have a negative impact on ruby/ruby testing," I tried to pass locale to assert_history.
e368f3e

[st0012]

Thank you for the fix as well as the detailed explanation!