{load|intersect}_word2vec_format: allow non-strict unicode error handling

[gojomo]

A word2vec.c-format file might not have perfectly legal unicode encodings. A reader might want to load them anyway. This adds a parameter to load_word2vec_format & intersect_word2vec_format, default 'strict', that is passed to the utils.to_unicode() method as its errors parameter. By choosing one of the more tolerant alternatives – ignore or replace – bad unicode won't cause a fatal ValueError. Instead, the problematic string will just ignore or replace the offending char(s), allowing progress to continue.

[piskvorky]

@tmylk I don't see this (important) change in the CHANGELOG. Did you miss it in the last release notes?

If so, can you (or @gojomo ) add it now? We'll announce it as part of the next release (0.12.4).

[gojomo]

Perhaps even 'ignore' should be the default, or we could catch the error and before re-throwing include a message pointing to the option?

[piskvorky]

Hmm, yes, a clearer error message may help. Plus a re-throw. Certainly not ignore.

[rsuwaileh]

How can we use this parameter of load_word2vec_format that skips or replaces the chars with different encoding? I can't find this in documentation!

[tmylk]

@ReemSuwaileh This question is better suited for the mailing list.

‘strict’, ‘replace’ (add U+FFFD, ‘REPLACEMENT CHARACTER’), or ‘ignore’ description in Python documentation.