Added encoding parameter to TextDirectoryCorpus

[Sandman-Ren]

Fixes #3316.

Added encoding='utf-8' keyword argument to TextDirectoryCorpus. Used smart_open to replace built-in open.
TextCorpus does not seem to have the problem. Nothing is changed for TextCorpus.

[piskvorky]

Looks good, thanks! Let's wait for the CI tests to finish, otherwise LGTM.

Is there any risk of backward compatibility issues? Do people save/load these objects to disk = new encoding parameter missing in old saves?

[Sandman-Ren]

Looks good, thanks! Let's wait for the CI tests to finish, otherwise LGTM.

Is there any risk of backward compatibility issues? Do people save/load these objects to disk = new encoding parameter missing in old saves?

I've tried saving the TextDirectoryCorpus using TextDirectoryCorpus.save() in the modified code and load it back using TextDirectoryCorpus.load() using code on the develop branch and did not get any errors.

Considering the current state of TextDirectoryCorpus where a windows user (like me) cannot handle some utf-8 characters, I think (think...) there should not be any issues with existing code except for the following case:

User 1 saves the TextDirectoryCorpus with the encoding optional parameter
User 2 (an unfortunate Windows user) loads the saved TextDirectoryCorpus, using a version of the code without the optional encoding parameter.
User 2 will receive a UniCodeDecodeError as seen in #3316 when they perform any operation that involves TextDirectoryCorpus.getstream()

The best solution I can think of overall is probably for gensim to check the operating system and set the 'PYTHONUTF8' environment variable (if possible).

[piskvorky]

I think it's fine now, thanks. Let's wait for @mpenkov 's review and we can merge :)

[mpenkov]

LGTM! Thank you @Sandman-Ren !