Error in moban 0.4.3

[ayan-b]

Using moban 0.4.3, we get this error: UnicodeDecodeError: 'utf-8' codec can't decode byte 0x89 in position 0: invalid start byte in the coala-mobans repo, although this error does not arise with moban 0.3.3.

CI log:

• moban 0.3.3: https://gitlab.com/ayan-b/mobans/-/jobs/199587253
• moban 0.4.3: https://gitlab.com/ayan-b/mobans/-/jobs/199585396

[chfw]

0.4.3 read and write

0.3.3 uses shutils to copy.

when reading in, if the file is not encoded in utf-8, this problem will rise.

[ayan-b]

when reading in, if the file is not encoded in utf-8, this problem will rise.

That is also what I suspected. So we can by default convert the file to utf-8 in this repo while reading?

[chfw]

https://github.com/moremoban/moban/blob/dev/moban/copy/__init__.py

Copy engine should have read with 'rb' flag and ignore encoding. Simple 'open' fuction should work.

When saving, it should use 'wb' flag, treating the content as binary throughout the journey within moban.

[ayan-b]

Ok, I will send a PR by tonight.

[ayan-b]

@chfw This error is not yet fixed: https://gitlab.com/ayan-b/mobans/-/jobs/201319687 . It comes from https://github.com/moremoban/moban/pull/267/files#diff-4ce3d7a5af974a09dae6519b5a45cc44R65. If we ignore decoding python3 tests fail. Interestingly, the test_copy_encoding does not fail here.

[ayan-b]

@chfw what do you think?

[chfw]

I think you have just shifted decoding from copy engine to write function. In python 3, cannot we write bytes directly to a file?

[chfw]

And it also means that the unit test case failed to capture the bug.

[ayan-b]

By the way, I wonder if we can simply copy in these scenarios (where we are not touching the contents of the file) using os module without reading or writing. This will greatly enhance the performance.

[ayan-b]

I will try to solve the issue over the weekend.