UTF8 BOM encoded as part of the first field

[dsimunic]

When a csv file with header row contains the UTF BOM (0xEFBBBF), the BOM characters become the part of the first field name.

For example:

$ hexdump -C file-with-utf-bom.csv
00000000  ef bb bf 74 69 74 6c 65  2c 66 69 72 73 74 2c 6c  |...title,first,l|
00000010  61 73 74 2c 65 6d 61 69  6c 2c 70 6f 73 69 74 69  |ast,email,positi|
00000020  6f 6e 2c 6f 72 67 61 6e  69 7a 61 74 69 6f 6e     |on,organization |

$ mlr --icsv --ojson file-with-utf-bom.csv | hexdump -C | head
00000000  7b 20 22 ef bb bf 74 69  74 6c 65 22 3a 20 22 22  |{ "...title": ""|
00000010  2c 20 22 66 69 72 73 74  22 3a 20 22 54 61 6b 65  |, "first": "Take|

$ mlr --version
Miller 5.1.0

OS is macOS Sierra 10.12.4

[johnkerl]

Easy to strip off on input. Do you want to have the option to produce the BOM on output as well?

[dsimunic]

I’d expect that most things one outputs json to are expecting UTF-8 without BOM, so no need to produce it on the output.

…

On 26 Apr 2017, at 05:38, John Kerl ***@***.***> wrote: Easy to strip off on input. Do you want to have the option to produce the BOM on output as well? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#138 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABJn2nCIh45Kurb4Sf3hTXb8Q1M-5fLBks5rzrw-gaJpZM4NHcA9>.

[johnkerl]

ok

[johnkerl]

i mean for csv output

[dsimunic]

No idea—my use case is always csv-->json. It’s easy to imagine that some people would want to do transformations without changing the file format markers, especially if they have to round-trip it back to windows (that’s where most encoding issues usually come from).

…

On 26 Apr 2017, at 11:40, John Kerl ***@***.***> wrote: i mean for csv output — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#138 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABJn2kcTom4yvJNDr5f_2Msfr1B7B8lCks5rzxEkgaJpZM4NHcA9>.

[johnkerl]

This is 2baf206 which is in head, and will go out in the next release.

@dsimunic please re-open if I missed anything -- thank you for the request!!