Improvements to BOM detection:

[st-pasha]

if BOM is found then set encoding to UTF8; print a message about BOM in verbose mode; detect BOM in GB18030 and UTF16 encodings and issue warning / error appropriately

[mattdowle]

Many thanks for this improvement.
Can you remove whitespace diffs please.
Please retain the nice comment with references to #1087 and #1465.
If a function is only ever called once, then I prefer it not to be a function but inside alongside where it opens the file in one place. That saves hopping around to helper functions. If a function is called twice or more, then it can be a function.
What's variable size? Yes I see it's not compiling for that reason.
There also needs to be tests added (e.g. issue_1087_utf8_bom.csv is tested in inst/tests) and a line added to NEWS.md all in the same commit please.

[codecov-io]

Codecov Report

Merging #2084 into master will increase coverage by 0.06%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #2084      +/-   ##
==========================================
+ Coverage   90.52%   90.59%   +0.06%     
==========================================
  Files          59       59              
  Lines       11332    11340       +8     
==========================================
+ Hits        10258    10273      +15     
+ Misses       1074     1067       -7

Impacted Files	Coverage Δ
src/fread.c	91.59% <100%> (+0.08%)	⬆️
src/forder.c	94.47% <0%> (+0.92%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c9f2de7...2f80fee. Read the comment docs.