IDE 1.6.6: non-ascii characters are no longer displayed properly

[romandubrov]

I'm working with TFT displays and need to have readable cyrillic strings in code. Prior 1.6.6, having my files in win-1251 encoding and adding preproc.substitute_unicode=false in preferences.txt worked fine for me. But since upgrade, IDE shows squares instead of non-ascii chars that makes it impossible to read. Please fix.

Cyrillic in IDE 1.6.6

Same file in Sublime text editor

[lmihalkovic]

I think i will be adding an explicit "convert from windows encoding" menu entry under Tools so that people on mac/linux who receive for some odd reason a old sketch encoded in win1251 can still read & concert it to utf8 (otherwise it can only be done on a windoze box)...
I think this makes even more sense for my branch as i am working on supporting zipped sketches (i.e working directly from a single zip that contains the code without having to unzip it anywhere)

[romandubrov]

actually the point is to allow human-friendly work with non-unicode strings in IDE because various peripherals like displays, printers, etc, do not support utf-8 at all. Now only possible workaround is to use 1.6.5 with adding preproc.substitute_unicode=false in preferences.txt

[matthijskooijman]

@romandubrov, another workaround would be to use \xnn escape codes in you strings. It's certainly not ideal, but might help you get things working now. I'm not sure what broke this and what would be needed to fix it again (encodings generally suck), though.

[romandubrov]

@matthijskooijman well I was talking about human-friendly solution. When working with large projects with lot of UI and UI strings, it's quite important to don't lose track, and these \xnn are messing everything up

[cmaglie]

@romandubrov
can you provide your source file (with cyrillic strings) to test this?

[romandubrov]

@cmaglie you can use this file from my test project https://github.com/romandubrov/myWeatherStation/blob/master/main_board/main_board.ino

[cmaglie]

@romandubrov
the file you linked seems to be already UTF-8, also opening it in the Arduino IDE works fine even if I set the preproc.substitute_unicode=false preference:

Are you sure that https://github.com/romandubrov/myWeatherStation/blob/master/main_board/main_board.ino is win-1251 encoded?

[lmihalkovic]

@cmaglie @matthijskooijman you might want to give some thinking to this one. IMHO the current code is less than optimum. \xnn is not a viable solution. You might really want to look into what i suggested...

@romandubrov ... I am writing code while my wife is driving... but i did get my hands on a list of months in cyrillic that i added to a testcase which i encoded as w1251, so in my case there is no doubt i am dealing with win.

@cmaglie it may very well be... depending on how you fetched it (git / curl) you'll see it or not...

[lmihalkovic]

@matthijskooijman

(encodings generally suck),

We programmers suck... encodings dont, and java is remarkably flexible in that respect.

[romandubrov]

@cmaglie well it is in win1251 on my local drive. may be github does some conversion as it is not a binary file. Please find the file attached, it is in 1251 for sure
main_board.ino.txt

@lmihalkovic if I fully got you idea, you mean converting from non-unicode encoding to utf-8. Yeah that's great and I'm also a big fan of utf-8. But when you compite utf-8 .ino file with cyrillic, you'll get utf-8 strings in your program - that's exactly what I wanted to avoid. So may be you have any idea about how to compile utf-8 sketch file into non-unicode strings in the program?

[lmihalkovic]

@romandubrov Utf8 or win1251 are just conventions stating which binary value will be used to store any given 'character'. There are several problem the ide has to address

• finding a consistsnt way to store non-ascii characters accross OSs (windows, linux, mac used to have different default ways to encode characters making - utf8 has become the defacto standard)
• finding a way to provide compilable source files for gcc (haven't checked recently, but gcc used to choke on nonascii identifiers)
• finding a way to pass store nonascii char inside program as PROGMEM & std vars (this is where the ide does a hidden rewriting of the source code)

Idealy the ide should be able to use utf8 to ensure consistancy and cross-platform convenience, wighout impacting the target binary (interestingly, pre-builder code used to only re-write strings in .ino and leave .cpp unchanged)

Nonascii refers to non usascii

[romandubrov]

@lmihalkovic yeah brilliant, moving all code with strings to separate .cpp file with my favorite 1-byte encoding is the idea I didn't came to on my own and seems it should help there. However, if I finally have to move almost all my code outside .ino (and thus edit it with separate code editor) - only thing IDE is still needed for is compiling/flashing (I don't assume setup() and loop() essential features of IDE). A good reason for me to start looking close to AVR studio btw

[lmihalkovic]

A good reason for me to start looking close to AVR studio btw

Hmmm... This is precisely my motivation for starting a fork

[romandubrov]

@lmihalkovic looks interesting...