ngSanitize cant escape all unicode characters

[jtangelder]

ngSanitize has a bug escaping unicode chars that arent in the range of charCodeAt, see the fiddle below. Removing this replace function fixes the problem, but i was wondering why this is done in the first place. Unicode chars can be placed inside documents safely, especially since utf-8 became a standard charset these days.

http://jsfiddle.net/jtangelder/SQf7w/

angular.js/src/ngSanitize/sanitize.js

Line 370 in a7e12b7

replace(NON_ALPHANUMERIC_REGEXP, function(value){

I can do a PR if it's ok!

[IgorMinar]

I can't think of a reason why we still need to do this.

@mhevery any idea?

I suggest that you send us a PR and we'll evaluate it there. At first glance it seems that it should be safe to remove, but we need to have a careful look at this before merging any change. In any case, this is a legitimate bug IMO, I'm just not sure of the constraints for the right solution.

[mhevery]

is it possible that you could construct a valid and safe UTF8 string which would be an unsafe string in another encoding? Not sure I would be that eager to remove this.

[jtangelder]

Another fix could be to check if document.charset is at UTF-8 or UTF-16... Maybe someone with more knowledge of charsets/encoding could take a look at this? This could fix the issues @mhevery brings up.

[anders]

The problem is JavaScript uses surrogate pairs, see: http://stackoverflow.com/questions/3744721/javascript-strings-outside-of-the-bmp

[memolog]

Hi,

I got the same issue recently.
It's the surrogate pairs issue as @anders mentioned in the above.

We could handle it before escaping unicode chars as the following diff
https://gist.github.com/memolog/79c88598b12d309368e5

see also http://mdn.beonex.com/en/Core_JavaScript_1.5_Reference/Global_Objects/String/charCodeAt.html

thanks!