Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Port from master to RB-2.1 - Fixes Unicode paths on Windows (#1363) #1527

Merged
merged 1 commit into from
Nov 1, 2021

Conversation

hodoulp
Copy link
Member

@hodoulp hodoulp commented Oct 29, 2021

  • Fixes Unicode paths on Windows

Unlike most other platforms, Windows' Unicode is standardized around
UTF-16, an encoding not compatible with "char *" arrays common in
C/C++. As such, to support Unicode correctly when using Win32 APIs,
strings must be converted to and from UTF-16 and the Unicode
versions of the APIs must be used over the ANSI versions.

This commit introduces the following:

  • Utility funcions for converting between UTF-8 and UTF-16LE on all
    platforms:

    • Platform::Utf8ToUtf16
    • Platform::Utf16ToUtf8
  • Adds test for these conversion functions to ensure the conversion
    to and from UTF-8 and UTF-16LE is correct.

  • Utility wrappers for "std::ifstream" that automatically convert
    to and from UTF-16 so that filenames requiring Unicode encoding
    function correctly:

    • Platform::CreateInputFileStream
    • Platform::OpenInputFileStream
  • Moves the file default compute hash function to the Platform
    class (Platform::CreateFileContentHash) and switches to the
    Win32 UTF-16 variant on Windows.

  • Adds the "UNICODE" macro before including "Windows.h" which
    ensures all functions called are the Unicode variants instead of
    the default ANSI variants.

    • Implicitly changes functions such as GetEnvironmentVariable and
      SetEnvironmentVariable to their Unicode variants.
  • Changes the following environment variable related functions to
    their Win32 Unicode variants on Windows:

    • environ -> _wenviron
    • _putenv_s -> _wputenv_s
  • Updates tests using SetEnvironmentVariable to use wide string
    literals since that function has been switched to the Unicode
    variant.

Signed-off-by: itsmattkc itsmattkc@gmail.com

  • Moved UNICODE and _UNICODE definitions to CMake

Signed-off-by: itsmattkc 34096995+itsmattkc@users.noreply.github.com

  • handle empty strings and use const references

Signed-off-by: itsmattkc 34096995+itsmattkc@users.noreply.github.com

  • only use wenviron when unicode is enabled

Signed-off-by: itsmattkc itsmattkc@gmail.com

  • add ANSI Win32 version of Getenv

Signed-off-by: itsmattkc itsmattkc@gmail.com

  • Add CMake option for compiling with Win32 Unicode support

Signed-off-by: itsmattkc itsmattkc@gmail.com

  • Throw exception if UTF functions are called on non-Windows platforms

Signed-off-by: itsmattkc itsmattkc@gmail.com

  • Throw OCIO Exception

Signed-off-by: itsmattkc itsmattkc@gmail.com

  • Don't run UTF-8/16 conversion test on non-Windows

Signed-off-by: itsmattkc 34096995+itsmattkc@users.noreply.github.com

  • Fix CMake error

Signed-off-by: itsmattkc 34096995+itsmattkc@users.noreply.github.com

  • minor CMake adjustment

Signed-off-by: itsmattkc itsmattkc@gmail.com

  • Clarified CMake option for Win32 unicode

Signed-off-by: itsmattkc 34096995+itsmattkc@users.noreply.github.com

  • minor improvements

Signed-off-by: itsmattkc 34096995+itsmattkc@users.noreply.github.com

  • fix macro in cpu test cmake

Signed-off-by: itsmattkc 34096995+itsmattkc@users.noreply.github.com

* Fixes Unicode paths on Windows

Unlike most other platforms, Windows' Unicode is standardized around
UTF-16, an encoding not compatible with "char *" arrays common in
C/C++. As such, to support Unicode correctly when using Win32 APIs,
strings must be converted to and from UTF-16 and the Unicode
versions of the APIs must be used over the ANSI versions.

This commit introduces the following:

- Utility funcions for converting between UTF-8 and UTF-16LE on all
  platforms:
  - Platform::Utf8ToUtf16
  - Platform::Utf16ToUtf8

- Adds test for these conversion functions to ensure the conversion
  to and from UTF-8 and UTF-16LE is correct.

- Utility wrappers for "std::ifstream" that automatically convert
  to and from UTF-16 so that filenames requiring Unicode encoding
  function correctly:
  - Platform::CreateInputFileStream
  - Platform::OpenInputFileStream

- Moves the file default compute hash function to the Platform
  class (Platform::CreateFileContentHash) and switches to the
  Win32 UTF-16 variant on Windows.

- Adds the "UNICODE" macro before including "Windows.h" which
  ensures all functions called are the Unicode variants instead of
  the default ANSI variants.
  - Implicitly changes functions such as GetEnvironmentVariable and
    SetEnvironmentVariable to their Unicode variants.

- Changes the following environment variable related functions to
  their Win32 Unicode variants on Windows:
  - environ -> _wenviron
  - _putenv_s -> _wputenv_s

- Updates tests using SetEnvironmentVariable to use wide string
  literals since that function has been switched to the Unicode
  variant.

Signed-off-by: itsmattkc <itsmattkc@gmail.com>

* Moved UNICODE and _UNICODE definitions to CMake

Signed-off-by: itsmattkc <34096995+itsmattkc@users.noreply.github.com>

* handle empty strings and use const references

Signed-off-by: itsmattkc <34096995+itsmattkc@users.noreply.github.com>

* only use wenviron when unicode is enabled

Signed-off-by: itsmattkc <itsmattkc@gmail.com>

* add ANSI Win32 version of Getenv

Signed-off-by: itsmattkc <itsmattkc@gmail.com>

* Add CMake option for compiling with Win32 Unicode support

Signed-off-by: itsmattkc <itsmattkc@gmail.com>

* Throw exception if UTF functions are called on non-Windows platforms

Signed-off-by: itsmattkc <itsmattkc@gmail.com>

* Throw OCIO Exception

Signed-off-by: itsmattkc <itsmattkc@gmail.com>

* Don't run UTF-8/16 conversion test on non-Windows

Signed-off-by: itsmattkc <34096995+itsmattkc@users.noreply.github.com>

* Fix CMake error

Signed-off-by: itsmattkc <34096995+itsmattkc@users.noreply.github.com>

* minor CMake adjustment

Signed-off-by: itsmattkc <itsmattkc@gmail.com>

* Clarified CMake option for Win32 unicode

Signed-off-by: itsmattkc <34096995+itsmattkc@users.noreply.github.com>

* minor improvements

Signed-off-by: itsmattkc <34096995+itsmattkc@users.noreply.github.com>

* fix macro in cpu test cmake

Signed-off-by: itsmattkc <34096995+itsmattkc@users.noreply.github.com>
@michdolan michdolan merged commit b252372 into RB-2.1 Nov 1, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants