clang++ failed create output file on windows if path more 247 chars #478

leanid · 2017-08-03T15:57:24Z

Description

clang++.exe failed to compile file on windows 10 if output object file path length more then 247 chars on my machine. I prepare minimal test case. Also I try to enable LongPathSupport but it not influence on clang++.exe.

Minimal test case:

put main.cpp - empty file into deep deep path folder
call clang++.exe from last NDK to produce objective file
add to output objective file name more charecters and try again
see result if output path more then 247 chars got error.

Environment Details

command line:

C:\Users\l_chayka>d:\android-sdk\ndk-bundle\toolchains\llvm\prebuilt\windows-x86_64\bin\clang++ -c D:/j/client/WoTBlitzAndroid/WotBlitz/build/intermediates/ndkBuild/debug/obj/local/x86/objs-debug/DavaNetworkServices/__/__/__/dava.framework/Modules/MemoryProfilerService/Sources/MemoryProfilerService/Private/main.cpp -o D:/j/client/WoTBlitzAndroid/WotBlitz/build/intermediates/ndkBuild/debug/obj/local/x86/objs-debug/DavaNetworkServices/__/__/__/dava.framework/Modules/MemoryProfilerService/Sources/MemoryProfilerService/Private/M11111111111111111111111111111111111.o

C:\Users\l_chayka>d:\android-sdk\ndk-bundle\toolchains\llvm\prebuilt\windows-x86_64\bin\clang++ -c D:/j/client/WoTBlitzAndroid/WotBlitz/build/intermediates/ndkBuild/debug/obj/local/x86/objs-debug/DavaNetworkServices/__/__/__/dava.framework/Modules/MemoryProfilerService/Sources/MemoryProfilerService/Private/main.cpp -o D:/j/client/WoTBlitzAndroid/WotBlitz/build/intermediates/ndkBuild/debug/obj/local/x86/objs-debug/DavaNetworkServices/__/__/__/dava.framework/Modules/MemoryProfilerService/Sources/MemoryProfilerService/Private/M111111111111111111111111111111111112.o
error: unable to open output file
      'D:/j/client/WoTBlitzAndroid/WotBlitz/build/intermediates/ndkBuild/debug/obj/local/x86/objs-debug/DavaNetworkServices/__/__/__/dava.framework/Modules/MemoryProfilerService/Sources/MemoryProfilerService/Private/M111111111111111111111111111111111112.o':
      'No such file or directory'
1 error generated.

C:\Users\l_chayka>

Pkg.Desc = Android NDK Pkg.Revision = 15.1.4119039
directly call compiler (minimal exemple)
Host OS: Windows 10 64 bit
Compiler: Clang++.

The text was updated successfully, but these errors were encountered:

DanAlbert · 2017-08-03T16:13:55Z

I assume you rebooted after changing that registry key? (It should only require a new run of clang, but ¯\_(ツ)_/¯)

Assuming that is the case: https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx#maxpath

Only a small family of functions is affected by that registry key. @stephenhines @pirama-arumuga-nainar: I assume that mingw is using multibyte strings. Any idea if there's a way to get it moved over to wide strings? (Or can we build Clang with msvc in the new system?)

leanid · 2017-08-03T16:50:48Z

Yes, I rebooted after change registry value, nothing happened, and after that start counting how many characters in path and it was not MAX_PATH limit of 260 only 248. Also this bug present in last windows build of clang++-4.0.1 too(just checked), maybe report there.

leanid · 2017-08-04T10:21:06Z

Found problem in clang in path convertion. If input file path not fit into MAX_PATH then clang use on windows long path names started from \\\\?\\ all directory separators converted from unix / to windows \\ BUT the first one NOT!
In my previous example resulted tmp path name converted inside clang.exe to:

\\\\?\\D:/j\\client\\WoTBlitzAndroid\\WotBlitz\\build\\intermediates\\ndkBuild\\debug\\obj\\local\\x86\\objs-debug\\DavaNetworkServices\\__\\__\\__\\dava.framework\\Modules\\MemoryProfilerService\\Sources\\MemoryProfilerService\\Private\\M111111111111111111111111111111111112.o-54d9ec6

look at first after D: should be:

\\\\?\\D:\\j\\client\\WoTBlitzAndroid\\WotBlitz\\build\\intermediates\\ndkBuild\\debug\\obj\\local\\x86\\objs-debug\\DavaNetworkServices\\__\\__\\__\\dava.framework\\Modules\\MemoryProfilerService\\Sources\\MemoryProfilerService\\Private\\M111111111111111111111111111111111112.o-54d9ec6

incoming parameter in command line was:

 D:/j/client/WoTBlitzAndroid/WotBlitz/build/intermediates/ndkBuild/debug/obj/local/x86/objs-debug/DavaNetworkServices/__/__/__/dava.framework/Modules/MemoryProfilerService/Sources/MemoryProfilerService/Private/main.cpp -o D:/j/client/WoTBlitzAndroid/WotBlitz/build/intermediates/ndkBuild/debug/obj/local/x86/objs-debug/DavaNetworkServices/__/__/__/dava.framework/Modules/MemoryProfilerService/Sources/MemoryProfilerService/Private/M111111111111111111111111111111111112.o

All directory separators in unix style. And after internal clang conversion only first not changed.

leanid · 2017-08-04T12:06:28Z

look at the: llvm/lib/Support/Path.cpp:427 (current trunk version)
look at START FIX and END FIX

void append(SmallVectorImpl<char> &path, Style style, const Twine &a,
            const Twine &b, const Twine &c, const Twine &d) {
  SmallString<32> a_storage;
  SmallString<32> b_storage;
  SmallString<32> c_storage;
  SmallString<32> d_storage;

  SmallVector<StringRef, 4> components;
  if (!a.isTriviallyEmpty()) components.push_back(a.toStringRef(a_storage));
  if (!b.isTriviallyEmpty()) components.push_back(b.toStringRef(b_storage));
  if (!c.isTriviallyEmpty()) components.push_back(c.toStringRef(c_storage));
  if (!d.isTriviallyEmpty()) components.push_back(d.toStringRef(d_storage));

  for (auto &component : components) {
    bool path_has_sep =
        !path.empty() && is_separator(path[path.size() - 1], style);
    bool component_has_sep =
        !component.empty() && is_separator(component[0], style);
    bool is_root_name = has_root_name(component, style);

    if (path_has_sep) {
      // Strip separators from beginning of component.
      size_t loc = component.find_first_not_of(separators(style));
      StringRef c = component.substr(loc);

      // Append it.
      path.append(c.begin(), c.end());
      continue;
    }

    if (!component_has_sep && !(path.empty() || is_root_name)) {
      // Add a separator.
      path.push_back(preferred_separator(style));
    }
// START FIX
    else if (component_has_sep && component.size() == 1)
    {
	// component itself is a separator '/' on windows after split: [c:/j/file] ->[{c:}, {/}, {j}, {/file}]
        // we have to change '/' into '\\' on windows on path start with "\\\\?\\"
	path.push_back(preferred_separator(style));
	continue;
    }
// END FIX
    path.append(component.begin(), component.end());
  }
}

pirama-arumuga-nainar · 2017-08-04T23:27:16Z

@leanid: Thanks for the diagnosis. It is indeed correct. Only suggestion would be to defensively invoke the fix only on Windows and only when path[-1] is a :. I'll put together a patch for upstream review (unless you want to contribute yourself, considering you did pretty much did the entire investigation).

leanid · 2017-08-07T08:10:37Z

Nice, please do the patch to upstream yourself.

pirama-arumuga-nainar · 2017-08-22T05:19:11Z

https://reviews.llvm.org/rL311382 is the commit from upstream. We'll cherry-pick it to r16 (hopefully in one of the betas).

DanAlbert · 2018-02-21T00:50:27Z

If this didn't make r16 then it's in r17 since that has another update.

imReker · 2018-05-29T08:41:02Z

Problem seems not solved totally in r17.
In my case, some long path libraries can be built successfully with r17 (which failed on r16 or earlier due to this issue), but still get error on another library in same project.

[armeabi-v7a] Compile thumb  : ancillary <= fd_recv.c
[armeabi-v7a] Compile thumb  : ancillary <= fd_send.c
[armeabi-v7a] StaticLibrary  : libancillary.a
D:/android-sdk/ndk-bundle/build//../toolchains/arm-linux-androideabi-4.9/prebuilt/windows-x86_64/bin/arm-linux-androideabi-ar: E:/AndroidStudioProjects/Project/Client-Android-Demo/core/build/intermediates/ndkBuild/debug/obj/local/armeabi-v7a/objs-debug/ancillary/E_/AndroidStudioProjects/Project/Client-Android-Demo/core/src/main/jni/android/Tester/__/libancillary/__/__/depends/libancillary/fd_recv.o: No such file or directory
make: *** [E:/AndroidStudioProjects/Project/Client-Android-Demo/core/build/intermediates/ndkBuild/debug/obj/local/armeabi-v7a/libancillary.a] Error 1

The fd_recv.o is built successfully (I've checked the file, it exist in correct place) but can't be found in linking.

pirama-arumuga-nainar · 2018-05-29T18:40:46Z

The path is longer than 260 characters, but the failure is from arm-linux-androideabi-ar. @DanAlbert should the build system be prepending long paths with '\?' when invoking tools?

imReker · 2018-06-04T06:19:32Z

@pirama-arumuga-nainar Any update? Should I open a new issue on this problem?

pirama-arumuga-nainar · 2018-06-04T06:23:35Z

IIUC, the failure is when invoking arm-linux-androideabi-ar and not clang itself. If that's the case, then open a new issue.

enh assigned pirama-arumuga-nainar Aug 10, 2017

DanAlbert added the clang label Oct 3, 2017

DanAlbert closed this as completed Feb 21, 2018

imReker mentioned this issue Jun 4, 2018

Build failed on Windows when path more than MAX_PATH #711

Closed

ej3 mentioned this issue Jul 6, 2018

buildReactNdkLib error while building from source post react-native@0.55.0 facebook/react-native#18833

Closed

enh mentioned this issue Nov 8, 2018

Linker runs into Windows 260 character path limit #839

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

clang++ failed create output file on windows if path more 247 chars #478

clang++ failed create output file on windows if path more 247 chars #478

leanid commented Aug 3, 2017

DanAlbert commented Aug 3, 2017 •

edited

Loading

leanid commented Aug 3, 2017

leanid commented Aug 4, 2017

leanid commented Aug 4, 2017 •

edited

Loading

pirama-arumuga-nainar commented Aug 4, 2017

leanid commented Aug 7, 2017 •

edited

Loading

pirama-arumuga-nainar commented Aug 22, 2017

DanAlbert commented Feb 21, 2018

imReker commented May 29, 2018 •

edited

Loading

pirama-arumuga-nainar commented May 29, 2018

imReker commented Jun 4, 2018

pirama-arumuga-nainar commented Jun 4, 2018

clang++ failed create output file on windows if path more 247 chars #478

clang++ failed create output file on windows if path more 247 chars #478

Comments

leanid commented Aug 3, 2017

Description

Environment Details

DanAlbert commented Aug 3, 2017 • edited Loading

leanid commented Aug 3, 2017

leanid commented Aug 4, 2017

leanid commented Aug 4, 2017 • edited Loading

pirama-arumuga-nainar commented Aug 4, 2017

leanid commented Aug 7, 2017 • edited Loading

pirama-arumuga-nainar commented Aug 22, 2017

DanAlbert commented Feb 21, 2018

imReker commented May 29, 2018 • edited Loading

pirama-arumuga-nainar commented May 29, 2018

imReker commented Jun 4, 2018

pirama-arumuga-nainar commented Jun 4, 2018

DanAlbert commented Aug 3, 2017 •

edited

Loading

leanid commented Aug 4, 2017 •

edited

Loading

leanid commented Aug 7, 2017 •

edited

Loading

imReker commented May 29, 2018 •

edited

Loading