Link.create only works for directories on windows

[jakemac53]

example program:

import 'dart:io';

main() async {
  var  link = new Link('link.txt');
  await link.create('original.txt');
}

This will "succeed" on both linux and windows, but on windows it will create a directory symlink which doesn't work (since its actually pointing at a file). You end up getting a permissions error if you try and open the symlink.

As far as I can tell windows has supported symlinks for files as well as directories for a while now (at least since vista), so the above should be able to work, although I am not sure how complex it would be to implement.

It looks like the windows file implementation assumes a directory, although this is not documented or validated.

[bkonyi]

Currently the Link implementation for Windows relies on Junctions which, as you've pointed out, can only be used for directories. We'll have to rewrite the majority of the Link implementation to use symlinks. Is this blocking you on something in particular?

[jakemac53]

It creates a significantly degraded experience for windows users when using the new build_runner package - especially with vm tests due to #33952.

We have to create a merged output directory containing all the generated and original sources before running tests always - and some users are doing this after each build so they can run custom dev servers in that directory.

For the pub package tests for instance the generated directory currently is ~6GB in size, so copying that over each time before running tests is obviously quite slow. Symlinking the files would be much faster.

Note that we can't symlink entire directories because we have to merge files from two different places (source tree and build cache) into the same logical directory.

[bkonyi]

Ah gotcha. I'll look into how difficult it would be to update the existing implementation and see if I can spare a few cycles to switch to real symlinks in the next week or so.

[bkonyi]

I've spent some time investigating this, and unfortunately I'm not sure that we can move to the Windows symlink implementation. According to this blog post creating a symbolic link requires that either:

1. The process creating the symbolic link must be running in a console with elevated privileges
2. The user invoking the process creating the symbolic link must have Windows 10 Developer Mode enabled for those on Insider Preview versions with builds >= 14972 (this is also included in all stable releases since the Creators update).

Both of these options require that the user have admin rights at one point or another, which seems like quite a lot to ask to simply create symbolic links for files. cc'ing @zanderso for his thoughts.

[jakemac53]

Thanks for looking into this @bkonyi, that is quite unfortunate indeed.

Would it be feasible for us to expose an environment variable which informs you if symlinks are supported on the current platform, which would check for developer mode? Then at least we could provide a warning on the command line to our users asking them to enable it.

This would also likely require some sort of optional argument to the Link.create method to tell it to use the newer windows symlinks as opposed to Junctions, which would throw if they weren't supported?

[zanderso]

The current implementation happily creates reparse points targeting files even though the underlying system calls don't traverse the reparse points when we try to access them as files. There are workarounds for this both in Dart code and in file_win.cc. In Dart code, you can do things like this:

import 'dart:io';

main() {
  var link = new Link('link.txt');
  link.createSync('original.txt');
  var linkFile = new File('link.txt');
  if (Platform.isWindows && FileSystemEntity.isLinkSync(linkFile.path)) {
    var linkLink = new Link(linkFile.path);
    var linkTarget = linkLink.targetSync();
    var linkTargetFile = new File(linkTarget);
    var contents = linkTargetFile.readAsStringSync();
    print(contents);
  }
}

Another fix is probably to use code like the following in file_win.cc to manually chase the reparse points when we're asked to interpret them as files:

File* File::Open(Namespace* namespc, const char* path, FileOpenMode mode) {
  // Manually chase symbolic links.
  const intptr_t kSymlinkLimit = 31;
  intptr_t links_followed = 0;
  File::Type type = GetType(namespc, path, false);
  while ((type == kIsLink) && (links_followed < kSymlinkLimit)) {
    path = LinkTarget(namespc, path);
    if (path == NULL) {
      return NULL;
    }
    type = GetType(namespc, path, false);
    links_followed++;
  }
  // Check if we reached the symlink limit, and bail if we did.
  if (type == kIsLink) {
    SetLastError(ERROR_TOO_MANY_LINKS);
    return NULL;
  }
  ...
}

We just have to be careful to chase the reparse points in all the required places (and not any extra places). I would prefer a solution like this until the Windows ecosystem is a little more uniform and we can switch over to proper symlinks.

[jakemac53]

The merged output directories we create need to be fully valid, they are intended for use by arbitrary executables, so a workaround like this won't work unfortunately.

[zanderso]

@jakemac53 Ack.

I think we should be very reluctant to add Windows specific named parameters and/or to set environment variables in the dart:io implementation. It is a breaking change, but I think we should do the following:

• Change Link.create to only make new-style symlinks
• Change the other Link calls to be able to understand both styles of symlink
• Update Link docs to indicate the need for an elevated console or dev mode on Windows.

I think the impact of the breaking change would be pretty small since Windows developers are already used to using an elevated console or dev mode for programs that make real symlinks. @bkonyi will correct me if I'm misremembering, but I believe this is the approach taken by Go, for example.

/cc @mit-mit for thoughts on the breaking change here.

[jakemac53]

Do you know what the exception we would end up getting looks like on the Dart side of things if you aren't in an elevated console? If it is something that is specific to this error then I think the suggested approach would be totally fine (modulo discussions about the breaking change). If the error is indistinguishable from other IO errors then I would prefer to have some api which can be used to detect support:

• While docs on the class/methods certainly are good, they aren't as actionable as me being able to explicitly detect the situation and give the user an error message that tells them they need to run in an elevated console.
• The people experiencing this issue in the wild will probably not be the people who actually wrote the code using the Link apis - and thus they probably haven't read the docs for Link to know they need to do this.

[zanderso]

A cursory glance at the Windows docs indicates that symlink creation failure due to lack of privileges will give ERROR_PRIVILEGE_NOT_HELD. I suspect when privileges are held, but there is a regular file system access error, we will continue to get ERROR_ACCESS_DENIED. It should be possible to bubble this up in the OS Error to the caller.

[bkonyi]

Friendly bump. @mit-mit, what are your thoughts?

[zanderso]

I discussed offline with @mit-mit, we agreed that we can proceed with this change, but we'll need to make a breaking change announcement on appropriate mailing lists before landing.

[bkonyi]

Fixed as of this commit.