Windows initiative 2018

[djdv]

This issue serves as an anchor to monitor the broader topic of go-ipfs on Windows.
Here I will be highlighting the current issues being focused on and the PR efforts to resolve them. Feel free to add anything to this topic that relates to go-ipfs on Windows as a whole.

Build problems

• make
  Fails to detect gx and gx-go properly
  make: check_gx & dist_get issues on Windows #4510
  Resolved in: Fix/win make #4682

• gx
  Fails to detect subtools properly (gx installs deps to wrong directory)
  inconsistent argument handling whyrusleeping/gx#153
  Resolved in: Fix/windows subtool suffix whyrusleeping/gx#154

• Build documentation
  A lot of people seem to have trouble building on Windows with the current document. I've proposed a revision here: Change Windows build documentation #4691

interface problems

• Pipes
  Commands like add and files write don't detect pipes on Windows #3266
  Resolved in: allow stdin on Windows go-ipfs-cmds#74

• get doesn't support NUL
  currently not tracked
  This will require fixes in Go itself and the tar-utils package
  os: Stat returns error with 'nul' on Windows golang/go#24482
  os: MkdirAll returns error with 'NUL' on Windows golang/go#24556
  Sanitize output paths whyrusleeping/tar-utils#2
  Resolved in: get: Handle symlink creation and sanitize paths on Windows #4956 (not yet merged)
  Resolved independently in tar-utils, rolled in via Extract: thirdparty/tar #4857

• Color/Colour output
  cli: Color on windows doesn't work #3068
  Windows: cli colour codes #2124
  Resolved in: Fix color output on Windows go-log#21 + routing fixes/refactor #5007

• File system character restrictions
  ipfs get cross platform paths #2013
  Resolved on Windows with: Sanitize output paths whyrusleeping/tar-utils#2 (merged) + get: Handle symlink creation and sanitize paths on Windows #4956 (not yet merged)

Windows Tests & CI
We need to pass tests on Windows so new issues can be caught by the CI before becoming a problem.
When Windows is skipped for testing, we suffer from regressions such as:
#4750
#4512
#3676

In addition to fixing existing tests, new tests should be made for testing platform specific behaviors.
Please comment about platform specific edge cases that should be considered.
For example, considering UNC paths (#1933 (comment))
Long prefix paths (https://github.com/ipfs/go-ipfs/pull/3328/files#diff-7ba0a8b1e26b8ed185c4824d4ee84164R503)
How :. behaves (#3328)
File system restrictions (#2013)
etc.

Feature parity with other platforms
The inability to pipe data, get certain data, mount IPFS, have readable output, etc. only pertain to Windows, platform native solutions for these features will have to be thought about, discussed, and implemented. The future issues and PRs will back-reference this issue so that development and discussion can be easily followed.

• mount is not implemented
  Windows mount support #5003
• Pubsub is not reliable
  pubsub sometimes hangs on Windows #4778
  This ended up being a js-ipfs issue.
  As of 2018.12.31 I have not personally experienced or heard reports of instability in regards to pubsub in go-ipfs.

Everything else
There are many outstanding and outdated issues
https://github.com/ipfs/go-ipfs/issues?utf8=%E2%9C%93&q=is%3Aissue+is%3Aopen+amd64%2Fwindows+
They need to be reviewed and tagged
https://github.com/ipfs/go-ipfs/issues?q=is%3Aissue+is%3Aopen+label%3Awindows
Ones that are no longer relevant (already fixed) need to be marked as closed.

[djdv]

Pipes and stdin seem to be partially working, it's fine if the daemon is running but gets caught up on the lockfile otherwise, needs investigating, tests, etc.

[djdv]

Back-referencing other Windows related patches that are helpful to look at.

Building on Windows documentation [wip]
#4691

Building with make now works reliably (this should be in 0.4.14+)
#4682

gx correctly handles subtools on Windows (will be in the next release of gx)
whyrusleeping/gx#154.
For now, if you depend on this, you should build and install gx from the master branch, set IPFS_GX_USE_GLOBAL in your environment, and proceed to build go-ipfs using the latest version of gx and gx-go.

Windows was returning an ambiguous "Access is denied" when trying to fetch gx deps
whyrusleeping/gx#163

[Stebalien]

Pipes and stdin seem to be partially working, it's fine if the daemon is running but gets caught up on the lockfile otherwise, needs investigating, tests, etc.

So, on Linux, the lockfile is automatically released when the process closes (by the OS). Unfortunately, windows appears to use the fallback which needs to be manually released. We may not be doing that somewhere.

Personally, I'd add some Windows specific logic to the lock package (https://github.com/camlistore/go4/tree/master/lock). Make it "succeed" if it can open the lock file with exclusive access, regardless of whether or not it already existed. Even better, have it mark the lock file for deletion on close.

[Kubuxu]

As @Stebalien said, it is same on Linux. The root of the issue is that shells start all processes in the pipe at the same time, no matter the pipe order so ipfs commands compete for repo lock.

[Stebalien]

Personally, I'd like to fix this by starting the daemon in the background in a special mode that:

1. Disables the DHT server.
2. Shuts down after some amount of idle time.
3. Has a really low connection limit?

[djdv]

I've modified the go4 lock library, adding Windows calls to flag the lock for deletion by the OS when all handles are closed, and to not fail immediately when a lock file exists, but I'm having trouble blocking on successive lock attempts after that.
I'm using the LockFileEx systemcall with flags LOCKFILE_EXCLUSIVE_LOCK and without LOCKFILE_FAIL_IMMEDIATELY, which should put additional lock calls in a blocked state.
However I'm running into issues with that right now. Currently calls to lock just succeed 100% of the time instead of blocking, I'm unsure why at the moment.

[Stebalien]

@djdv and I chatted over IRC. We've agreed to:

1. Match what happens on linux and not block, just fail. However, @djdv's still going to fix the lock library to automatically clean up the lock file on close.
2. Handle the pipe issue separately.

[djdv]

Lock reference: go4org/go4#31

[djdv]

ipfs/go-log#21

[djdv]

Making an effort to deal with file system restrictions:
#4850

[Stebalien]

@djdv we should also look into #3971. Users hit this all the time. We may need to set some kind of "no index" option on the datastore.

[djdv]

I'm going to collect information on #3971 here.
While this doesn't work 100% of the time this is a good way to instigate the problem: #3971 (comment)
I've been running repo gc and doing that to instigate this. I'm using a near full 5900RPM HDD for testing, people seem to not have issues on faster drives.

In addition sometimes this happens: #4527
I set my root to G:\tmp\.ipfs and got a lot of garbage in G:\tmp\ which is my working directory. This is bad because something is feeding some call with garbage path names. The garbage content looks like valid block data just with mangled names.
Memory corruption is suspected to be responsible for this, but it's not certain as the filepaths reported via errors are correct, and not mangled.

14:52:13.053 ERROR flatfs: Err in Get() PathError err:&os.PathError{Op:"open", Path:"G:\tmp\.ipfs\blocks\AN\CIQFG5PESHPZCNK2T6BEKAT4AXP7TFIHDHQ36NKX6YXKCHYM5J6VANY.data", Err:0x20} -> open G:\tmp.ipfs\blocks\AN\CIQFG5PESHPZCNK2T6BEKAT4AXP7TFIHDHQ36NKX6YXKCHYM5J6VANY.data: The process cannot access the file because it is being used by another process.

Instigating this while running ProcMon is causing ProcMon to crash for me which isn't helping.
Running a separate tool I noticed several instances of buffer overflows being reported but I'm unsure as to where they originate in the Go source. In addition to this there are over-reads(STATUS_END_OF_FILE) for NtReadFile and the expected STATUS_ACCESS_DENIED for NtSetInformationFile which looks like it's trying to rename a file.

Finding out why and where filenames are getting corrupted seems like the current goal as it may be the root cause of this.

[Stebalien]

I believe this was fixed (accidentally) by ipfs/go-ds-flatfs#34. go-os-rename wasn't properly keeping a temporary UTF16 buffers alive here: https://github.com/jbenet/go-os-rename/blob/3ac97f61ef67a6b87b95c1282f6c317ed0e693c2/rename_windows.go#L21-L40

Basically, it was taking the src and dst paths, converting them to UTF16 paths, and then converting those to uintptrs. However, this is only safe in two cases:

1. If the the uintptr is constructed directly as an argument to the call to Syscall due to super-special go magic.
2. If the buffers were marked as live (using runtime.KeepAlive) after it's last use.

Othewise, go will consider these buffers "free" after these lines:

https://github.com/jbenet/go-os-rename/blob/3ac97f61ef67a6b87b95c1282f6c317ed0e693c2/rename_windows.go#L31-L32

Congrats @djdv, you managed to fix a bug before looking into it (probably)!