IOError from cp with large file on Windows

[davidanthoff]

This is the error:

   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.0.3 (2018-12-18)
 _/ |\__'_|_|_|\__'_|  |  Official https://julialang.org/ release
|__/                   |

julia> cp("uniform_data_string.csv", "foo2.csv")
ERROR: IOError: sendfile: Unknown system error -756883004 (Unknown system error -756883004)
Stacktrace:
 [1] uv_error at .\libuv.jl:85 [inlined]
 [2] sendfile at .\filesystem.jl:111 [inlined]
 [3] sendfile(::String, ::String) at .\file.jl:749
 [4] #cp#12(::Bool, ::Bool, ::Function, ::String, ::String) at .\file.jl:336
 [5] cp(::String, ::String) at .\file.jl:330
 [6] top-level scope at none:0

I get this after I have rebooted my system, so I don't think this is related to some lingering stuff.

The file I'm trying to copy is relatively large:

-a----        1/15/2019  10:52 AM     3538084292 uniform_data_string.csv

The code actually does copy the file, as far as I can tell.

I can successfully copy files that are smaller than 2GB, so maybe there is something somewhere that makes this fail for files larger than 2GB?

The same code works on Linux and Mac.

[nalimilan]

Related: #14574.

[JockLawrie]

Hi there,
I am also experiencing this on Windows 10 with a 2.3GB file.
Using Julia 1.4.1.

[StefanKarpinski]

Seems like we should really do something about this. @vtjnash, I assume this is a libuv issue under the hood? Any way we can work around it and/or fix this in libuv?

[yakir12]

I have the same issue but on a mac (let me know if I should open/post on a different issue).

$ sw_vers
ProductName:    Mac OS X
ProductVersion: 10.15.4
BuildVersion:   19E287
$ uname -a
Darwin Ayses-MacBook-Pro.local 19.4.0 Darwin Kernel Version 19.4.0: Wed Mar  4 22:28:40 PST 2020; root:xnu-6153.101.6~15/RELEASE_X86_64 x86_64

julia> versioninfo()
Julia Version 1.4.2
Commit 44fa15b150* (2020-05-23 18:35 UTC)
Platform Info:
  OS: macOS (x86_64-apple-darwin18.7.0)
  CPU: Intel(R) Core(TM) i7-8850H CPU @ 2.60GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-8.0.1 (ORCJIT, skylake)

All the files that produced the same error were 3.1GB (while I had no errors for smaller files).

[jaakkor2]

I get the same stacktrace with Julia-1.5.0-rc1 (x86_64) both on Windows and MacOS when copying a file with 2^31 bytes

julia> Base.Filesystem.cp(filename_big, filename_big_copy)
ERROR: IOError: sendfile: Unknown system error -2147483647 (Unknown system error -2147483647)
Stacktrace:
 [1] uv_error at .\libuv.jl:97 [inlined]
 [2] sendfile at .\filesystem.jl:119 [inlined]
 [3] sendfile(::String, ::String) at .\file.jl:915
 [4] cp(::String, ::String; force::Bool, follow_symlinks::Bool) at .\file.jl:351
 [5] cp(::String, ::String) at .\file.jl:345
 [6] top-level scope at REPL[46]:1

on

ProductName:	Mac OS X
ProductVersion:	10.11.6
BuildVersion:	15G22010

and

Microsoft Windows Version 1909 (OS Build 18363.815)

[jaakkor2]

Problem still persists with Julia v1.6.0-beta1.0 on both Win and Mac.

[giordano]

My understanding is that this is the upstream fix of this bug: libuv/libuv#2533

[Yujie-W]

I have a similar problem as well:

ERROR: IOError: sendfile: Unknown system error -205546438 (Unknown system error -205546438)
Stacktrace:
  [1] uv_error
    @ ./libuv.jl:97 [inlined]
  [2] sendfile(dst::Base.Filesystem.File, src::Base.Filesystem.File, src_offset::Int64, bytes::Int64)
    @ Base.Filesystem ./filesystem.jl:119
  [3] sendfile(src::String, dst::String)
    @ Base.Filesystem ./file.jl:960
  [4] cp(src::String, dst::String; force::Bool, follow_symlinks::Bool)
    @ Base.Filesystem ./file.jl:355
  [5] cp
    @ ./file.jl:349 [inlined]
... ...

Julia version 1.6.2

When I try the same thing using Julia 1.5.1, it is working

[evetion]

A colleague of mine just had this error on Windows and Julia 1.9.3 for large files (>3GB). We worked around it now by using run(`cp from to`).

I can make a PR to warn not to use cp() for files > 2GB to https://docs.julialang.org/en/v1/base/file/#Base.Filesystem.cp?

[jaakkor2]

On Julia 1.10.0-beta2 on Windows 10 (10.0.19045), copying files sized

• 1...2^31-1 : ok
• 2^31...2^32-2 : IOError: sendfile: Unknown system error -214748364 (Unknown system error -214748364) [error code is filesize - 2^32]
• 2^32-1 : crash: Assertion failed: req->result != -1, file src/win/fs.c, line 2118
• 2^32... : spins the CPU, does not finish, no error, no crash

Not just copying files, but mving files across filesystems as well.

[evetion]

A colleague of mine just had this error on Windows and Julia 1.9.3 for large files (>3GB). We worked around it now by using run(`cp from to`).

To be precise, it's
run(`cmd /c copy $from $to`)
and it doesn't accept characters like + in filenames, while cp (and the filesystem do).

[vtjnash]

That is just because cmd doesn't parse command line arguments as normal. There is an option (windows_verbatim) to fix your program and a complete discussion of it with examples in the Base.shell_escape_wincmd docstring.

[bidhya]

I have observed the same error more than a few times on Linux (Slurm HPC system) when copying file larger than 3GB. The file (.tar.gz) will be copied successfully, but Julia will crash at this point with this error message.
Julia version: Julia-1.10.0

[StefanKarpinski]

@bidhya: if you wouldn't mind, opening a separate issue about that would be great since it's clearly not the same issue.