upgrade to libuv1.9.0-fork #12266

vtjnash · 2015-07-22T15:06:22Z

like the title says. we've been sitting on libuv-0.11.26 for awhile now, while libuv 1.x has some nice toys (in no particular order):

https://github.com/libuv/libuv/blob/v1.x/ChangeLog
uv_os_homedir
uv_fs_scandir_next
additional work on threading
improved uv_timer_t precision on windows
uv_fileno
uv_fs_mkdtemp
ability to replace the memory allocator
uv_fs_access

ViralBShah · 2015-07-22T18:24:27Z

@kpamnany Would we get the uv fixes you needed for threading with this update?

tkelman · 2015-07-22T23:15:53Z

src/init.c

@@ -960,7 +960,9 @@ static void jl_resolve_sysimg_location(JL_IMAGE_SEARCH rel)
    if (path_size >= PATH_MAX) {
        jl_error("fatal error: jl_options.julia_bin path too long");
    }
-    jl_options.julia_bin = strdup(free_path);
+    jl_options.julia_bin = malloc(path_size+1);


(char*) malloc

tkelman · 2015-07-22T23:35:43Z

Nice work here, didn't notice you went and did this. Still need to eventually figure out how to upstream what we need pipe-wise so we won't always need a fork, but this is a great start.

Thread affinity and detach are still open - libuv/libuv#280

tkelman · 2015-07-23T00:03:43Z

damn, this isn't starting under mintty at all

vtjnash · 2015-07-23T00:05:36Z

hm, worked for me?

tkelman · 2015-07-23T00:06:08Z

win10 vs win7?

tkelman · 2015-07-23T00:39:30Z

If I hit enter a few times, I end up getting this:

$ usr/bin/julia


ERROR: readcb: operation canceled (ECANCELED)
 in yieldto at task.jl:21
 in wait at task.jl:309
 in wait at task.jl:225
 in wait_readnb at stream.jl:359
 in eof at stream.jl:56
 in _start at client.jl:448

tkelman · 2015-07-23T00:41:00Z

attaching gdb to julia-debug:

$ gdb -p 7124
GNU gdb (GDB) 7.8
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-pc-cygwin".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word".
Attaching to process 7124
[New Thread 7124.0x1718]
[New Thread 7124.0x21e8]
[New Thread 7124.0x1c10]
[New Thread 7124.0x1f04]
[New Thread 7124.0x10b8]
[New Thread 7124.0x1f38]
[New Thread 7124.0x2208]
[New Thread 7124.0x1610]
[New Thread 7124.0x176c]
[New Thread 7124.0x1570]
[New Thread 7124.0x434]
[New Thread 7124.0x1f40]
Reading symbols from /home/Tony/julia/usr/bin/julia-debug.exe...done.
DW_FORM_strp pointing outside of .debug_str section [in module /home/Tony/julia/usr/lib/julia/sys-debug.DLL]
0x000000007735ce31 in ntdll!DbgBreakPoint () from /cygdrive/c/Windows/SYSTEM32/ntdll.dll
(gdb) thread 1
[Switching to thread 1 (Thread 7124.0x1718)]
#0  0x000000007735ef7a in ntdll!ZwRemoveIoCompletionEx ()
   from /cygdrive/c/Windows/SYSTEM32/ntdll.dll
(gdb) bt
#0  0x000000007735ef7a in ntdll!ZwRemoveIoCompletionEx ()
   from /cygdrive/c/Windows/SYSTEM32/ntdll.dll
#1  0x000007fefd11542f in KERNELBASE!GetQueuedCompletionStatusEx ()
   from /cygdrive/c/Windows/system32/KERNELBASE.dll
#2  0x000000007713c449 in KERNEL32!GetQueuedCompletionStatusEx ()
   from /cygdrive/c/Windows/system32/kernel32.dll
#3  0x000000006eab4b41 in uv_poll_ex (loop=0xffffffffffffffff, timeout=4294967295)
    at src/win/core.c:281
#4  0x000000006eab512a in uv_run (loop=0x6fa9f980 <default_loop_struct>, mode=UV_RUN_ONCE)
    at src/win/core.c:350
#5  0x000000006ea7f375 in jl_run_once (loop=0x6fa9f980 <default_loop_struct>) at jl_uv.c:106
#6  0x00000003191c9ecb in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

kpamnany · 2015-07-23T01:06:19Z

Yeah, as @tkelman pointed out, the PR for thread affinity and detach is still open on upstream libuv: libuv/libuv#280. So, no.

tkelman · 2015-07-28T08:48:46Z

Worth noting, I can reproduce the freeze on this branch using the Windows Server 2012 R2 image from contrib/windows/Vagrantfile. Have to start a GUI session of Virtualbox to use mintty though, I don't think vagrant ssh will do the right thing.

tkelman · 2015-07-31T12:34:00Z

Looks like we may need to revert libuv/libuv@1e59ab1. That unfortunately doesn't fix the mintty freeze for me though, so there are probably other changes needed either in libuv or Julia. Ref libuv/libuv#264

vtjnash · 2015-09-13T01:47:03Z

i've bumped this and included a possible fix for the pty detection issue. can you check whether jl_ispty is able to compute the right answer

tkelman · 2015-09-13T16:34:55Z

No change, still broken in mintty. The test failures are new I think.

vtjnash · 2015-11-01T20:39:34Z

"switch our flag to use enum value 2." JuliaLang/libuv#27

vtjnash · 2015-11-01T22:06:16Z

updated to v1.7.5 now

tkelman · 2015-11-01T22:20:07Z

deps/libuv.version

-LIBUV_BRANCH=julia-uv0.11.26
-LIBUV_SHA1=28f5f06b5ff6f010d666ec26552e0badaca5cdcd
+LIBUV_BRANCH=julia-v1.7.5
+LIBUV_SHA1=80fe49af0f19f82da811129220c9bdeb13b99ebb


checksum filename doesnt match sha

tkelman · 2015-11-01T22:54:41Z

Still doesn't start in mintty, though underlying cause might be the same as the test failures?

$ usr/bin/julia # freezes until I hit another key

ERROR: readcb: operation canceled (ECANCELED)
 in yieldto at task.jl:71
 in wait at task.jl:371
 in wait at task.jl:286
 in wait_readnb at stream.jl:357
 in eof at stream.jl:1210
 in _start at client.jl:414

Keno · 2015-12-28T16:47:50Z

@vtjnash did you have any idea what the problem was?

yuyichao · 2015-12-29T03:25:52Z

I've rebased this PR on the current master (with the original commit backed up at yyc/libuv1.6.1-bak) and fixed a few easy failing tests due to rearrangement on master.

Apart from the failing tests, the output at the REPL also seems strange. The frist line is aligned with the end of the input line on this branch.

julia> []
         0-element Array{Any,1}

julia> [[]]
           WARNING: [a] concatenation is deprecated; use collect(a) instead
 in depwarn at ./deprecated.jl:73
 in oldstyle_vcat_warning at ./abstractarray.jl:29
 in vect at ./abstractarray.jl:32
 in eval at ./boot.jl:265
while loading no file, in expression starting on line 0
0-element Array{Any,1}

julia> println(1)
                 1

tkelman · 2015-12-29T03:53:00Z

deps/Makefile

@@ -769,6 +769,8 @@ UV_OBJ_TARGET := $(build_libdir)/libuv.la
 UV_CFLAGS :=
 ifeq ($(USEMSVC), 1)
 UV_CFLAGS += -DBUILDING_UV_SHARED
+else
+UV_CFLAGS += -std=gnu99
 endif


I have a more recent commit on the main branch we use that may make this unnecessary

tkelman · 2015-12-29T03:53:53Z

deps/libuv.version

-LIBUV_BRANCH=julia-uv0.11.26
-LIBUV_SHA1=030481e9d659fd46702ab747caf2cbbe19d537ba
+LIBUV_BRANCH=julia-v1.7.5
+LIBUV_SHA1=80fe49af0f19f82da811129220c9bdeb13b99ebb


still does not match checksum file name

Should this be the commit SHA?

Right. But you need to do make NO_GIT=1 -C deps configure-libuv to generate the right checksums for no-git builds.

Like this?

yuyichao% cat deps/checksums/libuv-80fe49af0f19f82da811129220c9bdeb13b99ebb.tar.gz/md5 8fa6ed4dc950d3f4445a42c4b28724f9

yeah, the sha1 in the checksum filename should match the git commit sha1

vtjnash · 2015-12-29T06:02:49Z

The alignment is probably the change to the raw mode to be really unmodifying (previously, we probably set the ONLCR flag to fix \n on output and replace with \r)

tkelman · 2016-05-04T02:26:31Z

still no dice inside mintty (that why you cancelled?)

  | | |_| | | | (_| |  |  Version 0.5.0-dev+3869 (2016-05-04 01:16 UTC)
 _/ |\__'_|_|_|\__'_|  |  (HEAD detached at origin/jn/libuv1.6.1)/0c0b641* (fork: 1 commits, 0 days)
|__/                   |  x86_64-w64-mingw32

julia> fERROR: ArgumentError: stream is closed or unusable
 in check_open(::Base.TTY) at .\stream.jl:254
 in raw!(::Base.Terminals.TTYTerminal, ::Bool) at .\Terminals.jl:121
 in prompt!(::Base.Terminals.TTYTerminal, ::Base.LineEdit.ModalInterface, ::Base.LineEdit.MIState) at .\LineEdit.jl:1615
 in run_interface(::Base.Terminals.TTYTerminal, ::Base.LineEdit.ModalInterface) at .\LineEdit.jl:1565
 in run_frontend(::Base.REPL.LineEditREPL, ::Base.REPL.REPLBackendRef) at .\REPL.jl:881
 in run_repl(::Base.REPL.LineEditREPL, ::Function) at .\REPL.jl:166
 in _start() at .\client.jl:361

vtjnash · 2016-05-04T02:30:45Z

yeah, git made a rebase error and changed the meaning of one of the libuv commits. it only mattered if you can type faster than the OS can context switch to its thread pool though :P

tkelman · 2016-05-04T04:49:47Z

Ah, much better now. Basics seem to work so far. Something during stream.jl confuses the bootstrap printing causing it to look like it's frozen until it gets to the very end, but I guess that's livable.

(p.s. is any of libuv/libuv#852 relevant for things that are dealt with on our fork?)

vtjnash · 2016-05-04T05:02:22Z

i'm not sure anyone has been giving cmd.exe usage much testing since mintty is so much better, so that issue wouldn't apply

tkelman · 2016-05-04T08:15:05Z

Only for building and working with source builds. #7267 still had some open issues with BinDeps having trouble, so users of binaries generally use the native console. Windows 10's native console isn't that bad, but we'll need to support Windows 7 for some time. I should probably keep my Windows 7 laptop around and test with it more often.

vtjnash force-pushed the jn/libuv1.6.1 branch from f27d7dd to 5ded40d Compare July 22, 2015 15:43

tkelman reviewed Jul 22, 2015
View reviewed changes

tkelman mentioned this pull request Jul 31, 2015

Tag UUID v0.0.5 JuliaLang/METADATA.jl#3000

Merged

vtjnash mentioned this pull request Jul 31, 2015

tilde expansion in path names #1136

Closed

tkelman mentioned this pull request Aug 10, 2015

WIP: merge UUID.jl into Base #12531

Merged

vtjnash force-pushed the jn/libuv1.6.1 branch from 5ded40d to e762b94 Compare September 13, 2015 01:40

vtjnash force-pushed the jn/libuv1.6.1 branch from e762b94 to 2f41870 Compare November 1, 2015 22:05

vtjnash changed the title ~~WIP: upgrade to libuv1.6.1-fork~~ upgrade to libuv1.6.1-fork Nov 1, 2015

vtjnash changed the title ~~upgrade to libuv1.6.1-fork~~ upgrade to libuv1.7.5-fork Nov 1, 2015

tkelman reviewed Nov 1, 2015
View reviewed changes

tkelman mentioned this pull request Dec 12, 2015

Julia fails to build on arm64 due to missing __NR_epoll_wait #14373

Closed

yuyichao force-pushed the jn/libuv1.6.1 branch from 2f41870 to 4f153dc Compare December 29, 2015 03:18

tkelman reviewed Dec 29, 2015
View reviewed changes

yuyichao force-pushed the jn/libuv1.6.1 branch from 4f153dc to 7de33a6 Compare December 29, 2015 05:47

vtjnash force-pushed the jn/libuv1.6.1 branch 3 times, most recently from ef48b47 to 0c0b641 Compare May 4, 2016 01:17

vtjnash force-pushed the jn/libuv1.6.1 branch from 0c0b641 to ec1e03f Compare May 4, 2016 02:38

upgrade to libuv1.9.0-fork

d9cdda6

vtjnash force-pushed the jn/libuv1.6.1 branch from ec1e03f to d9cdda6 Compare May 4, 2016 02:45

pao changed the title ~~upgrade to libuv1.7.5-fork~~ upgrade to libuv1.9.0-fork May 4, 2016

vtjnash mentioned this pull request May 5, 2016

Merge ability to create sockets early from upstream JuliaAttic/libuv-archive#32

Closed

vtjnash merged commit d9cdda6 into master May 5, 2016

yuyichao deleted the jn/libuv1.6.1 branch May 5, 2016 03:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

upgrade to libuv1.9.0-fork #12266

upgrade to libuv1.9.0-fork #12266

vtjnash commented Jul 22, 2015

ViralBShah commented Jul 22, 2015

tkelman Jul 22, 2015

tkelman commented Jul 22, 2015

tkelman commented Jul 23, 2015

vtjnash commented Jul 23, 2015

tkelman commented Jul 23, 2015

tkelman commented Jul 23, 2015

tkelman commented Jul 23, 2015

kpamnany commented Jul 23, 2015

tkelman commented Jul 28, 2015

tkelman commented Jul 31, 2015

vtjnash commented Sep 13, 2015

tkelman commented Sep 13, 2015

vtjnash commented Nov 1, 2015

vtjnash commented Nov 1, 2015

tkelman Nov 1, 2015

tkelman commented Nov 1, 2015

Keno commented Dec 28, 2015

yuyichao commented Dec 29, 2015

tkelman Dec 29, 2015

tkelman Dec 29, 2015

yuyichao Dec 29, 2015

tkelman Dec 29, 2015

yuyichao Dec 29, 2015

tkelman Dec 29, 2015

vtjnash commented Dec 29, 2015

tkelman commented May 4, 2016

vtjnash commented May 4, 2016

tkelman commented May 4, 2016

vtjnash commented May 4, 2016

tkelman commented May 4, 2016

upgrade to libuv1.9.0-fork #12266

upgrade to libuv1.9.0-fork #12266

Conversation

vtjnash commented Jul 22, 2015

ViralBShah commented Jul 22, 2015

Choose a reason for hiding this comment

tkelman commented Jul 22, 2015

tkelman commented Jul 23, 2015

vtjnash commented Jul 23, 2015

tkelman commented Jul 23, 2015

tkelman commented Jul 23, 2015

tkelman commented Jul 23, 2015

kpamnany commented Jul 23, 2015

tkelman commented Jul 28, 2015

tkelman commented Jul 31, 2015

vtjnash commented Sep 13, 2015

tkelman commented Sep 13, 2015

vtjnash commented Nov 1, 2015

vtjnash commented Nov 1, 2015

Choose a reason for hiding this comment

tkelman commented Nov 1, 2015

Keno commented Dec 28, 2015

yuyichao commented Dec 29, 2015

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vtjnash commented Dec 29, 2015

tkelman commented May 4, 2016

vtjnash commented May 4, 2016

tkelman commented May 4, 2016

vtjnash commented May 4, 2016

tkelman commented May 4, 2016