Frequent Win32 segfaults on AppVeyor #10045

tkelman · 2015-02-03T05:51:30Z

This has been happening a bunch lately. Sometimes during bootstrap, sometimes in the tests. Not sure what's up.

tkelman · 2015-02-04T04:57:28Z

Possibly related, on 64 bit the tests have been intermittently taking much longer than usual. Linalg, fft, dsp, and parallel are occasionally several times slower than usual.

catawbasam · 2015-02-05T09:50:30Z

Just hit one in:
https://ci.appveyor.com/project/StefanKarpinski/julia/build/1.0.2247/job/rufdccmy70le2oti

At line 614 we have

Exception: EXCEPTION_ACCESS_VIOLATION at 0x693eac6a -- utf8proc_NFKC at C:\projects\julia\usr\bin\libjulia.dll (unknown line)

cc: @stevengj

tkelman · 2015-02-05T20:59:26Z

That function seems to always show up in the backtraces, not sure why. I suspect the problem is something unrelated to utf8proc. Maybe gensym related (based on merge timing of when this started getting frequent), maybe related to the 32-bit complex issues that have been happening on Linux (#10027).

tkelman · 2015-02-06T06:19:14Z

I can reproduce this locally, FWIW. I'm trying with various combinations of julia-debug, removing sys.dll, etc to see if I can get any more information here.

tkelman · 2015-02-06T09:16:44Z

@vtjnash any idea what's up with this? https://gist.github.com/tkelman/7e6aa9bf14d382ba5b65

tkelman · 2015-02-07T00:03:49Z

Looks like julia-debug.exe has been failing the spawn test ever since the changes to julia_init, and that's true on both win32 and win64 so I'll open a separate issue. Makes this issue harder to debug at the moment though.

tkelman · 2015-02-10T09:54:54Z

Unfortunately #10145 didn't appear to help here. Still getting segfaults.

… 4GB of ram

ihnorton · 2015-02-19T22:26:44Z

Is there a reliable way to reproduce this locally?

tkelman · 2015-02-19T22:29:19Z

Reliable? Not really. Run the tests a bunch of times and it will happen locally though.

tkelman · 2015-02-19T22:30:39Z

Or make clean && make -j4 a bunch of times to get the crash while building the sysimg to happen

ihnorton · 2015-02-22T18:17:08Z

@tkelman maybe we could run UserDump on appveyor: http://support.microsoft.com/kb/241215
Not sure whether such dumps would be usable in gdb or only in windbg.

tkelman · 2015-02-23T12:46:18Z

I'm going to tentatively call this fixed by #10275. Will reopen if it comes back.

catawbasam · 2015-02-23T12:57:33Z

👏

timholy · 2015-02-23T18:22:59Z

Sad news: https://ci.appveyor.com/project/StefanKarpinski/julia/build/1.0.2766/job/oxcc6ghee6br08i9

(Believe me, I'm as disappointed as you.)

tkelman · 2015-02-25T05:59:49Z

What's odd is I ran the tests 10 times in a row on your branch from #10275 and had no problems locally. Either some corruption can randomly happen at compile/bootstrap time, or something else changed on master that was not included in the history of that branch, or we're just unlucky.

timholy · 2015-02-25T10:20:19Z

Well, that's good news! So maybe there are multiple issues, and #10275 fixed one of them.

Feel free to close again.

tkelman · 2015-02-25T10:31:20Z

Maybe. We are still getting a lot of failures, so it would appear closing this was premature.

tkelman · 2015-02-25T12:01:35Z

Based on when this initially started getting frequent, my best guesses at the original cause are either a0266d4 or 5a4e1f8. The first 2 linked jobs up top were PR builds that may have been merging into a much newer state of master than their build number would indicate, depending on how long they were waiting in the queue for. The 3rd link was a doc commit to master (4b03533) and is probably the most useful point in the history to start looking from.

tkelman · 2015-02-25T20:55:41Z

@vtjnash I'm increasingly convinced this is gensym-related. I ran the tests repeatedly, several dozen times, on 51d5412, without any problems. 77d1394 appeared to be okay, but df0e099 and following commits segfault (sometimes during atomic_flag_test_and_set_explicit in libstdc++) during bootstrap or intermittently in the tests.

tkelman · 2015-02-26T13:14:07Z

@timholy you didn't happen to clone Jameson too while you were at it, did ya?

On the gensym merge commit 2d23b6f I get an odd broken module when running win32 julia-debug.exe on the arrayops test: https://gist.github.com/tkelman/8d0ada9bceacc36f3e9d

vtjnash · 2015-02-26T14:54:28Z

that was fixed in e41a507

tkelman · 2015-02-26T18:18:21Z

Yeah, I'm running a bunch of things at that commit right now. Running the tests except for spawn and repl (which were hitting #10111) using julia-debug in serial looked okay. This might be a little like #7942 and only happen when tests run in parallel.

tkelman · 2015-03-02T22:58:34Z

Can reproduce the problem at e41a507, even with julia-debug in serial when run enough times. Can't at 51d5412 (left things going overnight, repeatedly, several dozen more times - all fine).

vtjnash · 2015-03-03T01:04:36Z

the gensym commit is almost entirely a front-end compiler optimization, so it should have little effect on the final code generated. it should also have no win32-specific dependency, so we should see the same failures on linux32. you are still seeing a broken module after e41a507?

tkelman · 2015-03-03T01:16:35Z

Sorry, no, I should've been more specific. At e41a507 I do not see the broken module, but I do see intermittent segfaults that look very similar to what's been happening on AppVeyor.

It may be the case that this would also happen on 32 bit Linux, but we run the test suite less often on that architecture. Here's a possibly-related segfault on the 32 bit Linux buildbot: http://buildbot.e.ip.saba.us:8010/builders/build_ubuntu14.04-x86/builds/789/steps/shell_2/logs/stdio

My PR to turn on a 32 bit Linux build on Travis (#9153) works now and could be merge-able.

jakebolewski · 2015-03-12T02:05:18Z

Another instance?
https://ci.appveyor.com/project/StefanKarpinski/julia/build/1.0.3138/job/91g5xor5mrki8gka

tkelman · 2015-03-12T02:09:30Z

Certainly looks like it. The backtrace isn't always so ridiculously long, but I've seen a few different cases lately where it was.

This does happen locally, and with enough persistence I have even managed to catch it in gdb once or twice. But wasn't able to get any backtrace at all.

tkelman · 2015-03-12T03:44:23Z

Here's a tiny piece of backtrace from a local segfault at the parallel test:

     * parallel            [New Thread 10964.0x1fd8]
[New Thread 10964.0x1f3c]

Program received signal SIGSEGV, Segmentation fault.
0x0165e2c4 in llvm::APInt::initFromArray(llvm::ArrayRef<unsigned long long>)
    () from D:\cygwin64\home\Tony\julia32\usr\bin\libjulia-debug.dll
(gdb) bt
#0  0x0165e2c4 in llvm::APInt::initFromArray(llvm::ArrayRef<unsigned long long>) () from D:\cygwin64\home\Tony\julia32\usr\bin\libjulia-debug.dll
#1  0x71f152a0 in ?? ()
#2  0xac500415 in ?? ()
#3  0xe8057468 in ?? ()
#4  0xffedf097 in ?? ()
#5  0x5590c3c9 in ?? ()
#6  0x00000000 in ?? ()

tkelman · 2015-03-13T10:03:34Z

at 7e8b10c I was getting this repeatedly during bootstrap, same basic first-line of backtrace

precompile.jl
[New Thread 24200.0x6464]
[New Thread 24200.0x5090]

Program received signal SIGSEGV, Segmentation fault.
0x018ce2c4 in llvm::APInt::initFromArray(llvm::ArrayRef<unsigned long long>)
    () from D:\cygwin64\home\Tony\julia32\usr\bin\libjulia-debug.dll
(gdb) bt
#0  0x018ce2c4 in llvm::APInt::initFromArray(llvm::ArrayRef<unsigned long long>) () from D:\cygwin64\home\Tony\julia32\usr\bin\libjulia-debug.dll
#1  0x11863ff0 in ?? ()
#2  0x0f028118 in ?? ()
Backtrace stopped: previous frame inner to this frame (corrupt stack?)

tkelman · 2015-03-14T07:21:43Z

and, rebuilt with LLVM_DEBUG = 1

precompile.jl
[New Thread 20068.0x45bc]

Program received signal SIGSEGV, Segmentation fault.
0x627550e1 in llvm::APInt::initFromArray (this=0xd4dc68, bigVal=...)
    at /home/Tony/julia32/deps/llvm-3.3/lib/Support/APInt.cpp:93
(gdb) bt
#0  0x627550e1 in llvm::APInt::initFromArray (this=0xd4dc68, bigVal=...)
    at /home/Tony/julia32/deps/llvm-3.3/lib/Support/APInt.cpp:93
#1  0x627551a9 in llvm::APInt::APInt (this=0xd4dc68, numBits=32, bigVal=...)
    at /home/Tony/julia32/deps/llvm-3.3/lib/Support/APInt.cpp:108
#2  0x62069cec in julia_const_to_llvm (e=e@entry=0x11863ff8)
    at intrinsics.cpp:144
#3  0x6207cf71 in emit_unboxed (e=0x11863ff8, ctx=0xd4e280)
    at intrinsics.cpp:205
#4  0x6208d2d9 in emit_call_function_object (f=0xf9489c0, theF=theF@entry=
    0x7ee1cdc, theFptr=<optimized out>, specialized=true,
    args=args@entry=0x1111b25c, nargs=nargs@entry=6, ctx=ctx@entry=0xd4e280)
    at codegen.cpp:2554
#5  0x6207b60b in emit_call (expr=0x7ee1cdc, ctx=0xd4e280, arglen=7,
    args=0x1111b25c) at codegen.cpp:2645
#6  emit_expr (expr=expr@entry=0xfc25ae0, ctx=ctx@entry=0xd4e280,
    isboxed=isboxed@entry=true, valuepos=valuepos@entry=true,
    valuevar=valuevar@entry=0x0) at codegen.cpp:3147
#7  0x620851d2 in emit_assignment (bp=bp@entry=0x7eded58,
    r=r@entry=0xfc25ae0, declType=0xf72c780,
    isVolatile=isVolatile@entry=false, used=used@entry=true,
    ctx=ctx@entry=0xd4e280) at codegen.cpp:2900
#8  0x6207c825 in emit_assignment (ctx=0xd4e280, r=0xfc25ae0,
    l=<optimized out>) at codegen.cpp:2936
#9  emit_expr (expr=expr@entry=0xfc25ad0, ctx=ctx@entry=0xd4e280,
    isboxed=isboxed@entry=false, valuepos=valuepos@entry=false,
    valuevar=valuevar@entry=0x0) at codegen.cpp:3150
#10 0x6208f143 in _fu36___ZNSs4_Rep20_S_empty_rep_storageE ()
    at codegen.cpp:4662
#11 0x62091adb in to_function (li=li@entry=0x1111da78) at codegen.cpp:615
#12 0x62091bfb in jl_compile (f=f@entry=0xfc259a0) at codegen.cpp:768
#13 0x62052ae7 in jl_get_specialization (f=0xf03e8e0, types=0xfc252b0)
    at gf.c:1503
#14 0x62083d74 in emit_known_call (ff=<optimized out>, args=<optimized out>,
    nargs=<optimized out>, ctx=0xd4ea50, theFptr=0xd4e594, theF=0xd4e598,
    expr=0xfbe2460) at codegen.cpp:1897
#15 0x6207b95a in emit_call (expr=0xfbe2460, ctx=0xd4ea50, arglen=3,
    args=0x10464cac) at codegen.cpp:2595
#16 emit_expr (expr=expr@entry=0xfbe2460, ctx=ctx@entry=0xd4ea50,
    isboxed=isboxed@entry=true, valuepos=valuepos@entry=true,
    valuevar=valuevar@entry=0x0) at codegen.cpp:3147
#17 0x620851d2 in emit_assignment (bp=bp@entry=0x7ddc408,
    r=r@entry=0xfbe2460, declType=0xf036500,
    isVolatile=isVolatile@entry=false, used=used@entry=true,
    ctx=ctx@entry=0xd4ea50) at codegen.cpp:2900
#18 0x6207c825 in emit_assignment (ctx=0xd4ea50, r=0xfbe2460,
    l=<optimized out>) at codegen.cpp:2936
#19 emit_expr (expr=expr@entry=0xfbe2450, ctx=ctx@entry=0xd4ea50,
    isboxed=isboxed@entry=false, valuepos=valuepos@entry=false,
    valuevar=valuevar@entry=0x0) at codegen.cpp:3150
#20 0x6208f143 in _fu36___ZNSs4_Rep20_S_empty_rep_storageE ()
    at codegen.cpp:4662
#21 0x62091adb in to_function (li=li@entry=0x1111c9f8) at codegen.cpp:615
#22 0x62091bfb in jl_compile (f=f@entry=0xfbe0070) at codegen.cpp:768
#23 0x62052ae7 in jl_get_specialization (f=0xf284050, types=0xfbe0050)
    at gf.c:1503
#24 0x62083d74 in emit_known_call (ff=<optimized out>, args=<optimized out>,
    nargs=<optimized out>, ctx=0xd4f220, theFptr=0xd4ed64, theF=0xd4ed68,
    expr=0xfbde230) at codegen.cpp:1897
#25 0x6207b95a in emit_call (expr=0xfbde230, ctx=0xd4f220, arglen=3,
    args=0x1040202c) at codegen.cpp:2595
#26 emit_expr (expr=expr@entry=0xfbde230, ctx=ctx@entry=0xd4f220,
    isboxed=isboxed@entry=true, valuepos=valuepos@entry=true,
    valuevar=valuevar@entry=0x0) at codegen.cpp:3147
#27 0x620851d2 in emit_assignment (bp=bp@entry=0x7d912a0,
    r=r@entry=0xfbde230, declType=0xfbde010,
    isVolatile=isVolatile@entry=false, used=used@entry=true,
    ctx=ctx@entry=0xd4f220) at codegen.cpp:2900
#28 0x6207c825 in emit_assignment (ctx=0xd4f220, r=0xfbde230,
    l=<optimized out>) at codegen.cpp:2936
#29 emit_expr (expr=expr@entry=0xfbde210, ctx=ctx@entry=0xd4f220,
    isboxed=isboxed@entry=false, valuepos=valuepos@entry=false,
    valuevar=valuevar@entry=0x0) at codegen.cpp:3150
#30 0x6208f143 in _fu36___ZNSs4_Rep20_S_empty_rep_storageE ()
    at codegen.cpp:4662
#31 0x62091adb in to_function (li=li@entry=0x1111c738) at codegen.cpp:615
#32 0x62091bfb in jl_compile (f=f@entry=0xfbddfd0) at codegen.cpp:768
#33 0x62052ae7 in jl_get_specialization (f=0xf28cee0, types=0xfbddfb0)
    at gf.c:1503
#34 0x38b76f88 in ?? ()
#35 0x38b76d2a in ?? ()
#36 0x620524f6 in jl_apply (nargs=13956188, args=0xd4f50c, f=<optimized out>)
    at julia.h:1094
#37 jl_apply_generic (F=0xf0b0430, args=0xd4f50c, nargs=2) at gf.c:1686
#38 0x62097bc7 in jl_apply (nargs=2, args=0xd4f50c, f=0xf0b0430)
    at julia.h:1094
#39 do_call (f=f@entry=0xf0b0430, args=args@entry=0x10401790,
    nargs=nargs@entry=2, eval0=eval0@entry=0x0, locals=locals@entry=0x0,
    nl=nl@entry=0, ngensym=ngensym@entry=0) at interpreter.c:64
#40 0x6209720f in eval (e=<optimized out>, locals=locals@entry=0x0,
    nl=nl@entry=0, ngensym=0) at interpreter.c:214
#41 0x62097b05 in jl_interpret_toplevel_expr (e=<optimized out>)
    at interpreter.c:25
#42 0x620a9368 in jl_toplevel_eval_flex (e=<optimized out>, fast=fast@entry=1)
    at toplevel.c:503
#43 0x620a9d4e in jl_toplevel_eval_flex (fast=1, e=<optimized out>)
    at toplevel.c:547
#44 jl_parse_eval_all (fname=fname@entry=0xf4bdc0c "precompile.jl")
    at toplevel.c:551
#45 0x620a9fc9 in jl_load (fname=0xf4bdc0c "precompile.jl") at toplevel.c:590
#46 jl_load_ (str=0xf76c2a0) at toplevel.c:598
#47 0x003205e3 in ?? ()
#48 0x620524f6 in jl_apply (nargs=259441312, args=0xd4f888, f=<optimized out>)
    at julia.h:1094
#49 jl_apply_generic (F=0xf03e980, args=0xd4f888, nargs=1) at gf.c:1686
#50 0x0909205a in ?? ()
#51 0x62097bc7 in jl_apply (nargs=255901952, args=0xd4f8bc, f=0xf8868f0)
    at julia.h:1094
#52 do_call (f=f@entry=0xf8868f0, args=args@entry=0xf4bd8e0, nargs=255901952,
    nargs@entry=1, eval0=eval0@entry=0x0, locals=locals@entry=0x0,
    nl=nl@entry=0, ngensym=ngensym@entry=0) at interpreter.c:64
#53 0x6209720f in eval (e=<optimized out>, locals=locals@entry=0x0,
    nl=nl@entry=0, ngensym=0) at interpreter.c:214
#54 0x62097b05 in jl_interpret_toplevel_expr (e=<optimized out>)
    at interpreter.c:25
#55 0x620a9368 in jl_toplevel_eval_flex (e=<optimized out>, fast=fast@entry=1)
    at toplevel.c:503
#56 0x620a9912 in jl_toplevel_eval_flex (fast=1, e=<optimized out>)
    at toplevel.c:149
#57 jl_eval_module_expr (ex=0xf03c360) at toplevel.c:152
#58 0x620a9688 in jl_toplevel_eval_flex (e=<optimized out>, fast=fast@entry=1)
    at toplevel.c:395
#59 0x620a9d4e in jl_toplevel_eval_flex (fast=1, e=<optimized out>)
    at toplevel.c:547
#60 jl_parse_eval_all (fname=fname@entry=0xd4fda0 "sysimg.jl")
    at toplevel.c:551
#61 0x620a9f23 in jl_load (fname=0xd4fda0 "sysimg.jl") at toplevel.c:590
#62 0x0040161e in exec_program () at repl.c:375
#63 0x004017ff in true_main (argc=1, argv=<optimized out>) at repl.c:432
#64 0x00402389 in wmain (argc=1, argv=0xeeafbc, envp=0xee8dd8) at repl.c:494
#65 0x004013f0 in __tmainCRTStartup ()
    at /usr/src/debug/mingw64-i686-runtime-3.3.0-1/crt/crtexe.c:329
#66 0x7577338a in KERNEL32!BaseThreadInitThunk ()
   from C:\Windows\syswow64\kernel32.dll
#67 0x77a29f72 in ntdll!RtlInitializeExceptionChain ()
   from C:\Windows\system32\ntdll.dll
#68 0x77a29f45 in ntdll!RtlInitializeExceptionChain ()
   from C:\Windows\system32\ntdll.dll
#69 0x00000000 in ?? ()

Any clues in there? Anything more specific I should try to look at here?

tkelman · 2015-03-14T10:31:57Z

Interesting, the innermost emit_expr frame looks like this

(gdb) print jl_(expr)
Expr(:call, :call, :VersionNumber, 2147483647, 2147483647, 2147483647, (), Expr(
:call, top(:tuple), "")::(ASCIIString,))::Base.VersionNumber

which is really similar to something that was happening in #10235

tkelman · 2015-03-14T10:57:09Z

The initFromArray in LLVM is trying to dereference element 0 of the ArrayRef, which is constructed by casting jl_data_ptr(e) to (uint64_t*). This might be where the problem lies.

(gdb) print ((uint64_t*)jl_data_ptr(e))[0]
Cannot access memory at address 0x11863ffc
(gdb) print ((uint32_t*)jl_data_ptr(e))[0]
$65 = 2147483647

catawbasam · 2015-03-14T12:37:22Z

@tkelman I'm really grateful for the work you're doing here. Time to update your Amazon Wishlist or such!

vtjnash · 2015-03-14T16:04:48Z

Ah. That does explain it!

The win32 malloc returns 4-byte aligned values, whereas other malloc will return 16-byte aligned.

ihnorton · 2015-04-04T14:03:18Z

These utf8proc_NFKC calls we keep seeing in backtraces and code_native are actually calls to __chkstk_ms. I wonder why they are being mis-identified.

vtjnash · 2015-04-04T15:57:03Z

that'll be fixed in llvm3.5 when we switch to getting this info from llvm.

currently, we use the windows dbghelp library for this. it doesn't seem like microsoft has made any real improvements to that library in at least 10 years. it doesn't support the dwarf debugging format (although, why would it?), but that's the only format gcc knows how to create.

edit: note that it identifies most functions as utf8proc_NFKC, so you can't just make a simple 1-to-1 shift

tkelman · 2015-04-04T23:09:50Z

Also on the topic of this issue, I haven't been watching every appveyor failure lately but I think there's been an occasional 32-bit failure somewhat reminiscent of this one that has creeped back in.

edit: at least it's way less frequent https://ci.appveyor.com/project/StefanKarpinski/julia/build/1.0.3465/job/poidsluygaha8di6 https://ci.appveyor.com/project/StefanKarpinski/julia/build/1.0.3560/job/slad1jgx5m7yo8i0 https://ci.appveyor.com/project/StefanKarpinski/julia/build/1.0.3605/job/ac7kadcn4vlsbdy2

tkelman added the system:windows Affects only Windows label Feb 3, 2015

tkelman mentioned this issue Feb 5, 2015

Change precision to eps for TimeType #10090

Merged

This was referenced Feb 7, 2015

julia-debug.exe segfaults during spawn test ever since #9266 #10111

Closed

Faster setindex! for RHSs with inefficient linear indexing #10063

Merged

Add a julia-config script for configuration of externals #10069

Merged

jakebolewski mentioned this issue Feb 11, 2015

Win32 CI failures #10156

Closed

This was referenced Feb 11, 2015

Add getindex(::Pair) + tests for Pairs #10152

Merged

Add tests for some untested methods #9674

Merged

tkelman mentioned this issue Feb 13, 2015

Fix type instability in logdet(::Matrix{Complex64}) #10189

Merged

tkelman referenced this issue Feb 16, 2015

fix a computation error from running 32-bit julia on a machine with >…

a476458

… 4GB of ram

tkelman referenced this issue in JuliaData/DataFrames.jl Feb 16, 2015

Fix sort!(df), add tests. Closes #768.

35ad6a2

This was referenced Feb 19, 2015

Clean up tempfile in test/spawn.jl #10245

Merged

Add test for issue #10221 #10231

Merged

jiahao mentioned this issue Feb 19, 2015

fix #10234, and correct overflow scaling of vecnorm(x,p) for p < 0 #10250

Merged

timholy mentioned this issue Feb 22, 2015

A collection of GC rooting/write barrier additions #10275

Merged

tkelman closed this as completed Feb 23, 2015

tkelman added the system:32-bit Affects only 32-bit systems label Mar 3, 2015

This was referenced Mar 6, 2015

RFC: Move Colon translation out of the parser #10331

Merged

Greedy solver for Pkg (speeds up Pkg.publish) #10417

Merged

missing bounds checking for GenSym #10426

Closed

Bump MAX_TYPE_DEPTH #10230

Merged

Fix BigInt to Float conversions #10139

Merged

vtjnash closed this as completed in 8a777c2 Mar 14, 2015

Frequent Win32 segfaults on AppVeyor #10045

Frequent Win32 segfaults on AppVeyor #10045

Comments

tkelman commented Feb 3, 2015

tkelman commented Feb 4, 2015

catawbasam commented Feb 5, 2015

tkelman commented Feb 5, 2015

tkelman commented Feb 6, 2015

tkelman commented Feb 6, 2015

tkelman commented Feb 7, 2015

tkelman commented Feb 10, 2015

ihnorton commented Feb 19, 2015

tkelman commented Feb 19, 2015

tkelman commented Feb 19, 2015

ihnorton commented Feb 22, 2015

tkelman commented Feb 23, 2015

catawbasam commented Feb 23, 2015

timholy commented Feb 23, 2015

tkelman commented Feb 25, 2015

timholy commented Feb 25, 2015

tkelman commented Feb 25, 2015

tkelman commented Feb 25, 2015

tkelman commented Feb 25, 2015

tkelman commented Feb 26, 2015

vtjnash commented Feb 26, 2015

tkelman commented Feb 26, 2015

tkelman commented Mar 2, 2015

vtjnash commented Mar 3, 2015

tkelman commented Mar 3, 2015

jakebolewski commented Mar 12, 2015

tkelman commented Mar 12, 2015

tkelman commented Mar 12, 2015

tkelman commented Mar 13, 2015

tkelman commented Mar 14, 2015

tkelman commented Mar 14, 2015

tkelman commented Mar 14, 2015

catawbasam commented Mar 14, 2015

vtjnash commented Mar 14, 2015

ihnorton commented Apr 4, 2015

vtjnash commented Apr 4, 2015

tkelman commented Apr 4, 2015