REPL shell mode for Windows #23703

GregPlowman · 2017-09-13T22:59:04Z

Replaces stale PR #21294
cc: @stevengj

Summary:

JULIA_SHELL can be set to:
-- cmd
-- powershell
-- busybox
-- "" for no shell, execute command directly (current behavior)
cmd is default for Windows users.
For Windows, the raw line is processed, rather than converting to Cmd and then re-assembling into a Windows command line. Windows shell commands are passed as a string anyway, so this gives the user more control over any escaping. I think it works quite well.
Modified special handling of cd. When there are 2 or more arguments, command line is processed as normal, rather than throwing an error. This allows commands such as cd $dir && $command to be executed.

ararslan · 2017-09-13T23:00:41Z

Thanks for revisiting this. In general it's better to update PRs than open new ones but this is fine.

stevengj · 2017-09-19T14:50:01Z

LGTM, but probably someone with more Windows shell expertise like @tkelman should chime in.

stevengj · 2017-09-19T14:52:47Z

JULIA_SHELL should be documented in https://github.com/JuliaLang/julia/blob/master/doc/src/manual/interacting-with-julia.md#shell-modeid-man-shell-mode

tkelman · 2017-09-19T20:05:36Z

I don't think anything should ever default to cmd.

stevengj · 2017-09-20T13:22:20Z

@tkelman, isn't cmd by far the most familiar interactive command-line environment in Windows?

Wouldn't defaulting to e.g. PowerShell be even more confusing to someone who just wants dir to work (how many Windows users know to type Get-ChildItem)?

Alternatively, I'd be all for bundling something like busybox or toybox on Windows so that people can use the same Unix shell commands on all platforms (with the uniformity outweighing the non-native feel of ls vs. dir). Unfortunately, toybox doesn't support Windows, and people have raised licensing concerns over busybox. (What is the exact problem? The GPL explicitly doesn't extend to "mere aggregation", i.e. shipping two executables together, and the FSF also explicitly says that it doesn't affect one program calling another via fork/exec.)

tkelman · 2017-09-20T15:01:11Z

Familiar, maybe, but also the least useful to interact with programmatically.

Powershell has default aliases, both cmd and posix names, for many things. It does have an unfortunate first-time startup latency though.

We already do bundle busybox, but it only gets used by a few tests. I've never been bothered enough by "dir doesn't work in shell mode" to feel the need to rely on it more heavily though. It currently gets left out of a no-GPL build for uniformity's sake, though the viral parts of the GPL don't apply to how we use it.

stevengj · 2017-09-20T16:18:43Z

So can we default to busybox, to make the shell mode portable?

vtjnash · 2017-09-20T16:24:31Z

So can we default to busybox, to make the shell mode portable?

We could also default to Julia's shell mode everywhere, which is also portable.

stevengj · 2017-09-20T17:03:23Z

@vtjnash, but then we get an endless stream of complaints that ls and dir don't work, since getting a directory listing is one of the most common things to do in shell mode.

vtjnash · 2017-09-20T17:07:58Z

Anything with a binary (such as ls) will work. And as an additional benefit, you could actually use the REPL mode to run actual programs (#10120):

shell> ./julia -e 'print(1)'
bash: syntax error near unexpected token `('

stevengj · 2017-09-20T17:18:32Z

Yes, but the problem is that Windows does not have binaries for ls/dir or other commands that people typically expect in a shell. Hence the complaints and frustration.

One alternative option would be for the shell mode on Windows to preprocess the Cmd object and transform a few common shell functions like ls into calls to busybox.

The basic issue here is to make the default REPL shell mode more usable on Windows.

GregPlowman · 2017-09-20T22:28:02Z

Anything with a binary (such as ls) will work. And as an additional benefit, you could actually use the REPL mode to run actual programs (#10120):

Should also be able to run any binary with this PR.
Here using cmd as shell:

cmd> $JULIA_HOME/julia -e print(1)
1

One alternative option would be for the shell mode on Windows to preprocess the Cmd object and transform a few common shell functions like ls into calls to busybox.

This is effectively what windows cmd does. It "pre-processes" internal commands like dir and calls other programs directly.

It seems some people might want the familiarity of cmd , others want a more flexible and powerful shell (e.g. powershell), and yet others prefer portability (unix-like shell). Satisfying all of these with a single default seems unlikely.

Current PR allows users to select shell mode by setting JULIA_SHELL.
Could change default to "no shell", which is the current behaviour. This would be the least disruptive and would allow users to play around with other shell options, and see if a more popular default emerges over time.

stevengj · 2017-09-21T03:05:37Z

Making the default "no shell" means that the default will seem broken for many users. Furthermore, it is precisely the new users confused by this who will also have a hard time learning about an "unbreak me" JULIA_SHELL option.

StefanKarpinski · 2017-09-21T03:22:28Z

100% agree with @stevengj. As a purely interactive feature user-friendliness trumps other concerns.

GregPlowman · 2017-09-21T03:24:12Z

Making the default "no shell" means that the default will seem broken for many users.

Precisely. It will seem broken for those users that want cmd, powershell or unix-like behaviour.
Setting some other default will seem broken for those who want something other than that default.

I think cmd might be the better choice for a default. It's familiar, supports internal commands (e.g. dir), and supports running external programs.

Are there any examples of commands that can be run now, but you think can't be run via cmd?
If so, I'd be willing to try to make them work.

vtjnash · 2017-09-21T03:51:46Z

I have to 100% disagree. I entirely stopped being able to use shell mode once #4635 merged, and now never use it (and I originally implemented it, iirc).

StefanKarpinski · 2017-09-21T14:06:47Z

I entirely stopped being able to use shell mode once #4635 merged, and now never use it (and I originally implemented it, iirc).

Then you are clearly not the target demographic for this feature. You've never explained why this feature became "unusable" to you, so it's impossible to make progress towards something that works for you as well as for the many others who are happy with the way this works currently (except on Windows where it's unusable precisely because it has the behavior you claim is "unusable").

vtjnash · 2017-09-21T15:46:33Z

You've never explained why this feature became "unusable" to you

I opened an issue for that #10120 and you are self-assigned to solve it. Currently, I don't think it's possible to pass the character (.

vtjnash · 2017-09-21T15:46:56Z

base/client.jl

+    nothing
+end
+
+function repl_cmd(iswindows::Val{true}, line::AbstractString, out)


don't use Val

OK, I'm thinking I will just use a single run-time if statement within a common repl_cmd method.

Just for my own edification, why is using Val not recommended? Is it a performance issue?
I thought iswindows() would be known at compile time?
Are there any cases where using Val is OK?

StefanKarpinski · 2017-09-21T18:51:28Z

That's a bug – saying that it makes shell mode "unusable" is quite dramatic. Feel free to fix said bug, I'm not territorial.

vtjnash · 2017-09-21T18:59:47Z

If I thought it was fixable, don't you think I would have submitted a PR instead of ceasing to use it? My conclusion at the time was that its behavior was unpredictable-by-design, so I stopped attempting to use it.

tkelman · 2017-09-21T19:43:19Z

Calling it unusable on Windows is a bit of an exaggeration. Other than dir, what cmd builtins has anyone ever complained about wanting to use? People should probably use the Julia readdir anyway.

PallHaraldsson · 2017-09-25T09:19:13Z

@stevengj What is the exact problem? The GPL explicitly doesn't extend to "mere aggregation"

It's puzzling as "mere aggregation" should apply for busybox. We could do everything right, but someone downstream distributing Julia could get in trouble (unless we distribute source code of busybox with?):

https://torquemag.io/2013/03/busybox/

The interesting thing is the financial transaction. You see, it wasn’t enough, in the context of this issue, that Monsoon get away with just “being compliant” but rather also paying a settlement fee. As Ravicher stated,
"Simply coming into compliance now is not sufficient to settle the matter, because that would mean anyone can violate the license until caught, because the only punishment would be to come into compliance."
Sounds familiar on a lot of different fronts. Monsoon was also required to appoint an open source compliance officer on staff as well as publish the source code for the version of BusyBox that they were distributing.

PallHaraldsson · 2017-09-25T10:06:04Z

What I quoted on busybox sounds like FUD so I dug deeper, busybox copyright lawsuits are used as a lever against Linux kernel GPL violations (even if copyright owners of it don't want to sue).

My reading of this is that even if ~~"mere aggregation" should hold~~ (EDIT: see below, maybe wasn't like that) for the kernel on one hand and busybox on the other, in practice it doesn't (this doesn't apply to Julia.. but not all may realize that and be afraid of using busybox):

http://mjg59.dreamwidth.org/10437.html

The SFC will grant a new license, but on one condition - not only must you provide the source code to Busybox, you must provide the source code to all other works on the device that require source distribution.

https://lwn.net/Articles/478249/

PallHaraldsson · 2017-09-25T10:20:29Z

If you still want to use busybox, there's:

https://github.com/rmyorston/busybox-w32 "WIN32 native port of BusyBox."

If you distribute [binary] code meant for Windows, I guess the "Linux kernel lawsuit lever" is out.. or not as likely to lead to a lawsuit.

Just speculating on "mere aggregation" issue above, maybe it didn't apply, as busybox intentionally statically links shell command binaries together (we wouldn't link busybox as a whole with Julia.exe); maybe that wasn't the limit, not sure if a firmware would have one giant binary with the busybox + the Linux kernel statically linked..

PallHaraldsson · 2017-09-25T10:29:56Z

One question, since most users, at least developers will be on Windows 10, is busybox relevant? Can't you just use bash from the Linux subsystem for the shell mode only? Even keeping Julia (otherwise) a pure Windows build (not an ELF build for the subsystem).

stevengj · 2017-09-25T13:08:36Z

@PallHaraldsson, as I explicitly commented above, I don't think either shipping busybox with Julia ("mere aggregation") or executing it via fork/exec would be a problem. (That is, it wouldn't affect the licensing terms of Julia programs.)

The only question in my mind is whether Windows users would prefer a Windows shell (e.g. cmd), or if the superiority of the POSIX shell tools and the advantage of having the same Julia shell commands cross-platform would win. I favor the latter, but I'm obviously biased since I don't normally use Windows.

GregPlowman · 2017-09-28T01:38:45Z

I've tried to update some manual pages, but it seems like something isn't right, and tests are failing.
I wanted to provide a link to JULIA_SHELL in Environment Variables from Shell mode in Interacting With Julia manual page. Perhaps this is the problem?
Can anyone help with this?

EDIT: OK now fixed.
(I had JULIA_SHELL in backticks, which I learnt is for docstrings only, not for document headers)

ararslan · 2017-09-28T01:47:30Z

base/client.jl

+    elseif cmd[1] == "cd" && length(cmd) <= 2
+        repl_cd(cmd, shell, out)
+    else
+        run(ignorestatus(isa(STDIN, TTY) ? `$shell -i -c "$(shell_wrap_true(shell_name, cmd))"` : `$shell -c "$(shell_wrap_true(shell_name, cmd))"`))


This could be written a little more cleanly as

ttyopt = STDIN isa TTY ? "-i" : "" run(ignorestatus(`$shell $ttyopt -c "$(shell_wrap_true(shell_name, cmd))"`))

since -i is the only thing that differs between the two branches.

Yes of course, will update.
Although probably better to use command syntax rather than strings:

ttyopt = STDIN isa TTY ? `-i` : ``

https://discourse.julialang.org/t/interpolating-an-empty-string-inside-backticks/2428/3

Ah yes, good call. I had forgotten about that.

GregPlowman · 2017-10-02T21:45:56Z

docs have been updated
code updated to address comments (and no longer using Val)

stevengj · 2018-04-29T20:49:44Z

Would be nice to see this resurrected in some form…

stevengj · 2018-08-17T20:49:26Z

Ping. This didn't make it into 1.0, but it is not breaking (just a REPL feature), so it could go into 1.1.

musm · 2018-08-31T15:24:44Z

bump also looking forward to this. ref https://discourse.julialang.org/t/newbi-first-steps-in-julia-for-

stevengj · 2019-03-26T20:06:35Z

Bump? Looks like someone else has to take over this PR if @GregPlowman is too busy on other things these days…

vtjnash · 2019-03-26T20:30:03Z

c.f. #30614, which implements the escaping function (print_shell_escaped_windows) that's needed to shell out to cmd

vtjnash · 2019-03-26T20:30:23Z

base/client.jl


-    if isempty(cmd.exec)
+    cmd = cmd_gen(eval(Main, shell_parse(line)[1]))


there shouldn't be any calls to eval

eliminated in the updated PR

stevengj · 2019-03-26T20:54:15Z

Closing this in favor of the simplified and rebased #31490. I would have updated this PR, but it didn't look like I had the permissions?

coveralls · 2020-01-31T05:59:15Z

Coverage increased (+5.1%) to 92.5% when pulling 16afe32 on GregPlowman:glp/shell_mode_windows into 55ec9fe on JuliaLang:master.

GregPlowman added 2 commits September 12, 2017 14:48

REPL shell mode for Windows

ab83752

REPL shell mode for Windows

adaea4d

ararslan requested a review from stevengj September 13, 2017 23:01

ararslan added stdlib:REPL Julia's REPL (Read Eval Print Loop) system:windows Affects only Windows labels Sep 13, 2017

GregPlowman mentioned this pull request Sep 13, 2017

[WIP] REPL shell mode for windows #21294

Closed

stevengj approved these changes Sep 19, 2017

View reviewed changes

stevengj added the needs docs Documentation for this change is required label Sep 19, 2017

vtjnash reviewed Sep 21, 2017

View reviewed changes

GregPlowman added 2 commits September 25, 2017 11:00

Interacting with Julia docs - xref JULIA_SHELL

0d84a76

Update docs for JULIA_SHELL and shell mode

a5b762b

Fix whitespace JULIA_SHELL doc

792460a

ararslan reviewed Sep 28, 2017

View reviewed changes

GregPlowman added 2 commits September 30, 2017 16:51

Fix JULIA_SHELL cross-reference

5771888

Address code review comments

16afe32

GregPlowman mentioned this pull request Dec 11, 2017

string overhaul #24999

Merged

stevengj mentioned this pull request Apr 29, 2018

The "shell mode" in the REPL is not fully functional on Windows #23597

Open

stevengj mentioned this pull request Aug 17, 2018

cannot shell out to cmd's builtin commands #6316

Open

kdayday mentioned this pull request Dec 22, 2018

Shell commands to git in build.jl don't work on Windows NREL-Sienna/PowerSystems.jl#129

Closed

stevengj mentioned this pull request Mar 26, 2019

bundle Julia with a better terminal on Windows? #7267

Closed

vtjnash reviewed Mar 26, 2019

View reviewed changes

stevengj mentioned this pull request Mar 26, 2019

REPL shell mode for Windows #31490

Open

stevengj closed this Mar 26, 2019

musm mentioned this pull request Oct 11, 2019

REPL shell mode for Windows #33474

Closed

jaakkor2 mentioned this pull request Oct 6, 2023

Shell mode on Windows modernjuliaworkflows/modernjuliaworkflows.github.io#71

Closed


		if isempty(cmd.exec)
		cmd = cmd_gen(eval(Main, shell_parse(line)[1]))

REPL shell mode for Windows #23703

REPL shell mode for Windows #23703

Conversation

GregPlowman commented Sep 13, 2017

ararslan commented Sep 13, 2017

stevengj commented Sep 19, 2017

stevengj commented Sep 19, 2017 • edited Loading

tkelman commented Sep 19, 2017

stevengj commented Sep 20, 2017 • edited Loading

tkelman commented Sep 20, 2017

stevengj commented Sep 20, 2017

vtjnash commented Sep 20, 2017

stevengj commented Sep 20, 2017 • edited Loading

vtjnash commented Sep 20, 2017

stevengj commented Sep 20, 2017 • edited Loading

GregPlowman commented Sep 20, 2017

stevengj commented Sep 21, 2017 • edited Loading

StefanKarpinski commented Sep 21, 2017 • edited Loading

GregPlowman commented Sep 21, 2017

vtjnash commented Sep 21, 2017

StefanKarpinski commented Sep 21, 2017

vtjnash commented Sep 21, 2017

vtjnash Sep 21, 2017

Choose a reason for hiding this comment

GregPlowman Sep 25, 2017

Choose a reason for hiding this comment

StefanKarpinski commented Sep 21, 2017 • edited Loading

vtjnash commented Sep 21, 2017 • edited Loading

tkelman commented Sep 21, 2017

PallHaraldsson commented Sep 25, 2017

PallHaraldsson commented Sep 25, 2017 • edited Loading

PallHaraldsson commented Sep 25, 2017 • edited Loading

PallHaraldsson commented Sep 25, 2017

stevengj commented Sep 25, 2017 • edited Loading

GregPlowman commented Sep 28, 2017 • edited Loading

ararslan Sep 28, 2017

Choose a reason for hiding this comment

GregPlowman Sep 28, 2017 • edited Loading

Choose a reason for hiding this comment

ararslan Sep 28, 2017

Choose a reason for hiding this comment

GregPlowman commented Oct 2, 2017

stevengj commented Apr 29, 2018

stevengj commented Aug 17, 2018

musm commented Aug 31, 2018

stevengj commented Mar 26, 2019 • edited Loading

vtjnash commented Mar 26, 2019

vtjnash Mar 26, 2019

Choose a reason for hiding this comment

stevengj Mar 26, 2019

Choose a reason for hiding this comment

stevengj commented Mar 26, 2019

coveralls commented Jan 31, 2020

stevengj commented Sep 19, 2017 •

edited

Loading

stevengj commented Sep 20, 2017 •

edited

Loading

stevengj commented Sep 20, 2017 •

edited

Loading

stevengj commented Sep 20, 2017 •

edited

Loading

stevengj commented Sep 21, 2017 •

edited

Loading

StefanKarpinski commented Sep 21, 2017 •

edited

Loading

StefanKarpinski commented Sep 21, 2017 •

edited

Loading

vtjnash commented Sep 21, 2017 •

edited

Loading

PallHaraldsson commented Sep 25, 2017 •

edited

Loading

PallHaraldsson commented Sep 25, 2017 •

edited

Loading

stevengj commented Sep 25, 2017 •

edited

Loading

GregPlowman commented Sep 28, 2017 •

edited

Loading

GregPlowman Sep 28, 2017 •

edited

Loading

stevengj commented Mar 26, 2019 •

edited

Loading