Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

hist and fc: only run edited commands if changes are saved. #748

Closed
wants to merge 11 commits into from

Conversation

pghvlaans
Copy link

@pghvlaans pghvlaans commented May 1, 2024

According to The New KornShell, the full editor history editing commands are only meant to execute command lines that have had saved changes:

"The command is the ksh line that you were editing with the vi built-in editor if you omit n, or command n in the history file. When you exit the vi program, ksh displays and executes the command if you have changed it. Version: With some early releases of the 11/16/88 version of ksh, the file was executed even if you had not changed it." (Bolsky & Korn, pg. 116)

There is no other evidence that this functionality was ever part of release ksh93, however. Neither bash nor any existing open source ksh88 derivative behaves in this manner. Nevertheless, some distributions of ksh88 (such as the one shipped with UnixWare 7) did attempt to avoid running unchanged command lines using timestamps. The code in the new implementation is entirely different:

  • bltins/hist.c: When the full editor has been called, only execute the line being viewed if the contents have been changed and saved.
  • sh.1, data/builtins.c: Update the hist documentation accordingly.
  • tests/builtins.sh: Increase ulimit for the hist test to accommodate changes to hist.c.

@pghvlaans
Copy link
Author

Just a few quick notes about my testing round this time:

  • I no longer have access to a working macOS VM.
  • OmniOS and Haiku both experienced complete test failures on dev and my own branch. I'll try to look into this and open an issue.

@pghvlaans
Copy link
Author

Got rid of a couple of stray variables from testing and renamed some new variables I wasn't happy about.

@posguy99
Copy link

posguy99 commented May 1, 2024

Not documented in the Bolsky (as far as I know) are the ^X prefix commands supported by the emacs line editor. Chief among those is ^X ^E, as you know for bringing the current line into a full screen editor.

ksh, mksh, and bash all execute the line on return from the editor, even if there were no changes.

Since this patch isn't to either editor source file, it affects both?

@pghvlaans
Copy link
Author

ksh, mksh, and bash all execute the line on return from the editor, even if there were no changes.

Indeed. Personally, I think the book got it right and that's a bit of an anti-pattern, since the normal result of exiting an editor without saving is "nothing happens."

Since this patch isn't to either editor source file, it affects both?

Yes, that's right.

@posguy99
Copy link

posguy99 commented May 4, 2024

I tested it locally, works for me in vi or emacs mode.

@pghvlaans
Copy link
Author

Awesome, thanks!

@posguy99
Copy link

@McDutchie Would you consider merging this, since it brings ksh in line with the Bolsky?

@McDutchie
Copy link

Yes, I'm mulling it over and I've been doing some research.

I've researched the code for all the ksh93 versions in the ast-open-archive repo. And I have found no evidence that emacs ^X^E in ksh has ever acted like Bolsky & Korn describe.

So is ksh wrong, or the book? I lean towards the latter, particularly as all other shells also act like ksh does now.

That doesn't mean the change wouldn't be good to have, but perhaps it should be for the future 93u+m/1.1 release (the current dev branch) and not the 1.0 series.

@posguy99
Copy link

Bolsky doesn't mention the emacs variant at all, only the vi variant. Bolsky documents the vi variant, but says that the command line is only executed when the line was changed, while ksh always executes it.

The man page for ksh does document both the emacs variant and the vi variant, but doesn't state how either behaves.

@pghvlaans
Copy link
Author

So is ksh wrong, or the book? I lean towards the latter, particularly as all other shells also act like ksh does now.

That seems like a fair conclusion. The 1995 code I looked at when writing the original comment had neither mode doing this.

Since this change would apparently be a matter of personal taste rather than book accuracy, I'll make another commit to put it behind a shell option.

@pghvlaans pghvlaans changed the title hist and fc: only run edited commands if changes are saved. hist and fc: optionally, only run edited commands if changes are saved. Aug 24, 2024
@McDutchie
Copy link

I'm reluctant to add a shell option for something this minor. If we go that route we may eventually end up like zsh which has too many shell options. I think it would be better just to make a decision one way or another.

@pghvlaans
Copy link
Author

OK, I have a theory about what might've happened: checking for saved changes was a feature of some distributions of ksh88, but was written out of ksh93 altogether, either from the start or too early to get picked up by ksh-history.

In this UnixWare 7 repository, there is source for both ksh and ksh93. In ksh, builtin.c, line 1260:

/* if the file hasn't changed, treat this as a error */
if(*a1!='-' && fstat(fdo,&statb)>=0 && before==statb.st_mtime)
	sh.exitval = 1;

These lines are not present in the ksh93 version of hist.c in that same repository.

The Solaris 2.6 code also has some insight. Their (ksh88) builtin.c has the file check present but enclosed in notdef at line 1427. A code comment reads:

/*
 * Korn says this is the right thing to do, but our customers,
 * the ksh man page, the ksh book, and the POSIX shell spec
 * all disagree with Korn.  None of the docs make any mention
 * of checking for modification of the file.  And besides, the
 * mod check fails if you modify the file *very* quickly.
 */

If the Solaris people of that time are to be believed, the feature was controversial and (surprise, surprise) buggy. Although I do like it (and the code works differently), I'd be happy to drop this if you believe there might be user pushback.

@pghvlaans pghvlaans changed the title hist and fc: optionally, only run edited commands if changes are saved. hist and fc: only run edited commands if changes are saved. Aug 24, 2024
@pghvlaans pghvlaans marked this pull request as draft August 24, 2024 15:23
@posguy99
Copy link

It seems clear to me now that the current behavior is intentional, since the code was there, and was removed in the re-write. Currently it's the book that has the problem with how it works.

It just looks to me that editing a line and executing a line should be separate acts. If you bring back history into the line editor, you can do whatever you want to it, but never actually execute it, just by moving off of it to another line. With the full editor, SOMETHING is always going to get executed when you leave the editor, there's no way out unless you empty the buffer and save THAT.

@pghvlaans pghvlaans marked this pull request as ready for review August 25, 2024 06:46
@McDutchie
Copy link

McDutchie commented Dec 23, 2024

It seems clear to me now that the current behavior is intentional, since the code was there, and was removed in the re-write. Currently it's the book that has the problem with how it works.

I think it's difficult to guess what the intention was. Given how many decades-old experiments and other cruft were in this codebase when we forked it, it's clear to me that they had forgotten to remove or maintain loads of stuff. I wouldn't put it past them to also forget to include something they meant to.

Which leaves it up to us (ultimately, me) to make a sensible decision for the future 1.1 release. (I think it's too late to change this for the 1.0 series, except the manual page should be corrected.)

It just looks to me that editing a line and executing a line should be separate acts. If you bring back history into the line editor, you can do whatever you want to it, but never actually execute it, just by moving off of it to another line. With the full editor, SOMETHING is always going to get executed when you leave the editor, there's no way out unless you empty the buffer and save THAT.

Yes, but that is also not ideal -- even if that is what all the shells seem to have settled on. But it's why I rarely use the feature. I would like to be able to have a quick look in the full-screen editor and go "nah, never mind" and quit without having to delete everything and save the file.

I actually quite like the Unixware ksh88 approach that @pghvlaans found, two comments up: just check the timestamp of the file. That approach doesn't take so much new code to compare two temporary files. It doesn't technically check if the file has changed, but if the timestamp has, then usually so has the content. If the user does actively save the file without changing anything, I believe it is justified to take that as an intention to execute.

The old Unixware code checks with a one-second granularity because that's what systems supported then, and some of us may need less than a second to change a simple thing and save. But on modern systems, we can check timestamps with nanosecond granularity.

@McDutchie
Copy link

McDutchie commented Dec 23, 2024

Here is a patch that implements that. (It's against the dev branch, not this PR.)

It uses the tv library from libast to get the nanosecond modification time. POSIX only standardised this in Issue 7 (2018), but tv detects/supports four ways to read nanoseconds information from a struct stat, including the one POSIX ended up specifying.

Patch v1
diff --git a/src/cmd/ksh93/Mamfile b/src/cmd/ksh93/Mamfile
index a0bff1cf5..6a700b16c 100644
--- a/src/cmd/ksh93/Mamfile
+++ b/src/cmd/ksh93/Mamfile
@@ -406,6 +406,7 @@ make install virtual
 			prev include/io.h
 			prev include/variables.h
 			prev %{INCLUDE_AST}/error.h
+			prev %{INCLUDE_AST}/tv.h
 			prev %{INCLUDE_AST}/ls.h
 			prev include/defs.h
 			prev shopt.h
diff --git a/src/cmd/ksh93/bltins/hist.c b/src/cmd/ksh93/bltins/hist.c
index 8a69ce12c..b9164a50a 100644
--- a/src/cmd/ksh93/bltins/hist.c
+++ b/src/cmd/ksh93/bltins/hist.c
@@ -18,6 +18,7 @@
 #include	"shopt.h"
 #include	"defs.h"
 #include	<ls.h>
+#include	<tv.h>
 #include	<error.h>
 #include	"variables.h"
 #include	"io.h"
@@ -247,10 +248,20 @@ int	b_hist(int argc,char *argv[], Shbltin_t *context)
 	if(*arg != '-')
 	{
 		char *com[3];
+		struct stat statb;
+		Tv_t before = { 0 }, after;
+		if(stat(fname,&statb)>=0)
+			tvgetmtime(&before,&statb);
 		com[0] =  arg;
 		com[1] =  fname;
 		com[2] = 0;
 		error_info.errors = sh_eval(sh_sfeval(com),0);
+		/* if the file's timestamp hasn't changed, treat this as a error */
+		if(!error_info.errors && stat(fname,&statb)>=0)
+		{
+			tvgetmtime(&after,&statb);
+			error_info.errors = before.tv_sec==after.tv_sec && before.tv_nsec==after.tv_nsec;
+		}
 	}
 	fdo = sh_chkopen(fname);
 	unlink(fname);

@posguy99
Copy link

posguy99 commented Dec 23, 2024

The patch in #748 (comment) works for me. The original patch by @pghvlaans does differ in one respect... comparing timestamps means that if you save it, but haven't actually made changes to it, the timestamp gets updated and the line gets executed.

@dannyweldon
Copy link

dannyweldon commented Dec 23, 2024 via email

@pghvlaans
Copy link
Author

I actually quite like the Unixware ksh88 approach that @pghvlaans found, two comments up: just check the timestamp of the file. That approach doesn't take so much new code to compare two temporary files. It doesn't technically check if the file has changed, but if the timestamp has, then usually so has the content. If the user does actively save the file without changing anything, I believe it is justified to take that as an intention to execute.

With nanosecond accuracy, that sounds pretty robust to me. It really just comes down to whether going slightly off-book matters, and I'll defer on that point. I'd be happy to see either approach implemented.

@McDutchie
Copy link

McDutchie commented Dec 23, 2024

@dannyweldon wrote:

[…] a feature that I would like is that when control returns back to the shell from the external editor, the resultant text just replaces the current command line contents without executing it, and you can then choose to carry on editing, discard it or execute it as desired.

Thanks, that's an interesting idea. We could make that new behaviour subject to the histverify shell option, because that option already does the same thing for history expansions. Figuring out how to implement it for ^X^E and v is a whole new project, so it should probably be considered in a separate issue.

@McDutchie
Copy link

@pghvlaans wrote:

With nanosecond accuracy, that sounds pretty robust to me. It really just comes down to whether going slightly off-book matters, and I'll defer on that point.

The Solaris 2.6 comment that you found did note, in its disapproval, that David Korn said this was the right thing to do. This would not be the only instance where the book or other documentation uses slightly inexact language.

I'm sorry to throw away your work, though. Thank you for being gracious about it.

@pghvlaans
Copy link
Author

No worries! Checking the time stamp is a simpler solution, and executing on save is reasonable.

@McDutchie
Copy link

This version of the patch has more robust error handling.

Patch v2
diff --git a/src/cmd/ksh93/Mamfile b/src/cmd/ksh93/Mamfile
index a0bff1cf5..6a700b16c 100644
--- a/src/cmd/ksh93/Mamfile
+++ b/src/cmd/ksh93/Mamfile
@@ -406,6 +406,7 @@ make install virtual
 			prev include/io.h
 			prev include/variables.h
 			prev %{INCLUDE_AST}/error.h
+			prev %{INCLUDE_AST}/tv.h
 			prev %{INCLUDE_AST}/ls.h
 			prev include/defs.h
 			prev shopt.h
diff --git a/src/cmd/ksh93/bltins/hist.c b/src/cmd/ksh93/bltins/hist.c
index 8a69ce12c..227406409 100644
--- a/src/cmd/ksh93/bltins/hist.c
+++ b/src/cmd/ksh93/bltins/hist.c
@@ -18,6 +18,7 @@
 #include	"shopt.h"
 #include	"defs.h"
 #include	<ls.h>
+#include	<tv.h>
 #include	<error.h>
 #include	"variables.h"
 #include	"io.h"
@@ -246,11 +247,25 @@ int	b_hist(int argc,char *argv[], Shbltin_t *context)
 	}
 	if(*arg != '-')
 	{
+		int e; /* error flag */
+		struct stat statb;
+		Tv_t before, after;
 		char *com[3];
 		com[0] =  arg;
 		com[1] =  fname;
 		com[2] = 0;
-		error_info.errors = sh_eval(sh_sfeval(com),0);
+		if (!(e = stat(fname,&statb)<0))
+		{
+			tvgetmtime(&before,&statb);
+			/* invoke the editor */
+			if (!(e = sh_eval(sh_sfeval(com),0)) && !(e = stat(fname,&statb)<0))
+			{
+				/* if the file's timestamp hasn't changed, treat this as an error */
+				tvgetmtime(&after,&statb);
+				e = before.tv_sec==after.tv_sec && before.tv_nsec==after.tv_nsec;
+			}
+		}
+		error_info.errors = e;
 	}
 	fdo = sh_chkopen(fname);
 	unlink(fname);

@McDutchie
Copy link

McDutchie commented Dec 23, 2024

Unfortunately we're not done yet. This needs more consideration because (of course) it also affects the fc/hist command. While bowing out on not saving seems like the right thing to do for the ^X^E and v editor commands, which are not specified by POSIX, it isn't the right thing for the POSIX fc command, which has precisely specified behaviour. POSIX only says this:

If the editor returns a non-zero exit status, this shall suppress the entry into the history list and the command re-execution.

So, we need a new option to fc that introduces the new nonstandard behaviour, so that the edit commands can then use that.

@McDutchie
Copy link

This new patch does that.

Patch v3
diff --git a/src/cmd/ksh93/Mamfile b/src/cmd/ksh93/Mamfile
index a0bff1cf5..6a700b16c 100644
--- a/src/cmd/ksh93/Mamfile
+++ b/src/cmd/ksh93/Mamfile
@@ -406,6 +406,7 @@ make install virtual
 			prev include/io.h
 			prev include/variables.h
 			prev %{INCLUDE_AST}/error.h
+			prev %{INCLUDE_AST}/tv.h
 			prev %{INCLUDE_AST}/ls.h
 			prev include/defs.h
 			prev shopt.h
diff --git a/src/cmd/ksh93/bltins/hist.c b/src/cmd/ksh93/bltins/hist.c
index 8a69ce12c..fb8f439a7 100644
--- a/src/cmd/ksh93/bltins/hist.c
+++ b/src/cmd/ksh93/bltins/hist.c
@@ -18,6 +18,7 @@
 #include	"shopt.h"
 #include	"defs.h"
 #include	<ls.h>
+#include	<tv.h>
 #include	<error.h>
 #include	"variables.h"
 #include	"io.h"
@@ -52,6 +53,7 @@ int	b_hist(int argc,char *argv[], Shbltin_t *context)
 #if SHOPT_HISTEXPAND
 	int pflag = 0;
 #endif
+	int checktime = 0;
 	Histloc_t location;
 	NOT_USED(argc);
 	NOT_USED(context);
@@ -63,6 +65,9 @@ int	b_hist(int argc,char *argv[], Shbltin_t *context)
 	hp = sh.hist_ptr;
 	while((flag = optget(argv,sh_opthist))) switch(flag)
 	{
+	    case 'E':
+		checktime = 1;
+		break;
 	    case 'e':
 		edit = opt_info.arg;
 		break;
@@ -246,11 +251,25 @@ int	b_hist(int argc,char *argv[], Shbltin_t *context)
 	}
 	if(*arg != '-')
 	{
+		int e = 0; /* error flag */
+		struct stat statb;
+		Tv_t before, after;
 		char *com[3];
 		com[0] =  arg;
 		com[1] =  fname;
 		com[2] = 0;
-		error_info.errors = sh_eval(sh_sfeval(com),0);
+		if (checktime && !(e = stat(fname,&statb)<0))
+			tvgetmtime(&before,&statb);
+		/* invoke the editor */
+		if (!e)
+			e = sh_eval(sh_sfeval(com),0);
+		if (checktime && !e && !(e = stat(fname,&statb)<0))
+		{
+			/* if the file's timestamp hasn't changed, treat this as an error */
+			tvgetmtime(&after,&statb);
+			e = before.tv_sec==after.tv_sec && before.tv_nsec==after.tv_nsec;
+		}
+		error_info.errors = e;
 	}
 	fdo = sh_chkopen(fname);
 	unlink(fname);
diff --git a/src/cmd/ksh93/data/builtins.c b/src/cmd/ksh93/data/builtins.c
index 71ea688b8..ff78aa2ca 100644
--- a/src/cmd/ksh93/data/builtins.c
+++ b/src/cmd/ksh93/data/builtins.c
@@ -1017,7 +1017,7 @@ const char sh_opthash[] =
 
 #if !SHOPT_SCRIPTONLY
 const char sh_opthist[]	=
-"[-1cn?\n@(#)$Id: hist (AT&T Research) 2000-04-02 $\n]"
+"[-1cn?\n@(#)$Id: hist (ksh 93u+m) 2024-12-23 $\n]"
 "[--catalog?" SH_DICT "]"
 "[+NAME?\f?\f - process command history list]"
 "[+DESCRIPTION?\b\f?\f\b lists, edits, or re-executes, commands  "
@@ -1055,6 +1055,8 @@ const char sh_opthist[]	=
 "[+?If no editor is specified, then the editor specified by the \bHISTEDIT\b "
 	"variable will be used if set, or the \bFCEDIT\b variable will be "
 	"used if set, otherwise, \bed\b will be used.]"
+"[E?Only execute the edited command line if the file is saved before "
+	"exiting the editor.]"
 "[e]:[editor?\aeditor\a specifies the editor to use to edit the history "
 	"command. A value of \b-\b for \aeditor\a is equivalent to "
 	"specifying the \b-s\b option.]"
diff --git a/src/cmd/ksh93/data/msg.c b/src/cmd/ksh93/data/msg.c
index 3d193e204..438d7d02b 100644
--- a/src/cmd/ksh93/data/msg.c
+++ b/src/cmd/ksh93/data/msg.c
@@ -39,7 +39,7 @@
 
 /* error messages */
 const char e_timewarn[]		= "\r\n\ashell will timeout in 60 seconds due to inactivity";
-const char e_runvi[]		= "\\hist -e \"${VISUAL:-${EDITOR:-vi}}\" ";
+const char e_runvi[]		= "\\fc -Ee\"${VISUAL:-${EDITOR:-vi}}\" ";
 const char e_timeout[]		= "timed out waiting for input";
 const char e_mailmsg[]		= "you have mail in $_";
 const char e_query[]		= "no query process";

McDutchie added a commit that referenced this pull request Dec 24, 2024
src/cmd/ksh93/data/msg.c:
- Change e_runvi, the command used by ed_fulledit() (called by ^X^E
  in emacs and v in vi) to invoke a fullscreen editor:
  - 'fc' is the same as 'hist' but is shorter and standard. :)
  - Add a 'command' prefix to bypass a possible fc shell function.

src/cmd/ksh93/sh.1:
- Remove mention of the specific shell command executed by the
  emacs and vi editor commands; that's an internal implementation
  detail.
- Remove the incorrect claim that a command line edited by fc/hist
  is only executed if the file is saved.

Related: #748
@McDutchie McDutchie closed this in bb83575 Dec 24, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants