DECSDM (private mode 80) is interpreted incorrectly #41
Comments
|
Thank you for gathering all that evidence. How do you explain the "mistake" in the DEC documentation? It's not a simple transposition as it discusses both behaviors. |
|
Reading the VT3xx manual here, my thought was that the documentation writer probably got confused between the concept of the "Sixel Scrolling" feature in the Graphics Set-Up screen, and "Sixel Display Mode", which are really the opposite of each other. When you enable "Sixel Display Mode", you're requesting Sixel images be displayed full-screen, starting in the top left corner. You're not asking to enable scrolling. If Anyway, I don't want to claim that I'm absolutely certain about this, but it just seems very likely, given the evidence, that XTerm got it wrong. |
|
The other thing I meant to ask, was do you really need to set this mode at all? You'll probably have tested this on more terminals than I have, but all the ones I've tried have scrolling enabled by default, regardless of how they interpret |
|
Please share with me the list of terminals you've seen that have scrolling enabled by default. It makes sense that I see how you could think the DEC VT3xx manual is in error. I have been working off of a more thorough reference from DEC, EL-SM070-00 DEC STD 070 Video Systems Reference Manual Dec91 . |
XTerm (360), MLterm (3.8.4), Contour (0.1.1), WezTerm (20210502), Mintty (3.5.0), RLogin (2.25.3), and Reflection Desktop (16.2.0).
I figured that might have been the case, but hopefully whoever that was will have fixed the issue by now.
Yeah I've got the STD 070 manual, but that has no mention of |
|
Thank you for that research. I think you're right that if all modern terminals start with scrolling on, we might be able to skip the sequence. [Reminder to self: if we do this, check TERM and send it for those terminals that start with scrolling off (vt125?).] However, there is no way I know of to poll the current setting, so lsix cannot know what state the last sixel application has left the terminal in. Maybe this is worrying about nothing, but I notice other projects, like notcurses seem to think it is important to always enable sixel scrolling. |
|
By the way, as I reread the specs, I became less sure I read them correctly. I'd like to know what the actual hardware behaviour is, so I have purchased a VT340+. Hopefully we'll get this resolved in two weeks when it arrives. |
This is brilliant news! Once you've got it set up, do you think you might be interested in running some other Sixel compatibility tests too? I've been putting together a small set of tests covering edge cases in the spec that I was hoping to be able to run on a VT340 one day. The idea being to create a repository of examples with expected output which terminal developers could refer to if they want to be VT340 compatible. Each test is just a small text file that can be output to the terminal, and the result then photographed as a reference. For example, this is my current And this is what I would expect as output (assuming I'm correct in my understanding of the VT340 behaviour). Click to view
|
Yeah, I've been slowing raising issues with other apps that I've seen doing this. So far I've yet to find anyone that had a legitimate reason for it. I haven't got around to notcurses yet because I wanted to be able to actually test their code first, and I couldn't get it to build. I should probably just raise the issue anyway.
The VT125 (and the VT240) don't support Sixel scrolling at all. They're effectively always in "display mode", so the image will always start in the top left corner of the screen, and will be clipped when it reaches the bottom. |
ghost
mentioned this issue
|
@dankamongmen seems like the kind of person that would happily join this discussion even without a formal issue being filed :) |
|
i'm familiar with my valued comrade @j4james 's DECSDM march: i think he has the right idea, but i think in practice it would turn out flawed, as the (over)eager merging of my PR to mlterm kinda supports IMHO: in the end, pretty much no one got everything they wanted, and all parties seemed a bit less happy than they were before. i apologize for my role in this expansion of sadness, but have high hopes for a brighter tomorrow. |
|
Yes, I was thinking of asking @dankamongmen his opinion about omitting the scroll initialization. I have a guess, though, given his frequently expletive laden comments about other software packages. 😏 Like XTerm author @ThomasDickey, he clearly cares about good engineering. If @j4james is correct, his suggestion would work as long as no other application had changed the default setting. lsix might be able to get away with it as simplicity is one of its design goals. My guess is Nick Black would reject it out of hand since notcurses has to be more robust than that. Given that different terminals (XTerm and mlterm) interpret the same sequence in opposite ways, a reasonable solution might be a TERMCAP entry for "enable sixel scrolling". But I would want to get @ThomasDickey's opinion first. |
a terminfo entry definitely would be the preferred way to get this information (aside from issues with
=D i'm not sure i've ever spoken with you, @hackerb9 , but i dig your style |
|
I can wait til you've investigated the VT340. The existing documentation is too sketchy to say whether one was wrong or whether DEC happened to implement the feature differently: The PDF for the first editing of the VT330/VT340 graphics manual doesn't mention the feature, and the VT382 manual is undated. |
For the record, the Thai VT382 user guide is dated August 1989, and the second edition VT330/340 programmer reference is from May 1988. |
|
Preliminary tests with an actual VT340+ vindicate @j4james and mlterm. I will likely remove the sixel scrolling sequence from |
|
appreciate you getting a firm answer! |
|
so now the question becomes: is a great effort to be made in reforming other terminals? i know many of them were "copied from XTerm in the way of lemmings" in @j4james 's approximate words, but i doubt they're so broadly actively pulling in fixes. either way, what i would like to see is a new terminfo capability covering this. i would then be inclined to (in Notcurses) assume the inverse meaning for terminals lacking the entry in their terminfo databases (since that seems to be the most widespread interpretation currently), possibly special-casing the {small} number of terminals known to use the faithful interpretation. where the terminfo capability was defined, i'd of course use that. @ThomasDickey, especially if you end up changing XTerm behavior to reflect @hackerb9 's confirmation of @j4james 's , I'd very much appreciate a terminfo entry abstracting away DECSDM. thoughts? |
for what it's worth, i already have ghastly |
|
If you have |
well, sure, but (especially given the widespread differences in implementation) it means i (and you, as NCURSES maintainer, should NCURSES ever use this capability) have to effectively classify, record, and track every terminal in existence =]. if we can't have it in terminfo base -- iirc, you've stated that terminfo will "only hold those capabilities needed/used by NCURSES" (apologies if i'm misquoting) -- i will move to introduce it as a terminfo extension capability. it seems better for all involved if we can have it as a base capability, but i am of course ignorant of the costs involved in maintaining terminfo. i try to use |
and there are of course many (most?) terminals which don't support a meaningful |
|
Alacritty (reading the source code...) identifies itself as a |
I'm pretty sure that quote about terminfo being limited to only what ncurses uses is out of context since IIRC the manpage references capabilities that ncurses does not use. I hope @ThomasDickey corrects me if I'm wrong, but I believe he has taken the Noah Webster approach to maintaining terminfo: description, not prescription. He doesn't add capabilities because they are potentially useful, he adds them because they are actually used. While he has good insights, since he is an engineer and a developer, it would be up to us to coin a new capability and start writing code that uses it. As for Augean task of adding the behaviour of every sixel capable terminal to terminfo, I do not think that is part of the terminfo maintainer's job. If people care about a terminal, they will submit a patch for it. Terminals that people don't use don't need to get updated. Terminfo has never been "complete", it just looks that way because it's been accreting like a coral reef for eons. |
i certainly meant at no time to impugn Mr. Dickey, whose torch we merely carry, and of whom i am a great admirer. i cannot find this quote, and thus i cheerfully withdraw it. in any case, it was not meant to be a judgment of Mr. Dickey's maintainership, but a recognition of his prerogative. i would be happy to collect this information and prepare it; i did not mean to imply that any other free software volunteers ought do the work. i did not, however, want to do this if there was no chance of having the compiled data entered into terminfo. i apologize if any of my communication was unclear. i have no place nor desire to dictate what ought be done with terminfo. i simply think this would be useful to all terminfo clients. |
looking back over the exchange, i think this was insufficiently surfaced. @ThomasDickey , by no means did i mean that you ought go collect this information for my benefit =] i would be more than happy to perform the investigation if the data will be admitted to terminfo (i've submitted patches to terminfo--and xterm, and ncurses--before, so i hope i have some small cachet with the Grandmaster from the Invisible Island), |
|
Just FYI, as soon as XTerm has shipped a build with It still baffles me that you think the solution to this problem requires a complex rigmarole of terminal version detection and terminfo queries, when the whole issue could be avoided simply by not sending the mode in the first place. You're focusing on the hypothetical problem of somebody possibly changing the mode from the default (which they could easily fix themselves), and ignoring the very real problem of your code just not working on multiple terminals. |
|
agreed: adding a feature to |
Are you saying there were no physical terminals that used DECSDM in the (backwards) way DEC documented for the VT340? If you think nobody needs this behavior, why bother asking terminal emulators to interpret it correctly? (Note: I'd like to see them interpret it correctly because I think DECSDM can be useful. For example, the sixel screen-dumps from the VT340 always end with a graphic newline that would scroll the top of the image off the screen. It's handy to disable scrolling before displaying such images.)
I get what you're saying, and it will be removed from the next release of lsix, but it is a bit harsh to say it is a hypothetical problem. One very real example: As you may recall, I added that escape sequence to lsix because some terminals were defaulting to not scrolling. Now that same sequence may actually cause the problem I was trying to fix. I have another tool (sixgif, never released) which does the opposite, using the no-scroll mode. A person using either tool would not necessarily know how to fix scrolling problems in other sixel programs they run later.
Your proposal to not use that sequence (and hope no one else does) is good enough for lsix, but I have no illusions that it is a proper solution. I am coming to the conclusion that the "correct answer" is to define a way to query the current DECSDM state so it can be restored when a program exits instead of having lingering effects. That way, for example, lsix might have scrolling issues, but it wouldn't affect anything run afterward. Of course, the hard part is convincing all the terminal emulator developers to implement it. If that were to happen, there would be real benefit in asking them to also swap how they handle DECSDM. |
To the best of my knowledge, yes. The VT125 and VT240 didn't support scrolling at all. And both the VT340 and VT382 work as documented in the VT382 manuals (as originally pointed out by @ttdoda and confirmed by you). I don't know of any other hardware terminals that natively support sixel images.
Sorry, that wasn't very clear. I'm not saying nobody needs There are a few more things that need to be fixed to really take advantage of it though - like most terminals don't handle background select correctly,
I'm sorry my message above came across as harsh. I completely understand why you may have chosen to do that initially, but there's no evidence that that problem still exists - certainly not to the point that it outweighs all the terminals that you now know are broken by enabling
I know it's not ideal, but this can happen with any number of modes, and users can always ask for help and get the problem solved. As things stand now, though, there is absolutely nothing they can do if they're trying to use an app like lsix or notcurses and it doesn't work on their terminal (well lsix I suppose could reasonably be patched by a user, but that's really not feasible for notcurses).
There is already a well-defined way to do that - the But as you say, lsix is still going to have scrolling issues if you aren't setting the mode correctly in the first place - whatever you want it to be. If you have a genuine use case for setting |
That is precisely what I was thinking. If I recall correctly, the VT340 manual does not list DECSDM as queryable, which may explain why terminal emulators haven't implemented it yet. Here's a little tool you can use to check if DECRQM works for detecting DECSDM: checksixelmode.sh. I'll test my VT340 hardware when I get home. |
I actually meant that they don't implement As for why it's not in the VT340 manual, remember that Btw, can I ask again if you would be willing to run some other compatibility tests on your VT340 when you have time? It's totally OK if the answer is no - I just want to know if I need to look elsewhere for help with that. |
|
@j4james Sure, I can help with running some tests. I'll start a new project for that so we don't spam everyone on this bug. Once you get your results, are you thinking of adding your VT340 compatibility tests to @ThomasDickey's vttest? |
My initial plan was just to produce a simple set of text files that you can Producing tests along the lines of vttest is more complicated, though, because you'd ideally want to break things down into many more test cases that just cover one issue at a time, and you need to be able to describe the expected output in text. We can maybe progress to something like that eventually if you're keen, but it'll take a lot more work. Anyway, I'll put together a PR on your new repo tomorrow with the tests I have so far, and we can discuss further there. |
Okay, I've got a skeleton up here: https://github.com/hackerb9/vt340test . We'll talk more about non-DECSDM matters there. Speaking of which, I can confirm that on the VT340 DECSDM is always exactly the opposite of "Sixel Scrolling" in the Setup Screen. The VT340 does respond correctly to DECRQM (as does XTerm, of course). So, eventually it may be possible for lsix (and notcurses) to limit the damage caused by using DECSDM if it does the opposite of what is expected. |
i can dig it! |
In hackerb9#41, it was shown that enabling DECSDM, disables sixel scrolling. Something many terminals, including foot, has gotten wrong. In 1.8.2 this was changed in foot, so that setting DECSDM now disables scrolling. Thus, lsix needs to emit \E[?80l, not \E[?80h.
After purchasing a genuine DEC VT340 terminal, hackerb9 has learned that the DEC VT340 graphics programmer's manual was mistaken and the sense of DECSDM should be inverted. Nearly every terminal emulator which built upon that reference over the last thirty years is incorrect. Some are correcting the error now, some never were wrong, but we are at a time when it is not safe to presume what setting DECSDM means. For now, lsix will simply presume that sixel scrolling is enabled and hope that no program has been run before it that disables it. See #41 for details. Thanks go to @j4james for pointing out the mistake and for testing multiple terminal emulators to show that none of them currently default to having sixel scrolling disabled.
|
Release 1.8 is out and, as @j4james suggested, Thank you very much, @j4james, and to all who helped with this. |
|
fwiw, I made the suggested change to disable SIXEL scrolling on August 10, but have another feature improvement to complete before patch 369 (and am at the moment deep in a change for ncurses). The comment here about |
you feel relatively certain that the scrolling change will be present in Patch 369? if so, i'll likely add code to anticipate that. |
|
By the by, I've written a little shell script, testdecsdm.sh that attempts to detect if DECSDM is reversed on a terminal or not by checking if the cursor row moves down when sixels are displayed. It works for me, but has not been extensively tested. |
There has been some confusion whether enabling DECSDM (private mode 80) enables or disables sixel scrolling. Foot currently enables scrolling when DECSDM is set, and this patch changes this, such that setting DECSDM now *disables* scrolling. The confusion is apparently due to a documentation error in the VT340 manual, as described in dankamongmen/notcurses#1782 (comment). And that makes sense, in a way: the SDM in DECSDM stands for Sixel Display Mode. I.e. it stands to reason that enabling that disables scrolling. Anyway, this lead to hackerb9/lsix#41, where it was eventually proven (by testing on a real VT340), that foot, and a large number of other terminals (including XTerm) has it wrong: hackerb9/lsix#41 (comment).
|
While we're all here, is DECSDM also included with the other flags in save/restore cursor DECSC/DECRC? |
|
I strongly suspect the answer to that is no. The only mode that |
|
This is a bit out there, but on a hardware terminal would DECSDM permit displaying pixels all the way down to the bottom text row? It sounds like yes, and I have a test that worked in foot and partially in xterm. If this is what is supposed to happen, then I would like to encourage those terminals that choose to support DECSDM to also support writing fully into the bottom row without scrolling. |
It should, yes. There's a
When you say "partially in xterm", is this related to the
I think most of them already do. I probably need to go back and double check this, because some still had |
Probably a different bug. The image is being drawn from the top-left corner as expected without scrolling, but only the first sixel row (6 pixels) is written to the bottom text line. Example: https://gitlab.com/klamonte/jexer/uploads/8e957be7b082702c6f6370a1a10e62d8/xterm_sixel_bottom.png . The "text cursor" on the bottom row is actually an image, and if you zoom a lot you can see that a bit of the bottom text row is getting pixels. |
Interesting. I hadn't noticed that before, but it's easily reproducible just catting a largish sixel image with |
I'm fairly certain that you've got the
DECSDMmode backwards. I think it is XTerm that has implemented it incorrectly, and MLterm that has it the right way around.CSI ?80lshould enable sixel scrolling, andCSI ?80hshould disable it.Other terminals that match MLterm's behaviour include RLogin (you can see the source here), Tanasinn (see here), and WRQ Reflection (it's a commercial app, but you can see the documentation describing the behaviour of
DECSDMhere).I suspect the misunderstanding came about because the VT330/340 manual documented
DECSDMthe way XTerm has currently implemented it, but I believe that was a mistake. The various VT382 manuals have it the other way around. See here, and here.The author of Tera Term, who has a collection of DEC terminals, actually tested this on a VT330 and VT382 and reported the results here: https://twitter.com/ttdoda/status/479053314412126208
According to Google translate:
Which I understand to mean that sixel scrolling mode is enabled when
DECSDMis reset.Now I guess it's possible that everyone else is wrong about this, and XTerm is right, but all the evidence appears to point in the other direction.
The text was updated successfully, but these errors were encountered: