Slow command execution on Ubuntu using Kaldi engine.

[m-spisiak]

Hi,

I'm trying to use dragonfly for simple commands like copy and paste, but I'm experiencing a substantial lag between recognition and execution of the command. that is, For example I take kaldi-grammar-main, And I say “Charlie”, the recognized command is printed in terminal correctly and fast, but then the execution, during which the cursor is frozen and waits, takes around a whole second before printing “c”. The same happens for any other command.

I asked @daanzu whether he knows about this issue being in kaldi-active-grammar, but he said it looks more like a general dragonfly problem.

I'm on Ubuntu 20.4, but I asked a friend to test it on windows and he reported the same issue.

Do you know about this problem? or can you guide me how to fix it, or where to look?

Thanks!

[m-spisiak]

I have found that it's an xdotool issue:
jordansissel/xdotool#281

any ideas, if there is a workaround?

[kendonB]

@m-spisiak did you find the workaround here? i.e. run setxkbmap jp

Does that work for you?

[drmfinlay]

This only occurs for some users. The author of Xdotool goes into much detail on this in the issue Kendon linked to above (jordansissel/xdotool/issues/10).

I would recommend trying a different desktop environment and seeing if that helps. This particular problem doesn't occur for me running Xfce.

I'm not sure how your friend was able to reproduce this on Windows. That would be a separate issue; Xdotool is not used on Windows.

[m-spisiak]

Hi,
@kendonB I havent found any workaround, only that of using EN-US layout for which the problem disappears, but I need my keyboard differently mapped for work, so this doesnt solve the issue really. At least not for me. Same for your advice with jp map, it messes up the keys from the keyboard...

@Danesprite it may well be as you say, that it doesnt happen to everyone, but it is not easy to just change the environment, since other things depend on it.
The fact that it happens on Windows was evidently a mistake. Later I tried myself and it does not happen on Windows. Evidently it is only Linux and maybe just Ubuntu.

[m-spisiak]

I wonder if it would be easy to rewrite the xdotools keystroke emulation to some other emulator. Before I was using https://github.com/aidygus/LinVAM and I didnt have this problem there.
I am very far from understanding how these things work, so I would try to hire someone for that, but I would need to know if it is a lot of work or not.

[kendonB]

@m-spisiak that platform references xdotool so it would be good to learn what they did to work around this issue (or if indeed they also experience it).

[drmfinlay]

Hello m-spisiak, You are quite right to say that it is not so easy to change the environment. Please forgive that oversight on my part. I do have an alternative solution that should work for Dragonfly, which is to use the library's pynput keyboard implementation instead of Xdotool. If you like, I can add a way to enable it. I cannot guarantee that this will solve your problem as I am unable to replicate it, but pynput does not use Xdotool. The pynput implementation does have some quirks, which is why Xdotool is used instead. (See Dragonfly issue #79). However, the quirks are considerably less bothersome than this lag issue. Thank you for verifying that this does not occur on Windows. Your friend may have observed the system blocking input if simulating long text -- a behaviour of the Win32 *SendInput()* function Dragonfly utilises. The Xdotool/libxdo code-base would be hard to rewrite if that's what you had in mind in your second post. Luckily, this should not be necessary.

[codebold]

Hello, i experienced the same issue, and also switched from xdotool to pynput on Ubuntu #323. However, there are some issues:

• The Key action does not work in Eclipse; Text is fine.
• The Gnome interpreter (ALT+F2) does not receive any input.
  Has anyone a solution/workaround for that?

[drmfinlay]

Hello Johannes, The issues you mention are what I referred to above as quirks and are related to pynput sending synthetic key events. There is a related pynput issue on this: moses-palmer/pynput#59. One solution is to have pynput send non-synthetic key events like Xdotool. As far as I know, there is no switch in the API for that. Adding one would be up to the author of the library and it might not be so easy. Another is to change Dragonfly to allow specifying the keyboard implementation used by individual *Key* and *Text* actions. This would allow working around problems on a case by case basis. The *use_hardware* argument of both action classes is used for a similar purpose on Windows. This solution seems better to me and should not be difficult to implement. As I suggested in #323, an environment variable will be added for setting the default keyboard implementation to use in an X11 environment. There are currently three: Xdotool, libxdo and pynput. It is strange that, on your system, the *Text* action works in one instance where the *Key* action does not, especially since both use the same pynput function internally. This sounds like a separate issue with pynput.

[codebold]

Hey Dane,

thanks for your reply! Yesterday i tried to work around these issue with individual actions using individual keyboard, as you also suggested. However, i got stuck because of an error. I will give it another try today.

I also have to reformulate my problem a little bit. Most keys are also fine with pynput. Only, if a modifier is pressed, the key combination is not recognized in Eclipse (all versions i tried). For all other applications that I tested, this behavior does not become evident.

So right now i failed with libxdo, so what's the difference between libxdo and xdotool?

[codebold]

Okay, i fixed my issue with xdotool. I had some weird wiring in the class hierarchy, so that the xdotool key action was trying to send pynput typeables. Now i can use both libraries concurrently, but it is really some hack for now... I i would prefer to get the modifiers running with pynput I guess at some future point in time...

[m-spisiak]

@m-spisiak that platform references xdotool so it would be good to learn what they did to work around this issue (or if indeed they also experience it).

Hi @kendonB, I am not sure here, xdotool is an optional requirement there and I searched and didn't see it in the code anywhere. But if you looked closer than me then you're probably right.

[m-spisiak]

Hello m-spisiak, You are quite right to say that it is not so easy to change the environment. Please forgive that oversight on my part. I do have an alternative solution that should work for Dragonfly, which is to use the library's pynput keyboard implementation instead of Xdotool. If you like, I can add a way to enable it. I cannot guarantee that this will solve your problem as I am unable to replicate it, but pynput does not use Xdotool. The pynput implementation does have some quirks, which is why Xdotool is used instead. (See Dragonfly issue #79). However, the quirks are considerably less bothersome than this lag issue. Thank you for verifying that this does not occur on Windows. Your friend may have observed the system blocking input if simulating long text -- a behaviour of the Win32 SendInput() function Dragonfly utilises. The Xdotool/libxdo code-base would be hard to rewrite if that's what you had in mind in your second post. Luckily, this should not be necessary.

@Danesprite Thanks a lot for the reply! I would give the pynput a go. Can you direct me to some docs that would explain how to make it work? I had a search of the official doc and didn't see it.

[codebold]

@m-spisiak https://github.com/dictation-toolbox/dragonfly/blob/master/dragonfly/actions/keyboard/__init__.py initializes the keyboard. It selects xdotool if you are on Linux and pynput if you are on Mac. So something like the code below should do the trick. Feel free to implement it in a nicer or more flexible way...

if os.environ.get("DISPLAY"):
    from ._pynput import (
        PynputKeyboard as Keyboard,
        PynputTypeable as Typeable,
        DarwinKeySymbols as KeySymbols
    )

[drmfinlay]

Hello all,

My apologies for failing to reply to the most recent comments on this issue.

To answer your question @codebold, the difference between libxdo and xdotool is essentially that the former performs the work that the latter does but in the same process. Hence, it is more efficient. I would prefer to use libxdo by default, but, last I checked, it doesn't work with Python 3. Or, at least the python-libxdo package doesn't.

Late last year I investigated a similar slowness issue with xdotool input. It might have been the same one discussed here, I'm not sure. Anyhow, I tracked that particular issue to a general sub-process latency problem present when the Kaldi or sphinx backends were used on POSIX, or POSIX-like, systems. (The issue was not present on Windows.) If libxdo were to be made available to Python 3 users, and Dragonfly were changed to use it for input instead of xdotool, the slow typing issue would probably go away.

(Incidentally, a Python package for libxdo could be installed with Dragonfly instead of separately, through the system's package manager.)

Other issues with xdotool could be avoided by allowing the keyboard implementation to be selected via an environment variable: DFLY_KB_IMP. That feature will be available in the next version. I also hope to add an additional dotool implementation, too, which will (probably) be a little more reliable. See issue #374.

[drmfinlay]

My environment variable idea would be a bit of a hack, really. The only real solution is to use libxdo, or something like it, by default.

[drmfinlay]

Just an update on this.

I would like to fix this properly, but I must admit that it is a little beyond me. I think the simplest thing to do, assuming this only affects some desktop environments, is to have in the documentation lists of which ones work well with Dragonfly and Kald/Sphinx, and which ones don't. I'll try to do some testing soon and put these lists together.