Xlib error clues

[RedBearAK]

This is an extension of issue thread #206, which was getting too long to have a useful conversation.

I've discovered a clue that might help lead to a resolution of the persistent Xlib error talked about in that thread and encountered mostly in relation to IceWM but also on other systems.

The Xlib error NEVER seems to occur when logging directly into IceWM straight from a boot/reboot. But it ALWAYS occurs after logging out of the DE and logging back in.

A secondary phenomenon that I've been observing for a long time (that now appears to be somehow related) is that when entering my password after logging out from the desktop, the FIRST CHARACTER of my password is ALWAYS "eaten" by something, thus causing my first password attempt to frequently fail before I figured out what was happening.

This phenomenon as well DOES NOT appear to happen on a fresh reboot, and appears to ALWAYS happen after logging out and trying to log back in.

For a long time I assumed this was just some quirk with the old MX Linux laptop I was using, or the operating system. But I switched to Linux Mint 19.3 on that same system and it occurred again. It doesn't seem to happen prior to the first install of Kinto (this was difficult to discern because Kinto would almost always be one of the very first things installed), and it also doesn't seem to happen when Kinto is installed but completely disabled.

I just tried to verify this one more time and noticed that systemctl shows two things being removed when disabling the xkeysnail service:

Removed /etc/systemd/system/graphical.target.wants/xkeysnail.service.
Removed /etc/systemd/system/xkeysnail.service.

I guess the first is a symlink that gets re-created when re-enabling the service.

Anyway, I disabled Kinto and logged out/in a few times, and the first character was never "eaten" while Kinto was inactive. So it's definitely Kinto behind that phenomenon.

I also had another thought, so I stopped Kinto before logging out, and manually started it after logging in (still in IceWM). In this scenario, I DO NOT have to run the xhost fix to get Kinto working. It just starts right up. I think that's another important clue.

Kinto also appears to be in a different state just prior to logging in to the desktop for the first time after a reboot. This is a state that I never see under other circumstances:

● xkeysnail.service - xkeysnail
   Loaded: loaded (/lib/systemd/system/xkeysnail.service; enabled; vendor preset: enabled)
   Active: activating (auto-restart) (Result: exit-code) since Sun 2020-07-12 15:09:22 AKDT; 2s ago
  Process: 1575 ExecStartPre=/bin/bash -c /usr/bin/xhost +SI:localuser:root && /sbin/runuser -l kris -c /home/kris/.config/kinto/prexk.sh (code=exited, status=1/FAILURE)

Basically, prior to the very first login attempt, Kinto appears to be "activating", or not actually running yet, but after the initial login and logout, it will instead show the Xlib error or "Watches established" and be "active" rather than "activating".

So the logical conclusion is that Kinto is remaining active inappropriately when it should actually be going back into the "activating" or "failed" state shown above, in which case it would ALWAYS WORK upon logging back into IceWM (or, presumably, any other DE).

Is there something already existing that is supposed to trigger Kinto to go inactive upon the user logging out of the graphical target? Can that maybe be enhanced to more reliably "stop" Kinto when the user logs out, so that Kinto can start up more reliably when the user logs back in? And if we can get Kinto to "stop" upon logout, it should stop "eating" the first keystroke of my password.

I've also observed the password keystroke being eaten on another physical machine running elementary OS. Again, only when logging out and back in, not on a fresh boot. So it's not just some weird quirk of this old laptop.

[RedBearAK]

To follow up with something I probably should have tried a long time ago but it just never occurred to me, I am able to log into IceWM and then manually stop and start xkeysnail via systemctl, and running the xhost command to "fix" Kinto is not necessary at all. But it is necessary if I'm attempting to "restart" Kinto with systemctl. That always fails until I run xhost.

So something is getting reset when "stopping" xkeysnail and "starting" it, that isn't getting reset properly during a "restart" of the xkeysnail service. If we can figure that out...

[rbreaves]

Makes sense and yea that issue should be resolved. I have also noticed the character eating bug as well and had not realized that was related to Kinto. The difficulty in fixing it is that the service runs on the sudo level as it needs to - but ending a sudo process on a user initiated logout is an unexpected task for a systemd service not initiated by the user (even though it only initiates on user login regardless).

I will look for a place to put a script in place to end the process on logout properly.

A very simple bash script that runs on the destruction of the x11 user session will be sufficient. This command can run just fine - despite it calling sudo due to the limitedadmins addition.

sudo systemctl stop xkeysnail

Looks like the key will be to anticipate the login manager in use and install the proper logoff script accordingly. If I can find a universal way of doing it though then I will.

[RedBearAK]

Ha, I googled for a bit and was about to ask if you knew a sort of universal way of triggering a logout script. Guess this won't necessarily be a simple task.

Try not to fall into the idea that you can just detect whatever DE is in use at Kinto install time and install that version of the shutdown trigger. It needs to work for those of us that hop around playing with different DEs and WMs, which last time I checked is very common among Linux users. If there are multiple ways to trigger a shutdown script for each of the common DEs, they should probably all be installed together.

No more specific ideas about what the difference is between systemctl stop/start vs. restart? I feel like that's really at the heart of understanding the root cause of what's going wrong.

[rbreaves]

It's even a little more complicated than that.. a DE can be used with a number of login managers and I don't currently check for what your login manager is - if I had a reason to have done that before then adding this would be well not simple still but easier. As it stands I'd have to support at least the 2 or 3 most popular login managers, and detect them properly - whichever is in current use and not just detect the existence of them.

[RedBearAK]

Seems a much simpler path is just figuring out what causes (code=exited, status=1/FAILURE) on boot, and why that doesn't come back when returning to the login manager. After all, the same login manager is running on first booting up, asking for the password, but that error doesn't return after logging out of the first (or any subsequent) session. Only on first booting up.

[rbreaves]

I think I can probably fix it with a 2nd systemd service - but making it a local service instead that will run on logoff. That will likely be the shortest path to success on this.

update well that did not work execstop does not actually execute on the user logging off - the 2nd service ends but that doesn't really do me any good.

[RedBearAK]

This is probably a ludicrous idea but what about putting the main xkeysnail process inside some kind of "wrapper" process that just runs as the user? Then you kill the wrapper process and it automatically kills the child process. Sort of like running a GUI app from a terminal and then closing the terminal, which usually kills the program you started from the terminal (unless you launched it in a way that makes it independent). No idea how well that would work when the child process has elevated privileges.

Just trying to use my imagination.

[RedBearAK]

Possibly a less ludicrous idea: The processes in the autostart directory, like in my case Albert, those are all normally supposed to be terminated when the user logs out, right? It's just that the xkeysnail service has to run as sudo and systemd just tries to keep it running.

What about creating a "heartbeat" process in a separate autostart file, that runs with user permissions and keeps running until the user logs out, and have xkeysnail keep itself inactive unless that "heartbeat" process is detected in memory? So sort of the reverse of trying to kill xkeysnail externally. Let it kill or stop itself based on external detected criteria.

Obviously we'd have to watch out for the case where the heartbeat process dies for some reason while the user is still logged in, but I think it could work in general.

But I'd also have to figure out how to get the autostart stuff to actually run at login when logging into IceWM. All I know about the autostart folder so far is that it is related to xdg. Whatever that is.

[rbreaves]

That's sorta what I was trying to do but that failed.. the ExecStop command never executed and then I even tried placing a trap command in the ExecStart field of the user based systemd service to detect the process being ended so it could then end the actual sudo systemd level process but that also failed.

I think the answer instead is in updating the xkeysnail service to actually grep or listen for an Xlib error - which does reliably appear on log off - if I can suss that out then I can pipe in the command for the process to terminate itself.

[RedBearAK]

Yes, that sounds like a good step to take.

I got tired of trying to figure out how to run the autostart stuff so for the moment I just figured out how to run a startup and shutdown script in IceWM. Turns out to be relatively easy, and because of your limitedadmins file I can make it work without requiring any authorization.

touch ~/.icewm/startup ~/.icewm/shutdown
chmod +x ~/.icewm/startup ~/.icewm/shutdown

Contents of startup script:

#!/bin/bash

sudo /bin/systemctl stop xkeysnail && sudo /bin/systemctl start xkeysnail

albert &

Contents of shutdown script:

#!/bin/bash

sudo /bin/systemctl stop xkeysnail

This works for now but of course I'll have to remember to disable it all to test whatever else you might come up with.

With this in place, there's no more keystrokes being eaten at the login manager screen, and Kinto is working right away after logging into IceWM.

[rbreaves]

Well it looks like everything that I have tried did not work and I am about to call it a night..

I think implementing this apparmor changes will be the only thing that'll fix this and I had actually wanted to avoid this because it sorta lowers the permissions to low level system components - just so that the user can run them without elevation that requires a password. Not ideal - but every other means I have tried has result in failure pretty much. Although I never explicitly tried allowing lightdm, the login manager to disable the service. I did find another way to end the service - but only after it actually crashed without x11, and I am afraid the login managers will likely do the same - in which case "eating" a character will still be present. The service has to end BEFORE the x11 error or a character "eating" type of event will occur on the next login attempt.

mooz/xkeysnail#12

Only other thing I can think of would be to modify a DE so that just before it uses, probably, dbus to fire off a logoff command it will instead end the service first and then logoff like normal - but that would be a lot to expect - recompiling a DE. Be better if intercepting dbus was possible.

[rbreaves]

Also tbh - it might be better to try and move this functionality directly into xkeysnail instead of the service - as the character eating issue I believe to also be a problem in how xkeysnail crashes. If xkeysnail simply hooked into dbus to monitor for a logoff event then this issue would be avoided imo. Not all DE's make use of dbus signals though.. and they can and will vary so that's likely part of the complication there as well. Finding a universal fix for this no matter the direction appears to be rather difficult.

https://unix.stackexchange.com/questions/28181/how-to-run-a-script-on-screen-lock-unlock
https://stackoverflow.com/questions/14322688/in-gnome-what-signal-on-dbus-monitor-indicates-the-user-is-logging-out/15009892
https://askubuntu.com/questions/720380/how-can-i-run-a-local-command-to-run-a-script-on-just-before-log-out-of-a-un

[rbreaves]

This might be a universal path forward.. where a custom module gives the pam - login/authentication component the ability to run scripts as the root user. A potential attack vector as well.. but not entirely sure it would be any worse than someone know how to execute a script via some other means. May still be preferable to loosening restrictions on low level input components to the normal privileged user which I am also trying to avoid.

http://tlgp.sourceforge.net/hints/files/execute-session-scripts-using-pam.txt

Other things to note directly related to Ubuntu Budgie
https://discuss.getsol.us/d/19-one-click-shutdown/11
https://askubuntu.com/questions/92550/how-to-log-out-with-a-command-in-a-gnome-less-environment

[rbreaves]

Ok it looks like I solved it for Ubuntu Budgie and probably Gnome. Having a script like the following to monitor for logoff is all that is required really, so this will become part of the solution for Gnome and Budgie, but I will need to suss it out for other DE's. I believe this will likely be the simplest solution to this problem going forward.

I know that this works because the character eating issue goes away when the monitor script is running. It definitely works.

dbus-monitor --session "type='signal',interface='org.gnome.SessionManager.Logout'" | grep '1' |
while read x; do
  echo "$x"
  sudo systemctl stop xkeysnail
  break
done

[rbreaves]

The alpha branch now has a working solution implemented for Ubuntu Budgie and probably other Gnome DE's. Also removed experimental support for Back/Forward buttons in browsers as it was never stable enough and needs to implemented via changes in xkeysnail any ways, not kinto imo.

Will need to do further testing with other DE's before inclusion into Dev, also need to ensure that if I cannot hook into dbus that the user is made aware. I get the feeling you and others may use some obscure DE's that do not make use of dbus calls for logoff and I am not sure what the best way of dealing with them may be. The PAM script module method I mentioned above may be doable, but then again there might be a better method still as I don't really want to compile and include a 3rd party module if that can be avoided.

d727489

Not sure but I am also wondering about the possibility of monitor syslog for "lost session" or anything related to the session ending - although that might not register as quickly as monitoring dbus, assuming dbus is available.

For future reference in supporting xfce via dbus.
https://forum.xfce.org/viewtopic.php?pid=48508#p48508

[rbreaves]

Alpha branch now has a Budgie Applet fully implemented with the ability to shut down the xkeysnail service on logoff. Additionally it has the ability to set autostart on and off, toggle xkeysnail, edit config, open system shortcuts, and open language settings.

These features will also be built out into a more standard gnome, and xfce, and possibly kde compatible plugin extension.

[rbreaves]

And the Budgie Applet is now pretty much defunct.. I reused large chunks of the code however to rebuild it as purely just an AppIndicator3 system tray app. It is now in the alpha branch. I will try to test it in a few distros in the coming days, but it should work with any distro that supports AppIndicator3 system tray apps.

The main issue with Budgie Applets is that users have to manually add the Applet after it installs which I didn't like. Of course it is also limited to just Budgie.

[rbreaves]

Partially fixed in the current dev branch. The fix applies to Gnome3, Budgie, and Mate & includes a system tray. The more universal fix is less ideal and has been left commented out since it failed in one of my early tests.. non-official flavor known as Ubuntu Unity. More testing & commits needs to occur before I can say this has been resolved across all currently support distros.

4f56235

[rbreaves]

Adding support for additional DE's ought to be a little easier in v1.2 as I have methodologies configured for applying different keymaps per distro, but not certain about xlib errors. Try the latest and let me know either in a new ticket or add on to this one if relevant.