fix screen capture tools when used with xpra

[totaam]

Issue migrated from trac ticket # 988

component: core | priority: major | resolution: fixed

2015-09-24 23:54:34: Dweller created the issue

---

This bit of java should save a screengrab.. but all it saves is black rectangles where the Xpra windows are..

I suspect the underlying cause is the same one why my java based instant messaging client is unable to capture part of my screen to send to other participants. And likely also why screen casting in java based meeting software isn't functional.

import java.awt.AWTException;
import java.awt.GraphicsDevice;
import java.awt.GraphicsEnvironment;
import java.awt.Rectangle;
import java.awt.Robot;
import java.awt.image.BufferedImage;
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import javax.imageio.ImageIO;

public class Test {

	public static void main(String[] args) throws AWTException, IOException {
		GraphicsEnvironment ge = GraphicsEnvironment.getLocalGraphicsEnvironment();  
		GraphicsDevice[] screens = ge.getScreenDevices();       
		Rectangle allScreenBounds = new Rectangle();  
		for (GraphicsDevice screen : screens) {  
		       Rectangle screenBounds = screen.getDefaultConfiguration().getBounds();        
		       allScreenBounds.width += screenBounds.width;  
		       allScreenBounds.height = Math.max(allScreenBounds.height, screenBounds.height);
		       allScreenBounds.x=Math.min(allScreenBounds.x, screenBounds.x);
		       allScreenBounds.y=Math.min(allScreenBounds.y, screenBounds.y);
		      } 
		Robot robot = new Robot();
		System.out.println("Grabbing screen using awt robot.");
		BufferedImage bufferedImage = robot.createScreenCapture(allScreenBounds);
		File file = new File("screengrab.png");
		if(!file.exists())
		    file.createNewFile();
		FileOutputStream fos = new FileOutputStream(file);
		ImageIO.write( bufferedImage, "png", fos );
		fos.close();
		System.out.println("Done. File has been written to "+file.getAbsolutePath());
	}
}

[totaam]

2015-09-25 10:21:10: antoine changed status from new to closed

[totaam]

2015-09-25 10:21:10: antoine set resolution to wontfix

[totaam]

2015-09-25 10:21:10: antoine commented

---

We never composite the windows onto the virtual display, we send them to the client instead.

If you want the capture, run it where things are being displayed: on the client, not the server.

Not going to fix this, sorry.

[totaam]

2015-09-25 13:05:41: Dweller commented

---

Running the capture on the client OS isn't an option, because the chat client is running on the machine with the xpra server. So although yes, the capture code is running on the xpra server, it's doing so as part of an app being rendered via an xpra client.

I'm using xpra to display apps running on my remote server, one of those apps is an instant messaging client written in java, and one of it's features is to allow capturing part of the screen to send to a chat participant.

That feature of the chat client is non functional when the client is run via xpra, because all it can grab is black rectangles.

The same issue occurs when using e-meeting tools that allow sharing of the desktop, or part of the desktop.

I'm not expecting it to be able to grab windows that are not rendered by xpra, but did kinda expect it to be able to grab images of the ones that were.

I'm working round this today by launching an additional XServer on the client OS, and killing off & retargetting apps to that x-display when I need to share images from them.. which is less than ideal.

[totaam]

2015-09-25 13:12:07: antoine changed status from closed to reopened

[totaam]

2015-09-25 13:12:07: antoine removed resolution (was wontfix)

[totaam]

2015-09-25 13:12:07: antoine commented

---

We could add an option to paint onto the vfb.
This would need to be off by default, but I guess it could also be useful for debugging (the rare cases where "xpra screenshot" is not a good solution?)

The right place to do this is probably in the x11 server's damage function: call window.get_image and paint onto the root window at the window's coordinates.
Then we also need to handle window moves, resizes and destroy events: we have to repaint the vfb, initially we can just repaint everything in window z-order when that happens - and maybe do it via a timer so we don't overload the UI thread. The new option could be something like sync-vfb=off|delay-in-ms.

[totaam]

2015-09-25 16:48:44: antoine uploaded file paint-root-overlay.patch (1.3 KiB)

PoC

[totaam]

2015-09-25 16:48:57: antoine commented

---

Turns out not to be terribly difficult I think.
r10681 adds the bindings we need, and the patch above paints the root overlay window. It's leaky and inefficient, doesn't deal with window positions or resizes or moves... but screengrabs show the updated window contents.

This is yet another reason for dealing with the geometry bits from #907.

Came across this ticket: JDK-6903034 : java.awt.Robot.createScreenCapture() doesn't work for translucent windows

[totaam]

2015-09-25 17:23:39: antoine uploaded file paint-root-overlay-v2.patch (7.2 KiB)

much better implementation: with timer, command line option, etc..

[totaam]

2015-09-25 17:26:04: antoine changed status from reopened to new

[totaam]

2015-09-25 17:26:04: antoine changed owner from antoine to Dweller

[totaam]

2015-09-25 17:26:04: antoine commented

---

@Dweller: apply the v2 patch on top of trunk and run the server with:

xpra start --sync-vfb=50

You should then be able to do screengrabs.

If that works for you then re-assign to me and I'll fix it up.
(ie: the blue and red colours are swapped because we paint via gdk which expects RGBX and we feed it BGRX from the xshm image.. ideally we should be painting with xshm too for better performance, meh)

[totaam]

2015-09-25 20:27:31: Dweller changed owner from Dweller to antoine

[totaam]

2015-09-25 20:27:31: Dweller commented

---

Yes, it worked.. as far as I could test it..

A side effect of the patch seemed to be to kill off the ability for me to click on things.. I can click to activate windows, but not click on buttons, mouseover seemed to stop triggering.. and generally apps seemed to be missing my mouse messages.

So I could test with my sample code, as that's all pure command line, and yes, that grabs a screenshot that shows the xpra windows all present. But I couldn't actually test with the real chat client / emeeting stuff, because I although I could trigger their screen grab from a menu via the keyboard, I couldn't then dragselect the area of screen to capture as it ignored the mouse clicks.

Backported to 0.15.6, in case the lost mouse clicks were related to my build of trunk, but had the same behavior there. Tried setting the flag value to 0 and the function disables as expected, and then mouse clicks work just fine.

Intruigingly, the background for wbar changed from it's default grey to black while the patch was active.. (if wbar is run via a normal xserver, it's background is transparent) .. so looks like there could be ui effects beyond just screen grabs.

Sooo close !!

[totaam]

2015-09-26 07:55:06: antoine changed owner from antoine to Dweller

[totaam]

2015-09-26 07:55:06: antoine commented

---

Merged in r10682. Note: the flag has been renamed to sync-xvfb to make it more consistent with the rest of the options and docs.

It is far from perfect, it is still very slow for a start, but it is usable and as good as it is going to get for now.

I have added a one pixel border around the windows to make it a bit clearer. If we really wanted to, we could also paint a window top bar since we have the client's frame extents now #919 (meh, we don't have the top bar style or buttons to paint anyway).

[[BR]]

A side effect of the patch seemed to be to kill off the ability for me to click on things
[[BR]]
Fixed thanks to the excellent page so you want to build a compositor.

[[BR]]

Intruigingly, the background for wbar changed from it's default grey to black while the patch was active..
[[BR]]
Fixed in r10686, which also fixes the z-ordering of windows by keeping track of focus events. (the ones that had focus more recently are painted on top of others)

Potential future work (feel free to create a new ticket):

• performance: should be using xshm / xcopy and not roundtrip via GTK which is terribly slow
• could repaint only what has changed (very hard to do - meh)
• this could be enabled / disabled at runtime via control channel

@Dweller: please test and close if this works for you.

[totaam]

2015-10-05 12:21:53: antoine changed owner from Dweller to afarr

[totaam]

2015-10-05 12:21:53: antoine commented

---

@afarr: FYI as this may be of interest to you, we can now capture the virtual screen contents using regular capture tools and screen sharing tools should now work. Please close.

[totaam]

2015-10-05 13:26:23: Dweller commented

---

Did some testing.. it does function as far as screen grabbing IS possible when this patch is active.. but it also impacts the stability of the xpra server.. some apps that use screen grabbing seem capable of crashing the Xvfb process entirely.. (ok, not strictly xpra.. but same effect when all your apps are killed off) after limited investigation it would seem the kernel is killing off the Xvfb process due to memory constraints.

I'll try to find time to test again with a modified version of the test code that grabs in a loop.. if there's a leak somewhere maybe that can drive it enough to debug where..

Or maybe I should try and put Xdummy onto ubuntu instead of Xvfb ?

[totaam]

2015-10-05 13:35:20: antoine commented

---

@Dweller: Xvfb is only used as a workaround on Ubuntu, and since the leak seems to come from this process... this really does not look like an xpra problem.

[totaam]

2015-10-05 13:48:21: Dweller commented

---

I'll try the instructions at Xdummy for ubuntu, and see how I get on =)

[totaam]

2015-10-11 17:07:43: antoine changed title from Java cannot capture window contents when windows are displayed via Xpra to fix screen capture tools when used with xpra

[totaam]

2015-10-11 17:07:43: antoine commented

---

Some updates: r10797 (fix for trays), r10798 (paints workarea and monitor bounds)

[totaam]

2015-10-22 19:18:11: afarr changed status from new to closed

[totaam]

2015-10-22 19:18:11: afarr set resolution to fixed

[totaam]

2015-10-22 19:18:11: afarr commented

---

Yup, tested with 0.16.0 r10922 (OSX) client against a 0.16.0 r10916 server (using scrot, which was a new trick for me).

Worked as expected on the second try, but the first try only captured one of two xterms as well as a chromium window that was started from the xterm that wasn't captured.

I'm going to guess that this is a matter of the "It is far from perfect" stipulation of comment:4, and close.

[totaam]

2015-10-23 05:28:49: antoine commented

---

I can't reproduce the "second xterm not captured" - which should have worked, so leaving this closed.