webcam and scanner forwarding

[totaam]

Issue migrated from trac ticket # 1030

component: core | priority: major | resolution: fixed

2015-11-13 06:41:26: antoine created the issue

---

So that applications that want to access the webcam can be made to work (ie: skype, etc). Scanners use the same framework as webcams, so maybe we can handle those too? (means potentially more than one device...)

We will need a simplified "window video source" class that the client can use to send pixel data with the same adaptability as the server (but maybe simplified a bit when it comes to batch delay, speed and quality tuning). It would be good if we could know when frames are being consumed from the server device so that we don't bother updating it when no-one is watching.
It will only deal with full frames, and only using a video encoder (if one is available) and no mmap.
It should have the same controls as sound: on, off or disabled.

The server should only need to pass the frames to a virtual video device for this to work:

• v4l2loopback
• Motion - Video Four Linux Loopback Device API
• v4l2vd "V4L2 Virtual Device"
• webcamstudio

The initial version should not worry about:

• switching resolutions and formats: just one format will do
• controls such as showing the whole desktop instead of the video input
• sound sync!
  etc..

[totaam]

2016-02-01 06:30:43: antoine uploaded file webcam-forwarding.patch (35.4 KiB)

almost working patch using opencv, png encoding only for now

[totaam]

2016-02-01 20:31:21: antoine commented

---

Proof of concept code merged in r11808 with some severe limitations:

• we feed the wrong image format so the picture looks messed up
• png and jpeg only via PIL
• framerate is rate limited by packet roundtrip speed, and it may still cause bandwidth issues...
• should be able to specify the virtual device to use on the server using the command line option
• we should have a tray menu to enable/disable the feature and allow the user to select the device
• detect hardware changes (both on client and server): hotplug events

---

Build updates.

These things will need to be added to Dependencies.
We don't actually use or need some of those, but installing them anyway is easier than excising them out of matplotlib / opencv.

• Linux (generic): just install [https://github.com/umlaeute/v4l2loopback]

• OSX needs:

• [https://pypi.python.org/pypi/pyparsing]

• [https://pypi.python.org/pypi/Cycler]

• [https://pypi.python.org/pypi/pytz/]

• matplotlib on 10.5.x needs these patches: [https://trac.macports.org/browser/trunk/dports/python/py-matplotlib/files?rev=144224]

• opencv (huge download at 75MB!), then build with: cmake . -DWITH_IPP=OFF && make
  Almost gets you there, except it fails on directx.. (why on earth is it even trying to build directx on osx??)

• win32:

• C:\Python27\Scripts\pip.exe install E:\matplotlib-1.5.1-cp27-none-win32.whl

• C:\Python27\Scripts\pip.exe install E:\opencv-python-3.1.0-cp27-none-win32.whl

---

Still TODO:

• new wiki page
• documenting the installation of the virtual video device (rpmfusion has it, debian has it)
• documenting the configuration of video devices (groups and permissions)
• detecting device changes: dbus hotplug events for linux?
• packaging: add recommends opencv for Debian (no depends for rpms?)

[totaam]

2016-02-04 19:27:34: antoine commented

---

Added a wiki page with details: Webcam.
Will follow up in #1113.
This is good enough as a proof-of-concept only, it will need more work - but not for this release.

@afarr: ready for an initial round of testing. Please re-assign to me for bugs or to smo so that he can deal with the required win32 / osx / Fedora build updates.

[totaam]

2016-02-05 21:41:24: maxmylyn commented

---

• Built and installed v4l2loopback
• Upped client and server to r11861

Attempting --webcam=yes outputs the following error:

2016-02-05 13:40:09,266 Warning: virtual video directory /sys/devices/virtual/video4linux not found
2016-02-05 13:40:09,266  webcam forwarding disabled

And, obviously, the webcam forwarding does not work. Am I missing something?

I'll keep looking at it to see if I need to do anything else, but I'm pretty sure all the bits are there and ready.

[totaam]

2016-02-05 22:19:34: maxmylyn commented

---

Update:

• one must first load the device modprobe v4l2loopback

No more error.

However whatever I try, I cannot get it to detect a webcam on the client side and successfully forward it (even using a Win8.1 r11850 client).

[totaam]

2016-02-05 22:20:55: maxmylyn commented

---

Oh, also some logs:

Client:

C:\Program Files (x86)\Xpra>Xpra_cmd.exe attach tcp:10.0.32.189:2200 --webcam=ye
s -d webcam
2016-02-05 14:19:57,086 Xpra gtk2 client version 0.17.0-[r11850](../commit/1832392c746dc201fc58c508e597b285c6a7863a)
2016-02-05 14:19:57,086  running on Microsoft Windows 8.1
2016-02-05 14:19:57,086 webcam forwarding: yes
2016-02-05 14:19:57,275 GStreamer version 1.4 for Python 3.4
2016-02-05 14:19:57,634 OpenGL_accelerate module loaded
2016-02-05 14:19:57,711  detected keyboard: layout=us
2016-02-05 14:19:57,727  desktop size is 2560x1440 with 1 screen:
2016-02-05 14:19:57,727   Default (677x381 mm - DPI: 96x96) workarea: 2560x1400
2016-02-05 14:19:57,727     DISPLAY1 (597x336 mm - DPI: 108x108)
2016-02-05 14:19:57,727  upscaled by 125%, virtual screen size: 2048x1152
2016-02-05 14:19:57,727   Default (677x381 mm - DPI: 76x76) workarea: 2048x1120
2016-02-05 14:19:57,727     DISPLAY1 (597x336 mm - DPI: 87x87)
2016-02-05 14:19:57,775 Xpra X11 server version 0.17.0-11382
2016-02-05 14:19:57,775  running on Linux Fedora 23 Twenty Three
2016-02-05 14:19:57,775 webcam server support: True (1 devices)
2016-02-05 14:19:57,789 Attached to tcp:10.0.32.189:2200 (press Control-C to det
ach)

2016-02-05 14:19:57,789 webcam_state() webcam forwarding=True, server virtual vi
deo devices=1
2016-02-05 14:20:00,556 sound output using audio codec: Vorbis
2016-02-05 14:20:14,983 setting scaling to 100%:
2016-02-05 14:20:14,983 sending updated screen size to server: 2560x1440 with 1
screens
2016-02-05 14:20:14,983   Default (677x381 mm - DPI: 96x96) workarea: 2560x1400
2016-02-05 14:20:14,983     DISPLAY1 (597x336 mm - DPI: 108x108)

Server:

2016-02-05 13:57:41,569 init_virtual_video_devices
2016-02-05 13:57:41,570 found 1 virtual video device

(the only webcam related print)

[totaam]

2016-02-06 05:56:19: antoine commented

---

one must first load the device modprobe v4l2loopback
[[BR]]
Yes, it is on the wiki page Webcam.
r11862 tries to make it clearer in the server warning message.

[[BR]]

C:\Program Files (x86)\Xpra>Xpra_cmd.exe attach tcp:10.0.32.189:2200 --webcam=yes -d webcam
[[BR]]
As per the wiki page.. use the value "on" if you want it to start when you connect.
Unless you disable webcam completely (values "no", "false"), the webcam support will be available but needs to be started from the systray.

[[BR]]

r11863 and r11864 improve logging, command line parsing and error handling.
New beta builds with those changes are available.

[totaam]

2016-02-09 17:17:50: maxmylyn commented

---

Upped client to 11892 Win8.1 and Server to trunk 11892:

• Managed to get a random menu asking which camera to use on the laptop (two webcams, crappy Apple internal and a Logitech camera via USB)
• Not before realizing that I was running an old version(r11850)...

• Now with 11892: Unable to get that menu again. Clicking the webcam button on the tray only prints the following on the client side:

2016-02-09 09:15:54,519 webcam_toggled(<TrayCheckMenuItem object at 0x88600f8 (G
tkCheckMenuItem at 0x6dc5160)>,) active=False, menu=True
2016-02-09 09:15:54,519 do_start_sending_webcam(auto)
2016-02-09 09:15:54,519 do_start_sending_webcam(auto) device=0
2016-02-09 09:15:56,131 test capture using <VideoCapture 04597E60>: True, True
2016-02-09 09:15:56,131 webcam started
2016-02-09 09:15:56,137 webcam_changed(gtk2.client,) active=True, menu=True
2016-02-09 09:15:56,137 stop_sending_webcam()
2016-02-09 09:15:56,137 do_stop_sending_webcam() device=<VideoCapture 04597E60>
2016-02-09 09:15:56,263 webcam_changed(gtk2.client,) active=False, menu=True
2016-02-09 09:15:56,263 webcam_toggled(<TrayCheckMenuItem object at 0x88600f8 (G
tkCheckMenuItem at 0x6dc5160)>,) active=False, menu=False
2016-02-09 09:16:01,857 webcam_toggled(<TrayCheckMenuItem object at 0x88600f8 (G
tkCheckMenuItem at 0x6dc5160)>,) active=False, menu=True
2016-02-09 09:16:01,857 do_start_sending_webcam(auto)
2016-02-09 09:16:01,857 do_start_sending_webcam(auto) device=0
2016-02-09 09:16:19,670 test capture using <VideoCapture 04597FF0>: True, True
2016-02-09 09:16:19,670 webcam started
2016-02-09 09:16:19,671 check_acks: webcam_last_ack=-1
2016-02-09 09:16:19,671 Warning: no acknowledgements received from the server, s
topping webcam
2016-02-09 09:16:19,671 stop_sending_webcam()
2016-02-09 09:16:19,671 do_stop_sending_webcam() device=<VideoCapture 04597FF0>
2016-02-09 09:16:19,750 webcam_changed(gtk2.client,) active=False, menu=True
2016-02-09 09:16:19,750 webcam_toggled(<TrayCheckMenuItem object at 0x88600f8 (G
tkCheckMenuItem at 0x6dc5160)>,) active=False, menu=False
2016-02-09 09:16:19,750 webcam_changed(gtk2.client,) active=False, menu=False
2016-02-09 09:16:29,683 check_acks: webcam_last_ack=-1
2016-02-09 09:16:29,683 Warning: no acknowledgements received from the server, s
topping webcam
2016-02-09 09:16:29,683 stop_sending_webcam()
2016-02-09 09:16:29,683 do_stop_sending_webcam() device=None
2016-02-09 09:16:33,137 webcam_toggled(<TrayCheckMenuItem object at 0x88600f8 (G
tkCheckMenuItem at 0x6dc5160)>,) active=False, menu=True
2016-02-09 09:16:33,137 do_start_sending_webcam(auto)
2016-02-09 09:16:33,137 do_start_sending_webcam(auto) device=0
2016-02-09 09:16:34,720 test capture using <VideoCapture 04597EC0>: True, True
2016-02-09 09:16:34,720 webcam started
2016-02-09 09:16:34,721 webcam_changed(gtk2.client,) active=True, menu=True
2016-02-09 09:16:34,726 stop_sending_webcam()
2016-02-09 09:16:34,726 do_stop_sending_webcam() device=<VideoCapture 04597EC0>
2016-02-09 09:16:34,898 webcam_changed(gtk2.client,) active=False, menu=True
2016-02-09 09:16:34,898 webcam_toggled(<TrayCheckMenuItem object at 0x88600f8 (G
tkCheckMenuItem at 0x6dc5160)>,) active=False, menu=False
2016-02-09 09:16:44,723 check_acks: webcam_last_ack=-1
2016-02-09 09:16:44,723 Warning: no acknowledgements received from the server, s
topping webcam
2016-02-09 09:16:44,723 stop_sending_webcam()
2016-02-09 09:16:44,723 do_stop_sending_webcam() device=None

(Clicked it a few times hoping to maybe get that menu)

Will put server side prints in a second (have to restart session to get logs)

[totaam]

2016-02-09 17:21:04: maxmylyn commented

---

Server side -d webcam print:

2016-02-09 09:20:50,363 v4l2 using device /dev/video0
2016-02-09 09:20:50,363 Error setting up webcam forwarding:
2016-02-09 09:20:50,363  [Errno 13] Permission denied: '/dev/video0'
2016-02-09 09:20:50,386 stopping webcam device 0:
2016-02-09 09:20:50,386 Warning: cannot stop webcam device 0: no such context!

Looks like it can't access the camera? Is there a group in which I should be a member?

[totaam]

2016-02-09 18:06:49: maxmylyn commented

---

Update:

• added myself to the video and audio groups. (one must also log out and back in)

• it now sends video frames to the server (also takes a massive performance hit)

• However, applications on the system do not see the virtual camera.(Cheese, Google Chrome, will see about a few more)

[totaam]

2016-02-09 19:22:31: maxmylyn commented

---

Another update:

Turns out it just doesn't like cheese or Chrome. If you install camorama and launch it with camorama -d /dev/$cameradeviceid (usually is video0), then it works. (greyscale and chopped oddly, but 30fps)

Passing back to you, antoine.

[totaam]

2016-02-09 20:58:56: antoine commented

---

Specifying the device with camorama should not be necessary as long as the device is the first one (video0).

The fps will be lower, much lower, when running on a slower connection as we wait for the server ack before sending the next frame. ie: on a 100ms link, that's going to limit things to 10fps maximum.

As for compatibility with other applications like cheese and chrome, I will try to take a look but this will not be considered a blocker for the 0.17 release as this is mostly a feature preview at this point. This can be dealt with in the next release: #1113.

[totaam]

2016-02-26 05:37:54: antoine commented

---

Note we will need a newer gcc to build webcam support on OSX because of this compiler crash: Internal compiler error in directx.cpp on Mac OS X 10.6.
Getting around this by hacking the source files only leads to more gcc pain down the road. (tbb and gcd don't build, etc)

[totaam]

2016-03-28 06:51:36: antoine commented

---

I think this will do for this release.

[totaam]

2019-10-21 03:37:36: antoine commented

---

Follow up: #1231