Experimental hand detection and tracking from single depth camera.
Note: this is a two-week quick patch following the hand pose estimation project located in the main repo. The purpose is to make live capture and hand tracking possible, but currently no plan to make the code waterproof (might be a future research project).
The code was tested using SR300 and D400 series (D415/D435). Please follow the official instructions to install necessary drivers.
Note: the code was updated to use Intel® RealSense™ SDK 2.0, so the minimum supported camera model is SR300.
Also note: SR300 should work nicely after installing the drivers (hit 'realsense_viewer' to test if anything went wrong). But for D400 series, additional firmware update is necessary: go to the official site and following instructions there ('realsense_viewer' will also find any firmware issues and show you the latest update link).
Default is using the 'super_edt2m' model. Please check the main repo for finding the full list of models.
Note: tracking code does not require the prepared data, only need to download your favorite pretrained model. See README.md in the pose estimation part for more details.
cd depth-hand/code
python -m camera.capture \
--data_root=$HOME/data \
--out_root=$HOME/data/univue/output \
--model_name=super_edt2m
Key responses:
- q: quit the current window.
- b: save a single frame.
- Navigation Keyboard Shortcuts of matplotlib.
- If you want to see more debug information:
python -m camera.capture \ --data_root=$HOME/data \ --out_root=$HOME/data/univue/output \ --show_debug=True \ --model_name=super_edt2m - Save detection results:
python -m camera.capture \ --data_root=$HOME/data \ --out_root=$HOME/data/univue/output \ --save_det=True \ --model_name=super_edt2m - Save raw stream data (for replay and test):
NOTE: previous data will be overwritten!
python -m camera.capture \ --data_root=$HOME/data \ --out_root=$HOME/data/univue/output \ --save_stream=True --save_det=True \ --model_name=super_edt2m - Read saved raw stream data (instead of live capture):
This is very useful for debug: after making changes to the code, replay the same sequence and see if your idea works better. For example, you can play with the test capture sequence used for generating the teaser figure on the top.
python -m camera.capture \ --data_root=$HOME/data \ --out_root=$HOME/data/univue/output \ --read_stream=True --save_det=True \ --model_name=super_edt2m
Please check the project webpage to see how it works or why not working as promised.
The Tensorflow pretrained model is loaded on the first frame when something interesting show up in the range. On my laptop it roughly takes 2 seconds. After that, the detection should be almost as fast as the capture rate of your depth camera.
A: That sometimes happens with the "TkAgg" backend of matplotlib, which is unfortunately the default.
Tkinter is Python's de-facto standard GUI (Graphical User Interface) package."
So make sure Tcl/Tk is installed correctly on your system. Or switching to another backend might help (See code/camera/capture.py around the top), e.g. "Qt5Agg" backen is more stable (IMO), but extremely slow.
A: Mostly due to noise of depth image, which may relates to (in addition to algorithm complexity): hardware sensibility, lighting condition, reflective objects within range (in one funny test case, our tester's metal watch makes detection very random 😂), etc.