User Tools

Site Tools


misc:hacks:kinect

This page has deprecated and will be archived. Please go to https://www.bitcraze.io/.

Piloting the Crazyflie with the Kinect

This is documentation for the proof-of-concept where the Kinect is used to automatically pilot the Crazyflie. The code is available in the Crazyflie Python client repo on the dev-kinect branch.

The application is intended as a proof-of-concept to show that this is possible and also to act as a starting point for anyone that wants to do some development using the Kinect and the Crazyflie. The application will probably not work out of the box for everyone since the image processing is still pretty basic and some tuning might be needed to detect the Crazyflie.

The application uses libfreenect and OpenCV.

What you need

A Crazyflie, a Xbox30 Kinect and lib libfreenect (code) and the libfreenect python wrappers.

How to get it working

These instructions are valid for Ubuntu 13.04 but similar steps should work with other OSes.

git clone https://github.com/OpenKinect/libfreenect.git
cd libfreenect
mkdir build
cd build
cmake ..
make
cd libfreenect/wrappers/python
python setup.py install
  • Clone the crazyflie-clients-python repository and switch to the dev-kinect branch
git clone https://github.com/bitcraze/crazyflie-clients-python.git
git checkout dev-kinect
  • Connect your Kinect and set up the test as explained below
  • Start the cfkinect client
cd crazyflie-clients-python/bin
./cfkinect
  • The Crazyflie should now fly itself but it will probably require some tweaking… Note that as soon as the application loses track of the Crazyflie it will kill the motors.

The set-point can be moved when holding down the mouse button in the RGB window.

Tweaking and development

To tweak and develop have a look at the following files:

crazyflie-clients-python/lib/kinect/cfkinect.py
crazyflie-clients-python/lib/kinect/kinect.py
crazyflie-clients-python/lib/kinect/pid.py

Tracking

The tracking is done using the depth image in combination with the normal RGB image. The depth image is used to get the distance from the Crazyflie to the Kinect and the RGB image is used to get the X/Y position (the yaw is not tracked at all). In order to make the detection in the RBG image easier a red ball should be attached to the Crazyflie. We used a styrofoam ball and painted it read with a filt-tip pen.

In order for the tracking to work no other objects should be in the image that will be picked up by the depth or the RGB image. The best is use a white wall as a background.

Depth

Since the Kinect's RGB camera and the camera used to read the IR laser is not aligned it's hard to find what is the Crazyflie in the depth image. Therefore it s not done at all. The image processing uses a range in the middle (i.e not farthest away and not closest) and calculates a mean of the depth value. If other objects are picked up then this calculation will not work as well.

X/Y

The RGB image is used to pick out the red ball using a red tint in the HSV space. This can tuned to detect other colors instead. Obviously it's not a good idea to have other objects picked up with similar tint.

Conclusions

Depth

The most tricky part to detect is the depth which is why we used the Kinect. The problem is that, with the current implementation, the effective range of the IR laser detection is about 0.8 - 2 meters, which is a bit too small. At two meters the RGB image is about 1.5×1.5 meters which is a bit too small to do flight maneuvers. To do more fancy stuff a larger area would be required which means that the depth measurement would be lost with the Kinect.

Yaw

Only using one visual marker makes it impossible to detect the yaw. So more visual markers would have to be added and tracked to fix this. One idea investigated is to add a black dot in the center of the red ball and to try to keep this dot in the middle as seen from the camera.

FPS

Currently the application is running at ~30 FPS which is too low when the Crazyflie moves fast. The image becomes smeared and with a small operating area of 1.5×1.5 meters the Crazyflie quickly goes out of the frame.

Regulation

The model for the regulation has be be improved a lot to get the Crazyflie more stable.

Improvements

The image processing should be improved to be more robust and generic.

The regulation has to be improved.

Better result could be achieved by calibrating the kinect(?).

misc/hacks/kinect.txt · Last modified: 2021-06-24 16:45 by kimberly