Kenneth M. Cruikshank
Contour Kinect Depth Frame

Introduction

For this project I used the following components (see main Kinect Projects page for more details)

• Dell PC with Intel i7 processor, and Intel USB 3.0 dedicated to Kinect
• Microsoft Kinect for XBox One with Kinect for Windows adapter
• Microsoft Visual Studio Community Edition
• Microsoft Kinect SDK 2.0
• Emgu CV

This example modifies the basic "Depth Sensor Essentials" code provided in the Microsoft Kinect SDK. The SDK example captures a Kinect frame, and then displays it as a greyscale image. We will modify it so that the displayed image is a contour map (and add a few controls to change the contoured range). This involves adding the Emgu.CV, Emgu.CV.Structure, and Emgu.CV.Util namespaces. You should follow the instructions on Emgu CV Web site on how to load these libraries into your project (specifics depends on the IDE that you are using, I am using Visual Studio).

The Kinect Depth Sensor returns the distance away from the sensor in millimeters (mm). The effective working range of the sensor is 500mm to 4500 mm (although it can return values up to 8000 mm). The sensor returns a vector of distances from the sensor. This can be converted to an array, representing a grid. Each value in the array is the depth at that location. One thing we may want to do is contour this image. Contouring can be a slow process, we we can seed it up using the CV package. The logic is a follows

1. Start with a 1-dimensional array containing the depth data
2. Create a grid representing a "frame" of vision, each cell in the grid has x,y coordinates; the cell value is the distance from the sensor in mm.
3. Convert the image to a grid of 1-byte values. The distances can be from 500 to 4500 mm, so a range of 4000 mm, which requires 2-byte integers to hold the range of values. For convenience of speed, we will convert these to 1-byte integers (each cell has a numeric range of 0-255).
4. The grid in step 2 can be though of as an image, with each cell representing a shade of gray (between 0 and 255).
5. Pick a distance (in mm), and convert that range to a pixel value (using the same scale factor as on step (3).
6. Change all pixels in the image that are not equal to the value from step (5) to "Black", and convert the pixels equal to that value to "White"
7. Trace the white pixels with a series of point (coordinates), this will define a contour line (now a vector)
8. Draw the vectors as a colored line on a white background (or as an option, a filled shape with color)
9. Repeat steps 5-8 for all the contour values you want to see
10. Display the contour image, then return to step 1 for the next frame.

Steps 7 and 8 are very efficiently done with OpenCV functions. Using those functions, you can usually draw 10-15 contours per frame in real time on a normal PC (at 30 frames per second).

If we were making real-time contour maps of say a stream table, we could easily update the contour map every second, and do quite a bit of processing between frames. For example, it is easy to compare frames and display only changes, or do volume calculations (this may be best done on the original matrix of data).

Sample Code

Below I discuss a couple of the key processing steps, and some of the things to watch out for.

There are two main frameworks to create Microsoft Windows programs. One is the (older) Windows Forms, the other is the (newer) Windows Presentation Framework (WPF). For us, the importance is is what format of bitmaps work well. Many of the examples I have seen in Emgu CV were written for Windows Forms, we are writing for WPF. I chose this because (1) it is the newer framework, and (2) because it just needs one additional bitmap translation. So, the code we are writing will work with Windows Forms if you omit the last bitmap translation. If you want to use WPF, then you will learn the additional step needed. Bitmaps in Windows Forms are of type System.Drawing.Bitmap, in WFP they are of type System.Windows.Media.Media.Imaging.WriteableBitmap. One does not work quite the same way as the other.

A couple of notes on the contouring. Much of the contouring is done using integer math. For that reason, we need at least a difference of 250 mm between the maximum and minimum distance (specified with a couple of input boxes). Failure to do this results in a run-time error because of contour values becoming zero. For this reason, the maximum value is adjusted if the difference is not large enough. For that reason, adjust the lower bound before the upper bound to minimize the amount of times you have to change the input box

Complete Project: DepthContour.zip

Not much more to say about the code. The 10 steps above outline what is done, and the code contains enough comments that you should be able to work through (remember, windows is an event-driven program, so it may not look like a traditional top-to-bottom code that many "simple" programs in other languages may have.

If the code there are a couple of #defines that change the behavior of the code

#define IMAGE_ONLY
#define COLOR_MAP

IMAGE_ONLY will enable the display of an image rather than a contour map. If you use this option, you must also make the following change in the XAML file:

 

COLOR_MAP will display a color contour map. No other changes should be needed.