Kinect integration

Here we are going to discuss the path from the handling of the Kinect to the spray painting on the canvas. This is separated in 3 intuitive steps. First we get the raw input from the player, then we compute the necessary inputs to paint and finally we paint!

First we have to chose what information we are going to extract from the Kinect’s representation of the player. We found a library for Unity that gives us the raw body joints positions from the Kinect's camera perspective. As first input, we take the arm vectors which are the hand position minus the shoulder position. After some filtering and project instance related adjustments (mainly depending on where is the Kinect camera), we get the arm vectors in the scene coordinate.

Now the tricky part was to, from those, get the input painting parameters namely the color as hue, saturation and value (or brightness) and the aimed position in the canvas. We choose the hue wheel to represent the choice of color because this is a more « human-friendly » way of choosing a color than directly pick the RGB components. With wisely used vector and dot products, we construct the virtual coordinate system for color selection and project the left arm vector inside. It can be seen as a cylinder going from the left shoulder of the user to the visible hue wheel. the brightness is computed from the distance from the shoulder to the hand. The position in the image is computed with a raycast from the camera into a plan fitting the 3D canvas in the scene. We then project the raycast hit in the canvas' left and upper borders (following image axis) and finally map the coordinates from unity units to pixels.

The last challenge is to apply the color that the user is selecting, the blend color, over the base color already on the painting. The algorithm that will be used from those 2 colors will determine the look of the virtual brush and is applied to each pixels within a circle. As suggests the name of the project, we wanted to modelize a spray. This can be achieved in the following way : To get the resulting color from the blend and the base color, we will do a simple linear interpolation with a certain factor. In our implementation, the factor depends on the blend factor or opacity parameter of the spray, the distance from the center and a random factor between 0.5 and 1 to give some irregularity and realism.

All the source files of the project can be found on github: [https://github.com/maximehulliger/SprayIt]