Crayon Tracking  

The first thing that was done was separating out the different colors so that we could track different colored crayons. Each pixel is analyzed to check that its depth is closer to the Kinect then at some initial calibration phase. Then that pixel is analyzed based on Hue, Saturation, and Brightness to determine if it is a color that looks like a crayon. To cut back on noise an averaging method was used. When a pixel was found of a certain color, say blue, a corresponding blue mask layer was altered slightly so that the corresponding pixel was altered slightly but all the pixels around it also increased slightly. This means that single or even double pixels that seem blue from the original registered image will be averaged out but large sections of blue crayon will show up with a good strong signal.

Once the colors have been separated out into different layers the screen is scanned for the closest point to the wall for each x-value. All the points that fall within some threshold are stored inside a tip object. This stores the basic tip information which is based on an averaging of these points.

One problem that arises with this method is that the tip when it is against the wall is lost. This is due to the angle of the Kinect relative to the wall and crayon and due to the noise of the sensor itself. The program uses a threshold for distance to check if a point seems closer then the calibration point. This made it difficult to track when a crayon is in contact with the wall. Essentially the signal for contact (distance between the wall and the tip) is dampened as the tip approaches the wall. This forced the threshold for what is considered to be contact higher and the results less consistent.

To make the tip tracking more accurate, the pixels near the tip are tested for hue, saturation, and brightness to check if the tip is closer than what could be sense with the depth information. This allows for a much more accurate testing particularly in the ability of the program to detect proximity to the wall.

Other touchscreen walls

We researched other software’s that use a Kinect to turn a wall into a touchscreen mainly touchless touch and Ubi. Touchless touch is the less popular software and turned out to be very difficult to set up and not particularly accurate but the calibration process is similar to the one we used in the end which involves mapping the Kinect to the projector by touching the wall at the corner points. Ubi projects a grid of points and the Kinect maps itself as long as the whole grid is in view to the Kinect. Ubi is very accurate and gave us a promising backup plan. Ubi can be used for our project with some hardware installed. We experimented with the use of buttons on a platform that would hold the crayons. The program would respond to a released button, choosing the color of that crayon, and Ubi would pickup the location of the crayon and allow the user to draw. The downside of this would be the user’s inability to use multiple crayons at once, draw with multiple users, or allow the user to draw if the rest of the crayons are not in their set location on their buttons. It is preferable if using buttons to put them on the tip of the crayon with a Bluetooth inside telling the program to draw in that crayon’s color when the mouse is pressed.

Graphics

In order to create an interaction that feels as real as possible, it was important to us that the lines drawn look like crayon lines. We experimented with a few different options. The first was taking a circular jpg image of a crayon texture and having it repeated following the mouse. This looks repetitive and unrealistic. Our second option was to create a texture layer with a crayon texture in black and change the color of the background based on the color crayon being used. There was a black square on the top layer and moving the crayon would erase that black layer exposing the color and texture underneath. This worked but it would require saving the image as pixels every time the background was going to change colors so that the rest of the drawing wouldn’t be lost or change colors as well and the edges didn’t look very natural. What we ultimately did was draw multiple lines with different angles at the location of the crayon creating a star shape that would repeat itself as the user draws. These shapes create a solid center but the edges are jagged much like a crayon.

The Crayons

We wanted the crayons themselves to feel playful and natural. The use of oversize crayons served two purposes, the first being that adults can feel more childlike holding a crayon that makes their hands feel smaller, and the second being that the location of contact with the wall and the lines drawn don’t need to be as exact because the crayon tip is larger. We found plastic crayons that were the size that we wanted but they felt too light and flexible. We added insulation to the interior of the crayons and paper around them to give them a more solid feel. We then had to address the sound that they make while dragging against a wall. We received a broad range of feedback while user testing. It seems that what sounds pleasant to one person sounds like nails on a chalkboard to others. Some people enjoyed the sound of plastic and disliked felt or paper and some felt the opposite. The negative reaction to plastic dragging on the wall was more drastic than that of felt. Some users would only wave the crayon in front of the wall because it was so unpleasant to draw, so we put felt tips on the crayons.