Claude + Tyler Update

Making the transition from 2D to 3D was not as easy as previously thought. First off, I realized I had forgotten toproperly render the skeleton data in 3D to begin with. I was simply taking the X and Y world coordinates of the skeleton, linearly scaling them, and drawing them on the 2D canvas, effectively ignoring the Z component. This "approximation" worked when just viewing the skeleton (or data based directly on the skeleton's location such as the energy ball in Web-Kamehameha), but in order to incorporate Tyler in 3D and have its scale and proportions match the skeleton, I would have to do proper projection. Fortunately, the Microsoft SDK provides a projection function that is used in one of its samples (the one Claude was based off of). It can convert a skeleton point (world coordinate) into a screen coordinate on a 640x480 resolution. Unfortunately, this means my web application for Tyler + Claude would have an extra dependency on Claude, a projection service. It was necessary though to ensure that Tyler's world coordinates (which were computed from the skeleton world coordinates, more on that in a minute) would be projected in the same way that the skeleton was. I also added another service to Claude to return the skeleton in screen coordinates so I could avoid an unnecessary projection call.

To get the world coordinates of Tyler in the first place, I had to implement a calibration function). The calibration function reads the location of the left foot and stores that as the corner of the Tyler grid. The function needs to be run twice to get the two opposite corners of the grid. In the video below, you can see me calibrating the grid.

The grid initially displayed is based off of old calibration points. It has no bearing on where the current grid is. You can see me walk over to the edge of the screen and calibrate the first point. After doing so, one of the grid corners snaps to that location. After calibrating the other corner, the grid grows to the appropriate size. What I am doing is using the displacement of the two world coordinates divided by the number of sensors along the X and Z axes (width and height) to figure out the dimensions of each individual sensor. I then compute the individual world coordinates of each sensor, aggregate them, and send them to Claude's project service to project all the points onto the 2D plane. When they come back, I draw them connected as polygons and fill them with a color based on the pressure data. This does mean there is a slight inaccuracy in the rendered grid as the actual sensor location is the lower left corner of the polygon, not the center.

In the video, I am using static pressure data of a single foot. The purpose of the video was merely to show calibrating the grid size. The application can currently handle live data however do to limitations of the Tyler service, I have been unable to run a full test with it and record the results. That will hopefully happen soon. In the video I also tried to turn off the grid lines so you could see the pressure colors better but I was using teamviewer on a tablet to control my desktop and got really buggy when I tried to use backspace.

One last interesting piece of information is how tilting affects the Kinect, or rather how it doesn't. First, consider the image midway down this page. It shows the three axes in respect to the Kinect's sensors. When the Kinect tilts, the axes do as well. In reality, we would want the axes to always remain in the same position relative to the world, not the tilt of the Kinect. Supposedly the Kinect has a sensor that detects it's orientation so perhaps there is a feature in the Kinect SDK to translate world coordinates back to their real orientation so that when the Kinect is tilted, the axes aren't. Currently, we can only run Claude + Tyler when the Kinect has 0 degrees of tilt, otherwise the tiles will no longer be at a constant Y but will appear to rise on the Y axis.