Server Performance Improvements

Today I did a little experiment by changing every function on the server that takes an object, array, or struct as an argument so that they only accept pointers to those parameters. After going through and making the necessary changes to the objects in question, I found that the running time of THEA before the sensor grid freezes when running the same test as in the previous post is ~8 minutes (compared to the 2.5 minutes when passing static objects). I'll run an officially timed comparison soon and post the results.

In addition, I have an old version by which we can run comparisons to show the drastic difference this change makes. For now, though, I'm enjoying the extended run time (>15 minutes when only receiving pressure centers).

THEA's Data Freeze

In testing THEA and her embedded Mongoose server more with AJAX calls, I've come across an issue where the Tactonic frame will stop transferring data to the viewer program. The time it takes for this phenomenon to occur seems inversely proportional to the amount of data being transmitted and the number of processes my computer is running. In the video above, the only non-kernel applications running were the two servers necessary to transmit data from THEA and to run my client, Google Chrome, and Quicktime; in this case, THEA stopped transmitting new data after 1:33. I tried again without running Quicktime, and THEA lasted for 2:28.360 (other trials with as close to the same conditions as I could create ran for about the same amount of time). When transmitting pressure centers only, THEA can keep running for ~10 minutes at a time.

Activity monitor shows that the viewer program consumes 53-55% of my CPU once it freezes. This is consistent with Tactonic's viewer's CPU consumption at freeze time. Tactonic's viewer also sees this issue, though it does not seem to follow any trend. On one run, Tactonic's viewer froze after 10 minutes; on another, it ran with no issue for more than 15.

AppleUSBFTDI and Running Tyler/THEA

The content of this post deals with moving/removing kernel files in OSX to accommodate the needs of a specific application. Be very careful when following these steps: There is some potential to severely damage your system.

My MacBook has been set up to run Tyler and THEA for around six months now, so when I applied some critical updates to the OS it was a point of frustration that the system no longer recognized any Tactonic device plugged into it. After confirming that the problem was with my computer and not the sensors, I immediately suspected that the update had placed new AppleUSBFTDI files in the System Library. Getting my system re-set up for working with Tactonic devices took a good 30 minutes of my life, and that was with some idea of what to do, so here are the exact steps I followed to get back in working order.

Note: A good way to ensure that this is the problem encountered is by plugging the device in and running

> sudo dmesg

right away. If something that looks like

sage.domain com.apple.commssw.ftdi.device] [com.apple.message.signature AppleUSBFTDI] [com.apple.message.signature2 0x403] [com.apple.message.signature3 0x6010] [com.apple.message.summarize YES]
AppleUSBFTDI: Version number - 1.0.1b3, Input buffers 8, Output buffers 16
         0 [Level 5] [com.apple.message.domain com.apple.commssw.ftdi.device] [com.apple.message.signature AppleUSBFTDI] [com.apple.message.signature2 0x403] [com.apple.message.signature3 0x6010] [com.apple.message.summarize YES]
AppleUSBFTDI: Version number - 1.0.1b3, Input buffers 8, Output buffers 16
[0xffffff801e8d8000](1)/(5) Device not responding
com_apple_driver_AppleUSBCardReaderUMC:: Stop::Controller Reset
USBMSC Identifier (non-unique): 000000009833 0x5ac 0x8403 0x9833, 2
         0 [Level 5] [com.apple.message.domain com.apple.commssw.ftdi.device] [com.apple.message.signature AppleUSBFTDI] [com.apple.message.signature2 0x403] [com.apple.message.signature3 0x6010] [com.apple.message.summarize YES]
AppleUSBFTDI: Version number - 1.0.1b3, Input buffers 8, Output buffers 16
         0 [Level 5] [com.apple.message.domain com.apple.commssw.ftdi.device] [com.apple.message.signature AppleUSBFTDI] [com.apple.message.signature2 0x403] [com.apple.message.signature3 0x6010] [com.apple.message.summarize YES]
AppleUSBFTDI: Version number - 1.0.1b3, Input buffers 8, Output buffers 16

comes up near the bottom of the list, this guide could save you a lot of time. This is probably correct for many USB inputs, but for devices that require their own FTDI (or ftdi), you don't want to see this.

1. Update the locate Database

In the terminal, run

> locate.updatedb

which will update locate's file database so that it includes everything on the filesystem. Supposedly the OS will run this periodically, but it can't hurt to manually update it when looking for a specific file. If the command does not execute, try running

> /usr/libexec/locate.updatedb

2. Locate FTDIs

Next, run

> locate FTDI

This should return a list of filepaths with "FTDI" (case-sensitive) in their names. Find the one mentioned in the dmesg output (in my case, AppleUSBFTDI.kext; Tactonic's instructions reference FTDIUSBSerialDriver.kext; the one to deal with really depends on the situation).

The paths you're looking for should be somewhere in /System/Library/Extensions. My locate output looked like

/System/Library/Extensions/IOUSBFamily.kext/Contents/PlugIns/AppleUSBFTDI.kext
/System/Library/Extensions/IOUSBFamily.kext/Contents/PlugIns/AppleUSBFTDI.kext/Contents
/System/Library/Extensions/IOUSBFamily.kext/Contents/PlugIns/AppleUSBFTDI.kext/Contents/Info.plist
/System/Library/Extensions/IOUSBFamily.kext/Contents/PlugIns/AppleUSBFTDI.kext/Contents/MacOS
/System/Library/Extensions/IOUSBFamily.kext/Contents/PlugIns/AppleUSBFTDI.kext/Contents/MacOS/AppleUSBFTDI
/System/Library/Extensions/IOUSBFamily.kext/Contents/PlugIns/AppleUSBFTDI.kext/Contents/_CodeSignature
/System/Library/Extensions/IOUSBFamily.kext/Contents/PlugIns/AppleUSBFTDI.kext/Contents/_CodeSignature/CodeResources
/System/Library/Extensions/IOUSBFamily.kext/Contents/PlugIns/AppleUSBFTDI.kext/Contents/version.plist

When dealing with .kext (Kernel Extension) files, especially when moving/deleting them, ONLY deal with .kext files that you absolutely have to: Moving or removing them may have unintended consequences.

3. Move it to a different directory or remove it entirely

Now that we know where the problem file is, we have to deal with it. Tactonic recommends removing the file entirely; I'm hesitant about removing Kernel Extensions without knowing how their removal will affect my system. Either case seems to work without ruining the USB ports' abilities to function, so this step is up to you.

It may be useful to keep a record of which FTDIs you remove if you go that route, so just copy their filepaths to a text file and save it. Then, run

> sudo mv /path/to/FTDI/problemFTDI.kext /path/to/wherever/you/want/to/move/it

or

> sudo rm /path/to/FTDI/problemFTDI.kext

 As I said above, this step is largely up to you. Just bear in mind my warning about dealing with .kext files.

4. Locate ftdis

This process likely has to be repeated with filepaths with "ftdi" (case-sensitive) in the path, so run

> locate ftdi

Once again, you'll get a list of all of the filepaths containing "ftdi," but this time find the ones in /Library/Receipts. My output didn't have any files in that directory, but it did return some Ruby scripts in another directory. Ignore any filepaths not containing /Library/Receipts. The files you're looking for should be .pkg files.

5. Move It/Them to Another Directory or Remove It/Them

Now repeat step 3 with the new paths. Once again, you may want to keep a record of which files you're dealing with. Run

> sudo mv /path/to/ftdi/problemftdi.pkg /path/to/wherever/you/want/to/move/it

or

> sudo rm /path/to/ftdi/problemftdi.pkg

6. Reboot Your Computer

The changes that we have made so far will not go into effect until your computer is restarted. This is because of the .kext files, which are loaded at startup. Once your computer is back on, plug in the device you were trying to use and run

> sudo dmesg

again. If you don't see anything about the FTDI files you moved/removed, that's a good sign.

Recap

The short version of this process is:

> sudo dmesg
> locate.updatedb
> locate FTDI
> sudo mv /path/to/FTDI/problemFTDI.kext /path/to/wherever/you/want/to/move/it
> locate ftdi
> sudo mv /path/to/ftdi/problemftdi.pkg /path/to/wherever/you/want/to/move/it
> sudo dmesg

This is a specific case of looking for a file using FTDI, but replace the FTDI with whatever the problem name is, and you have a process for dealing with application-specific kernel needs.

New Spatial Audio Example

Dr. Remy wanted to test the limits of the WebAudio API and see what further interesting things we could do with spatial audio. I have been developing a testing framework that will let us play around with several sources and sounds. Dr. Remy's main question is how well the WebAudio API handles a moving audio source and a moving listener. The answer, as seen below, is fairly well (note that you must wear headphones to notice much of any spatialization).

I think it handles motion just fine, but the overall realism of the audio visualization is still of a poor quality. Left versus right is very distinguishable however up/down and front/back have very weak.

Introducing THEA and Server Orientation

This is THEA, the Tactile Hand Evaluation Apparatus, a cousin of Tyler's from Tactonic Technologies. Tyler is experiencing frequent technical difficulties these days, so THEA is filling in while we await a new USB adapter.

THEA, like Tyler, is a pressure sensor made up of sensels. Unlike Tyler, THEA's sensels are packed much more tightly together. Each of Tyler's sensels is spaced about 1in. from every adjacent sensel, with 24x24 sensels in one panel. THEA, on the other hand, packs 34x44 sensels into a single panel, each spaced about 1/4in. from every adjacent sensel.

Due to Tyler's injury, I'll be using THEA to move forward with the Sensory Translation project. The first phase is to make a simple keyboard (as in piano) by dividing THEA's sensels into groups that each emit a distinct tone. I'll be implementing this using the WebAudio API and data from the Mongoose server I've been working on.

THEA has already helped in adding features to the server: now, with a query string, users can specify a tile orientation in one of four directions. Since the tiles are only usable in straight lines, swapping orientations was not difficult, but being able to rapidly test the code with THEA saved quite a bit of time.

Now, if a user wants to specify an orientation, they just add

?orientation=x 

where x may be any 32-bit integer value. The value is taken mod(4) if it is greater than three, then passed into the function that transforms the force grid into a string. That function has been modified to be:

std::string NumViews::getForceGrid(TactonicFrame *frame, int orient, bool usespacer) {
    std::string out = "";
    int xmax, ymax, f;
    uint8_t xref, yref;

    ymax = (!(orient & 1)?device.rows:device.cols) - 1;
    xmax = (!(orient & 1)?device.cols:device.rows) - 1;

    yref = (orient == 0 || orient == 3?0:1);
    xref = (orient == 0 || orient == 1?0:1);


    for (int i = 0; i <= ymax; i++) {
        for (int j = 0; j <= xmax; j++) {
            std::string spacer = "";

            if (orient == 0 || orient == 2) {
                f = frame->forces[(yref&1?(ymax-i):i) * device.cols + (xref&1?(xmax-j):j)];
            }
            else {
                f = frame->forces[(yref&1?(ymax-i):i) + (xref&1?(xmax-j):j) * device.cols];
            }

            out.append(std::to_string(f));
            if (!usespacer && j < xmax) {
                spacer = " ";
            }
            else if (!usespacer){
                spacer = "";
            }

            else {
                spacer = (f>=1000?"":f>=100 && f < 1000?" ":f>=10 && f < 100?"  ":"   ");
            }


            out.append(spacer);
        }
        if (i < ymax) {
            out.append("\n");
        }
    }
    return out;
}

Essentially, what has happened is the frame of reference for iterating over the force grid is changed depending upon the orientation. I'll explain the new parts piece-by-piece.

Deciding Which Coordinate Represents Rows vs. Columns

    ymax = (!(orient & 1)?device.rows:device.cols) - 1;
    xmax = (!(orient & 1)?device.cols:device.rows) - 1;

In orientation 0 or 2, the grid is iterated over with x as the column dimension and y as the row dimension. However, in orientation 1 or 3, this is swapped. This is then used in the nested for loops to set the bounds for iterating over each dimension.

    for (int i = 0; i <= ymax; i++) {
        for (int j = 0; j <= xmax; j++) {

This allows the parameters of the loop to change without having to do any work in the loop setup or having to create another set of loops.

Deciding Which Point is Each Dimension's Origin

    yref = (orient == 0 || orient == 3?0:1);
    xref = (orient == 0 || orient == 1?0:1);

To change the orientation, we also need to know if each dimension starts at the beginning or end of its range of values. xref and yref are just flags that indicate whether or not the iterator must read from front-to-back or back-to-front over a given dimension.

Pulling the Correct Values from the Array

 for (int i = 0; i <= ymax; i++) {
        for (int j = 0; j <= xmax; j++) {
            std::string spacer = "";

            if (orient == 0 || orient == 2) {
                f = frame->forces[(yref&1?(ymax-i):i) * device.cols + (xref&1?(xmax-j):j)];
            }
            else {
                f = frame->forces[(yref&1?(ymax-i):i) + (xref&1?(xmax-j):j) * device.cols];
            }

The final piece of this data is to pull the correct values out of the array. This was an interesting case because the array is 1-dimensional, so all of the math had to be done in one shot. Essentially, the same thing is happening in both cases:

index = (current_row * column_count) + current_column

The first case deals with orientations 0 and 2, which are 180-degree rotations of each other. The inline conditionals check the flags to see whether or not they are set. If so, the dimension is the difference between the maximum index for that dimension and the current iterator value (read from the back); if not, the value is the current iterator value (read from the front).

The second case deals with orientations 1 and 3, which are 90- and 270-degree rotations of orientation 0, respectively. The conditionals do the same thing as in the first case, but here the x- and y-dimensions are swapped in the for loops, making the loops iterate over all y for a particular x before moving on to the next x.

The output of this code is as follows:

Orientation 0

Orientation 1

Orientation 2

Orientation 3

 These four screenshots are all of my left hand. I did not move my hand between screenshots. The image looks like my right hand because of the location of (0, 0) on THEA's sensor grid and the fact that it's more convenient to put THEA to my left on my desk.

Essentially, I've made an in-place row/column-major determining function. Careful observers will notice that the first indexing case above may actually be done with one check, since in orientation 2 both flags are set to 1. However, by leaving it as-is, we actually allow for four other orientations to be implemented, if desired, by swapping the flags of orientations 0 and 1 and orientations 2 and 3. These will produce mirror-images of the currently implemented orientations across their major axis, so I'm leaving it as-is in the event that somebody wants to use them in the future.

Currently, no drop in performance has been observed as a result of these changes, and since Daniel has been making AJAX calls to the server using this new setup, we should have seen some slow-down in the tests if this was going to be an issue.

The completion of this task means that my work with the code that interprets Tyler's and THEA's data is complete until somebody needs new functionality, so I'm moving on to other tasks.

Claude + Tyler Video

Here is a video of Claude and Tyler both working with live data.

I was going to use Firefox to record the video since it had better performance in the past but as it turns out it is currently much worse. Firefox fails to properly double buffer the canvas element and the result is a flashing screen. Though this SO article doesn't provide an official reference, I believe it is correct in stating that the browser is responsible for double buffering. It is possible to do it at the user level by creating 2 canvas elements and hiding one to be the buffer, but as in the case of Chrome this is not necessary to achieve smooth animations. Firefox didn't have this problem before the addition of Tyler.

I found this article on optimizing code for the canvas particularly interesting. Unfortunately, Chrome's performance issues are due to slow get requests rather than slow canvas drawing, as evidenced by a simple timer on the get requests.

Claude + Tyler Update

Making the transition from 2D to 3D was not as easy as previously thought. First off, I realized I had forgotten toproperly render the skeleton data in 3D to begin with. I was simply taking the X and Y world coordinates of the skeleton, linearly scaling them, and drawing them on the 2D canvas, effectively ignoring the Z component. This "approximation" worked when just viewing the skeleton (or data based directly on the skeleton's location such as the energy ball in Web-Kamehameha), but in order to incorporate Tyler in 3D and have its scale and proportions match the skeleton, I would have to do proper projection. Fortunately, the Microsoft SDK provides a projection function that is used in one of its samples (the one Claude was based off of). It can convert a skeleton point (world coordinate) into a screen coordinate on a 640x480 resolution. Unfortunately, this means my web application for Tyler + Claude would have an extra dependency on Claude, a projection service. It was necessary though to ensure that Tyler's world coordinates (which were computed from the skeleton world coordinates, more on that in a minute) would be projected in the same way that the skeleton was. I also added another service to Claude to return the skeleton in screen coordinates so I could avoid an unnecessary projection call.

To get the world coordinates of Tyler in the first place, I had to implement a calibration function). The calibration function reads the location of the left foot and stores that as the corner of the Tyler grid. The function needs to be run twice to get the two opposite corners of the grid. In the video below, you can see me calibrating the grid.

The grid initially displayed is based off of old calibration points. It has no bearing on where the current grid is. You can see me walk over to the edge of the screen and calibrate the first point. After doing so, one of the grid corners snaps to that location. After calibrating the other corner, the grid grows to the appropriate size. What I am doing is using the displacement of the two world coordinates divided by the number of sensors along the X and Z axes (width and height) to figure out the dimensions of each individual sensor. I then compute the individual world coordinates of each sensor, aggregate them, and send them to Claude's project service to project all the points onto the 2D plane. When they come back, I draw them connected as polygons and fill them with a color based on the pressure data. This does mean there is a slight inaccuracy in the rendered grid as the actual sensor location is the lower left corner of the polygon, not the center.

In the video, I am using static pressure data of a single foot. The purpose of the video was merely to show calibrating the grid size. The application can currently handle live data however do to limitations of the Tyler service, I have been unable to run a full test with it and record the results. That will hopefully happen soon. In the video I also tried to turn off the grid lines so you could see the pressure colors better but I was using teamviewer on a tablet to control my desktop and got really buggy when I tried to use backspace.

One last interesting piece of information is how tilting affects the Kinect, or rather how it doesn't. First, consider the image midway down this page. It shows the three axes in respect to the Kinect's sensors. When the Kinect tilts, the axes do as well. In reality, we would want the axes to always remain in the same position relative to the world, not the tilt of the Kinect. Supposedly the Kinect has a sensor that detects it's orientation so perhaps there is a feature in the Kinect SDK to translate world coordinates back to their real orientation so that when the Kinect is tilted, the axes aren't. Currently, we can only run Claude + Tyler when the Kinect has 0 degrees of tilt, otherwise the tiles will no longer be at a constant Y but will appear to rise on the Y axis.