AppleUSBFTDI and Running Tyler/THEA

The content of this post deals with moving/removing kernel files in OSX to accommodate the needs of a specific application. Be very careful when following these steps: There is some potential to severely damage your system.

My MacBook has been set up to run Tyler and THEA for around six months now, so when I applied some critical updates to the OS it was a point of frustration that the system no longer recognized any Tactonic device plugged into it. After confirming that the problem was with my computer and not the sensors, I immediately suspected that the update had placed new AppleUSBFTDI files in the System Library. Getting my system re-set up for working with Tactonic devices took a good 30 minutes of my life, and that was with some idea of what to do, so here are the exact steps I followed to get back in working order.

Note: A good way to ensure that this is the problem encountered is by plugging the device in and running

> sudo dmesg

right away. If something that looks like

sage.domain com.apple.commssw.ftdi.device] [com.apple.message.signature AppleUSBFTDI] [com.apple.message.signature2 0x403] [com.apple.message.signature3 0x6010] [com.apple.message.summarize YES]
AppleUSBFTDI: Version number - 1.0.1b3, Input buffers 8, Output buffers 16
         0 [Level 5] [com.apple.message.domain com.apple.commssw.ftdi.device] [com.apple.message.signature AppleUSBFTDI] [com.apple.message.signature2 0x403] [com.apple.message.signature3 0x6010] [com.apple.message.summarize YES]
AppleUSBFTDI: Version number - 1.0.1b3, Input buffers 8, Output buffers 16
[0xffffff801e8d8000](1)/(5) Device not responding
com_apple_driver_AppleUSBCardReaderUMC:: Stop::Controller Reset
USBMSC Identifier (non-unique): 000000009833 0x5ac 0x8403 0x9833, 2
         0 [Level 5] [com.apple.message.domain com.apple.commssw.ftdi.device] [com.apple.message.signature AppleUSBFTDI] [com.apple.message.signature2 0x403] [com.apple.message.signature3 0x6010] [com.apple.message.summarize YES]
AppleUSBFTDI: Version number - 1.0.1b3, Input buffers 8, Output buffers 16
         0 [Level 5] [com.apple.message.domain com.apple.commssw.ftdi.device] [com.apple.message.signature AppleUSBFTDI] [com.apple.message.signature2 0x403] [com.apple.message.signature3 0x6010] [com.apple.message.summarize YES]
AppleUSBFTDI: Version number - 1.0.1b3, Input buffers 8, Output buffers 16

comes up near the bottom of the list, this guide could save you a lot of time. This is probably correct for many USB inputs, but for devices that require their own FTDI (or ftdi), you don't want to see this.

1. Update the locate Database

In the terminal, run

> locate.updatedb

which will update locate's file database so that it includes everything on the filesystem. Supposedly the OS will run this periodically, but it can't hurt to manually update it when looking for a specific file. If the command does not execute, try running

> /usr/libexec/locate.updatedb

2. Locate FTDIs

Next, run

> locate FTDI

This should return a list of filepaths with "FTDI" (case-sensitive) in their names. Find the one mentioned in the dmesg output (in my case, AppleUSBFTDI.kext; Tactonic's instructions reference FTDIUSBSerialDriver.kext; the one to deal with really depends on the situation).

The paths you're looking for should be somewhere in /System/Library/Extensions. My locate output looked like

/System/Library/Extensions/IOUSBFamily.kext/Contents/PlugIns/AppleUSBFTDI.kext
/System/Library/Extensions/IOUSBFamily.kext/Contents/PlugIns/AppleUSBFTDI.kext/Contents
/System/Library/Extensions/IOUSBFamily.kext/Contents/PlugIns/AppleUSBFTDI.kext/Contents/Info.plist
/System/Library/Extensions/IOUSBFamily.kext/Contents/PlugIns/AppleUSBFTDI.kext/Contents/MacOS
/System/Library/Extensions/IOUSBFamily.kext/Contents/PlugIns/AppleUSBFTDI.kext/Contents/MacOS/AppleUSBFTDI
/System/Library/Extensions/IOUSBFamily.kext/Contents/PlugIns/AppleUSBFTDI.kext/Contents/_CodeSignature
/System/Library/Extensions/IOUSBFamily.kext/Contents/PlugIns/AppleUSBFTDI.kext/Contents/_CodeSignature/CodeResources
/System/Library/Extensions/IOUSBFamily.kext/Contents/PlugIns/AppleUSBFTDI.kext/Contents/version.plist

When dealing with .kext (Kernel Extension) files, especially when moving/deleting them, ONLY deal with .kext files that you absolutely have to: Moving or removing them may have unintended consequences.

3. Move it to a different directory or remove it entirely

Now that we know where the problem file is, we have to deal with it. Tactonic recommends removing the file entirely; I'm hesitant about removing Kernel Extensions without knowing how their removal will affect my system. Either case seems to work without ruining the USB ports' abilities to function, so this step is up to you.

It may be useful to keep a record of which FTDIs you remove if you go that route, so just copy their filepaths to a text file and save it. Then, run

> sudo mv /path/to/FTDI/problemFTDI.kext /path/to/wherever/you/want/to/move/it

or

> sudo rm /path/to/FTDI/problemFTDI.kext

 As I said above, this step is largely up to you. Just bear in mind my warning about dealing with .kext files.

4. Locate ftdis

This process likely has to be repeated with filepaths with "ftdi" (case-sensitive) in the path, so run

> locate ftdi

Once again, you'll get a list of all of the filepaths containing "ftdi," but this time find the ones in /Library/Receipts. My output didn't have any files in that directory, but it did return some Ruby scripts in another directory. Ignore any filepaths not containing /Library/Receipts. The files you're looking for should be .pkg files.

5. Move It/Them to Another Directory or Remove It/Them

Now repeat step 3 with the new paths. Once again, you may want to keep a record of which files you're dealing with. Run

> sudo mv /path/to/ftdi/problemftdi.pkg /path/to/wherever/you/want/to/move/it

or

> sudo rm /path/to/ftdi/problemftdi.pkg

6. Reboot Your Computer

The changes that we have made so far will not go into effect until your computer is restarted. This is because of the .kext files, which are loaded at startup. Once your computer is back on, plug in the device you were trying to use and run

> sudo dmesg

again. If you don't see anything about the FTDI files you moved/removed, that's a good sign.

Recap

The short version of this process is:

> sudo dmesg
> locate.updatedb
> locate FTDI
> sudo mv /path/to/FTDI/problemFTDI.kext /path/to/wherever/you/want/to/move/it
> locate ftdi
> sudo mv /path/to/ftdi/problemftdi.pkg /path/to/wherever/you/want/to/move/it
> sudo dmesg

This is a specific case of looking for a file using FTDI, but replace the FTDI with whatever the problem name is, and you have a process for dealing with application-specific kernel needs.

New Spatial Audio Example

Dr. Remy wanted to test the limits of the WebAudio API and see what further interesting things we could do with spatial audio. I have been developing a testing framework that will let us play around with several sources and sounds. Dr. Remy's main question is how well the WebAudio API handles a moving audio source and a moving listener. The answer, as seen below, is fairly well (note that you must wear headphones to notice much of any spatialization).

I think it handles motion just fine, but the overall realism of the audio visualization is still of a poor quality. Left versus right is very distinguishable however up/down and front/back have very weak.

Introducing THEA and Server Orientation

This is THEA, the Tactile Hand Evaluation Apparatus, a cousin of Tyler's from Tactonic Technologies. Tyler is experiencing frequent technical difficulties these days, so THEA is filling in while we await a new USB adapter.

THEA, like Tyler, is a pressure sensor made up of sensels. Unlike Tyler, THEA's sensels are packed much more tightly together. Each of Tyler's sensels is spaced about 1in. from every adjacent sensel, with 24x24 sensels in one panel. THEA, on the other hand, packs 34x44 sensels into a single panel, each spaced about 1/4in. from every adjacent sensel.

Due to Tyler's injury, I'll be using THEA to move forward with the Sensory Translation project. The first phase is to make a simple keyboard (as in piano) by dividing THEA's sensels into groups that each emit a distinct tone. I'll be implementing this using the WebAudio API and data from the Mongoose server I've been working on.

THEA has already helped in adding features to the server: now, with a query string, users can specify a tile orientation in one of four directions. Since the tiles are only usable in straight lines, swapping orientations was not difficult, but being able to rapidly test the code with THEA saved quite a bit of time.

Now, if a user wants to specify an orientation, they just add

?orientation=x 

where x may be any 32-bit integer value. The value is taken mod(4) if it is greater than three, then passed into the function that transforms the force grid into a string. That function has been modified to be:

std::string NumViews::getForceGrid(TactonicFrame *frame, int orient, bool usespacer) {
    std::string out = "";
    int xmax, ymax, f;
    uint8_t xref, yref;

    ymax = (!(orient & 1)?device.rows:device.cols) - 1;
    xmax = (!(orient & 1)?device.cols:device.rows) - 1;

    yref = (orient == 0 || orient == 3?0:1);
    xref = (orient == 0 || orient == 1?0:1);


    for (int i = 0; i <= ymax; i++) {
        for (int j = 0; j <= xmax; j++) {
            std::string spacer = "";

            if (orient == 0 || orient == 2) {
                f = frame->forces[(yref&1?(ymax-i):i) * device.cols + (xref&1?(xmax-j):j)];
            }
            else {
                f = frame->forces[(yref&1?(ymax-i):i) + (xref&1?(xmax-j):j) * device.cols];
            }

            out.append(std::to_string(f));
            if (!usespacer && j < xmax) {
                spacer = " ";
            }
            else if (!usespacer){
                spacer = "";
            }

            else {
                spacer = (f>=1000?"":f>=100 && f < 1000?" ":f>=10 && f < 100?"  ":"   ");
            }


            out.append(spacer);
        }
        if (i < ymax) {
            out.append("\n");
        }
    }
    return out;
}

Essentially, what has happened is the frame of reference for iterating over the force grid is changed depending upon the orientation. I'll explain the new parts piece-by-piece.

Deciding Which Coordinate Represents Rows vs. Columns

    ymax = (!(orient & 1)?device.rows:device.cols) - 1;
    xmax = (!(orient & 1)?device.cols:device.rows) - 1;

In orientation 0 or 2, the grid is iterated over with x as the column dimension and y as the row dimension. However, in orientation 1 or 3, this is swapped. This is then used in the nested for loops to set the bounds for iterating over each dimension.

    for (int i = 0; i <= ymax; i++) {
        for (int j = 0; j <= xmax; j++) {

This allows the parameters of the loop to change without having to do any work in the loop setup or having to create another set of loops.

Deciding Which Point is Each Dimension's Origin

    yref = (orient == 0 || orient == 3?0:1);
    xref = (orient == 0 || orient == 1?0:1);

To change the orientation, we also need to know if each dimension starts at the beginning or end of its range of values. xref and yref are just flags that indicate whether or not the iterator must read from front-to-back or back-to-front over a given dimension.

Pulling the Correct Values from the Array

 for (int i = 0; i <= ymax; i++) {
        for (int j = 0; j <= xmax; j++) {
            std::string spacer = "";

            if (orient == 0 || orient == 2) {
                f = frame->forces[(yref&1?(ymax-i):i) * device.cols + (xref&1?(xmax-j):j)];
            }
            else {
                f = frame->forces[(yref&1?(ymax-i):i) + (xref&1?(xmax-j):j) * device.cols];
            }

The final piece of this data is to pull the correct values out of the array. This was an interesting case because the array is 1-dimensional, so all of the math had to be done in one shot. Essentially, the same thing is happening in both cases:

index = (current_row * column_count) + current_column

The first case deals with orientations 0 and 2, which are 180-degree rotations of each other. The inline conditionals check the flags to see whether or not they are set. If so, the dimension is the difference between the maximum index for that dimension and the current iterator value (read from the back); if not, the value is the current iterator value (read from the front).

The second case deals with orientations 1 and 3, which are 90- and 270-degree rotations of orientation 0, respectively. The conditionals do the same thing as in the first case, but here the x- and y-dimensions are swapped in the for loops, making the loops iterate over all y for a particular x before moving on to the next x.

The output of this code is as follows:

Orientation 0

Orientation 1

Orientation 2

Orientation 3

 These four screenshots are all of my left hand. I did not move my hand between screenshots. The image looks like my right hand because of the location of (0, 0) on THEA's sensor grid and the fact that it's more convenient to put THEA to my left on my desk.

Essentially, I've made an in-place row/column-major determining function. Careful observers will notice that the first indexing case above may actually be done with one check, since in orientation 2 both flags are set to 1. However, by leaving it as-is, we actually allow for four other orientations to be implemented, if desired, by swapping the flags of orientations 0 and 1 and orientations 2 and 3. These will produce mirror-images of the currently implemented orientations across their major axis, so I'm leaving it as-is in the event that somebody wants to use them in the future.

Currently, no drop in performance has been observed as a result of these changes, and since Daniel has been making AJAX calls to the server using this new setup, we should have seen some slow-down in the tests if this was going to be an issue.

The completion of this task means that my work with the code that interprets Tyler's and THEA's data is complete until somebody needs new functionality, so I'm moving on to other tasks.

Claude + Tyler Video

Here is a video of Claude and Tyler both working with live data.

I was going to use Firefox to record the video since it had better performance in the past but as it turns out it is currently much worse. Firefox fails to properly double buffer the canvas element and the result is a flashing screen. Though this SO article doesn't provide an official reference, I believe it is correct in stating that the browser is responsible for double buffering. It is possible to do it at the user level by creating 2 canvas elements and hiding one to be the buffer, but as in the case of Chrome this is not necessary to achieve smooth animations. Firefox didn't have this problem before the addition of Tyler.

I found this article on optimizing code for the canvas particularly interesting. Unfortunately, Chrome's performance issues are due to slow get requests rather than slow canvas drawing, as evidenced by a simple timer on the get requests.