Friday, April 18, 2014

Generating Audio from Arbitrary Pressure Groups

Today, I wrote the last leg of this project: generating audio from arbitrary pressure data. The purpose of this is not only to take some random data and make audio from it--that's easy to do. By simply throwing numbers at WebAudio I can make random, meaningless noise. Now I'm searching for some means of making the sound meaningful in some way, for instance making the tone sharper or flatter as the pressure group moves up or down the x-axis. If changes in the tone are made meaningful, then this project has applications in both human and machine learning, assuming that the agent learning from the device knows how to interpret the changes in tone.

I'll post information on the process of actually making the tone meaningful as that happens, but for now I have something basic that accomplishes this imperfectly:

freq = 110 * (Math.pow(2, max_data / (1000 * area)) + (center_y) + (center_x / 10))

This takes the frequency of an A2 (110), scales it up by 2 to the power of the local pressure maximum divided by the area adjusted to be more even with the max_data point, then offsets the resulting value by the center coordinates. The multiplication by a power of 2 was chosen because the same note in different octaves represents the frequency of that note in octave 1 * 2^the current octave number (e.g. A1 = 55Hz, A2=110Hz, A3=220Hz, etc.).

This method, as mentioned above, needs tweaking. Many different groups sound similar by virtue of the
 
max_data / (1000 * area) 

calculation, and THEA is actually so sensitive to pressure input that even a small change in pressure can cause the max_data variable to skyrocket, resulting in high-frequency spikes.

On the Javascript end of today's code, and if anyone from W3C sees this, WebAudio's AudioBufferSourceNode object could benefit from a function to check whether or not the node is in start() or stop() state. Javascript throws up an error when a node is started or stopped twice while in the each respective state, and while it isn't hard to implement a check using boolean flags, I was surprised to see that an internal state is not set when start or stop is called. Perhaps it is and that state is simply kept private. In any case, that's a minor issue at best.

Wednesday, April 16, 2014

Spatial Audio + Robots

Since the last post, I have implemented two new features in the Web Audio GUI I have been working on. First, I adjusted the audio start time so that webpages will all play at the same time in the audio source. Previously, the audio source simply started from the beginning when the page was loaded. With some simple modulus, audio sources are now synchronized and start at a various points in the audio depending on what time the webpage is opened. The key is that now two people on separate devices can have the same experience.



Secondly, I have connected the playground to a ROS service Dr. Remy had previously set up. The service provides the location and direction of robots in a virtual world. In a separate application, you can control one of the robots and view the world. The idea was to have our WebAudio GUI (dubbed WebAudio playground) mimic the virtual world. Robots are given sounds so you can see and here what is going on.


Comparison of Group and Hull Results

With the functions to grab a group's concave hull up and running, now seems a good time to stop and compare the results of the holistic group grabbing functions versus those of the hull grabbing functions. To preface, none of the results were in the least bit surprising, and they have shown me that further modifications are needed to accurately represent a group with the least amount of data possible. If, however, only the hull is needed, the hull functions provide no more or less than the tightest concave hull possible.

Below is the numeric output representation of one of the groups I used to compare the two function sets. This group was made by cupping my left and and placing it horizontally on the sensor, pinky-side down. It took several tries to get the shape just right such that the top was a straight horizontal line, one of the necessary testing conditions for determining whether or not the hull functions perform to my specifications.

0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 135228141617349 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 427 1153168628043562224115 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 240838351020442 56 474 15641568598 17 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 43 1394920 0 0 0 0 28 904 1249870 357 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 115910410 0 0 0 0 0 0 559 125214151153725 3 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 317 52 0 0 0 0 0 0 0 0 1247185614361648124538 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 434 1597196417731189369 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 905 190718851292353 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 834 14161579117623 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 20 691 1317124134 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 15 27 23 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0


Center: 22.099086 14.843965


I'll begin by describing the reason for creating this particular shape. As mentioned above, the top of the shape contains a stretch of linear, horizontal hull points. As only the end points of that line are necessary to determine its shape, the algorithm checks for and excludes these kinds of points. It does so by excluding points that contain a linear sequence of adjacent edge points wherein the excluded points are adjacent to exactly three zero-points and those three zero-points are vertically or horizontally linear. If a point is adjacent to exactly three linear zero-points, then it is guaranteed to have one edge-point neighbor on each side in the same direction; thus, that point lies within a linear edge. If, on the other hand, the point has four adjacent zero-points, it is guaranteed to be an end point. If the edge is adjacent to exactly one or exactly two points, then it is in a concave portion of the shape. The only case that needs be excluded is the case in which a point is adjacent to exactly three linear zero-points.

This shape also affords the opportunity to determine whether or not the hull-grabbing algorithm truly generates a concave hull, due to the curvature on the bottom. Convex hull wrapping would ignore the entire concave portion of the shape, leaving too many extraneous zero-points in the group data set. It would also give an inaccurate impression of the group's shape. This particular group allowed me to see if the algorithm actually did what I intended: picking up the concave hull of the group.

Since only one group was picked up from this pressure array, the results are easy to see side-by-side. The group centers and points within the group are listed below.

  Holistic Group Grab  Hull Grab

Centers (x y):
22.099086 14.843965 17.866549 13.251164

Data Points (x y data):
13 15 1159 13 15 1159
13 16 317 13 16 317
14 14 434 14 14 434
14 15 1041 14 15 1041
14 16 52 14 16 52
15 13 2408 15 13 2408
15 14 1394 15 14 1394
16 13 3835 16 13 3835
16 14 920 16 14 920
17 12 427 17 12 427
17 13 1020 17 13 1020
18 12 1153
18 13 442 18 13 442
19 11 1352 19 11 1352
19 12 1686
19 13 56 19 13 56
20 11 2814 20 11 2814
20 12 2804
20 13 474
21 11 1617 21 11 1617
21 12 3562
21 13 1564
21 14 28 21 14 28
22 11 349 22 11 349
22 12 2241
22 13 1568
22 14 904
22 15 559 22 15 559
23 12 15 23 12 15
23 13 598
23 14 1249
23 15 1252
23 16 1247
23 17 434 23 17 434
24 13 17 24 13 17
24 14 870
24 15 1415
24 16 1856
24 17 1597
24 18 905 24 18 905
25 14 357 25 14 357
25 15 1153
25 16 1436
25 17 1964
25 18 1907
25 19 834
25 20 20 25 20 20
26 15 725
26 16 1648
26 17 1773
26 18 1885
26 19 1416
26 20 691
26 21 15 26 21 15
27 15 3 27 15 3
27 16 1245
27 17 1189
27 18 1292
27 19 1579
27 20 1317
27 21 27 27 21 27
28 16 38 28 16 38
28 17 369 28 17 369
28 18 353
28 19 1176
28 20 1241
28 21 23 28 21 23
29 19 23 29 19 23
29 20 34 29 20 34


The blank lines were intentionally place in the hull group point list to underscore the difference between these two point lists. The hull point set is, as expected, much smaller than the holistic set (33 points, as compared to 69).


Monday, April 14, 2014

Fixed Static Parameters Version of the Server, Set Up Tests of Concave Hull

The version of the server that passes parameters by value instead of reference (mentioned in the last post) is ready but not yet uploaded to Buffet. The Buffet server isn't loading pages, so there must be maintenance, development, or a server issue preventing its use. The repos will be updated as soon as possible.

Also, minor testing of the concave hull groups has been set up, and I'll post the results later this week. The algorithm is as follows.

Given a set of adjacent points (a Group as defined in earlier posts), for each point in the group:
  • Check all adjacent points, including diagonal points. Record the coordinates of all 0-datapoints adjacent.
  • If there are only one or two adjacent 0-datapoints, or more than three, add the point being evaluated to the list of hull points.
  • Else, if there are three 0-datapoints, check to see if the set of adjacent 0-datapoints is linear. If it is not linear, add the point being evaluated to the list of hull points. Otherwise, it is part of the hull but is unnecessary to determine the shape of the hull, so do not add it to the point list.
  • Else, if there are no adjacent 0-datapoints, do not add it to the point list.

The implementation has not been tested yet, and improvements are probably necessary, but having worked this out by hand on several data sets, it seems to be a sufficient starting point. In addition, it can readily be extended to higher dimensions with a concrete definition of adjacency in that dimension.

For two dimensions, the current C++ implementation is as follows.


void NumViews::addHullGroup(TactonicFrame *frame, uint8_t * checkedArray, int x, int y) {
    Group * g = new Group();
    Point * p = NULL;
    int zeropoints[] = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0};
    std::queue<Point *> * q = new std::queue<Point *>();
    int cols = device.cols, rows = device.rows, data, zerocount;

    q->push(new Point(x, y, frame->forces[y * cols + x]));

    while (q->size() > 0) {
        p = q->front();
        q->pop();

        x = p->getX();
        y = p->getY();

        if (!checkedArray[y * cols + x]) {
            data = p->getData();
            zerocount = 0;
            checkedArray[y * cols + x] = 1;



            if (x - 1 >= 0 && frame->forces[y * cols + (x - 1)] > 0 && !checkedArray[y * cols + (x - 1)]) {
                q->push(new Point(x - 1, y, frame->forces[y * cols + (x - 1)]));
            }
            else {
                zeropoints[zerocount] = x - 1;
                zeropoints[8 + zerocount++] = y;
            }

            if (x - 1 >= 0 && y - 1 >= 0 && frame->forces[(y - 1) * cols + (x - 1)] > 0 && !checkedArray[(y - 1) * cols + (x - 1)]) {
                q->push(new Point(x - 1, y - 1, frame->forces[(y - 1) * cols + (x - 1)]));
            }
            else {
                zeropoints[zerocount] = x - 1;
                zeropoints[8 + zerocount++] = y - 1;
            }

            if (y - 1 >= 0 && frame->forces[(y - 1) * cols + x] > 0 && !checkedArray[(y - 1) * cols + x]) {
                q->push(new Point(x, y - 1, frame->forces[(y - 1) * cols + x]));
            }
            else {
                zeropoints[zerocount] = x;
                zeropoints[8 + zerocount++] = y - 1;
            }

            if (x + 1 < cols && y - 1 >= 0 && frame->forces[(y - 1) * cols + (x + 1)] > 0 && !checkedArray[(y - 1) * cols + (x + 1)]) {
                q->push(new Point(x + 1, y - 1, frame->forces[(y - 1) * cols + (x + 1)]));
            }
            else {
                zeropoints[zerocount] = x + 1;
                zeropoints[8 + zerocount++] = y - 1;
            }

            if (x + 1 < cols && frame->forces[y * cols + (x + 1)] > 0 && !checkedArray[y * cols + (x + 1)]) {
                q->push(new Point(x + 1, y, frame->forces[y * cols + (x + 1)]));
            }
            else {
                zeropoints[zerocount] = x + 1;
                zeropoints[8 + zerocount++] = y;
            }

            if (x + 1 < cols && y + 1 < rows && frame->forces[(y + 1) * cols + (x + 1)] > 0 && !checkedArray[(y + 1) * cols + (x + 1)]) {
                q->push(new Point(x + 1, y + 1, frame->forces[(y + 1) * cols + (x + 1)]));
            }
            else {
                zeropoints[zerocount] = x + 1;
                zeropoints[8 + zerocount++] = y + 1;
            }

            if (y + 1 < rows && frame->forces[(y + 1) * cols + x] > 0 && !checkedArray[(y + 1) * cols + x]) {
                q->push(new Point(x, y + 1, frame->forces[(y + 1) * cols + x]));
            }
            else {
                zeropoints[zerocount] = x;
                zeropoints[8 + zerocount++] = y + 1;
            }

            if (x - 1 >= 0 && y + 1 < rows &&frame->forces[(y + 1) * cols + (x - 1)] > 0 && !checkedArray[(y + 1) * cols + (x - 1)]) {
                q->push(new Point(x - 1, y + 1, frame->forces[(y + 1) * cols + (x - 1)]));
            }
            else {
                zeropoints[zerocount] = x - 1;
                zeropoints[8 + zerocount++] = y + 1;
            }



            if (zerocount == 3) {
                for (int i = 0; i < zerocount - 1; i++) {
                    if (zeropoints[i] != zeropoints[i + 1] || zeropoints[i + 8] != zeropoints[i + 9]) {
                        g->addPoint(p, 'x');
                    }
                }
            }
            else if (zerocount != 0) {
                g->addPoint(p, 'x');
            }
        }
        else {
            std::cerr << "Deleting a point in NumViews::addHullGroup" << std::endl;
            delete p;
        }

        p = NULL;
    }
    std::cerr << "Deleting q in NumViews::addHullGroup" << std::endl;
    delete q;

    if (g->size() > 0) {
        g->calculateCenter();
        pressure_groups.push_back(g);
    }
    else {
        delete g;
    }
}
(Everett) Coming to the close of the semester, I have used many tools and resources to accomplish goals this semester. Among those tools are webrtc, node.js, web.py, and other resources mentioned in recent posts. Here I will specify how to acquire or implement those tools.

Webrtc

The webrtc tool is an API now incorporated into most browsers to allow communication through the browser alone without additional plugins. A description of the API can be found here. For more usages it is helpful to review some existing projects. Muaz Khan has a multitude of projects in development on github.

Web.py

To run many of my first projects I used web.py to deploy my code. Web.py is relatively simple to implement, and the details of it can be found on the website.

Node.js

Node.js was used to create a signaling server for my most recent text chat application. Node.js can be used to run javascript from the command line, and with that capability, I used a signaling javascript to handle communications between users of the text chat application. The installation instructions for node.js an be found here. Also, for more detailed information on using node.js as a signaling server Muaz uses it in his experiments as well.

My Code

To view my code on github click the links below
  • Using the webcam and gestures
  • Text chat via peer-to-peer connections
  • Text chat via rooms with router connection

Friday, April 11, 2014

Server Performance Improvements

Today I did a little experiment by changing every function on the server that takes an object, array, or struct as an argument so that they only accept pointers to those parameters. After going through and making the necessary changes to the objects in question, I found that the running time of THEA before the sensor grid freezes when running the same test as in the previous post is ~8 minutes (compared to the 2.5 minutes when passing static objects). I'll run an officially timed comparison soon and post the results.

In addition, I have an old version by which we can run comparisons to show the drastic difference this change makes. For now, though, I'm enjoying the extended run time (>15 minutes when only receiving pressure centers).

Monday, April 7, 2014

THEA's Data Freeze


In testing THEA and her embedded Mongoose server more with AJAX calls, I've come across an issue where the Tactonic frame will stop transferring data to the viewer program. The time it takes for this phenomenon to occur seems inversely proportional to the amount of data being transmitted and the number of processes my computer is running. In the video above, the only non-kernel applications running were the two servers necessary to transmit data from THEA and to run my client, Google Chrome, and Quicktime; in this case, THEA stopped transmitting new data after 1:33. I tried again without running Quicktime, and THEA lasted for 2:28.360 (other trials with as close to the same conditions as I could create ran for about the same amount of time). When transmitting pressure centers only, THEA can keep running for ~10 minutes at a time.

Activity monitor shows that the viewer program consumes 53-55% of my CPU once it freezes. This is consistent with Tactonic's viewer's CPU consumption at freeze time. Tactonic's viewer also sees this issue, though it does not seem to follow any trend. On one run, Tactonic's viewer froze after 10 minutes; on another, it ran with no issue for more than 15.