Python Anywhere - What it is and how can it be used

PythonAnywhere is an online IDE with the options of Python 2.7-3.4, iPython, PyPy 2.7, Bash terminal, and MySQL terminal. For a free account, you have the ability to create a python web-app that's hosted on the main site, up to two shells, and the ability to share your console with anyone. There's a private file storage of up to 512mb which for CS students is more than enough.

Advantages:
-This website has a python shell and a bash terminal. This can open up students gently to the idea of using a shell (then using vim, pico, etc).
-Students can use the online text editor which has a push-button option to run code.
-Accounts have file management, so files can be stored and downloaded to local machines.
-Consoles can be shared, which opens up opportunities for teamwork or student/teacher sessions.
-Has tons of modules

Disadvantages:
-Consoles are limited based on processor time used (100 seconds for a free user)
-Site server performance is a concern, but not much of an issue.
-Online only.

Brython - Adaption to class tool

Before going into specifics on how best to utilize Brython, I'll be covering some advantages/disadvantages on the onset of using this as a classroom tool.

Advantages:

Disadvantages:
-Useless if offline. Python's native GUI and countless other applications make this inadequate.
-No file sharing. If you want to publish something, you will have to do it on your own server.
-The constant need to build a HTML webpage in order to actually use Brython.
-If you're not familiar with ECMAscript, brython under the hood will seem weird.
-Limited to its own libraries (Python 3.x) and its ECMAscript libraries.
-Suspect translation of python to ECMAscript. 
-Questionable speeds due to python to ECMAscript translations.
-Python 3 maybe not be the standard for many applications.
-If being taught, would need to implement HTML template source code. 
-There's not immersive way to use this as a teaching tool.

Observing a computer science 101 class

Observed on Oct 30, 2014 - a CPSC1010 class

This professor used a projector setup with her own personal laptop.
The subject language was C and a swap function with pointers was being dissected and discussed. I noted that she was using Word to review code in C.
Next lesson was a quiz review. She went over the questions and answers using the above technique.

At a particular point in the classroom she had to use the faculty computer which was a Windows machine which necessitated the use of SSH. The SSH program for Windows might be difficult to see because of the white background. Unsure if there are program options to change the text font or size.

Using a SSH shell, she demonstrated the use of command line arguments with argc and argv. She would alternate between using VI and then using the command line compiler (GCC) to compile source and observe output.

I noticed most of the students (save a few for Mac users) were using SSH/Putty to use the compiler on school computers. This would be an excellent use of an online IDE as it would make it a lot easier to students/teachers to view/edit/compile source code. Instead of using the cumbersome SSH program, class exercises could easily be done using koding or even cloud9. There was also a special needs student and I wondered how this affected the teacher's methods.

Personal Thoughts:
I have discussed using cloud-based IDEs to help demonstrate programming exercises with the professor. However, I had to be mindful because techniques cannot be easily changed overnight and many teachers have a method to how they teach (e.g. using Word to go over C code) and their personal preferences. I personally think programming code would be easier to view/learn from in terms of it being shown in an IDE environment or an application where a teacher could change the font/size. Students could vastly benefit from using cloud-based IDEs as well. For those who don't know exactly (or yet) how to use command line based editors or SSH/Putty on Windows, they could easily use koding to type out and compile example code. This eliminates the hassle of having to log on to a specific computer (which can be confusing to the first year student). This also effectively cuts the users who are signed on to school workstations thus eliminating extemporaneous workloads on said workstations that could be utilized by later year students.

Summary of powerpoint presentation: Web Based IDEs and their place in the classroom

Overview of IDEs:

Brython:

• Designed to replace JaveScript as the web scripting language.
• Pros:
  Can be used to create custom, interactive lessons in python.
  Can be used to make websites documenting the power of Python as a web script.
• Cons:
  Its scope is limited to being a script object in HTML.
  Limited by its JavaScript library.
  Can be utilized offline. However, if teaching Python offline, perhaps it’s better to use the official Python Idle.

PythonAnywhere:

• PythonAnywhere is an online IDE/Web Host based on Python.
• Pros:
  Implements many versions of python, bash, and even MySQL
  Accounts are free and have 512mb of storage
  Console can be shared -- cooperative work environment
  Python scripts can be saved on the server.
  Many python modules supported.
  On-site script editor alongside python interpreter.
• Cons:
  Dropbox integration is down.
  Use is limited by CPU allowance (processor time used).

koding:

• Koding is an online IDE that allows real-time collaboration alongside access to a full Ubuntu linux terminal that supports many languages.
• Pros:
  A free Ubuntu terminal with many developer tools.
  Extensive help section alongside other users that can answer questions.
  Can get files from Google Drive, BitBucket, DropBox, etc.
  Ability to collaborate in real time plus integrated Google hangouts
• Cons:
  Downtime of terminals (sometimes unscheduled)

Student use:
As a student, it’s simple to log into PythonAnywhere or Koding.
Brython might require the student to be familiar with HTML just to program in python. The student can follow along any online course (youtube, udemy, coursera, etc.) and program in the same window. Collaboration and the ability to save files onto site host servers makes it easy for students to work from home and school on different machines. To turn in homework or assignments, students can invite the teacher into a workspace so he/she can explain the program in real time. No need to install software; less headache about hardware limitations. Internet and a compatible web browser is only needed.

Teacher use:
No administrative access to websites so students will have need a system to turn in completed work. PythonAnywhere and koding can be used to demonstrate concepts while in class. While using a centralized IDE, students will worry less how to do things in the IDE and concentrate on actual programming.
Teachers could write half-completed programs as activities and have students fill in the blanks. Teachers could plan and utilize Brython and create a webpage that demonstrates Python concepts. No need for books or if books are necessary students can follow along and be less distracted by switching between two windowed applications.

Areas to improve for these IDEs (in terms of teaching/learning):
Compiler:

• Providing a compiler that compiles as code is being written.

• A compiler with clearer error messages (more similar to clang)

• If the compiler is providing hints and the student still doesn’t understand what’s wrong, maybe a compiler that just provides answers

Drag and Drop:
Allowing the student to drag and drop things. For instance, if compiler says “missing semi-colon” then student can drag and drop a semi-colon. Or drag and drop variables and initializations to these variables. (Basically like Scratch) (A survey of literature on the teaching of introductory programming)

Help Guide/Forum:
If need help finding certain function names, or functions. Or need help figuring out how to actually do something in code. This can be a forum, where they ask people, or be like a huge database in the system, that just so happens to be intelligent enough to do this.

Built In Code:
Providing code for the student to play around with. Code can be specific examples of if statements, loops, recursion, etc. that encourages the student to change things around, asks them specific questions about what they’re doing and what the code is doing.

Brython -- an introduction

In this blog post, I am going to briefly do an overview on Brython, how to install several versions of it, and ways to implement it as a student.

What is Brython?
Brython is a way to use python instead of javascript for browsers.
However, the only way to implement brython on web pages is to source it from a JavaScript file. The convenience is able to write python script inside a HTML file or to have a python program (.PY file) executed by HTML controls.

How do I use Brython?
Taken from the github repository of Brython:
To use Brython, all there is to do is .

1. load the script brython.js.
2. run the function brython() on page load.
3. write Python code inside tags <script type="text/python">.

Note that Brython implements Python 3 so several 2.7 methods/techniques won't be applicable.

Where to download Brython:
https://github.com/PierreQuentel/brython/releases

There are two different files:
The latest version of brython is the necessary javascript files to code Python into a html file.
The site mirror file contains a clone of the brython.info website that includes documentation and an online script editor console (similar to CPython).

How does Brython work?
Python scripts are converted to JavaScript code, which might explain why it compiles 3-5 times slower. In the FAQ (http://brython.info/doc/en/index.html) , the developers explain circumstances concerning run-time:

• the time to translate Python into Javascript, on the fly in the browser. To give an idea, the module datetime (2130 lines of Python code) is parsed and converted to Javascript in 0,5 second on an ordinary PC
• JavaScript code generated by Brython must be compliant with the specifications of Python, including the dynamic nature of the search attributes, which leads to unoptimized Javascript code

Infrastructure as a Service as a teaching tool

On upcoming blog posts I will be documenting (technically and figuratively) several website that can be used in Computer Science classes. These websites have been chosen for their ability to emulate a Unix environment (primarily for python scripting use) and the ability to share and collaborate the code with instructors and other students.

The following websites:
www.koding.com
brython.info
Cloud9
IDEone
Python Anywhere

Of these three websites, I will document and assess their ability for use for beginning software developer students.

The criteria for assessing these abilities:

• Ease of use in usage.
• Ease of use in installing (brython).
• The capabilities of each website in terms of development environments.
• Reliability of each website.
• The ability to collaborate and/or evaluate source code.
• The aesthetic design of each website.
• The overall usability for each website.

The websites will then be compared with their strengths and weaknesses and it will ultimately be up to the reader to decide which platform best fits their needs in a classroom.

When Benchmarking Goes Bad

For some reason, the benchmarking utility has started going into an infinite loop whenever I run it. Any fixes I try mess up the sample collection (e.g. out of 5000 successful XHRs in one loop I got 12 data points). I don't exactly know what changed--I certainly haven't touched the benchmarking code (though I re-downloaded it for good measure), and I got two successful benchmarks out of it (read: two on each client for a total of six) after the last change I made to the server, meaning it was definitely working. The oddest part is it's still receiving good data from the server. I've tracked the received data several times, and it's dynamic with what the sensor is seeing.

Whatever is happening here, it's clear that I can't be certain of the reliability of any of my previous benchmarks until I figure out what's wrong.


UPDATE:
Apparently I've found some means of controlling the number of requests again. Everything seems to be an order of magnitude slower now, though, which is moderately irritating.

UPDATE 2:
Apparently this is only really an issue over localhost. Walter slowed some, but that's more likely to be a result of network traffic in our lab than anything else (~2ms slower than usual). Dragon had the same time as usual. It's only on localhost that I'm forced to sit through ~6000 requests with a slowdown of an order of magnitude--meaning that the Air on localhost is about as slow as Walter concerning asynchronous sever calls. I'm going to keep looking into this.

Cloud of Computers

When trying to define cloud computing, it's fairly easy to be confused. The simplest explanation I could find is that something as simple as a Google search using cloud computing principles. The principle being you, the user, is using Google's many connected computer network to search for cat images, check your email, or type up your blog. This is also known as SaaS (Software as a Service) where software is not stored/ran locally on your computer. As a student, it's much easier to type up a quick document in Google's text editor as opposed to waiting for Microsoft Word to open up. Also, email via SaaS is the only way I know how to access email! Even if you use an email client, like Thunderbird, you still are drawing data from servers that handle emails like software data.

Preliminary Results

So far, I've conducted benchmarks on my MacBook Air, Walter (the lab's Pi running Raspbian), and Dragon6 (one of the CS lab computers running Ubuntu). The results are telling me that data transmission times involve negligible server turnaround time--I feel confident saying this because the different client computers are averaging different amounts of time.

Air: ~.8ms
Walter: ~1.1ms
Dragon: ~.3ms

Walter and Dragon are running on ethernet, and the Air runs over wifi. The Tactonic server is also situated on the Air, so it has been running on localhost. What all of this seems to indicate is that transmission time is largely client-bound.

In previous AJAX-based web pages that tried to render the Tactonic device's data in "real-time" I encountered a noticeable lag when I set AJAX requests to occur at intervals less than 100ms. Based on the data above, this behavior may be more on the browser-end, dealing with replacing currently displayed content. At that point it becomes a client-side issue of optimizing rendering.

Testing will continue into next week due to a late start on Walter and the Dragon machine.

What is a robot?

What is a robot?

As defined by Merriam-Webster, a robot is "a machine that can do the work of a person and that works automatically or is controlled by a computer." Basically, one can think of robots as tool that can do many varieties of work that humans can do. An autonomous extension of the tools created by mankind. A robot seems to be restricted to do only what it's instructed/programmed to do, however advances in machine learning (and A.I.) are closing the gap between human-like intelligence and robots, albeit slowly.

As a burgeoning programmer, I feel there is strong bond between the mechanical parts of a robot and the central hub of instructions that tell how these mechanical parts work. At the simplest level, there may not be much need for complicated instruction in robots but that ideal might be disappearing fast as computer integration is becoming more and more vital. Robots can be used to clear fields of mines or improvised explosive devices are controlled through a computer by a human user. Robots can be used to clear out incredibly hazardous radioactive accident sites by their human counterpart miles ago. However the software part dictates what can be done with the robot! This opens up infinite possibilities because anyone can program anything (well, almost anything yet).

In closing, I wanted to share a childhood experience of what I still think is a great example of a robot. Although still incredibly advanced for today, one can dream right?
http://www.youtube.com/watch?v=HVtojNukkA0

The Plan for Benchmarking

Now that I have the benchmarking script written up (using Benchmark.js), I'll be running a series of benchmarks on three of the services the Tactonic Server provides:

• Getting the whole grid in JSON format
• Getting entire group pointsets in JSON format
• Getting group hull pointsets in JSON format

After some deliberation, Dr. Remy and I agreed that these would be ideal to test because they contain no interpretation of the data on the sensor (aside from the definition of adjacency), meaning they return raw point data and let the client do what they like with it. The plan is to run a series of benchmarks on each of these functions consisting of 20 samples each over localhost and over the Clemson University network using the CS linux machines.

JavaScript Benchmarking Tools and Why I Chose Benchmark.js

I mentioned in the last post that I'll be benchmarking the Tactonic Server's web services soon, and how I plan to do that bears mentioning. Initially, I planned on writing my own benchmarking utility, and it actually contains (based on looking at other benchmarking utilities) and fairly complete benchmark suite. However, after more thought, it seems to go with the idea of leveraging web standards to use a different benchmarking suite. Because of this, the benchmarking will be done using Benchmark.js.
Benchmark.js is not a web standard, so why does this choice fit with leveraging web standards? First, I was convinced by this article written by the creators of Benchmark.js that describes how it works and their reasoning for writing the library the way they did. They pose well-thought-out criticisms of other approaches to JavaScript function benchmarking and describe how their solution overcomes some of the issues they noted. They also acknowledge the difficulties in designing a browser-based benchmarking system. I have yet to find any major criticism of the library, but it is fairly young yet. If any reader knows of legitimate criticisms of the library, by all means post in the comments.
The point here is that I could have (and did) make a benchmarking utility that does return timing and statistical results, but this would not be useful as a measure of the web services provided by the Tactonic Server. The more useful approach is to use a library where the design and implementation choices are both documented and have justification by the creators, as with Benchmark.js. This, in my opinion, makes it a useful utility and a potential standard because when others look at my results they can find other examples of functions benchmarked by this utility and they can understand why I chose the utility.
Time permitting, I would run the functions through several different benchmarking utilities to provide better coverage and more trustworthy results. If results are consistent across libraries implemented using different methodologies, then it can be reasoned that they are reliable and accurate (assuming the libraries themselves are all working correctly and reporting accurate results, though that's a different bag of worms).
I'm going to try to have the results of a run through Benchmark.js over localhost posted by Monday. I'm working on a local test across Clemson University's network. The fact that I'm bounded to my computer (due to resource constraints and the fact that Tactonic designed their hardware to run on OSX only) is making it difficult, but the results will come.

--EDIT--

For a good explanation of the Benchmark.js process, see this article. The author goes in depth and explains how a benchmark proceeds.

Also, for those who are curious, in order to accurately benchmark a function making an AJAX call, you have to tell Benchmark.js to run asynchronously:

new Benchmark(name, fn, 'async' : true, ...).run();

Where the Tactonic Project Went

The recent posts have been about Heroku, so it's time for an update on the in-progress project involving the Tactonic sensors. The last thing that I posted on regarding this was the concave hull search and optimizing the whole process. The past three weeks have been (in addition to Heroku work) been spent continuing to optimize the Tactonic server and setting up benchmarks for the client end of this process.
The server has been updated to include mutual exclusion around accesses to the shared force grid, JSON output, and a re-vamped group-grabbing operation that eliminates some redundant/repeated calculations. When last I posted about this system, there was a data freeze that affected the system after running it for a period of time. After these upgrades, I am fairly confident that the freeze is on the hardware end, especially since the copy of the force grid that the server works with does not persist between client requests. Logs show that the group grabbing process still occurs after the freeze, and the force grid continues to be copied to a fresh array for processing after the freeze. This tells me that the device driver stops reading in new values at some point, or else the device stops registering new values. Whatever the case, I can get about 15 minutes of uptime (it varies widely, but on average about 15 minutes) before having to restart the server.
I switched to JSON output from plain text because JavaScript has built in facilities to read JSON, and it allows the server to send named fields to the client. This should hopefully reduce some of the overhead in creating new variables to hold parsed plain text, and in having to do the actual text parsing and transformations. The benchmarks will yield the results of these changes, and as soon as I've analyzed the results they will be posted here. I may also implement the client using Node.js and benchmark on that platform, time permitting.

Generating Audio from Arbitrary Pressure Groups

Today, I wrote the last leg of this project: generating audio from arbitrary pressure data. The purpose of this is not only to take some random data and make audio from it--that's easy to do. By simply throwing numbers at WebAudio I can make random, meaningless noise. Now I'm searching for some means of making the sound meaningful in some way, for instance making the tone sharper or flatter as the pressure group moves up or down the x-axis. If changes in the tone are made meaningful, then this project has applications in both human and machine learning, assuming that the agent learning from the device knows how to interpret the changes in tone.

I'll post information on the process of actually making the tone meaningful as that happens, but for now I have something basic that accomplishes this imperfectly:

freq = 110 * (Math.pow(2, max_data / (1000 * area)) + (center_y) + (center_x / 10))

This takes the frequency of an A2 (110), scales it up by 2 to the power of the local pressure maximum divided by the area adjusted to be more even with the max_data point, then offsets the resulting value by the center coordinates. The multiplication by a power of 2 was chosen because the same note in different octaves represents the frequency of that note in octave 1 * 2^the current octave number (e.g. A1 = 55Hz, A2=110Hz, A3=220Hz, etc.).

This method, as mentioned above, needs tweaking. Many different groups sound similar by virtue of the

max_data / (1000 * area) 

calculation, and THEA is actually so sensitive to pressure input that even a small change in pressure can cause the max_data variable to skyrocket, resulting in high-frequency spikes.

On the Javascript end of today's code, and if anyone from W3C sees this, WebAudio's AudioBufferSourceNode object could benefit from a function to check whether or not the node is in start() or stop() state. Javascript throws up an error when a node is started or stopped twice while in the each respective state, and while it isn't hard to implement a check using boolean flags, I was surprised to see that an internal state is not set when start or stop is called. Perhaps it is and that state is simply kept private. In any case, that's a minor issue at best.

Spatial Audio + Robots

Since the last post, I have implemented two new features in the Web Audio GUI I have been working on. First, I adjusted the audio start time so that webpages will all play at the same time in the audio source. Previously, the audio source simply started from the beginning when the page was loaded. With some simple modulus, audio sources are now synchronized and start at a various points in the audio depending on what time the webpage is opened. The key is that now two people on separate devices can have the same experience.

Secondly, I have connected the playground to a ROS service Dr. Remy had previously set up. The service provides the location and direction of robots in a virtual world. In a separate application, you can control one of the robots and view the world. The idea was to have our WebAudio GUI (dubbed WebAudio playground) mimic the virtual world. Robots are given sounds so you can see and here what is going on.

Comparison of Group and Hull Results

With the functions to grab a group's concave hull up and running, now seems a good time to stop and compare the results of the holistic group grabbing functions versus those of the hull grabbing functions. To preface, none of the results were in the least bit surprising, and they have shown me that further modifications are needed to accurately represent a group with the least amount of data possible. If, however, only the hull is needed, the hull functions provide no more or less than the tightest concave hull possible.

Below is the numeric output representation of one of the groups I used to compare the two function sets. This group was made by cupping my left and and placing it horizontally on the sensor, pinky-side down. It took several tries to get the shape just right such that the top was a straight horizontal line, one of the necessary testing conditions for determining whether or not the hull functions perform to my specifications.

0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   
0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   
0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   
0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   
0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   
0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   
0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   
0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   
0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   
0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   
0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   
0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   135228141617349 0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   
0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   427 1153168628043562224115  0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   
0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   240838351020442 56  474 15641568598 17  0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   
0   0   0   0   0   0   0   0   0   0   0   0   0   0   43  1394920 0   0   0   0   28  904 1249870 357 0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   
0   0   0   0   0   0   0   0   0   0   0   0   0   115910410   0   0   0   0   0   0   559 125214151153725 3   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   
0   0   0   0   0   0   0   0   0   0   0   0   0   317 52  0   0   0   0   0   0   0   0   1247185614361648124538  0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   
0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   434 1597196417731189369 0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   
0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   905 190718851292353 0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   
0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   834 14161579117623  0   0   0   0   0   0   0   0   0   0   0   0   0   0   
0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   20  691 1317124134  0   0   0   0   0   0   0   0   0   0   0   0   0   0   
0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   15  27  23  0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   
0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   
0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   
0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   
0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   
0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   
0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   
0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   
0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   
0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   
0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   
0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   
0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   0   


   Center: 22.099086 14.843965

I'll begin by describing the reason for creating this particular shape. As mentioned above, the top of the shape contains a stretch of linear, horizontal hull points. As only the end points of that line are necessary to determine its shape, the algorithm checks for and excludes these kinds of points. It does so by excluding points that contain a linear sequence of adjacent edge points wherein the excluded points are adjacent to exactly three zero-points and those three zero-points are vertically or horizontally linear. If a point is adjacent to exactly three linear zero-points, then it is guaranteed to have one edge-point neighbor on each side in the same direction; thus, that point lies within a linear edge. If, on the other hand, the point has four adjacent zero-points, it is guaranteed to be an end point. If the edge is adjacent to exactly one or exactly two points, then it is in a concave portion of the shape. The only case that needs be excluded is the case in which a point is adjacent to exactly three linear zero-points.

This shape also affords the opportunity to determine whether or not the hull-grabbing algorithm truly generates a concave hull, due to the curvature on the bottom. Convex hull wrapping would ignore the entire concave portion of the shape, leaving too many extraneous zero-points in the group data set. It would also give an inaccurate impression of the group's shape. This particular group allowed me to see if the algorithm actually did what I intended: picking up the concave hull of the group.

Since only one group was picked up from this pressure array, the results are easy to see side-by-side. The group centers and points within the group are listed below.

  Holistic Group Grab  Hull Grab
  
Centers (x y):
  22.099086 14.843965  17.866549 13.251164

Data Points (x y data):
  13 15 1159           13 15 1159
  13 16 317            13 16 317
  14 14 434            14 14 434
  14 15 1041           14 15 1041
  14 16 52             14 16 52
  15 13 2408           15 13 2408
  15 14 1394           15 14 1394
  16 13 3835           16 13 3835
  16 14 920            16 14 920
  17 12 427            17 12 427
  17 13 1020           17 13 1020
  18 12 1153
  18 13 442            18 13 442
  19 11 1352           19 11 1352
  19 12 1686
  19 13 56             19 13 56
  20 11 2814           20 11 2814
  20 12 2804
  20 13 474
  21 11 1617           21 11 1617
  21 12 3562
  21 13 1564
  21 14 28             21 14 28
  22 11 349            22 11 349
  22 12 2241
  22 13 1568
  22 14 904
  22 15 559            22 15 559
  23 12 15             23 12 15
  23 13 598
  23 14 1249
  23 15 1252
  23 16 1247
  23 17 434            23 17 434
  24 13 17             24 13 17
  24 14 870
  24 15 1415
  24 16 1856
  24 17 1597
  24 18 905            24 18 905
  25 14 357            25 14 357
  25 15 1153
  25 16 1436
  25 17 1964
  25 18 1907
  25 19 834
  25 20 20             25 20 20
  26 15 725
  26 16 1648
  26 17 1773
  26 18 1885
  26 19 1416
  26 20 691
  26 21 15             26 21 15
  27 15 3              27 15 3
  27 16 1245
  27 17 1189
  27 18 1292
  27 19 1579
  27 20 1317
  27 21 27             27 21 27
  28 16 38             28 16 38
  28 17 369            28 17 369
  28 18 353
  28 19 1176
  28 20 1241
  28 21 23             28 21 23
  29 19 23             29 19 23
  29 20 34             29 20 34

The blank lines were intentionally place in the hull group point list to underscore the difference between these two point lists. The hull point set is, as expected, much smaller than the holistic set (33 points, as compared to 69).

Fixed Static Parameters Version of the Server, Set Up Tests of Concave Hull

The version of the server that passes parameters by value instead of reference (mentioned in the last post) is ready but not yet uploaded to Buffet. The Buffet server isn't loading pages, so there must be maintenance, development, or a server issue preventing its use. The repos will be updated as soon as possible.

Also, minor testing of the concave hull groups has been set up, and I'll post the results later this week. The algorithm is as follows.

Given a set of adjacent points (a Group as defined in earlier posts), for each point in the group:

• Check all adjacent points, including diagonal points. Record the coordinates of all 0-datapoints adjacent.
• If there are only one or two adjacent 0-datapoints, or more than three, add the point being evaluated to the list of hull points.
• Else, if there are three 0-datapoints, check to see if the set of adjacent 0-datapoints is linear. If it is not linear, add the point being evaluated to the list of hull points. Otherwise, it is part of the hull but is unnecessary to determine the shape of the hull, so do not add it to the point list.
• Else, if there are no adjacent 0-datapoints, do not add it to the point list.

The implementation has not been tested yet, and improvements are probably necessary, but having worked this out by hand on several data sets, it seems to be a sufficient starting point. In addition, it can readily be extended to higher dimensions with a concrete definition of adjacency in that dimension.

For two dimensions, the current C++ implementation is as follows.

void NumViews::addHullGroup(TactonicFrame *frame, uint8_t * checkedArray, int x, int y) {
    Group * g = new Group();
    Point * p = NULL;
    int zeropoints[] = {0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0};
    std::queue<Point *> * q = new std::queue<Point *>();
    int cols = device.cols, rows = device.rows, data, zerocount;

    q->push(new Point(x, y, frame->forces[y * cols + x]));

    while (q->size() > 0) {
        p = q->front();
        q->pop();

        x = p->getX();
        y = p->getY();

        if (!checkedArray[y * cols + x]) {
            data = p->getData();
            zerocount = 0;
            checkedArray[y * cols + x] = 1;



            if (x - 1 >= 0 && frame->forces[y * cols + (x - 1)] > 0 && !checkedArray[y * cols + (x - 1)]) {
                q->push(new Point(x - 1, y, frame->forces[y * cols + (x - 1)]));
            }
            else {
                zeropoints[zerocount] = x - 1;
                zeropoints[8 + zerocount++] = y;
            }

            if (x - 1 >= 0 && y - 1 >= 0 && frame->forces[(y - 1) * cols + (x - 1)] > 0 && !checkedArray[(y - 1) * cols + (x - 1)]) {
                q->push(new Point(x - 1, y - 1, frame->forces[(y - 1) * cols + (x - 1)]));
            }
            else {
                zeropoints[zerocount] = x - 1;
                zeropoints[8 + zerocount++] = y - 1;
            }

            if (y - 1 >= 0 && frame->forces[(y - 1) * cols + x] > 0 && !checkedArray[(y - 1) * cols + x]) {
                q->push(new Point(x, y - 1, frame->forces[(y - 1) * cols + x]));
            }
            else {
                zeropoints[zerocount] = x;
                zeropoints[8 + zerocount++] = y - 1;
            }

            if (x + 1 < cols && y - 1 >= 0 && frame->forces[(y - 1) * cols + (x + 1)] > 0 && !checkedArray[(y - 1) * cols + (x + 1)]) {
                q->push(new Point(x + 1, y - 1, frame->forces[(y - 1) * cols + (x + 1)]));
            }
            else {
                zeropoints[zerocount] = x + 1;
                zeropoints[8 + zerocount++] = y - 1;
            }

            if (x + 1 < cols && frame->forces[y * cols + (x + 1)] > 0 && !checkedArray[y * cols + (x + 1)]) {
                q->push(new Point(x + 1, y, frame->forces[y * cols + (x + 1)]));
            }
            else {
                zeropoints[zerocount] = x + 1;
                zeropoints[8 + zerocount++] = y;
            }

            if (x + 1 < cols && y + 1 < rows && frame->forces[(y + 1) * cols + (x + 1)] > 0 && !checkedArray[(y + 1) * cols + (x + 1)]) {
                q->push(new Point(x + 1, y + 1, frame->forces[(y + 1) * cols + (x + 1)]));
            }
            else {
                zeropoints[zerocount] = x + 1;
                zeropoints[8 + zerocount++] = y + 1;
            }

            if (y + 1 < rows && frame->forces[(y + 1) * cols + x] > 0 && !checkedArray[(y + 1) * cols + x]) {
                q->push(new Point(x, y + 1, frame->forces[(y + 1) * cols + x]));
            }
            else {
                zeropoints[zerocount] = x;
                zeropoints[8 + zerocount++] = y + 1;
            }

            if (x - 1 >= 0 && y + 1 < rows &&frame->forces[(y + 1) * cols + (x - 1)] > 0 && !checkedArray[(y + 1) * cols + (x - 1)]) {
                q->push(new Point(x - 1, y + 1, frame->forces[(y + 1) * cols + (x - 1)]));
            }
            else {
                zeropoints[zerocount] = x - 1;
                zeropoints[8 + zerocount++] = y + 1;
            }



            if (zerocount == 3) {
                for (int i = 0; i < zerocount - 1; i++) {
                    if (zeropoints[i] != zeropoints[i + 1] || zeropoints[i + 8] != zeropoints[i + 9]) {
                        g->addPoint(p, 'x');
                    }
                }
            }
            else if (zerocount != 0) {
                g->addPoint(p, 'x');
            }
        }
        else {
            std::cerr << "Deleting a point in NumViews::addHullGroup" << std::endl;
            delete p;
        }

        p = NULL;
    }
    std::cerr << "Deleting q in NumViews::addHullGroup" << std::endl;
    delete q;

    if (g->size() > 0) {
        g->calculateCenter();
        pressure_groups.push_back(g);
    }
    else {
        delete g;
    }
}

(Everett) Coming to the close of the semester, I have used many tools and resources to accomplish goals this semester. Among those tools are webrtc, node.js, web.py, and other resources mentioned in recent posts. Here I will specify how to acquire or implement those tools.

Webrtc

The webrtc tool is an API now incorporated into most browsers to allow communication through the browser alone without additional plugins. A description of the API can be found here. For more usages it is helpful to review some existing projects. Muaz Khan has a multitude of projects in development on github.

Web.py

To run many of my first projects I used web.py to deploy my code. Web.py is relatively simple to implement, and the details of it can be found on the website.

Node.js

Node.js was used to create a signaling server for my most recent text chat application. Node.js can be used to run javascript from the command line, and with that capability, I used a signaling javascript to handle communications between users of the text chat application. The installation instructions for node.js an be found here. Also, for more detailed information on using node.js as a signaling server Muaz uses it in his experiments as well.

My Code

To view my code on github click the links below

• Using the webcam and gestures
• Text chat via peer-to-peer connections
• Text chat via rooms with router connection

Server Performance Improvements

Today I did a little experiment by changing every function on the server that takes an object, array, or struct as an argument so that they only accept pointers to those parameters. After going through and making the necessary changes to the objects in question, I found that the running time of THEA before the sensor grid freezes when running the same test as in the previous post is ~8 minutes (compared to the 2.5 minutes when passing static objects). I'll run an officially timed comparison soon and post the results.

In addition, I have an old version by which we can run comparisons to show the drastic difference this change makes. For now, though, I'm enjoying the extended run time (>15 minutes when only receiving pressure centers).

THEA's Data Freeze

In testing THEA and her embedded Mongoose server more with AJAX calls, I've come across an issue where the Tactonic frame will stop transferring data to the viewer program. The time it takes for this phenomenon to occur seems inversely proportional to the amount of data being transmitted and the number of processes my computer is running. In the video above, the only non-kernel applications running were the two servers necessary to transmit data from THEA and to run my client, Google Chrome, and Quicktime; in this case, THEA stopped transmitting new data after 1:33. I tried again without running Quicktime, and THEA lasted for 2:28.360 (other trials with as close to the same conditions as I could create ran for about the same amount of time). When transmitting pressure centers only, THEA can keep running for ~10 minutes at a time.

Activity monitor shows that the viewer program consumes 53-55% of my CPU once it freezes. This is consistent with Tactonic's viewer's CPU consumption at freeze time. Tactonic's viewer also sees this issue, though it does not seem to follow any trend. On one run, Tactonic's viewer froze after 10 minutes; on another, it ran with no issue for more than 15.

AppleUSBFTDI and Running Tyler/THEA

The content of this post deals with moving/removing kernel files in OSX to accommodate the needs of a specific application. Be very careful when following these steps: There is some potential to severely damage your system.

My MacBook has been set up to run Tyler and THEA for around six months now, so when I applied some critical updates to the OS it was a point of frustration that the system no longer recognized any Tactonic device plugged into it. After confirming that the problem was with my computer and not the sensors, I immediately suspected that the update had placed new AppleUSBFTDI files in the System Library. Getting my system re-set up for working with Tactonic devices took a good 30 minutes of my life, and that was with some idea of what to do, so here are the exact steps I followed to get back in working order.

Note: A good way to ensure that this is the problem encountered is by plugging the device in and running

> sudo dmesg

right away. If something that looks like

sage.domain com.apple.commssw.ftdi.device] [com.apple.message.signature AppleUSBFTDI] [com.apple.message.signature2 0x403] [com.apple.message.signature3 0x6010] [com.apple.message.summarize YES]
AppleUSBFTDI: Version number - 1.0.1b3, Input buffers 8, Output buffers 16
         0 [Level 5] [com.apple.message.domain com.apple.commssw.ftdi.device] [com.apple.message.signature AppleUSBFTDI] [com.apple.message.signature2 0x403] [com.apple.message.signature3 0x6010] [com.apple.message.summarize YES]
AppleUSBFTDI: Version number - 1.0.1b3, Input buffers 8, Output buffers 16
[0xffffff801e8d8000](1)/(5) Device not responding
com_apple_driver_AppleUSBCardReaderUMC:: Stop::Controller Reset
USBMSC Identifier (non-unique): 000000009833 0x5ac 0x8403 0x9833, 2
         0 [Level 5] [com.apple.message.domain com.apple.commssw.ftdi.device] [com.apple.message.signature AppleUSBFTDI] [com.apple.message.signature2 0x403] [com.apple.message.signature3 0x6010] [com.apple.message.summarize YES]
AppleUSBFTDI: Version number - 1.0.1b3, Input buffers 8, Output buffers 16
         0 [Level 5] [com.apple.message.domain com.apple.commssw.ftdi.device] [com.apple.message.signature AppleUSBFTDI] [com.apple.message.signature2 0x403] [com.apple.message.signature3 0x6010] [com.apple.message.summarize YES]
AppleUSBFTDI: Version number - 1.0.1b3, Input buffers 8, Output buffers 16

comes up near the bottom of the list, this guide could save you a lot of time. This is probably correct for many USB inputs, but for devices that require their own FTDI (or ftdi), you don't want to see this.

1. Update the locate Database

In the terminal, run

> locate.updatedb

which will update locate's file database so that it includes everything on the filesystem. Supposedly the OS will run this periodically, but it can't hurt to manually update it when looking for a specific file. If the command does not execute, try running

> /usr/libexec/locate.updatedb

2. Locate FTDIs

Next, run

> locate FTDI

This should return a list of filepaths with "FTDI" (case-sensitive) in their names. Find the one mentioned in the dmesg output (in my case, AppleUSBFTDI.kext; Tactonic's instructions reference FTDIUSBSerialDriver.kext; the one to deal with really depends on the situation).

The paths you're looking for should be somewhere in /System/Library/Extensions. My locate output looked like

/System/Library/Extensions/IOUSBFamily.kext/Contents/PlugIns/AppleUSBFTDI.kext
/System/Library/Extensions/IOUSBFamily.kext/Contents/PlugIns/AppleUSBFTDI.kext/Contents
/System/Library/Extensions/IOUSBFamily.kext/Contents/PlugIns/AppleUSBFTDI.kext/Contents/Info.plist
/System/Library/Extensions/IOUSBFamily.kext/Contents/PlugIns/AppleUSBFTDI.kext/Contents/MacOS
/System/Library/Extensions/IOUSBFamily.kext/Contents/PlugIns/AppleUSBFTDI.kext/Contents/MacOS/AppleUSBFTDI
/System/Library/Extensions/IOUSBFamily.kext/Contents/PlugIns/AppleUSBFTDI.kext/Contents/_CodeSignature
/System/Library/Extensions/IOUSBFamily.kext/Contents/PlugIns/AppleUSBFTDI.kext/Contents/_CodeSignature/CodeResources
/System/Library/Extensions/IOUSBFamily.kext/Contents/PlugIns/AppleUSBFTDI.kext/Contents/version.plist

When dealing with .kext (Kernel Extension) files, especially when moving/deleting them, ONLY deal with .kext files that you absolutely have to: Moving or removing them may have unintended consequences.

3. Move it to a different directory or remove it entirely

Now that we know where the problem file is, we have to deal with it. Tactonic recommends removing the file entirely; I'm hesitant about removing Kernel Extensions without knowing how their removal will affect my system. Either case seems to work without ruining the USB ports' abilities to function, so this step is up to you.

It may be useful to keep a record of which FTDIs you remove if you go that route, so just copy their filepaths to a text file and save it. Then, run

> sudo mv /path/to/FTDI/problemFTDI.kext /path/to/wherever/you/want/to/move/it

or

> sudo rm /path/to/FTDI/problemFTDI.kext

 As I said above, this step is largely up to you. Just bear in mind my warning about dealing with .kext files.

4. Locate ftdis

This process likely has to be repeated with filepaths with "ftdi" (case-sensitive) in the path, so run

> locate ftdi

Once again, you'll get a list of all of the filepaths containing "ftdi," but this time find the ones in /Library/Receipts. My output didn't have any files in that directory, but it did return some Ruby scripts in another directory. Ignore any filepaths not containing /Library/Receipts. The files you're looking for should be .pkg files.

5. Move It/Them to Another Directory or Remove It/Them

Now repeat step 3 with the new paths. Once again, you may want to keep a record of which files you're dealing with. Run

> sudo mv /path/to/ftdi/problemftdi.pkg /path/to/wherever/you/want/to/move/it

or

> sudo rm /path/to/ftdi/problemftdi.pkg

6. Reboot Your Computer

The changes that we have made so far will not go into effect until your computer is restarted. This is because of the .kext files, which are loaded at startup. Once your computer is back on, plug in the device you were trying to use and run

> sudo dmesg

again. If you don't see anything about the FTDI files you moved/removed, that's a good sign.

Recap

The short version of this process is:

> sudo dmesg
> locate.updatedb
> locate FTDI
> sudo mv /path/to/FTDI/problemFTDI.kext /path/to/wherever/you/want/to/move/it
> locate ftdi
> sudo mv /path/to/ftdi/problemftdi.pkg /path/to/wherever/you/want/to/move/it
> sudo dmesg

This is a specific case of looking for a file using FTDI, but replace the FTDI with whatever the problem name is, and you have a process for dealing with application-specific kernel needs.

New Spatial Audio Example

Dr. Remy wanted to test the limits of the WebAudio API and see what further interesting things we could do with spatial audio. I have been developing a testing framework that will let us play around with several sources and sounds. Dr. Remy's main question is how well the WebAudio API handles a moving audio source and a moving listener. The answer, as seen below, is fairly well (note that you must wear headphones to notice much of any spatialization).

I think it handles motion just fine, but the overall realism of the audio visualization is still of a poor quality. Left versus right is very distinguishable however up/down and front/back have very weak.

Introducing THEA and Server Orientation

This is THEA, the Tactile Hand Evaluation Apparatus, a cousin of Tyler's from Tactonic Technologies. Tyler is experiencing frequent technical difficulties these days, so THEA is filling in while we await a new USB adapter.

THEA, like Tyler, is a pressure sensor made up of sensels. Unlike Tyler, THEA's sensels are packed much more tightly together. Each of Tyler's sensels is spaced about 1in. from every adjacent sensel, with 24x24 sensels in one panel. THEA, on the other hand, packs 34x44 sensels into a single panel, each spaced about 1/4in. from every adjacent sensel.

Due to Tyler's injury, I'll be using THEA to move forward with the Sensory Translation project. The first phase is to make a simple keyboard (as in piano) by dividing THEA's sensels into groups that each emit a distinct tone. I'll be implementing this using the WebAudio API and data from the Mongoose server I've been working on.

THEA has already helped in adding features to the server: now, with a query string, users can specify a tile orientation in one of four directions. Since the tiles are only usable in straight lines, swapping orientations was not difficult, but being able to rapidly test the code with THEA saved quite a bit of time.

Now, if a user wants to specify an orientation, they just add

?orientation=x 

where x may be any 32-bit integer value. The value is taken mod(4) if it is greater than three, then passed into the function that transforms the force grid into a string. That function has been modified to be:

std::string NumViews::getForceGrid(TactonicFrame *frame, int orient, bool usespacer) {
    std::string out = "";
    int xmax, ymax, f;
    uint8_t xref, yref;

    ymax = (!(orient & 1)?device.rows:device.cols) - 1;
    xmax = (!(orient & 1)?device.cols:device.rows) - 1;

    yref = (orient == 0 || orient == 3?0:1);
    xref = (orient == 0 || orient == 1?0:1);


    for (int i = 0; i <= ymax; i++) {
        for (int j = 0; j <= xmax; j++) {
            std::string spacer = "";

            if (orient == 0 || orient == 2) {
                f = frame->forces[(yref&1?(ymax-i):i) * device.cols + (xref&1?(xmax-j):j)];
            }
            else {
                f = frame->forces[(yref&1?(ymax-i):i) + (xref&1?(xmax-j):j) * device.cols];
            }

            out.append(std::to_string(f));
            if (!usespacer && j < xmax) {
                spacer = " ";
            }
            else if (!usespacer){
                spacer = "";
            }

            else {
                spacer = (f>=1000?"":f>=100 && f < 1000?" ":f>=10 && f < 100?"  ":"   ");
            }


            out.append(spacer);
        }
        if (i < ymax) {
            out.append("\n");
        }
    }
    return out;
}

Essentially, what has happened is the frame of reference for iterating over the force grid is changed depending upon the orientation. I'll explain the new parts piece-by-piece.

Deciding Which Coordinate Represents Rows vs. Columns

    ymax = (!(orient & 1)?device.rows:device.cols) - 1;
    xmax = (!(orient & 1)?device.cols:device.rows) - 1;

In orientation 0 or 2, the grid is iterated over with x as the column dimension and y as the row dimension. However, in orientation 1 or 3, this is swapped. This is then used in the nested for loops to set the bounds for iterating over each dimension.

    for (int i = 0; i <= ymax; i++) {
        for (int j = 0; j <= xmax; j++) {

This allows the parameters of the loop to change without having to do any work in the loop setup or having to create another set of loops.

Deciding Which Point is Each Dimension's Origin

    yref = (orient == 0 || orient == 3?0:1);
    xref = (orient == 0 || orient == 1?0:1);

To change the orientation, we also need to know if each dimension starts at the beginning or end of its range of values. xref and yref are just flags that indicate whether or not the iterator must read from front-to-back or back-to-front over a given dimension.

Pulling the Correct Values from the Array

 for (int i = 0; i <= ymax; i++) {
        for (int j = 0; j <= xmax; j++) {
            std::string spacer = "";

            if (orient == 0 || orient == 2) {
                f = frame->forces[(yref&1?(ymax-i):i) * device.cols + (xref&1?(xmax-j):j)];
            }
            else {
                f = frame->forces[(yref&1?(ymax-i):i) + (xref&1?(xmax-j):j) * device.cols];
            }

The final piece of this data is to pull the correct values out of the array. This was an interesting case because the array is 1-dimensional, so all of the math had to be done in one shot. Essentially, the same thing is happening in both cases:

index = (current_row * column_count) + current_column

The first case deals with orientations 0 and 2, which are 180-degree rotations of each other. The inline conditionals check the flags to see whether or not they are set. If so, the dimension is the difference between the maximum index for that dimension and the current iterator value (read from the back); if not, the value is the current iterator value (read from the front).

The second case deals with orientations 1 and 3, which are 90- and 270-degree rotations of orientation 0, respectively. The conditionals do the same thing as in the first case, but here the x- and y-dimensions are swapped in the for loops, making the loops iterate over all y for a particular x before moving on to the next x.

The output of this code is as follows:

Orientation 0

Orientation 1

Orientation 2

Orientation 3

 These four screenshots are all of my left hand. I did not move my hand between screenshots. The image looks like my right hand because of the location of (0, 0) on THEA's sensor grid and the fact that it's more convenient to put THEA to my left on my desk.

Essentially, I've made an in-place row/column-major determining function. Careful observers will notice that the first indexing case above may actually be done with one check, since in orientation 2 both flags are set to 1. However, by leaving it as-is, we actually allow for four other orientations to be implemented, if desired, by swapping the flags of orientations 0 and 1 and orientations 2 and 3. These will produce mirror-images of the currently implemented orientations across their major axis, so I'm leaving it as-is in the event that somebody wants to use them in the future.

Currently, no drop in performance has been observed as a result of these changes, and since Daniel has been making AJAX calls to the server using this new setup, we should have seen some slow-down in the tests if this was going to be an issue.

The completion of this task means that my work with the code that interprets Tyler's and THEA's data is complete until somebody needs new functionality, so I'm moving on to other tasks.

Claude + Tyler Video

Here is a video of Claude and Tyler both working with live data.

I was going to use Firefox to record the video since it had better performance in the past but as it turns out it is currently much worse. Firefox fails to properly double buffer the canvas element and the result is a flashing screen. Though this SO article doesn't provide an official reference, I believe it is correct in stating that the browser is responsible for double buffering. It is possible to do it at the user level by creating 2 canvas elements and hiding one to be the buffer, but as in the case of Chrome this is not necessary to achieve smooth animations. Firefox didn't have this problem before the addition of Tyler.

I found this article on optimizing code for the canvas particularly interesting. Unfortunately, Chrome's performance issues are due to slow get requests rather than slow canvas drawing, as evidenced by a simple timer on the get requests.

Claude + Tyler Update

Making the transition from 2D to 3D was not as easy as previously thought. First off, I realized I had forgotten toproperly render the skeleton data in 3D to begin with. I was simply taking the X and Y world coordinates of the skeleton, linearly scaling them, and drawing them on the 2D canvas, effectively ignoring the Z component. This "approximation" worked when just viewing the skeleton (or data based directly on the skeleton's location such as the energy ball in Web-Kamehameha), but in order to incorporate Tyler in 3D and have its scale and proportions match the skeleton, I would have to do proper projection. Fortunately, the Microsoft SDK provides a projection function that is used in one of its samples (the one Claude was based off of). It can convert a skeleton point (world coordinate) into a screen coordinate on a 640x480 resolution. Unfortunately, this means my web application for Tyler + Claude would have an extra dependency on Claude, a projection service. It was necessary though to ensure that Tyler's world coordinates (which were computed from the skeleton world coordinates, more on that in a minute) would be projected in the same way that the skeleton was. I also added another service to Claude to return the skeleton in screen coordinates so I could avoid an unnecessary projection call.

To get the world coordinates of Tyler in the first place, I had to implement a calibration function). The calibration function reads the location of the left foot and stores that as the corner of the Tyler grid. The function needs to be run twice to get the two opposite corners of the grid. In the video below, you can see me calibrating the grid.

The grid initially displayed is based off of old calibration points. It has no bearing on where the current grid is. You can see me walk over to the edge of the screen and calibrate the first point. After doing so, one of the grid corners snaps to that location. After calibrating the other corner, the grid grows to the appropriate size. What I am doing is using the displacement of the two world coordinates divided by the number of sensors along the X and Z axes (width and height) to figure out the dimensions of each individual sensor. I then compute the individual world coordinates of each sensor, aggregate them, and send them to Claude's project service to project all the points onto the 2D plane. When they come back, I draw them connected as polygons and fill them with a color based on the pressure data. This does mean there is a slight inaccuracy in the rendered grid as the actual sensor location is the lower left corner of the polygon, not the center.

In the video, I am using static pressure data of a single foot. The purpose of the video was merely to show calibrating the grid size. The application can currently handle live data however do to limitations of the Tyler service, I have been unable to run a full test with it and record the results. That will hopefully happen soon. In the video I also tried to turn off the grid lines so you could see the pressure colors better but I was using teamviewer on a tablet to control my desktop and got really buggy when I tried to use backspace.

One last interesting piece of information is how tilting affects the Kinect, or rather how it doesn't. First, consider the image midway down this page. It shows the three axes in respect to the Kinect's sensors. When the Kinect tilts, the axes do as well. In reality, we would want the axes to always remain in the same position relative to the world, not the tilt of the Kinect. Supposedly the Kinect has a sensor that detects it's orientation so perhaps there is a feature in the Kinect SDK to translate world coordinates back to their real orientation so that when the Kinect is tilted, the axes aren't. Currently, we can only run Claude + Tyler when the Kinect has 0 degrees of tilt, otherwise the tiles will no longer be at a constant Y but will appear to rise on the Y axis.

Sampling-Based Motion Planning Lecture

In the lecture "Sampling-Based Motion Planning: from Intelligent CAD to Crowd Simulation to Protein Folding" here at Clemson, Dr. Nancy Amato of Texas A&M presented improvements to Probabilistic Roadmap Methods (PRM) for solving motion planning problems.

The problems that Dr. Amato described are all set in a n-dimensional Constraint Space (C-Space), where each constraint is one dimension. This could be as simple as a set of Cartesian coordinates or as complicated as bond angles between each carbon in a protein. The motion of the actor within its C-Space is determined by the presence of C-Obstacles, places where the actor cannot move in the C-Space.

PRMs build a set of possible routes through a C-Space by

1. Randomly generating a list of points in the C-Space and discarding invalid points (i.e. those within a certain proximity of an obstacle)
2. Connecting each remaining point to all other points and discarding any invalid connection

PRMs generate the path before the actor even begins moving, making querying paths a matter of looking up results instead of calculating them at the time of the query. Dr. Amato claimed that these methods are probabilistically complete and easily applied to high-dimensional spaces. The major drawback, though, comes from the fact that points are sampled randomly: it is unlikely that the actor will find a path through narrow passages.

The problem, then, is finding ways to sample nodes close to obstacles so that actors may better traverse narrow passages. PRMs that attempt to solve the problem in this way are called Obstacle-Based PRMs (OBPRM). The idea here is to

1. Find a point in an obstacle
2. Select a random direction
3. Find a free point in that direction
4. Determine the the boundary between the obstacle and the free point

The second problem is efficiency of mapping. By the nature of their design, PRMs are relatively easily parallelized at each stage, boosting their efficiency. In the case of moving constraints, PRMs must redo the entire pathfinding process for each distinct configuration of the C-Space. In this case, it is more efficient to repair approximate paths instead of redoing the entire pre-query stage.

In some cases, efficiency can be improved using hybrid human/planner systems in which a human agent traces a path through the space and the planner uses that path data to make decisions. The given example of this system had the human agent trace a path on a haptic device, then the path was passed to the planner. Dr. Amato's group used these systems in CAD designs and deformable object modelling, but she noted that this is only useful when the solution is fairly obvious to humans.

PRMs are also used to aid flocking (coordinated) behavior by helping individual members of the flock to find a path. Dr. Amato's group successfully applied this system to architectural problems.

The most interesting application (thanks to my bias toward biological applications of computing) was protein-folding modelling. While computers have successfully been used to determine the normal state of a protein for some time, they only look at the final state and not how the protein folded into that state. PRMs can model the actual folding process to show how a protein reaches its normal state. The presentation had some of the data from these experiments, and it looked promising, though I can't say I know enough about the results to talk about them in detail.

Overall, PRMs are an interesting concept. Randomness and probability in computing have always interested me because they allow for degrees of flexibility that rigidly logical systems do not provide and make behavior more realistic. These concepts are not necessarily useful in every case, but they provide an alternate way of thinking about how we solve problems that breaks out of the computer science shell.

Tyler Update

February 10th, I took a look at the documentation for Mongoose and easily found a function ("mg_send_response") to directly insert a header into the server's response. I downloaded the latest version of Mongoose and the hello.c example and tested it out on another machine. The hello.c example worked fine as-is but when I tried to add the response header to the code, the page would take at least 30s to load. It turns out I was trying to add a response header after already returning data from the function handler in Mongoose. I needed to set the response header first thing in the function. Once I had the CORS response headers working on localhost, it was time to actually try a cross-domain request with ajax. From a separate machine I tested an ajax call but sadly it still failed. Part of the error message clued me in to the problem. It turns out I was referencing the server by IP only and I needed to prefix the HTTP protocol to the IP in order to get the cross-domain request to work. With cross-domain requests working, I had my co-worker Jeff make the necessary modifications to his Tyler server and then run a test. He connected his macbook to the tile sensors and ran the server while I ran my tyler visualization on a separate machine. The test worked and the results of him walking across the tiles can be observed in the video below:

Grabbing Patterns

The pattern-grabbing code is now functional and can read from the Mongoose server.

The Pattern Grabber

from queue import LifoQueue
def grab_group(graph, rows, cols, x, y):
    q = LifoQueue()
    
    group = Group()

    q.put([x, y])

    while not q.empty():
        tempx, tempy = q.get()
        
        if not group.contains(tempx, tempy, None):
            group.add_point(tempx, tempy, graph[tempy][tempx])
            graph[tempy][tempx] = 0
            
            if tempx > 0 and graph[tempy][tempx - 1] != 0:
                q.put([tempx - 1, tempy])
                
            if tempx < cols and graph[tempy][tempx + 1] != 0:
                q.put([tempx + 1, tempy])

            if tempy > 0 and graph[tempy - 1][tempx] != 0:
                q.put([tempx, tempy - 1])

            if tempy < rows and graph[tempy + 1][tempx] != 0:
                q.put([tempx, tempy + 1])

    group.calculate_center()
    return group

This code makes use of Python's queue.LifoQueu, which is essentially a stack. The function takes an iterable storage data type containing data points, a row size, a column size, and starting coordinates.

Once everything is initialized, the initial points are placed in the queue and the following loop runs until the queue is empty. At each iteration, the next point in the queue is pulled and the group is checked to see if it contains the point.

This check required a small modification to class Point's compare method.

    def compare(self, p):
        '''
        Compares the point passed to self. For coordinate-only comparison, pass
        a point with p.data = None.
        '''
        return (self.coords == p.coords and \
                (self.data == p.data or p.data is None))

Before this change, grab_group() would occasionally grab the same point a second time after the data had been set to zero. This was due to the fact that a previously enqueued data point had since been set to zero, bypassing the checks for a zero data point. It was more useful to give the Point class the option to compare coordinates only than to add a second condition to grab_group().

If the point is not in the group, it is added to the group and that point in the graph is set to zero. Then, each adjacent point is checked for nonzero values. If the adjacent point contains a nonzero data point and is within range of the graph, it is added to the queue.

After all points are added to the group, the center point is calculated as a pair of floating-point values. The group is then returned.

This algorithm is useful despite the potential for checking the same point multiple times because it follows the shape of the data instead of needing information about the shape. For out purposes, it seems fast enough, though it has not been tested using fast, consecutive requests to the Mongoose server.

The Server Request

This is only being documented here because it took some time to work out how to do this properly. Initially, Python's urllib2 module was being used to no effect, and it took some time afterward to work out exactly what needed to be done to retrieve the information.

import httplib
def read_graph_data(f):
    '''Reads pressure data from site f into an array.
    '''
    try:
        #text = open('data.txt')
        conn = httplib.HTTPConnection("127.0.0.1:8080")
        conn.request("GET", "/data")
    except IOError:
        print "Could not open the file."

    text = conn.getresponse()
    #text = text.read()
    conn.close()
    del conn

    #print text
    
    graph = text.split('\n')
    #graph.remove('')
    del text

    for y in xrange(0, len(graph)):
        graph[y] = graph[y].split(' ')
        for x in xrange(0, len(graph[y])):
            graph[y][x] = int(graph[y][x])
    

    return graph

This uses httplib to talk with the server. It opens a connection and requests a specific URI extension. The only response at the page '\data' is the raw data, so no extra formatting work is needed. After this, the formatting and typecasting is straightforward.

The response is split into a list of strings, then each string in the list is split into its own list of strings, and each member of the 2d list is converted into an integer.

Next Steps

Daniel and I were able to get his AJAX calls to the server working today, and he took a video of the result.

I will continue by looking into Python audio libraries and work on a simple program to output audio.