More Papers

Dr. Remy took an interest in The Web Browser As Synthesizer And Interface and requested I do further research on it. The opening sentence from the abstract of the paper reads "our research examines the use and potential of native web technologies for musical expression". This is quite similar to our research on using the web browser as a mediator for other technologies. As such, it would be interesting to see what other papers this author has written, who has cited his paper, and what other papers share some of his sources.

The paper actually has three authors: Charlie Roberts, Graham Wakefield, and Matthew Wright. Wakefield is from a graduate school in Daejon, Republic of Korea while the other two are in the Media Arts and Technology program at University of California at Santa Barbara.

Charlie Roberts appears in another paper titled Enabling Multimodal Mobile Interfaces for Interactive
Musical Performance. The paper is about a mobile application called Control that provides a prototyping environment for defining multimodal interfaces that control real-time artistic and musical performances. He later "broadly" defines multimodal to be the combination of multiple signals from different modalities in order to generate a single coherent action. Control allows user to quickly create interfaces that use multiple modalities such as: speec recognition, computer vision, musical feature extraction, touchscreen widges, and inertial sensor data. The application uses JSON to create its interfaces and can be extended with JavaScript.

After discovering that Charlie Roberts did this second paper while at UCSB and with yet another person in the Media Arts program at USCB, I decided to find their publications page and see what else was going on in the program. A quick search on the page for "web" turns up the first paper I posted above as well as two others of potential interest. The first and most promising of which is www.lovelyweather.com: A Web-Based Interactive Audio-Visual Environment featuring two other UCSB students, Salman Bakht and Christopher Jette. The abstract explains that lovelyweather.com is a web-based hypermedia work created by the two authors and a third person, Alenjandro Casazi. Straight from the abstract "[the website] combines digital photographs with environmental audio recordings to form a virtual audio-visual space navigable by computer mouse. www.lovelyweather.com examines the relationsihp between the methdos of navigation used in web-based media and the process of exploration that an artist and audience experience in the creation, presentation, and interpretation of artwork". I found the abstract intriguing and visited www.lovelyweather.com myself. The project is quite interesting and essential starts you off in an environment, meaning the sounds and images displayed are related to the same physical environment. Certain "squares" or smaller images focus on a section of that environment and mousing over them appears to slowly change the audio to focus on that part of the environment. Likewise clicking them brings up a larger image of that section and re-focuses the audio. Spatial audio is also used in both the individual environment recordings themselves and when you are choosing a new environment to switch too.

The third paper from UCSB that mentions web browsers is Instructional Tools in Educational Measurement and Statistics (ITEMS) for School Personnel: Evaluation of Three Web-Based Training Modules. The paper focuses much more on the assessments themselves and the issues surrounding standardized testing rather than the medium, the browser, so I don't think this is very applicable to our research however it may still contain some relevant data or reflections so it is worth noting.

As for Charlie Robert's other publications from UCSB, he has quite a few, eleven to be exact. The latest two of which are those previously featured in this blog post. The third latest is Mobile Controls On-The-Fly:
An Abstraction for Distributed NIMEs which also makes mention of the application Control. The abstract notes that musical artists waste a lot of time on non-musical tasks such as network configuration, development environments for mobile devices, and OSC address spaces and environments. The authors' research is aimed at solving this problem by minimizing the code necessary to generate and read the output of interfaces on mobile devices.

Roberts also appears in two other papers of potential interest: Composition for Conductor and Audience:
New Uses for Mobile Devices in the Concert Hall and LuaAV: Extensibility and Heterogeneity for Audiovisual. The first is about an audience interaction piece called Composition for Conductor and Audience which leverages Control to allow audience members to become the "orchestra" by performing using different musical variables on the touchscreens of their mobile devices. The audience members actually downloaded the application just prior to the start of the concert and were given basic instructions on how to respond to certain gestures by the conductor in terms of an action on their device. I was able to find a video including some of the audio here. While it doesn't give a clear idea of what sound is being created by the conductor versus the audience, it shows visual examples of the application in use which is rather interesting. The second is a paper on the use of LuaAV, a runtime library that extends the Lua programing language, to support computational composition of temporal, sound, visual, spatial, and other elements. The abstract also states "In this paper we document how we have attempted to maintain several core principles of Lua itself... while providing the flexibility and temporal accuracy demanded by interactive audio-visual media".

Of the authors in the original paper that spawned this research tree, Graham Wakefield is featured in 24 papers from UCSB's Media Arts publications page. Titles of interest are "A Toolkit for Composing Immersive Audio-Visual Worlds of Agency and Autonomy", "Augmenting Computer Music with Just-In-Time Compilation", "The Allobrain: An Interactive, Stereographic, 3D Audio Immersive Virtual World", and "Experiencing Audio and Music in a Fully Immersive Environment".  Some of his others are shared with the Roberts papers previously discussed. Matthew Wright is also featured in 15 featuring "Sound Element Spatializer", a system for rendering and control of spatial audio. It operates on a cross-platform C++ application that can spatialize sound sources from other applications or live inputs in real-time.

I was unable to find any papers that cited the original paper that spurred this blogpost. It is relatively new (2013) and probably has not had time to engender others to further research.

As for the paper's sources, three other papers cite the source "C. Rogers. Web audio API. http://www.w3.org/TR/webaudio/" as seen here. The only paper of interest (other than the original) is yet another by Charlie Roberts. Gibber: Live coding audio in the browser is a paper that was mentioned on UCSB's Media Arts publications page but was not linked. Gibber is described as a javascript coding environment for web browsers that provides several synthesis options, audio effects, and sequencing objects to be controlled. It also enables networking of multiple users to perform simultaneously using a remotely controlled instance of Gibber.

Another source that I thought would have spawned other interesting papers is J. Young's J Using the web for live interactive music. The source itself is about a website that allows participants to navigate through a virtual interface and while manipulating elements receive a projection of their actions in the form of sounds in physical space. The live audio performance is streamed out to all participants and anyone could participate and hear the resulting music in real time.

Google scholar only finds two papers that cite this source, neither of which is C. Roberts original paper. I find "Public sound objects: a shared musical space on the web" by A. Barbosa the more interesting (the other also being in Spanish). Public Sound Objects is an on going project that allows users to collaborate as performers online. A public performance space is hosted along with the controls to create the audio. The mixture of all online collaborators is streamed back to the users as well any audience members and contextual data such as sound spatialization and metaphorical visual representations of current participants are added.

Finally, one of the sources of C. Roberts original paper that I found interesting was this one on Firefox and WebAudio. Though Mozilla doesn't support it yet, they are deprecating their Firefox specific Audio Data API and are in the process of adding support for the W3C standard Web Audio.