# Ubiquitous Interaction Devices

*2003-01-01 — project*


I investigated how to interface the new emerging class of smartphone devices
(circa 2002) with concepts from ubiquitous computing such as location-aware
interfaces or context-aware computing. I discovered the surprisingly positive
benefits of piggybacking on simple communications medium such as audible sound
and visual tags. Our implementations of some of these ended up with new audio
ringtone and visual smart tags that worked on the widely deployed mobile phones
of the era.

In 2003, the mobile phone market had grown tremendously and given the average
consumer access to cheap, small, low-powered and constantly networked devices
that they could reliably carry around. Similarly, laptop computers and PDAs
became a common accessory for businesses to equip their employees with when on
the move.  The research question then, was how to effectively interface them
with existing digital infrastructure and realise some of the concepts of
ubiquitous computing such as location-aware interfaces or context-aware
computing.


Ubiquitous Interaction Devices (see [original webpages](https://www.cl.cam.ac.uk/research/srg/netos/projects/archive/uid/))
was a project I started with [Dave Scott](https://dave.recoil.org) and [Richard Sharp](mailto:richard.sharp@gmail.com) in 2003 to work on this at
the Cambridge Computer Laboratory and Intel Research Cambridge (who had just
set up a "lablet" within our building and was a great source of free coffee).

## Audio Networking

The project kicked off when we randomly experimented with our fancy Nokia smartphones
and discovered that they didn't have anti-aliasing filters on the microphones.
This allowed us to record and decode ultrasound between the phones. The
2003 [Context-Aware Computing with Sound](https://anil.recoil.org/papers/audio-networking) in Ubicomp describes some of the applications it
allowed.

In a nutshell, audio networking used ubiquitously available sound hardware
(i.e, speakers, soundcards and microphones) for low-bandwidth wireless networking.
It has a number of characteristics which differentiate it from existing wireless technologies:

- fine-grained control over the range of transmission by adjusting the volume.
- walls of buildings are typically designed to attenuate sound waves, making it easy to contain transmission to a single room (unlike, for example, Wifi or Bluetooth).
- existing devices can play or record audio to be networked easily (e.g. voice recorders).

The *Audionotes* video below demonstrates an Audio Pick and Drop Interface: an Internet URL is recorded into a PDA via a standard computer speaker. The URL is later retrieved from the PDA by playing it into the computer, and also printed by carrying the PDA to a print server and playing back the recorded identifier.
A great advantage of this is that any cheap voice recorder is capable of carrying the audio information (e.g. mobile phone, PDA, dictaphone, etc).

One of the more interesting discoveries during this research is that most computer soundcards do not have good anti-aliasing filters fitted, resulting in them being able to send and receive inaudible sound at upto 24 Khz (assuming a 48 KHz soundcard).
We used this to create a set of inaudible location beacons that would allow laptop computers to simply listen using their microphones and discover their current location without any advanced equipment being required! The *location* video below demonstrates this.

<div class="video-center"><iframe title="" width="100%" height="315px" src="https://crank.recoil.org/videos/embed/f75b6a17-1dbb-450c-8938-8b5bcbd9896a" frameborder="0" allowfullscreen sandbox="allow-same-origin allow-scripts allow-popups allow-forms"></iframe></div>

<div class="video-center"><iframe title="" width="100%" height="315px" src="https://crank.recoil.org/videos/embed/cda6e8e1-6f64-4f6f-aab0-8dc615836a51" frameborder="0" allowfullscreen sandbox="allow-same-origin allow-scripts allow-popups allow-forms"></iframe></div>

We then devised a scheme of encoding data into mobile phone ringtones while still making them sound pleasant to the ear. This allows us to use the phone network for the video example above: an SMS 'capability' is transmitted to a phone, which can be played back to gain access to a building.
Since the data is just a melody, this allows for uses such as parents purchasing cinema tickets online for their children (who dont own a credit card), yet still allowing the children to gain access to the cinema by playing the ringtone capability back (via their mobile phones, common among children).

We observed that audio networking allows URLs to be easily sent over a telephone conversation, and retrieved over a normal high bandwidth connection by the recipient.
This gets over the real awkwardness of "locating" someone on the Internet, which is rife with restrictive firewalls and Network Address Translation which is destroying end-to-end connectivity. In contrast, it is trivial to locate someone on the phone network given their telephone number, but harder to perform high bandwidth data transfers (which the Internet excels at).  There's more detail on these usecases in [Using smart phones to access site-specific services](https://anil.recoil.org/papers/2005-ieee-smartphones).

<div class="video-center"><iframe title="" width="100%" height="315px" src="https://crank.recoil.org/videos/embed/c7da2f1a-1ae5-4c54-aaf7-4b6fed6c2773" frameborder="0" allowfullscreen sandbox="allow-same-origin allow-scripts allow-popups allow-forms"></iframe></div>

<div class="video-center"><iframe title="" width="100%" height="315px" src="https://crank.recoil.org/videos/embed/f75b6a17-1dbb-450c-8938-8b5bcbd9896a" frameborder="0" allowfullscreen sandbox="allow-same-origin allow-scripts allow-popups allow-forms"></iframe></div>

## SpotCode Interfaces

Once we'd had such success with audible networking, a natural next step was to use the new fancy cameras present on smartphones. [Eben Upton](mailto:eben@phlegethon.org) joined our project and knocked up a real-time circular barcode detector for Symbian operating system phones in short order that we demonstrated in [Using Camera-Phones to Enhance Human-Computer Interaction](https://anil.recoil.org/papers/2004-ubicomp-camera).

The phone app we built could detect our circular barcodes in realtime, unlike the ungainly "click and wait" experiences of the time.  Since the phone computes the relative rotation angle of detected tags in real-time, an intuitive "volume control" was possible by simple locking onto a tag and physically rotating the phone. The videos demonstrates a volume control in the "SpotCode Jukebox", and how further interfaces could be "thrown" to the phone for detailed interactions.

<div class="video-center"><iframe title="" width="100%" height="315px" src="https://crank.recoil.org/videos/embed/6b90fd6b-b88a-4bab-8943-371776dcda64" frameborder="0" allowfullscreen sandbox="allow-same-origin allow-scripts allow-popups allow-forms"></iframe></div>

<div class="video-center"><iframe title="" width="100%" height="315px" src="https://crank.recoil.org/videos/embed/67fb611f-42d0-43b2-9961-93b2b7425750" frameborder="0" allowfullscreen sandbox="allow-same-origin allow-scripts allow-popups allow-forms"></iframe></div>

We then built some more elaborate public applications (such as a travel booking shop) in [Using camera-phones to interact with context-aware mobile services](https://anil.recoil.org/papers/2004-spotcodes) and [Using smart phones to access site-specific services](https://anil.recoil.org/papers/2005-ieee-smartphones)).  When [Eleanor Toye Scott](https://www.cst.cam.ac.uk/people/eft20) joined the project, we subsequently conducted structured user studies to see how effective the tags are in [Interacting with mobile services: an evaluation of camera-phones and visual tags](https://anil.recoil.org/papers/2006-puc-tags). As a side-note, the whole zooming interface was written using OCaml and OpenGL.
We spun out a startup company called High Energy Magic Ltd., and got into the [New York Times](https://www.nytimes.com/2004/10/07/technology/circuits/connecting-paper-and-online-worlds-by-cellphone-camera.html) and [Wired](https://www.wired.com/2004/06/from-the-prawn-of-time/) (alongside a decaying prawn sandwich).

<div class="video-center"><iframe title="" width="100%" height="315px" src="https://crank.recoil.org/videos/embed/f2144cf5-16ee-4373-8a81-e5b8b2263f6b" frameborder="0" allowfullscreen sandbox="allow-same-origin allow-scripts allow-popups allow-forms"></iframe></div>

<div class="video-center"><iframe title="" width="100%" height="315px" src="https://crank.recoil.org/videos/embed/fa57ea22-6158-4143-b533-24bd439e0717" frameborder="0" allowfullscreen sandbox="allow-same-origin allow-scripts allow-popups allow-forms"></iframe></div>

It also became obvious that the technology was also really robust, since it worked fine on printed (and crumpled) pieces of paper, making it ideal for public signage.  We used this to experiment with more robust device discovery in [Using visual tags to bypass Bluetooth device discovery](https://anil.recoil.org/papers/2005-mc2r-visualtags). This subsequently lead us to show how smartphones could be trusted side-channels in [Enhancing web browsing security on public terminals using mobile composition](https://anil.recoil.org/papers/2008-mobisys-splittrust), an idea that is now (as of 2020) becoming realised with trusted computing chips in modern smartphones.

<div class="video-center"><iframe title="" width="100%" height="315px" src="https://crank.recoil.org/videos/embed/9ef069e9-0aee-4c51-87c3-e09625711f99" frameborder="0" allowfullscreen sandbox="allow-same-origin allow-scripts allow-popups allow-forms"></iframe></div>

<div class="video-center"><iframe title="" width="100%" height="315px" src="https://crank.recoil.org/videos/embed/cc75883b-379e-467b-81ca-736a13671f8e" frameborder="0" allowfullscreen sandbox="allow-same-origin allow-scripts allow-popups allow-forms"></iframe></div>

## Towards smarter environments

At the Computer Laboratory, we also happen to have one of the world's best indoor location systems (the [Active Bat](https://en.wikipedia.org/wiki/Active_Bat)), which we inherited from AT\&T Research when that shut down in Cambridge.  [Ripduman Sohan](mailto:ripduman.sohan@gmail.com), [Alastair Tse](https://liquidx.net) and [Kieran Mansley](mailto:kieran@recoil.org) joined forces with us to investigate how commodity hardware could interface with this smart location system to make really futuristic buildings possible.

A few really fun projects resulted from this:
- We used the BAT system to help with Bluetooth-based indoor location inference, in [A Study of Bluetooth Propagation Using Accurate Indoor Location Mapping](https://anil.recoil.org/papers/2005-ubicomp-bluetooth).
- Interfaced the audio networking with the AT\&T "broadband phones" in [The Broadband Phone Network: Experiences with Context-Aware Telephony](https://anil.recoil.org/papers/2005-bbphone).
- We constructed a real-time capture the flag game in the Computer Lab with augmented reality "windows" into the game in [Feedback, latency, accuracy: exploring tradeoffs in location-aware gaming](https://anil.recoil.org/papers/netgames04-ctf).
- I worked with my Recoil-in-chief [Nick Ludlam](https://nick.recoil.org) on digital TV interfaces in [Ubiquitious Computing needs to catch up with Ubiquitous Media](https://anil.recoil.org/papers/2005-ubiapp-ubimedia).

In around 2005, we sold High Energy Magic Ltd. to a [Dutch entrepreneur](https://www.youtube.com/watch?v=sN01wkRzsfk) so that [Richard Sharp](mailto:richard.sharp@gmail.com), [Dave Scott](https://dave.recoil.org) and I could join the [Xen Hypervisor](https://anil.recoil.org/projects/xen) startup company.   However, the ubiquitous computing ideals that drove much of our work continue to persist, and in 2018 I started thinking about this again as part of my [Interspatial OS](https://anil.recoil.org/projects/osmose) project.  The idea of building truely ubiquitous environments (without smartphones) is resurrected again there, and you can start reading about it in [An architecture for interspatial communication](https://anil.recoil.org/papers/2018-hotpost-osmose).
Period: 2003–2008

## Related

- [At the AI Impact Summit in Delhi: people, planet, progress](https://anil.recoil.org/notes/india-ai-summit) (note, 2026-02-21)
- [About](https://anil.recoil.org/notes/index) (note, 2025-02-15)
- [throw](https://anil.recoil.org/videos/67fb611f-42d0-43b2-9961-93b2b7425750) (video, 2021-08-12)
- [spinner](https://anil.recoil.org/videos/6b90fd6b-b88a-4bab-8943-371776dcda64) (video, 2021-08-12)
- [passive-spiderman](https://anil.recoil.org/videos/9ef069e9-0aee-4c51-87c3-e09625711f99) (video, 2021-08-12)
- [authentication](https://anil.recoil.org/videos/c7da2f1a-1ae5-4c54-aaf7-4b6fed6c2773) (video, 2021-08-12)
- [passive-crumpled-throw](https://anil.recoil.org/videos/cc75883b-379e-467b-81ca-736a13671f8e) (video, 2021-08-12)
- [location](https://anil.recoil.org/videos/cda6e8e1-6f64-4f6f-aab0-8dc615836a51) (video, 2021-08-12)
- [audionotes](https://anil.recoil.org/videos/e6138b22-483c-4158-8393-4e60c39ab217) (video, 2021-08-12)
- [wmap-plasma](https://anil.recoil.org/videos/f2144cf5-16ee-4373-8a81-e5b8b2263f6b) (video, 2021-08-12)
- [telephone](https://anil.recoil.org/videos/f75b6a17-1dbb-450c-8938-8b5bcbd9896a) (video, 2021-08-12)
- [wmap-projector](https://anil.recoil.org/videos/fa57ea22-6158-4143-b533-24bd439e0717) (video, 2021-08-12)
- [An architecture for interspatial communication](https://anil.recoil.org/papers/2018-hotpost-osmose) (paper, 2018-04-01)
- [Interspatial OS](https://anil.recoil.org/projects/osmose) (project, 2018-01-01)
- [Enhancing web browsing security on public terminals using mobile composition](https://anil.recoil.org/papers/2008-mobisys-splittrust) (paper, 2008-06-01)
- [Interacting with mobile services: an evaluation of camera-phones and visual tags](https://anil.recoil.org/papers/2006-puc-tags) (paper, 2007-02-01)
- [A Study of Bluetooth Propagation Using Accurate Indoor Location Mapping](https://anil.recoil.org/papers/2005-ubicomp-bluetooth) (paper, 2005-09-01)
- [Audio networking: the forgotten wireless technology](https://anil.recoil.org/papers/2005-ieee-audio) (paper, 2005-07-01)
- [Ubiquitious Computing needs to catch up with Ubiquitous Media](https://anil.recoil.org/papers/2005-ubiapp-ubimedia) (paper, 2005-07-01)
- [The Broadband Phone Network: Experiences with Context-Aware Telephony](https://anil.recoil.org/papers/2005-bbphone) (paper, 2005-01-01)
- [Using smart phones to access site-specific services](https://anil.recoil.org/papers/2005-ieee-smartphones) (paper, 2005-01-01)
- [Using visual tags to bypass Bluetooth device discovery](https://anil.recoil.org/papers/2005-mc2r-visualtags) (paper, 2005-01-01)
- [Using visual tags to bypass Bluetooth device discovery](https://anil.recoil.org/notes/2005-mc2r-visualtags-1) (note, 2005-01-01)
- [Using camera-phones to interact with context-aware mobile services](https://anil.recoil.org/papers/2004-spotcodes) (paper, 2004-12-01)
- [Connecting Paper and Online Worlds by Cellphone](https://anil.recoil.org/notes/spotcodes-nytimes) (note, 2004-10-07)
- [Using Camera-Phones to Enhance Human-Computer Interaction](https://anil.recoil.org/papers/2004-ubicomp-camera) (paper, 2004-09-01)
- [MIT Technology review covers SpotCodes](https://anil.recoil.org/notes/mit-spotcodes) (note, 2004-09-01)
- [Feedback, latency, accuracy: exploring tradeoffs in location-aware gaming](https://anil.recoil.org/papers/netgames04-ctf) (paper, 2004-08-01)
- [Cellphone as a virtual mouse/keyboard](https://anil.recoil.org/notes/hem) (note, 2004-05-27)
- [Context-Aware Computing with Sound](https://anil.recoil.org/papers/audio-networking) (paper, 2003-10-01)
- [Context-Aware Computing with Sound](https://anil.recoil.org/notes/audio-networking-1) (note, 2003-10-01)
- [Xen Hypervisor](https://anil.recoil.org/projects/xen) (project, 2002-01-01)

---
Canonical: https://anil.recoil.org/projects/ubiqinteraction
Type: project
Tags: ubicomp, systems, hci, audio