These are some of the public projects I've worked on over the years.
watchworks: exploring interaction models for the watch
We explored interaction models for small displays such as a smart
watch. One of the challenges is that your finger covers the display. In
one design, we use sensors on the side of the watch to move the input
area over to your arm. In another design, we flip up the display,
allowing you to look through it or point with it, or have a second
surface to touch.
Smart Watch Flip Up Display patent
Gesture Based Small Device Input patent
The web supports a different model for service integration, apps, and
layout management than discrete applications running on the native OS. You
should be able to search and run a calculator, not have to go to a store,
trust the download, save it, then find it on your phone and run it. With
Argos, you rapidly develop applications as web pages using a
hardware-enabled model. In addition to the native integration for
speed and phone-specific capabilities, Argos also stacks three transparent
windows so that you can render HTML on top of native OpenGL, on top of the
real world camera view. Our default layout puts a search bar at the top, but
you can simply change index.html to make the main page your own.
To use argos, you include argos.css, argos.js into
your HTML page to get access to sensors, GPS, voice, etc. The argosEmulator.js
allows you to test on the desktop before pushing the files to the phone.
Back in 2011, Yong Zhao developed a very fast way to track a person's face
on a tablet. I created several interaction models and applications using
that software. One advantage with head tracking is the direct coupling to
natural movement (e.g. to explore an object). Another advantage is that,
unlike accelerometers, head tracking works in moving vehicles. Some of
these apps were shown at the Google I/O Android keynote.
QuickSuggest improves the traditional remote control input (left,
right, up, down, select) by dynamically adjusting the onscreen keyboard
based on the current context and vocabulary. To reduce the amount of
clicking, we place the 4 most likely letters near the current selection
point, but we do not disturb the underlying keyboard. This reduces the
scan time when the 4 suggested letters do not hold the answer. This
interface was demonstrated to be more efficient mathematically as well
as through user testing. It was implemented as part of the YouTube TV
We presented a new CAPTCHA which is based on identifying an image's upright
orientation. This task requires analysis of the often complex contents of
an image, a task which humans usually perform well and machines generally
do not. Given a large repository of images, such as those from a web search
result, we use a suite of automated orientation detectors to prune those
images that can be automatically set upright easily. We then apply a social
feedback mechanism to verify that the remaining images have a
human-recognizable upright orientation. The main advantages of our CAPTCHA
technique over the traditional text recognition techniques are that it is
language-independent, does not require text-entry (e.g. for a mobile
device), and employs another domain for CAPTCHA generation beyond character
Gulliver was originally designed as a complete travel system, from the
planning stages on a desktop, to a notification and note-taking/discovery
system in situ on a mobile phone.
Tha ability to search, discover, gather and then organize the information
fluidly not only allowed the person to explore options rapidly, but also to
change them during the trip. Later the trip could be shared, reused, or
added as part of a larger recommendation system. Gulliver supported multiple
people collaborating to create a trip as well.
Gulliver was implemented and launched in partnership with Lonely Planet as
part of Google Wave and as an iGoogle gadget.
In 2006 Google designed and implemented a TV Show search system which
featured a dual-navigation control combining a powerful search system with
an interactive, personalize-able TV listings grid. The dual-control system
allowed users to move fluidly between explicit search and general
discovery, obtain detailed information without leaving the global context,
and construct a personal channel for easy access and recommendations. As a
complete system, TV Search integrated with Google's general search page
through a mini-listings onebox and also exported information out to Google
calendar, Gmail and TiVo.
Building this TV Search system as a large-scale
product meant addressing many technological, business, and user
experience details. This paper describes the design process,
user studies, and integration into Google.
Very, very early in the development of Google Glass, I worked with a
small team on what the hardware might be, the computer vision/machine
learning possibilities, and the user experience. We explored various
augmented reality models as well as simple notification icons and an
RSVP presentation of text in the small display.
SketchUp was an easy-to-use, 3D modeling company that Google bought back in
the day. I worked with a fantastic team in Boulder on how a 3D warehouse of
objects might work. We also developed a simple "swivel" interface that
allowed people to interactively explore the models. We made the catalog
have gamification elements such as achievements to encourage adding better
CustomTV integrated the web and television shows on TV hardware, not by
creating a web experience, but by creating a lean-back TV experience where
you watch channels of information. You could create new channels of content
that would automatically fill themselves with interesting content and then
play through them. These channels could be search results, recommended
topics of video (e.g., news or "how to", even personal content such as
vacation pictures and video.
Media2Go explored using bluetooth to send different amounts and types of
information from a public display to your phone. In our example, we sent
information about a movie. The longer you waited, the more information you
received. You could get the movie details, images, a coupon, a small video.
Back then, the idea of a cell phone with a camera was still novel. We added
a server, allowing a person to take an unlimited number of pictures and do
it with just two clicks. The server would then use time and location
information to create animated stories that could be shared with others,
printed on postcards, even made into a newspaper.
When several people took pictures of a fire in San Jose, we realized we
could create an automatic newspaper based on whether a lot of pictures were
taken within a radius of distance and time.
The MERBoard was a collaborative display system used by different mission
scientist teams during the Mars Explorer Rover Mission (MER). MERBoard
involved several distributed, large, touch-enabled, displays with custom
software that allowed scientists to gather, organize, share and present
mission information in order to determine what the rovers should do next.
This was also a great opportunity for me to work with IDEO and learn their
approach to projects and design.
Almaden Research Labs
The idea behind blueboard was to simplify collaboration in meeting
spaces. By walking up to a board, the board sensed your badge and brought
up the information you need for that meeting space. The touch screen and
digital whiteboard tools made it easy to operate without requiring a
keyboard. You could browse the web and drag and drop any information you
found into a space, making it easy to gather, organize and share
information. If two people were at the board, then it brought up
information trhat both of you were working on last. You could have a meeting
in California, then continue it in New York, just by meeting in front of a
Instead of complicated database models, dspace allows you to
programatically create a (tuple) space, put objects in the space, take
objects out, have them listen for actions from other objects, and emit
actions. We coupled this with physical objects such as bend sensors. So you
could put a sensor in the real world, then very quickly write programs to
react to sensors and tell emitters what to do.
A digital book made up of its content and the content of all of its
references and their references, etc. As you read the book, it recommends
and highlights related articles. The system used a ruler at the top so that
you could get a visual overview of the entire book, the chapters and the
highlighted articles. The ruler allowed you to quickly access different
parts of the book.
(PARC, circa 2002)
When purchasing stock, how do you trade-off risk and reward? When you are a
shop keeper, how much of a shelf should be dedicated to basic staples and
how much to extravagant items? By examining a time-window of sales, you can
adjust to improve profit, but not suffer wild variations in volitility. We
applied the same model to polygons for a graphics object. To improve
graphics rendering, we use an importance function as a value of an object
and the cost as the time to render it at one level of detail or another.
By representing file structures as cones, you can rotate a cone to see the
children and use the space to represent other information. By building a
heirarchy, you can then show links that travel outside of the
heirarchy. Tipping and slicing the view of the cone in various ways reveals
Side Impact was an explicit and implicit search and recommendation tool. As
you visited a page, it would recommend related pages. You could also put in
a search and get a minimized list of results without leaving the page.
Wrote a 3D platform in C++ that presents a simple, clean 3D API. We used
this platform to create several 3D information visualizations
(e.g. BookPlex). It stemmed from earlier work on Diver, but was designed
for the desktop and for information visualization. We even built fun
information "art" like a virtual fountain that sprinkled words, mimicing
the real world PARC fountain that presented stock prices as gushing water.
While at SGI I worked on cosmoWorlds, a 3D editing platform for building
virtual worlds on the web. It is interesting how much we've moved forward
and how much we have not over the years.
University of Virginia
DIVER (Distributed VR platform) and Alice were early VR systems we built
back in the early 1990s. We used headsets like the occulus, only a lot
heavier and we tracked movement with Polhemus magnetic trackers. Alice was
a platform for quickly creating VR worlds with more intuitive programming
than using quaternions to move around (later becoming an educational system
for learning how to program).
Using DIVER, a VR system, we were able to parallel real-world perception
studies and even extend the studies beyond what was difficult or impossible
to do in the real world. We measured how people estimate the steepness of a
hill both in the real world and in a virtual one. In the virtual perception
lab we could quickly adjust the slope to different heights. The values
matched very well and allowed us to measure very steep, tall hills not
found in Virginia.