Projects

We explored interaction models for small displays such as a smart watch. One of the challenges is that your finger covers the display. In one design, we use sensors on the side of the watch to move the input area over to your arm. In another design, we flip up the display, allowing you to look through it or point with it, or have a second surface to touch. Smart Watch Flip Up Display patent Gesture Based Small Device Input patent

The web supports a different model for service integration, apps, and layout management than discrete applications running on the native OS. You should be able to search and run a calculator, not have to go to a store, trust the download, save it, then find it on your phone and run it. With Argos, you rapidly develop applications as web pages using a hardware-enabled model. In addition to the native integration for speed and phone-specific capabilities, Argos also stacks three transparent windows so that you can render HTML on top of native OpenGL, on top of the real world camera view. Our default layout puts a search bar at the top, but you can simply change index.html to make the main page your own.

To use argos, you include argos.css, argos.js into your HTML page to get access to sensors, GPS, voice, etc. The argosEmulator.js allows you to test on the desktop before pushing the files to the phone.

Back in 2011, Yong Zhao developed a very fast way to track a person's face on a tablet. I created several interaction models and applications using that software. One advantage with head tracking is the direct coupling to natural movement (e.g. to explore an object). Another advantage is that, unlike accelerometers, head tracking works in moving vehicles. Some of these apps were shown at the Google I/O Android keynote.

QuickSuggest improves the traditional remote control input (left, right, up, down, select) by dynamically adjusting the onscreen keyboard based on the current context and vocabulary. To reduce the amount of clicking, we place the 4 most likely letters near the current selection point, but we do not disturb the underlying keyboard. This reduces the scan time when the 4 suggested letters do not hold the answer. This interface was demonstrated to be more efficient mathematically as well as through user testing. It was implemented as part of the YouTube TV interface.

We presented a new CAPTCHA which is based on identifying an image's upright orientation. This task requires analysis of the often complex contents of an image, a task which humans usually perform well and machines generally do not. Given a large repository of images, such as those from a web search result, we use a suite of automated orientation detectors to prune those images that can be automatically set upright easily. We then apply a social feedback mechanism to verify that the remaining images have a human-recognizable upright orientation. The main advantages of our CAPTCHA technique over the traditional text recognition techniques are that it is language-independent, does not require text-entry (e.g. for a mobile device), and employs another domain for CAPTCHA generation beyond character obfuscation.

Gulliver was originally designed as a complete travel system, from the planning stages on a desktop, to a notification and note-taking/discovery system in situ on a mobile phone. Tha ability to search, discover, gather and then organize the information fluidly not only allowed the person to explore options rapidly, but also to change them during the trip. Later the trip could be shared, reused, or added as part of a larger recommendation system. Gulliver supported multiple people collaborating to create a trip as well. Gulliver was implemented and launched in partnership with Lonely Planet as part of Google Wave and as an iGoogle gadget.

In 2006 Google designed and implemented a TV Show search system which featured a dual-navigation control combining a powerful search system with an interactive, personalize-able TV listings grid. The dual-control system allowed users to move fluidly between explicit search and general discovery, obtain detailed information without leaving the global context, and construct a personal channel for easy access and recommendations. As a complete system, TV Search integrated with Google's general search page through a mini-listings onebox and also exported information out to Google calendar, Gmail and TiVo. Building this TV Search system as a large-scale product meant addressing many technological, business, and user experience details. This paper describes the design process, user studies, and integration into Google.

Very, very early in the development of Google Glass, I worked with a small team on what the hardware might be, the computer vision/machine learning possibilities, and the user experience. We explored various augmented reality models as well as simple notification icons and an RSVP presentation of text in the small display.

SketchUp was an easy-to-use, 3D modeling company that Google bought back in the day. I worked with a fantastic team in Boulder on how a 3D warehouse of objects might work. We also developed a simple "swivel" interface that allowed people to interactively explore the models. We made the catalog have gamification elements such as achievements to encourage adding better models.

CustomTV integrated the web and television shows on TV hardware, not by creating a web experience, but by creating a lean-back TV experience where you watch channels of information. You could create new channels of content that would automatically fill themselves with interesting content and then play through them. These channels could be search results, recommended topics of video (e.g., news or "how to", even personal content such as vacation pictures and video.

Media2Go explored using bluetooth to send different amounts and types of information from a public display to your phone. In our example, we sent information about a movie. The longer you waited, the more information you received. You could get the movie details, images, a coupon, a small video.

Back then, the idea of a cell phone with a camera was still novel. We added a server, allowing a person to take an unlimited number of pictures and do it with just two clicks. The server would then use time and location information to create animated stories that could be shared with others, printed on postcards, even made into a newspaper. When several people took pictures of a fire in San Jose, we realized we could create an automatic newspaper based on whether a lot of pictures were taken within a radius of distance and time.

The MERBoard was a collaborative display system used by different mission scientist teams during the Mars Explorer Rover Mission (MER). MERBoard involved several distributed, large, touch-enabled, displays with custom software that allowed scientists to gather, organize, share and present mission information in order to determine what the rovers should do next. This was also a great opportunity for me to work with IDEO and learn their approach to projects and design.

The idea behind blueboard was to simplify collaboration in meeting spaces. By walking up to a board, the board sensed your badge and brought up the information you need for that meeting space. The touch screen and digital whiteboard tools made it easy to operate without requiring a keyboard. You could browse the web and drag and drop any information you found into a space, making it easy to gather, organize and share information. If two people were at the board, then it brought up information trhat both of you were working on last. You could have a meeting in California, then continue it in New York, just by meeting in front of a blueboard.

Instead of complicated database models, dspace allows you to programatically create a (tuple) space, put objects in the space, take objects out, have them listen for actions from other objects, and emit actions. We coupled this with physical objects such as bend sensors. So you could put a sensor in the real world, then very quickly write programs to react to sensors and tell emitters what to do.

A digital book made up of its content and the content of all of its references and their references, etc. As you read the book, it recommends and highlights related articles. The system used a ruler at the top so that you could get a visual overview of the entire book, the chapters and the highlighted articles. The ruler allowed you to quickly access different parts of the book. (PARC, circa 2002)

When purchasing stock, how do you trade-off risk and reward? When you are a shop keeper, how much of a shelf should be dedicated to basic staples and how much to extravagant items? By examining a time-window of sales, you can adjust to improve profit, but not suffer wild variations in volitility. We applied the same model to polygons for a graphics object. To improve graphics rendering, we use an importance function as a value of an object and the cost as the time to render it at one level of detail or another.

By representing file structures as cones, you can rotate a cone to see the children and use the space to represent other information. By building a heirarchy, you can then show links that travel outside of the heirarchy. Tipping and slicing the view of the cone in various ways reveals different relationships. ConeGraph Patent

Side Impact was an explicit and implicit search and recommendation tool. As you visited a page, it would recommend related pages. You could also put in a search and get a minimized list of results without leaving the page.

Wrote a 3D platform in C++ that presents a simple, clean 3D API. We used this platform to create several 3D information visualizations (e.g. BookPlex). It stemmed from earlier work on Diver, but was designed for the desktop and for information visualization. We even built fun information "art" like a virtual fountain that sprinkled words, mimicing the real world PARC fountain that presented stock prices as gushing water.

While at SGI I worked on cosmoWorlds, a 3D editing platform for building virtual worlds on the web. It is interesting how much we've moved forward and how much we have not over the years.

DIVER (Distributed VR platform) and Alice were early VR systems we built back in the early 1990s. We used headsets like the occulus, only a lot heavier and we tracked movement with Polhemus magnetic trackers. Alice was a platform for quickly creating VR worlds with more intuitive programming than using quaternions to move around (later becoming an educational system for learning how to program).

Using DIVER, a VR system, we were able to parallel real-world perception studies and even extend the studies beyond what was difficult or impossible to do in the real world. We measured how people estimate the steepness of a hill both in the real world and in a virtual one. In the virtual perception lab we could quickly adjust the slope to different heights. The values matched very well and allowed us to measure very steep, tall hills not found in Virginia.