iCandy: connecting physical objects to digital information with and without barcodes

What does it mean to connect physical objects to digital information? Well, we think it's really cool to point a device (e.g. a cell phone or webcam) at objects (newspapers, documents, signs, posters, etc.) and receive digital information. This type of technology is often referred to as "augmented reality" (AR) and is becoming increasingly possible with today's smart phones (e.g. Mobile Media Search).

Several years back, we started using paper as an interface for viewing and controlling video. Our technology was called "video paper" and it provided a newspaper-like interface for viewing TV shows, movies and recorded meetings. Our goal was to take advantage of the affordances of paper as a visualization, distribution and retrieval medium. The application provided three common items that most everyone knew how to use: paper (with images, text & barcodes on it), a remote control and a TV. Users pointed the remote control (which in this case was a bar code reader) at the parts of the paper they wished to view. After pointing at a section, the media would start playing on a TV set or computer screen. We referred to this style of interface as "point-and-play."

Video paper eventually evolved into the iCandy Project. With iCandy, we initially focused on providing a similar point-and-play interface for two popular forms of media: iTunes music & videos and YouTube content. We called the project "iCandy" (or "eye candy") because we printed pretty pictures on a page or card along with a barcode. The picture, in this case, was typically the album art associated with the iTunes media or the thumbnail image associated with the YouTube video. We believed that this kind of form+function interface both reminded users what the card referred to while at the same time provided access to the digital representation via the barcode. (more information is available here).

iCandy Beta

At the beginning of 2009 we launched the iCandy (beta) Project providing a desktop application for both the Mac and Windows environments. The application has three main features: scanning(recognizing) QR code barcodes, printing(authoring) paper tokens with QR codes, and analytics (collecting information on usage). The application uses a PC web cam to scan QR codes. Once a QR code has been recognized, the information in the code is interpreted and the associated media is launched. For example, most QR codes contain a URL. After recognizing the QR code, iCandy launches the associated application based on the media-type found in the QR code. Therefore, if a QR code contains only a URL, a web page is displayed with the URL found in the QR code. Unlike other QR code applications, iCandy also provides a unique connection to iTunes for controlling media playback. This gets back to the original idea of "point-and-play": Scan an iTunes QR code with iCandy and music or videos will begin playing in iTunes. The video below shows how a 4.5yr old child uses iCandy. She has her own book of music and videos and knows how to point a "webcam-on-a-stand" at the cards to initiate playback. The playback is displayed on the TV screen and heard through the TV speakers. She can use the system even though she doesn't know how to read.

Click to watch video of small child using iCandy

Any URL can be embedded in an iCandy QR code using the card creation feature in the desktop application. For example, a link to your LinkedIn web site, a flickr photo collection, a Slideshare presentation, a band's MySpace page, etc. There are several videos on the here on iCandy web site which explain how to author such cards. 

iCandyMobile for the iPhone

In December '09 we launched the iPhone version of iCandy called "iCandyMobile"  which is available on iTunes App Store for FREE. iCandyMobile provides an easy-to-use tool for scanning any QR code and displaying the result inside the app. iCandyMobile includes a embedded web browser so users can scan and view web pages without leaving the application.

The app also includes social networking tools for sharing information. We integrated both Facebook and Twitter into the application so that people can easily share the links they scan For instance, let's say you're at a new, hip restaurant where a bar code is printed on the menu. After completing your meal you decide to create a Facebook post to share the experience with your friends. You scan the QR code and then submit your post to Facebook and Twitter, all within the iCandy application. We think socially enabled applications like iCandy give users the power to share information with both their personal and professional social networks.

Analytics

We think the iCandy experience is not only useful but fun. Using physical cards to control media playback is something that can be quite rewarding, especially when you share it with friends or colleagues. However, there is more to iCandy than just a pretty face. Behind the scenes we provide tracking and analytics of the cards users author and scan.

We think analytics is a key ingredient for any application and so we provide this feature as part of the iCandy package. Here's how it works. Each time a card is created using the iCandy desktop software, the unique iCandy ID associated with the author is embedded in the barcode and a message is sent to our server telling us that this author just created a new code. When that code is scanned, by anyone, the embedded URL in the barcode first points to our server which provides a redirect to the actual target URL. Hence, we are "notified" when any iCandy-generated barcode is scanned. This helps us understand how our software is being used.

iCandy analytics also support our user community: users can track their analytics using a personal Dashboard on the iCandy website. Here is an example of how this works. Imagine a scenario where a small garage band wants to promote their work. They create business cards with a QR code pointing to a YouTube video or pointing to their music on iTunes. They can create posters to be placed around town, promoting an upcoming gig. Then, the band can track which QR codes are scanned most often and where (as in "where in the world") using the iCandy analytics. From a developer's standpoint, we also track how the application is being used. For instance, we can track everything from how many times the "Facebook post" feature was used to how often users print their iCandy cards to flickr instead of to an actual printer. We instrumented the desktop and mobile applications to help us understand which features are used most often so that we can improve the applications we build in the future.

iCandy in the workplace

iCandy is not just for iTunes and YouTube or even URL's on the web. For the last 10 months we have worked with a small civil engineering firm in Phoenix, AZ. This paper-centric office needed a way to connect the paper they use on a daily basis (think 24"x36" sheets of paper) to digital information on their servers. They consistently required a method for connecting from the paper sheets back to the digital media, e.g. the original PDF and all documents related to the project.

The civil engineering firm created three new layouts for their specific needs: a sticky label that could be applied to the target document, 4"x6" cards with one document thumbnail on them and 8.5" x 11" sheets with up to six document thumbnails and barcodes on them describing a particular project. As this firm continues to use iCandy, they have provided us with many valuable insights into how iCandy-like systems can improve the ability to access digital information in similar, paper-centric offices.

Visual search

There is a new kind of search that you will be hearing more and more about in the coming months: visual search. Visual search links physical objects with digital information without using barcodes. Instead, it uses the natural features of an object to create a unique "fingerprint."

Google recently announced their own visual search application called Goggles (http://www.google.com/mobile/goggles). With this application you can point at an object, like the Golden Gate bridge, with your cell phone and view supplemental information about that object. Ricoh has also been working in this area for years and we have our own visual search technology. This technology has been added to the next version of iCandyMobile for the iPhone. Our visual search technology connects documents containing text and pictures to digital information. For instance, users can point their camera phone at any area on a page and take a picture.

Our technology converts that picture into a link and retrieves the data it points to. We call this capability "clickable paper" since it makes paper documents as interactive as web pages.

Click to watch video of iCandy Visual Search

There are two parts to this exciting new technology: client applications running on mobile devices or desktops and the recognition technology, currently running on a server. As part of our effort to explore this area, we will soon be providing an API for developers to build their own visual search applications using our server technology. We look forward to working with our corporate partners to create some interesting new visual search applications.

Summary

As mobile devices and the networks that support such devices become more powerful and sophisticated, the opportunities to leverage technology like iCandy also become more appealing. Whether such applications include barcodes or "no-codes" as with visual search, bridging the gap from the physical to the digital world enables many new forms of connectivity, thereby potentially increasing productivity and (maybe more importantly) satisfaction.