Mobile Media Search: Has Media Search Finally Found its Perfect Platform?

No denying that mobile phones are on their way to becoming the most popular computing platforms and information portals. The computation and memory capabilities, battery life, and connection speeds of mobile phones are increasing at a rapid pace. Nevertheless mobile devices still have some limitations that are here to stay for the foreseeable future, such as small display size and limited peripherals for text input. These limitations make searching and browsing information on the mobile phones inconvenient and difficult.

Even though content-based media (Visual and Audio) search have been active research areas for the last 20+ years, these technologies have not been widely adopted outside of a few niche applications. I strongly believe this is bound to change because of mobile phones. It is possible that the limitations of mobile phones (difficult text entry and browsing) combined with the following advantages makes mobile phones a perfect platform for media search applications:

Integrated sensors: Mobile phones have sensors such as GPS, digital compass, and accelerators that give applications/services more metadata to make intelligent decisions on user’s intentions with search. For example, there are many mobile applications that rank search results based on location information.

Always connected: The main functionality of mobile phones is to provide connectivity. As a result, a mobile phone typically has connection to the Internet more often than other devices, such as laptops, providing the users ubiquitous access to information.

Instant on: When a user needs instant information, such as looking up a restaurant phone number or a location, it would take a user only a few seconds to turn on a mobile phone, where a computer would take several minutes to be turned on.

Personal: Mobile phone is personal. Therefore it is likely that mobile phones are customized with user’s preferences, information about user’s contacts, and preferred social networks that include links to friends. This information can be utilized in mobile search for better and more targeted results.

Always accessible: Most people carry their mobile phone with them all the time. When a user needs information, particularly outside of their home and work, it is much likely that they have access to a mobile phone than they have access to a computer.

Integrated audiovisual capture: Most smartphones have audio and visual capture capability, whereas submitting an image or an audio query to a personal computer requires many steps. Mobile phones make it simpler to submit audiovisual queries to content based retrieval systems.

Mobile Media Search Apps

Recently some great audio and visual media search applications were introduced on smartphones. One of my favorite apps is the Snaptell iPhone application that allows taking a picture of a book or CD cover and retrieve pricing information. Shazam application can find any song information by recording just few seconds of a music clip from an external audio source, such as radio. My unscientific personal experiments show that the accuracy of Snaptell and Shazam apps is very high. Google voice search application allows users to enter search keywords via voice entry, making it very convenient for on the go restaurant searches (though it still needs some work to improve accuracy for non-English speakers).

Shazam Music Search: http://www.shazam.com/
Google Mobile App with Voice Search: http://www.google.com/mobile/apple/app.html
Yahoo OneSearch with Voice: http://mobile.yahoo.com/onesearch/voice
DooG: http://www.doog.mobi/

Visual and Audio Search Research

At Ricoh Innovations, we have been researching mobile Visual Search algorithms and applications since 2004. We have a brief description of our research, some video clips and links to our published papers here. Below is a video clip that shows information retrieval using our Visual Search technology.

We also organized a couple of panel discussions (ICASSP 2009 and ACM Multimedia 2009) on mobile media search in 2009. A panel opinion paper, where Ricoh, NTT Docomo, Kodak, Google, Microsoft, and SRI researchers discuss the open questions and killer apps of Mobile Video Search can be found here. If you'd like to be notified of our Visual Search apps and APIs in the future, please sign up to our website.

Google, Yahoo, Microsoft, NTT Docomo and Nokia are among many research labs who have been developing technologies for mobile Visual and Audio search.

Future Has Arrived

Mobile phones are becoming a second set of eyes, ears, sensing even before we do with accuracy better than we have. They are aware of our environment (in both the physical and virtual worlds) and provide exact information to us exactly when we need it. One cannot help but think: Future has arrived and our relationship with information and our environment will be forever changed.