Signal rich media white paper
- 1. Mobile Computing & Signal-Rich Media Phil Leigh Senior Analyst Inside Digital Media, Inc. January 21, 2011 813.837.3631 phil@insidedigitalmedia.com
- 2. Table of Contents Executive Summary Mobile Computing Paradigm Ubiquity Intuitive & Integrated Touch Screen Silicon Eye Silicon Ear Location Aware Personalized Media Transmutation Terrestrial-to-Virtual Integration Barcodes Digital Fingerprints Digital Watermarks Unifying Platform Current Need Digimarc Discover Conclusion
- 3. Executive Summary Historically computers evolved from rarely seen mainframes, to personal models on every desktop, to laptops convenient for offices, students, business trips, and homes. But most recently smartphones and tablet models are pushing the industry into a new mobile computing paradigm signified by (1) ubiquity, (2) intuitive and integrated operation, and (3) personalization. Mobile computing devices are becoming cognitive prosthetics. Much as experienced amputees routinely use artificial members as mechanical limb extensions, habitual smartphone and tablet owners are starting to use the units as handy intelligent aids. For example smartphones can obtain price comparisons merely by scanning bar codes and other implanted signals of shelf merchandise. Gartner predicts smartphones will outnumber personal computers in two years. Forrester forecasts 82 million Americans will use tablet computers by 2015. Pew Research Center estimates 60% of Americans accessed the Net on their phones last year as compared to only 25% in 2009. Unlike tools such as calculators that are only used when needed, cognitive prosthetics are ever-present. They’re also integrated and intuitive owing to (1) touch screens, (2) cameras, (3) microphones, and (4) location-awareness, among other features. It’s been twenty five years since the icon-based interface rendered command-line prompts obsolete. Touch screens are a similar advance. On-board cameras function as intelligent silicon eyes and integrated microphones are metaphorical silicon ears. Location-awareness adds geographic context to web interactions. Ubiquity computing implies content can no longer be isolated to old platforms such a magazines, newspapers, radio, or TV. There’ll be an increasing transmutation of media-- especially from physical-to-digital. Digital-to-physical media transformation has been routine since laser printers gave us image-correct paper printouts. But owing to integrated sensors, such as cameras and microphones, mobile computing will prompt media transformation in the opposite -- physical-to-digital -- direction. For example, instead of printing digital files onto paper, cognitive prosthetics will more often scan printed media into the digital domain. Terrestrial-to-virtual transmutation is enhanced by digital signaling. Among the most popular methods are (1) bar codes, (2) digital fingerprints, and (3) digital watermarks. 2
- 4. While bar codes are widely used and reliably machine-readable, they’re incompatible with audio. Similarly, digital fingerprints are effective identification tools for unmarked content but they carry no intrinsic data. Digital watermarks impart unique information and survive media transformation from analog-to- digital, or the reverse. However, they must be intentionally embedded in applicable media. Much as bar codes transformed retailing with machine-readable packaging, mobile computing and embedded-signal content will transform media. For example, signals within media can trigger cognitive prosthetics to present consumers special offers in the context of their location, preferences, and prior behavior. However, widespread bar code use took many years, requiring discrete catalyzing events. Popular adoption of embedded-signal media awaits similar catalysts, such as unifying platforms and standards. Digimarc’s Discover platform contains such a vision. Demonstration versions of Digimarc Discover combine watermarking, music recognition from Gracenote, and bar code reading for price comparison. The platform enables developers to create their own implanted-signal applications. Ultimately, however, device-makers from tablet computers to smartphones might consider the merits of integrating the platform into their operating systems in order facilitate signal-rich media transmutation. Mobile Computing Paradigm Computing has evolved through four stages. First, computers were isolated in air-conditioned rooms with access limited to an elite priesthood of operators. In the second stage office workers typically used desktop units, generally for document creation. Third, computers became conveniently portable as laptops to accompany us on business trips. Document creation continued to be popular but laptop units also were used for presentations and media consumption. In the current era smartphones and tablet computers have become nearly ubiquitous. Ubiquity In the emerging era of ubiquity-computing devices are becoming cognitive prosthetics. Much like amputees eventually regard artificial limbs as mechanical extensions of their own truncated limbs, devoted smartphone and tablet users employ the devices as ever-present intelligent aids. They use them as communications and information gathering prosthetics as naturally as hockey players use sticks as extended limbs of their bodies. Furthermore, steadily advancing technology implies the trend will become even more obvious. By 2013 3
- 5. it is estimated that mobile phones will overtake PCs as the primary Internet access device. Intuitive & Integrated Unlike tools which are only used on applicable occasions, cognitive prosthetics are ever-present. We wear them like garments and rely upon them as though they are an integral part of us. Thus, a calculator is a tool whereas a wristwatch is a familiar, but aging, cognitive prosthetic. Just as we live in a familiar terrestrial world, we are also increasingly living in a cyberworld created by our computers and the Internet. Some fear the change will lead us to into less personal contact. Instead, it is more likely to merely change how we interact. For example, the telephone didn’t prevent us from talking to one another;; it just altered how we did so by enabling us converse remotely. The computer industry is evolving into an era of intuitive and integrated computing. In the present proto-era four device characteristics drive the evolutionary path: (1) touch screens, (2) cameras, (3) microphones, and (4) location-awareness. When the concept is entirely developed sensors will automatically integrate with technologies to immediately identify media and objects within the user’s surrounding. At its full-grown stage the primary enabler of intuitive computing is accurate content identification. Touch Screens. About 25 years ago the mouse and graphical user interface (GUI) forever changed computing. Until then commands were an arcane string of alphanumeric characters. The GUI made computing more intuitive. If we wanted to discard a document, we dragged it to a trash-can icon. The touch screen is an equally important interface paradigm popularized by the iPhone three years ago. Mobile computing could not reach its full potential without eliminating the mouse, which the touch screen does by substituting our fingers. The significance of the new interface is underscored by the fact that it has been extended to the tablet form-factor as well. Silicon Eye. David Pogue is a popular technology columnist at The New York Times who dwells in the realm of inventions. But even he was surprised by a development that underscores just how quickly smartphones are becoming intelligent aids. Initially he thought this video of an intelligent iPhone app was fake. However, eventually he learned it genuinely enables users to point the iPhone camera at Spanish text – like a sign or menu – and see the English translation on their screen. Users not only see a translation, they also see it superimposed on the original sign or menu background. 4
- 6. Another popular example is an app category enabling consumers to price-shop merchandise merely by scanning the ticketed bar code. In point of fact, as this video report documents, even some of the biggest terrestrial retailers were affected by such price-shopping during this past Christmas season. Digital Watermark Demo Finally, a digital watermark app enables publishers to transform the printed page into a web browser. The pole vaulting picture at the left is embedded with such a watermark. While the mark is imperceptible to the human eye, an app-equipped smartphone cameras recognize it. In this instance the Digimarc Discover app instructs smartphones to connect to an associated video of the athletic event. Readers may try it for themselves by downloading Digimarc’s Discover app from the iPhone App Store or Android Marketplace. (For best results first watch the animated instructions provided within the app.) Silicon Ear. Just as the camera is a silicon eye for a cognitive prosthetic, the microphone is its silicon ear. One illuminating example is music recognition. Computerized methods of identifying full compositions from audio samples have been available for about twenty years, but were little used by desktop computers. Services first appeared on mobile phones in 2003 but availability was limited on a carrier-by-carrier basis. Not until the advent of the app platform in 2008 did music recognition come-of-age. Today smartphone owners routinely use such apps to identify songs playing on the radio. Most permit users to purchase track downloads once songs are identified. Significantly, automated music identification only evolved from obscurity to popularity after smartphones became portable intelligent aids. Location Aware. Location-awareness provides added utility. For example, only a location-aware engine is able to isolate nearby suggestions when users search for “Italian restaurants”. Similarly, cognitive features may combine to provide amplified efficacy. For example, vacationers in a strange city may use a smartphone camera to photograph a prominent landmark and – with the aid of an app -- map their location. Other examples include applications that (1) locate nearby gas stations 5
- 7. with prices, (2) provide realtime identification of nearby mountains, elevations, and names, and (3) offer Wikipedia entries for points of interest within eyesight. Personalized Personalization is another well-established computer trend, probably even more applicable in the mobile environment. One example is the news reader. Instead of looking through a daily newspaper to learn of relevant items, today’s computer-savvy news consumers choose their own news feed. They only get news on topics they care about. Moreover, when the feed connects to their mobile phones they get realtime updates wherever they go. Another example is personalized radio. Instead of relying upon a conventional radio station that plays music in a selected genre, Internet radio services provide a better fit. They enable consumers to construct “stations” that only play tunes by their favorite, or similar, artists. Media Transmutation It’s increasingly evident that content cannot be isolated to old platforms such as newspapers, TV, and magazines. Consumers want to use them all, and sometimes conflate them into whatever form factor is most convenient. Thus, content should be optimized across all platforms. Yet the rise of smartphones and tablet computers enables consumers to more readily transform media between the analog and digital domains. For decades computer users routinely transformed virtual data into physical media. First computer printers created paper copies. Later CD and DVD-burners etched computer files onto laser discs. But the mobile computing paradigm enables media transmutation in the opposite direction by converting physical media into digital media. For example, smartphone owners can use integrated cameras to scan printed media embedded with watermarks or to read bar code information. When traditional media contains implanted signaling, content owners get two benefits. First they can engage customers in new ways. Second, they can transform the added engagement into revenue-producing opportunities. Terrestrial-to-Virtual Integration Methods of implanted signaling evolved gradually. 6
- 8. Barcodes One of the earliest is the common bar code illustrated here. Although frequently used today, the evolution of bar code adoption provides insight into how more sophisticated methods such digital fingerprinting and watermarking might ultimately achieve mass market acceptance. Pioneering optical recognition technologies in the 1950s led the National Association of Food Chains to investigate automated check-out in 1966. By the early 1970s they adopted a Universal Product Code (UPC). Unfortunately, seventy-five percent of grocery items first needed to be bar-coded in order for savings to be realized. By 1977 only 200 supermarkets used scanners. Stores didn’t want to buy expensive scanners unless a significant majority of trade products had pre-printed labels. But product-makers didn’t want to print labels until a critical mass of stores had scanners. Neither side wanted to move first. A change catalyst was the discovery of unexpected benefits in the stores actually using bar codes. Most importantly, the detailed product sales information enabled the stores to improve customer service with better stocked inventory. After installing scanners applicable grocery stores started to witness increasing sales that ultimately rose to 10% - 12% ahead of non-equipped stores. By the end of the decade 8,000 stores were using scanners. One barcode variety, termed the QR code, has been more popular overseas than in the USA. The two dimensional code is illustrated at the left. Recently a Denver bank used QRs on billboards at the city’s airport enabling smartphone users to download free copies of public domain e-books, such as Treasure Island. Being two dimensional, far more data can be represented by QR codes as compared to UPCs. Bar codes have four principal disadvantages. First, they are limited to visual media whereas digital fingerprints and watermarks can be both visual and audible. Second, they are ugly. Third, for many publishers the very fact that they require dedicated page space is objectionable. Fourth, they can be counterfeited merely by pasting a false code over a legitimate one. Nonetheless, bar codes have two important advantages: (1) mass market acceptance and (2) proven device-readability. 7
- 9. Digital Fingerprints Digital fingerprints are parametric representations of an original data file. For example, a CD music track is a digital manifestation of the song’s analog waveforms. A digital fingerprint is created by applying feature-extraction algorithms to a track sample. The process yields a unique feature set corresponding to the original file segment. Digital fingerprints are powerful identification tools when compiled in a machine- accessible online database. For example, when loaded with the proper app a smartphone can (1) “listen” to a song played on the radio and (2) transmit a digitized sample to an online database where it is (3) matched with the most similar fingerprint in the index. Fingerprints have two advantages. First, they can be constructed after the original files were created. Algorithms enable fingerprints to be derived for any data file. Furthermore, it is not necessary that the original content be in a digital form. Instead it might be a magazine article, 45-rpm record, or video taped TV show. Analog-to-digital conversion tools such as scanners, audio recording software, and camcorders are readily available to convert traditional media into digital versions. After format conversion, digital fingerprints may be created in the customary way. Thus, songs, photos, and motion pictures released to the market a half-century, or more, ago can be fingerprinted. Second, unlike bar codes, fingerprints are applicable to both visual and audible content. The principal disadvantage of fingerprints is that do not contain intrinsic information. The augmented information actually resides in the online database. For example, a smartphone owner using a fingerprint app to identify songs on the radio can likely learn the name of the song, its performing artist, and connect to actions linked to the database such as options to purchase the song. But the fingerprint likely cannot tell which radio station played the song, where the DJ got the CD, or other information that might have been added to the file by a digital watermark. Digital fingerprints can also provide “false positive” identification if the fingerprint in the database does not match the content sample precisely. In addition, as fingerprint databases and associated metadata get larger, identification and the accompanying triggering actions may process too slowly. Thus, some users might conclude the process is “not working” and abort the query. 8
- 10. Digital Watermarks A digital watermark is added to a digital file to impart additional information or features. Sometimes the mark is obvious and sometimes it’s imperceptible. An example of an obvious watermark is illustrated at the left. It enables a professional photographer to send sample action shots to bike race participants, but renders such copies undesirable for contestants wanting to own pictures documenting their participation. In contrast, imperceptible watermarks are un- noticeable to human eyes. Presently they are most commonly used in forensic or tracking applications. Since the watermark is added to the underlying content file, it can impart unique information. For example, unique watermarks can be added to free promotional music CDs shipped to radio stations. Thus if a station employee “rips” music tracks to place them on Internet websites without authorization, record labels can direct investigations to the correct station. In a similar manner, watermarks are added to copies of motion pictures and television programs. The ability to track illegal copies to their source is a powerful piracy deterrent. An important watermark advantage is the ability to survive digital-to-analog conversion. It’s one reason they’ve been adopted for automated audience measurement by broadcast radio and television. TV broadcasters cooperate with Nielsen by inserting unique watermarks into each show identifying both the program and the local affiliate broadcasting it. Similarly Arbitron and radio stations cooperate on watermark use for automated audience measurement. Watermarks also uniquely survive analog-to-digital conversion. Thus anyone recording a TV broadcast also records its watermark. If the recording is later posted on the Internet the forensic watermark gets posted as well. Like digital fingerprints, digital watermarks are suitable for both visual and audio content. Relative to fingerprints, the watermark’s principal disadvantage is a necessity to be added to original content before the media is released to market. The technology cannot be used to recognize content unless identifying watermarks were added. Unfortunately a great many older music and movie recordings entered the market before watermarking was even invented. 9
- 11. Unifying Platform As noted, it took many years and a number of catalyzing events before bar codes attained widespread use in grocery stores. The launch of signal-rich content for consumer markets has awaited similar catalysts. Undeniably, one such triggering event is the advent of mobile computing. Until the now-familiar app platform, few consumer devices were compliant with bar codes, digital fingerprints, or digital watermarks. Furthermore, compliance was largely device-specific (e.g. laser scanners). In contrast, today’s smartphones and tablets are normally made compliant by third party app developers. As a result, innovation is presently much faster. Current Need. Nonetheless, a number of obstacles remain. One is a fragmentation of standards owing to a proliferation of applications for bar codes, QR codes, watermarks, and fingerprints. As innovators focus on discrete opportunities individually with little regard to standards, consumers will eventually become overloaded with competing applications that might otherwise work cooperatively together. In short, signal-rich media would benefit if consumers could access a unified platform. Ideally it could lead to products providing integrated bar code, QR code, watermark, and fingerprint recognition from a number of cooperating app developers. Digimarc is pioneering an effort to address the need with its Discover platform. Digimarc Discover. The Discover platform is a tool designed to promote intuitive computing. Digimarc’s vision is that once users launch the application, their cognitive prosthetics (e.g. smartphones and tablet computers) will examine surrounding media with built-in cameras and microphones. When content is identified the application might eventually use context algorithms combining the user’s location, prior use history, and any known personal preferences to determine what the user most likely wants to know. Although Digimarc is a leader in watermarking, the platform is not intended to be limited to watermarking. Instead it envisions using the signaling technology best suited to environmental objects in order to provide users with the most relevant content or actions. Starting this calendar quarter a demonstration version of Discover is available from iTunes App Store and Android Market. It incorporates digital watermark detection from Digimarc along with Gracenote’s music recognition, and bar code reading for price comparisons. The following video shows Digimarc’s visualization for how the Discover platform could one day (1) transform a newspaper into a web browser, (2) interact with digital signage, (3) identify music, (4) enhance printed drug labels with additional online information, and (5) find nearby theaters exhibitors after “watching” the 10
- 12. applicable movie trailer on TV. The company already has a working prototype available demonstrating the concepts depicted in the video. It is able to detect watermarks in print, audio, and video media. Widespread adoption of watermarking could transform the vision into a reality. Digimarc Discover Video Demonstration Additionally Discover is a reference platform enabling developers to create their own signal-rich media applications. For example, big-box retailers can develop applications permitting customers to use smartphones as intelligent aids for gathering more information about shelf merchandise. Furthermore their apps might trigger spontaneous in-store promotions based upon (1) the prior shopping history of the smartphone customer, (2) available merchandise stocked, or (3) most any merchant-defined factor. Ultimately, however, smartphones and tablets will become so powerful that access to onboard sensors such as cameras and microphones will be integrated into the operating system. Digimarc Discover provides a viable pathway for device makers to implement such integration. Once such integration is achieved, cognitive prosthetic applications will function even more effectively and naturally. Conclusion Smartphones and tablet computers are inducing a new paradigm of mobile computing signified by three characteristics. First, it’s ubiquitous. Within two years Gartner estimates more mobile phones will be in use than personal computers. Second, smartphones and tablets are becoming cognitive prosthetics. Owners increasingly use them as intelligent aids as routinely as amputees use mechanical limbs for body extensions. Third, cognitive prosthetics are nearly continuously tethered to the nearly infinite computing resources of the Internet cloud. Much like Local Area Networks 11
- 13. (LANs) led users to the epiphany that “the network is the computer”, the corresponding mantra for mobile computing rises to “the Internet is the computer”. Embedded signaling will be crucial to methods of protecting media, engaging consumers in new ways, and generating incremental revenue for content owners. Phil Leigh Inside Digital Media, Inc. January 21, 2011 813-837-3631 phil@insidedigitalmedia.com 12
- 14. Inside Digital Media, Inc. 3911 San Pedro Tampa, FL 33629 813.837.3631