Automatic content recognition (ACR) is a technology to identify content played on a media device or present within a media file. Devices implementing ACR can allow the device or the manufacturer to collect content consumption information automatically at the screen level itself, without any user-based input or search efforts. This information may be collected for purposes such as personalized advertising, content recommendations, sale to customer data aggregators and other applications.

How it works

To start the recognition, a short media clip (audio, video, or both) is selected from within a media file or captured as displayed on a device such as a

smart TV A smart TV, also known as a connected TV (CTV), is a traditional television set with integrated Internet and interactive Web 2.0 features, which allows users to stream music and videos, browse the internet, and view photos. Smart TVs are a techn ...

. Using techniques such as

fingerprinting A fingerprint is an impression left by the friction ridges of a human finger. The recovery of partial fingerprints from a crime scene is an important method of forensic science. Moisture and grease on a finger result in fingerprints on surfac ...

and watermarking, the selected content is compared by the ACR software with a database of known recorded works. If the fingerprint of the media clip finds a match, the ACR software returns the corresponding

metadata Metadata is "data that provides information about other data", but not the content of the data, such as the text of a message or the image itself. There are many distinct types of metadata, including: * Descriptive metadata – the descriptive ...

regarding the media as well as other associated or recommended content back to the client application for display to the user or for collection by the device manufacturer or a data aggregator.

Fingerprints and watermarking

Two leading methodologies for audio-based ACR are

acoustic fingerprint An acoustic fingerprint is a condensed digital summary, a fingerprint, deterministically generated from an audio signal, that can be used to identify an audio sample or quickly locate similar items in an audio database. Practical uses of aco ...

ing and watermarking. Similarly,

video fingerprint Video fingerprinting or video hashing are a class of dimension reduction techniques in which a system identifies, extracts, and then summarizes characteristic components of a video as a unique or a set of multiple perceptual hashes, enabling tha ...

ing is used to facilitate ACR for visual media. Acoustic fingerprinting generates unique fingerprints from the audio content itself. Fingerprinting techniques are agnostic to content format, codec, bit rate and compression techniques. This makes it possible to employ across varying networks and channels and is widely used for interactive TV, second screen application, and content monitoring sectors. Popular apps like Shazam,

YouTube YouTube is a global online video sharing and social media platform headquartered in San Bruno, California. It was launched on February 14, 2005, by Steve Chen, Chad Hurley, and Jawed Karim. It is owned by Google, and is the second mo ...

Facebook Facebook is an online social media and social networking service owned by American company Meta Platforms. Founded in 2004 by Mark Zuckerberg with fellow Harvard College students and roommates Eduardo Saverin, Andrew McCollum, Dust ...

, Thetake,

WeChat WeChat () is a Chinese instant messaging, social media, and mobile payment app developed by Tencent. First released in 2011, it became the world's largest standalone mobile app in 2018, with over 1 billion monthly active users. WeChat has b ...

and

Weibo Weibo may refer to: * Microblogging in China, or China-based microblogging services (), including: ** NetEase Weibo (), launched by NetEase ** People's Weibo (), launched by ''People's Daily'' ** Phoenix Weibo (), launched by Phoenix Television ** W ...

reportedly use

audio fingerprinting An acoustic fingerprint is a condensed digital summary, a fingerprint, deterministically generated from an audio signal, that can be used to identify an audio sample or quickly locate similar items in an audio database. Practical uses of aco ...

methodology to recognize content played from a TV to trigger additional features like votes, lotteries, topics or purchases. In contrast to fingerprinting,

digital watermarking A digital watermark is a kind of marker covertly embedded in a noise-tolerant signal such as audio, video or image data. It is typically used to identify ownership of the copyright of such signal. "Watermarking" is the process of hiding digital in ...

require the inclusion of digital "tags" embedded within the digital content stream prior to distribution. For example, a broadcast encoder might insert a

watermark A watermark is an identifying image or pattern in paper that appears as various shades of lightness/darkness when viewed by transmitted light (or when viewed by reflected light, atop a dark background), caused by thickness or density variations ...

every few seconds that could be used to identify to broadcast channel, program id, and time stamp. This watermark is normally inaudible or invisible to the users. Display devices like phones or tablets however can read the watermarks to identify the content it is playing. Watermarking technology is also utilized in the media protection field to help identify where illegal copies originate.

History

In 2011, ACR technology was applied to TV content by the Shazam service, which captured the attention of the television industry. Shazam was previously a music recognition service which recognized music from sound recordings. By utilizing its own fingerprint technology to identify live channels and videos, Shazam extended their business to television programming. Also in 2011, Samba TV (at the time known as Flingo) introduced its patented video ACR technology, which uses video fingerprinting to identify on-screen content and power cross-screen interactive TV apps on Smart TVs. In 2012, satellite communications provider

DIRECTV DirecTV (trademarked as DIRECTV) is an American multichannel video programming distributor based in El Segundo, California. Originally launched on June 17, 1994, its primary service is a digital satellite service serving the United States. I ...

partnered with TV loyalty vendor Viggle to provide an interactive viewing experience on the second screen. In 2013, LG partnered with

Cognitive Networks In communication networks, cognitive network (CN) is a new type of data network that makes use of cutting edge technology from several research areas (i.e. machine learning, knowledge representation, computer network, network management) to ...

(later purchased by

Vizio Vizio Inc. (stylized as VIZIO) is an American publicly traded company that designs and sells televisions, sound bars, viewer data, and advertising. The company was founded in 2002 and is based in Irvine, California.Lawton, Christopher, Iwatani ...

and renamed Inscape), an ACR vendor, to provide ACR driven interaction. In 2015, ACR technology spread to even more applications and smart TVs. Social applications and TV manufacturers like

Twitter Twitter is an online social media and social networking service owned and operated by American company Twitter, Inc., on which users post and interact with 280-character-long messages known as "tweets". Registered users can post, like, and ...

Google Google LLC () is an American Multinational corporation, multinational technology company focusing on Search Engine, search engine technology, online advertising, cloud computing, software, computer software, quantum computing, e-commerce, ar ...

, LG,

Samsung The Samsung Group (or simply Samsung) ( ko, 삼성 ) is a South Korean multinational manufacturing conglomerate headquartered in Samsung Town, Seoul, South Korea. It comprises numerous affiliated businesses, most of them united under the ...

, and

TV have used ACR technology either developed by themselves or integrated from third party ACR providers. In 2016, additional applications and mobile OS embedded with automatic content recognition services were available including Peach, Omusic and Mi OS.

Applications

Advertising and customer data collection

Data collected on the media consumption habits of customers is of enormous value to device manufacturers, advertisers, and data aggregation companies. ACR technology helps these companies surveil the interests of customers and collect data so that they can be more precisely targeted with personalized marketing and advertising campaigns. This data is extremely valuable-- it was reported in Nov 2021 that smart television manufacturer

is more profitable from the sale of their customers' data than from the televisions they sold.

Audience measurement

Real-time

audience measurement Audience measurement measures how many people are in an audience, usually in relation to radio listenership and television viewership, but also in relation to newspaper and magazine readership and, increasingly, web traffic on websites. Sometim ...

metrics are now achievable by applying ACR technology into smart TVs, set top boxes and mobile devices such as smart phones and tablets. This measurement data is highly essential to quantify audience consumption to set advertising pricing policies.

Content identification

ACR technology helps audiences retrieve information about the content they watched or listened to. The identified video and music content can be linked to internet content providers for on-demand viewing, third parties for additional background information, or complementary media.

Content enhancement

Because devices can be "aware" of content being watched or listened to, second screen devices can feed users complementary content beyond what is presented on the primary viewing screen. ACR technology can not only identify the content, but also it can identify the precise location within the content. Thus, additional information can be presented to the user. ACR can enable a variety of interactive features such as polls, coupons, lottery or purchase of goods based on timestamp.

Privacy Concerns

Organizations ranging from consumer rights advocates

Electronic Frontier Foundation The Electronic Frontier Foundation (EFF) is an international non-profit digital rights group based in San Francisco, California. The foundation was formed on 10 July 1990 by John Gilmore, John Perry Barlow and Mitch Kapor to promote Internet ...

to tech web sites such as

PCMag ''PC Magazine'' (shortened as ''PCMag'') is an American computer magazine published by Ziff Davis. A print edition was published from 1982 to January 2009. Publication of online editions started in late 1994 and have continued to the present ...

have expressed serious objections to the collection of user viewing consumption habits by their devices on privacy grounds. However there is an option in almost every set to disable this feature.

Technology providers

ACR service providers includ
Beatgrid

ACRCloud ACRCloud (Formerly Syntec TV) is an automatic content recognition platform based on acoustic fingerprinting technology. Its creator intended to help media, broadcasters and app developers to identify, monitor and monetize content on the second sc ...

, Red Bee Media, Digimarc,

Gracenote Gracenote, Inc. is a company owned by Nielsen Holdings that provides music, video and sports metadata and automatic content recognition (ACR) technologies to entertainment services and companies, worldwide. Formerly CDDB ("Compact Disc Data Ba ...

, Kantar Media, Inscape Data Services, Zapr Media Labs, Shazam and Samba TV.

References

{{Reflist * Acoustic fingerprinting Music search engines Online music and lyrics databases Television technology Television terminology *