Pony Preservation Project

15.ai is a non-commercial freeware artificial intelligence web application that generates natural emotive high-fidelity text-to-speech voices from an assortment of fictional characters from a variety of media sources. Developed by an anonymous MIT researcher under the eponymous pseudonym 15, the project uses a combination of audio synthesis algorithms, speech synthesis deep neural networks, and sentiment analysis models to generate and serve emotive character voices faster than real-time, even those with a very small amount of data.

Launched in early 2020, 15.ai began as a proof of concept of the democratization of voice acting and dubbing using technology.

Its gratis and non-commercial nature (with the only stipulation being that the project be properly credited when used), ease of use, and substantial improvements to current text-to-speech implementations have been lauded by users; however, some critics and voice actors have questioned the legality and ethicality of leaving such technology publicly available and readily accessible.

Credited as the impetus behind the popularization of AI vocal reconstruction technology in content creation, 15.ai has had a significant impact on multiple Internet fandoms, most notably the ''My Little Pony: Friendship Is Magic'', ''Team Fortress 2'', and ''SpongeBob SquarePants'' fandoms. Several commercial alternatives have spawned with the rising popularity of 15.ai, leading to cases of misattribution and theft. In January 2022, it was discovered that Voiceverse NFT, a company that voice actor Troy Baker announced his partnership with, had plagiarized 15.ai's work as part of their platform.

Features

Available characters include GLaDOS and Wheatley from ''Portal'', characters from ''Team Fortress 2'', Twilight Sparkle and a number of main, secondary, and supporting characters from '' My Little Pony: Friendship Is Magic'', SpongeBob from ''SpongeBob SquarePants'', Daria Morgendorffer and Jane Lane from '' Daria'', the Tenth Doctor from ''Doctor Who'', HAL 9000 from '' 2001: A Space Odyssey'', the Narrator from ''The Stanley Parable'', the Wii U/3DS/ Switch Super Smash Bros. Announcer (formerly), Carl Brutananadilewski from '' Aqua Teen Hunger Force'', Steven Universe and the Crystal Gems from '' Steven Universe'', Dan from ''Dan Vs.'', Sans from '' Undertale, The Griffin's Family from Family Guy, Rick and Morty and DC Superhero Girls (2019)''

The deep learning model used by the application is nondeterministic: each time that speech is generated from the same string of text, the intonation of the speech will be slightly different. The application also supports manually altering the emotion of a generated line using ''emotional contextualizers'' (a term coined by this project), a sentence or phrase that conveys the emotion of the take that serves as a guide for the model during inference.
Emotional contextualizers are representations of the emotional content of a sentence deduced via transfer learned emoji embeddings using DeepMoji, a deep neural network sentiment analysis algorithm developed by the MIT Media Lab in 2017. DeepMoji was trained on 1.2 billion emoji occurrences in Twitter data from 2013 to 2017, and has been found to outperform human subjects in correctly identifying sarcasm in Tweets and other online modes of communication.

15.ai uses a ''multi-speaker model''—hundreds of voices are trained concurrently rather than sequentially, decreasing the required training time and enabling the model to learn and generalize shared emotional context, even for voices with no exposure to such emotional context. Consequently, the entire lineup of characters in the application is powered by a single trained model, as opposed to multiple single-speaker models trained on different datasets. The lexicon used by 15.ai has been scraped from a variety of Internet sources, including Oxford Dictionaries, Wiktionary, the CMU Pronouncing Dictionary, 4chan, Reddit, and Twitter. Pronunciations of unfamiliar words are automatically deduced using phonological rules learned by the deep learning model.

The application supports a simplified version of a set of English phonetic transcriptions known as ARPABET to correct mispronunciations or to account for heteronyms—words that are spelled the same but are pronounced differently (such as the word ''read'', which can be pronounced as either or depending on its tense). While the original ARPABET codes developed in the 1970s by the Advanced Research Projects Agency supports 50 unique symbols to designate and differentiate between English phonemes, the CMU Pronouncing Dictionary's ARPABET convention (the set of transcription codes followed by 15.ai) reduces the symbol set to 39 phonemes by combining allophonic phonetic realizations into a single standard (e.g. AXR/ER; UX/ UW) and using multiple common symbols together to replace syllabic consonants (e.g. EN/AH0 N). ARPABET strings can be invoked in the application by wrapping the string of phonemes in curly braces within the input box (e.g. to denote , the pronunciation of the word ''ARPABET'').

The following is a table of phonemes used by 15.ai and the CMU Pronouncing Dictionary:

Background

Speech synthesis

In 2016, with the proposal of DeepMind's WaveNet, deep-learning-based models for speech synthesis began to gain popularity as a method of modeling waveforms and generating human-like speech. Tacotron2, a neural network architecture for speech synthesis developed by Google AI, was published in 2018 and required tens of hours of audio data to produce intelligible speech; when trained on 2 hours of speech, the model was able to produce intelligible speech with mediocre quality, and when trained on 36 minutes of speech, the model was unable to produce intelligible speech.

For years, reducing the amount of data required to train a realistic high-quality text-to-speech model has been a primary goal of scientific researchers in the field of deep learning speech synthesis. The developer of 15.ai claims that as little as 15 seconds of data is sufficient to clone a voice up to human standards, a significant reduction in the amount of data required.

Copyrighted material in deep learning

A landmark case between Google and the Authors Guild in 2013 ruled that Google Books—a service that searches the full text of printed copyrighted books—was transformative, thus meeting all requirements for fair use. This case set an important legal precedent for the field of deep learning and artificial intelligence: using copyrighted material to train a discriminative model or a ''non-commercial'' generative model was deemed legal. The legality of ''commercial'' generative models trained using copyrighted material is still under debate; due to the black-box nature of machine learning models, any allegations of copyright infringement via direct competition would be difficult to prove.

Development

15.ai was designed and created by an anonymous research scientist affiliated with the Massachusetts Institute of Technology known by the alias ''15''.

The project began development while the developer was an undergraduate. The developer has stated that they are capable of paying the high cost of running the site out of pocket.

According to posts made by its developer on Hacker News, 15.ai costs several thousands of dollars per month to operate; they are able to support the project due to a successful startup exit. The developer has stated that during their undergraduate years at MIT, they were paid the minimum hourly rate to work on a related project (approximately $14 an hour in Massachusetts) that eventually evolved into 15.ai. They also stated that the democratization of voice cloning technology is not the only function of the website; in response to a user asking whether the research could be conducted without a public website, the developer wrote:

The algorithm used by the project to facilitate the cloning of voices with minimal viable data has been dubbed DeepThroat (a double entendre in reference to speech synthesis using deep neural networks and the sexual act of deep-throating). The project and algorithm—initially conceived as part of MIT's Undergraduate Research Opportunities Program—had been in development for years before the first release of the application.

The developer has also worked closely with the Pony Preservation Project from /mlp/, the ''My Little Pony'' board of 4chan. The Pony Preservation Project, which began in 2019, is a "collaborative effort by /mlp/ to build and curate pony datasets" with the aim of creating applications in artificial intelligence.
The ''Friendship Is Magic'' voices on 15.ai were trained on a large dataset crowdsourced by the Pony Preservation Project: audio and dialogue from the show and related media—including all nine seasons of ''Friendship Is Magic'', the 2017 movie, spinoffs, leaks, and various other content voiced by the same voice actors—were parsed, hand-transcribed, and processed to remove background noise. According to the developer, the collective efforts and constructive criticism from the Pony Preservation Project have been integral to the development of 15.ai.

In addition, the developer has stated that the logo of 15.ai, which features a robotic Twilight Sparkle, is an homage to the fact that her voice (as originally portrayed by Tara Strong) was indispensable to the implementation of emotional contextualizers.

Reception

15.ai has been met with largely positive reviews. Liana Ruppert of ''Game Informer'' described 15.ai as "simplistically brilliant." Lauren Morton of ''Rock, Paper, Shotgun'' and Natalia Clayton of '' PCGamer'' called it "fascinating," and José Villalobos of '' LaPS4'' wrote that it "works as easy as it looks." Users praised the ability to easily create audio of popular characters that sound believable to those unaware that the voices had been synthesized by artificial intelligence: Zack Zwiezen of ''Kotaku'' reported that " isgirlfriend was convinced it was a new voice line from GLaDOS' voice actor, Ellen McLain," while Rionaldi Chandraseta of '' Towards Data Science'' wrote that, upon watching a YouTube video featuring popular character voices generated by 15.ai, " isfirst thought was the video creator used cameo.com to pay for new dialogues from the original voice actors" and stated that "the quality of voices done by 15.ai is miles ahead of ts competitors"

Computer scientist and technology entrepreneur Andrew Ng commented in his newsletter ''The Batch'' that the technology behind 15.ai could be "enormously productive" and could "revolutionize the use of virtual actors"; however, he also noted that "synthesizing a human actor's voice without consent is arguably unethical and possibly illegal" and could potentially open up to cases of impersonation and fraud. In his blog '' Marginal Revolution'', economist Tyler Cowen deemed 15 one of the "most underrated talents in AI and machine learning."

Impact

Fandom content creation

15.ai has been frequently used for content creation in various fandoms, including the ''My Little Pony: Friendship Is Magic'' fandom, the ''Team Fortress 2'' fandom, the ''Portal'' fandom, and the ''SpongeBob SquarePants'' fandom. Numerous videos and projects containing speech from 15.ai have gone viral. However, some videos and projects that contain non-15.ai-generated speech have also gone viral, many of which do not properly credit the source(s) of the synthetic speech featured in them. As a consequence, many videos and projects that have been made with other speech synthesis software have been mistaken as being made with 15.ai, and vice versa. Due to this misattribution and absence of proper credit, 15.ai's terms of service has a rule that forbids having 15.ai-and-non-15.ai-generated speech in the same videos and projects.

The ''My Little Pony: Friendship Is Magic'' fandom has seen a resurgence in video and musical content creation as a direct result, inspiring a new genre of fan-created content assisted by artificial intelligence. Some fanfiction have been adapted into fully voiced "episodes": ''The Tax Breaks'' is a 17-minute long animated video rendition of a fan-written story published in 2014 that uses voices generated from 15.ai with sound effects and audio editing, emulating the episodic style of the early seasons of ''Friendship Is Magic''.

Viral videos from the ''Team Fortress 2'' fandom that feature voices from 15.ai include ''Spy is a Furry'' (which has gained over 3 million views on YouTube total across multiple videos) and ''The RED Bread Bank'', both of which have inspired Source Filmmaker animated video renditions. Other fandoms have used voices from 15.ai to produce viral videos. , the viral video '' Among Us Struggles'' (which uses voices from ''Friendship Is Magic'') has over 5.5 million views on YouTube; YouTubers, TikTokers, and Twitch streamers have also used 15.ai for their videos, such as FitMC's video on the history of 2b2t—one of the oldest running ''Minecraft'' servers—and datpon3's TikTok video featuring the main characters of ''Friendship Is Magic'', which have 1.4 million and 510 thousand views, respectively.

Some users have created AI virtual assistants using 15.ai and external voice control software. One user on Twitter created their own personal GLaDOS desktop assistant using the voice control system VoiceAttack that is able to boot up applications, utter corresponding random dialogues, and thank the user in response to actions.

Troy Baker / Voiceverse NFT plagiarism scandal

In December 2021, the developer of 15.ai posted on Twitter that they had no interest in incorporating non-fungible tokens (NFTs) into their work.

On January 14, 2022, it was discovered that Voiceverse NFT, a company that video game and anime dub voice actor Troy Baker announced his partnership with, had plagiarized voice lines generated from 15.ai as part of their marketing campaign. Log files showed that Voiceverse had generated audio of Twilight Sparkle and Rainbow Dash from the show '' My Little Pony: Friendship Is Magic'' using 15.ai, pitched them up to make them sound unrecognizable from the original voices, and appropriated them without proper credit to falsely market their own platform—a violation of 15.ai's terms of service.

A week prior to the announcement of the partnership with Baker, Voiceverse made a (now-deleted) Twitter post directly responding to a (now-deleted) video posted by Chubbiverse—an NFT platform with which Voiceverse had partnered—showcasing an AI-generated voice and claimed that it was generated using Voiceverse's platform, remarking ''"I wonder who created the voice for this? ;)"'' A few hours after news of the partnership broke, the developer of 15.ai—having been alerted by another Twitter user asking for his opinion on the partnership, to which he speculated that it "sounds like a scam"—posted screenshots of log files that proved that a user of the website (with their IP address redacted) had submitted inputs of the exact words spoken by the AI voice in the video posted by Chubbiverse, and subsequently responded to Voiceverse's claim directly, tweeting "Certainly not you :)".

Following the tweet, Voiceverse admitted to plagiarizing voices from 15.ai as their own platform, claiming that their marketing team had used the project without giving proper credit and that the "Chubbiverse team adno knowledge of this." In response to the admission, 15 tweeted "Go fuck yourself." The final tweet went viral, accruing over 75,000 total likes and 13,000 total retweets across multiple reposts.

The initial partnership between Baker and Voiceverse was met with severe backlash and universally negative reception. Critics highlighted the environmental impact of and potential for exit scams associated with NFT sales. Commentators also pointed out the irony in Baker's initial Tweet announcing the partnership, which ended with "You can hate. Or you can create. What'll it be?", hours before the public revelation that the company in question had resorted to theft instead of creating their own product. Baker responded that he appreciated people sharing their thoughts and their responses were "giving ima lot to think about." He also acknowledged that the "hate/create" part in his initial Tweet might have been "a bit antagonistic," and asked fans on social media to forgive him. Two weeks later, on January 31, Baker announced that he would discontinue his partnership with Voiceverse.

Resistance from voice actors

Some voice actors have publicly decried the use of voice cloning technology. Cited reasons include concerns about impersonation and fraud, unauthorized use of an actor's voice in pornography, and the potential of AI being used to make voice actors obsolete.

List of voices

All characters available on 15.ai (both currently and formerly) are listed in the table below.

See also

* Audio deepfake
* Character.ai
*ChatGPT
* DALL-E
*Deepfakes
* Midjourney
* Stable Diffusion
* Synthetic media
* WaveNet

Notes

References

;Notes

;Tweets

;YouTube (referenced for view counts and usage of 15.ai only)

;TikTok

External links

*
*

''The Tax Breaks (Twilight) (15.ai)''

{{My Little Pony: Friendship Is Magic

Speech synthesis
Deep learning software applications
Applications of artificial intelligence
Deepfakes
Massachusetts Institute of Technology alumni
My Little Pony: Friendship Is Magic
My Little Pony fandom
Computer-related introductions in 2020
Web applications
2020 in Internet culture
2020s fads and trends