Motion capture (sometimes referred as mo-cap or mocap, for short) is the process of recording the
movement
Movement may refer to:
Common uses
* Movement (clockwork), the internal mechanism of a timepiece
* Motion, commonly referred to as movement
Arts, entertainment, and media
Literature
* "Movement" (short story), a short story by Nancy Fu ...
of objects or people. It is used in
military
A military, also known collectively as armed forces, is a heavily armed, highly organized force primarily intended for warfare. It is typically authorized and maintained by a sovereign state, with its members identifiable by their distinct ...
,
entertainment
Entertainment is a form of activity that holds the attention and interest of an audience or gives pleasure and delight. It can be an idea or a task, but is more likely to be one of the activities or events that have developed over thousa ...
,
sports
Sport pertains to any form of competitive physical activity or game that aims to use, maintain, or improve physical ability and skills while providing enjoyment to participants and, in some cases, entertainment to spectators. Sports can, th ...
, medical applications, and for validation of computer vision and robots. In
filmmaking
Filmmaking (film production) is the process by which a motion picture is produced. Filmmaking involves a number of complex and discrete stages, starting with an initial story, idea, or commission. It then continues through screenwriting, casti ...
and
video game development
Video game development (or gamedev) is the process of developing a video game. The effort is undertaken by a developer, ranging from a single person to an international team dispersed across the globe. Development of traditional commercial PC a ...
, it refers to recording actions of
human actors, and using that information to animate
digital character
Character animation is a specialized area of the animation process, which involves bringing animated s to life. The role of a character animator is analogous to that of a film or stage actor and character animators are often said to be "actors wi ...
models in 2-D or 3-D
computer animation
Computer animation is the process used for digitally generating animations. The more general term computer-generated imagery (CGI) encompasses both static scenes (still images) and dynamic images (moving images), while computer animation refe ...
.
[Andrew Harris Salomon, Feb. 22, 2013, Backstage Magazine]
Growth In Performance Capture Helping Gaming Actors Weather Slump
Accessed June 21, 2014, "..But developments in motion-capture technology, as well as new gaming consoles expected from Sony and Microsoft within the year, indicate that this niche continues to be a growth area for actors. And for those who have thought about breaking in, the message is clear: Get busy...."[Ben Child, 12 August 2011, The Guardian]
Andy Serkis: why won't Oscars go ape over motion-capture acting? Star of Rise of the Planet of the Apes says performance capture is misunderstood and its actors deserve more respect
Accessed June 21, 2014 When it includes face and fingers or captures subtle expressions, it is often referred to as performance capture.
[Hugh Hart, January 24, 2012, Wired magazine]
When will a motion capture actor win an Oscar?
Accessed June 21, 2014, "...the Academy of Motion Picture Arts and Sciences’ historic reluctance to honor motion-capture performances .. Serkis, garbed in a sensor-embedded Lycra body suit, quickly mastered the then-novel art and science of performance-capture acting. ..." In many fields, motion capture is sometimes called motion tracking, but in filmmaking and games, motion tracking usually refers more to
match moving
In visual effects, match moving is a technique that allows the insertion of computer graphics into live-action footage with correct position, scale, orientation, and motion relative to the photographed objects in the shot. The term is used loose ...
.
In motion capture sessions, movements of one or more actors are sampled many times per second. Whereas early techniques used
images from multiple cameras to calculate 3D positions, often the purpose of motion capture is to record only the movements of the actor, not their visual appearance. This ''animation data'' is mapped to a 3D model so that the model performs the same actions as the actor. This process may be contrasted with the older technique of
rotoscoping
Rotoscoping is an animation technique that animators use to trace over motion picture footage, frame by frame, to produce realistic action. Originally, animators projected photographed live-action movie images onto a glass panel and traced ov ...
.
Camera movements can also be motion captured so that a virtual camera in the scene will pan, tilt or dolly around the stage driven by a camera operator while the actor is performing. At the same time, the motion capture system can capture the camera and props as well as the actor's performance. This allows the computer-generated characters, images and sets to have the same perspective as the video images from the camera. A computer processes the data and displays the movements of the actor, providing the desired camera positions in terms of objects in the set. Retroactively obtaining camera movement data from the captured footage is known as ''match moving'' or ''
camera tracking
In visual effects, match moving is a technique that allows the insertion of computer graphics into live-action footage with correct position, scale, orientation, and motion relative to the photographed objects in the Shot (filmmaking), shot. The ...
''.
The first virtual actor animated by motion-capture was produced in 1993 by Didier Pourcel and his team at Gribouille. It involved "cloning" the body and face of french comedian Richard Bohringer, and then animating it with still-nascent motion-capture tools.
Advantages
Motion capture offers several advantages over traditional
computer animation
Computer animation is the process used for digitally generating animations. The more general term computer-generated imagery (CGI) encompasses both static scenes (still images) and dynamic images (moving images), while computer animation refe ...
of a 3D model:
* Low latency, close to real time, results can be obtained. In entertainment applications this can reduce the costs of keyframe-based
animation
Animation is a method by which image, still figures are manipulated to appear as Motion picture, moving images. In traditional animation, images are drawn or painted by hand on transparent cel, celluloid sheets to be photographed and exhibited ...
.
The
Hand Over technique is an example of this.
* The amount of work does not vary with the complexity or length of the performance to the same degree as when using traditional techniques. This allows many tests to be done with different styles or deliveries, giving a different personality only limited by the talent of the actor.
* Complex movement and realistic physical interactions such as
secondary motion Secondary animation also known as secondary motion, is flat motions generated as a reaction to the movement of primary motion by a character. It is significant in animation because it amplifies the character's motion via effects that appear to be dr ...
s, weight and exchange of forces can be easily recreated in a physically accurate manner.
* The amount of animation data that can be produced within a given time is extremely large when compared to traditional animation techniques. This contributes to both cost effectiveness and meeting production deadlines.
*Potential for free software and third party solutions reducing its costs.
Disadvantages
* Specific hardware and special software programs are required to obtain and process the data.
* The cost of the software, equipment and personnel required can be prohibitive for small productions.
* The capture system may have specific requirements for the space in which it is operated, depending on camera field of view or magnetic distortion.
* When problems occur, it is easier to shoot the scene again rather than trying to manipulate the data. Only a few systems allow real-time viewing of the data to decide if the take needs to be redone.
* The initial results are limited to what can be performed within the capture volume without extra editing of the data.
* Movement that does not follow the laws of physics cannot be captured.
* Traditional animation techniques, such as added emphasis on anticipation and follow through, secondary motion or manipulating the shape of the character, as with
squash and stretch
Squash and stretch is the phrase used to describe "by far the most important" of the 12 basic principles of animation, described in the book ''The Illusion of Life'' by Frank Thomas and Ollie Johnston.
Basis
The principle is based on observatio ...
animation techniques, must be added later.
* If the computer model has different proportions from the capture subject, artifacts may occur. For example, if a cartoon character has large, oversized hands, these may intersect the character's body if the human performer is not careful with his or her physical motion.
Applications
Video games
Video games, also known as computer games, are electronic games that involves interaction with a user interface or input device such as a joystick, game controller, controller, computer keyboard, keyboard, or motion sensing device to gener ...
often use motion capture to animate athletes,
martial artists
Martial arts are codified systems and traditions of combat practiced for a number of reasons such as self-defense; military and law enforcement applications; competition; physical, mental, and spiritual development; entertainment; and the preserv ...
, and other in-game characters.
As early as 1988, an early form of motion capture was used to animate the
2D player characters
A player character (also known as a playable character or PC) is a fictional character in a video game or tabletop role-playing game whose actions are controlled by a player rather than the rules of the game. The characters that are not control ...
of
Martech
Martech Games was an early video game publisher based in Pevensey Bay between 1982 and 1989. It published a number of successful video games for the emerging home computer games marketplace, including BBC Micro, BBC Model B, ZX81, Sinclair ZX81, Z ...
's video game ''
Vixen
Foxes are small to medium-sized, omnivorous mammals belonging to several genera of the family Canidae. They have a flattened skull, upright, triangular ears, a pointed, slightly upturned snout, and a long bushy tail (or ''brush'').
Twelve sp ...
'' (performed by model
Corinne Russell
Corinne Russell (born 22 November 1963) is an English former Page 3 Girl, glamour model and dancer during the 1980s.
Modelling career
Russell made her Page 3 debut in ''The Sun'' on 22 August 1982, and first appearing in the '' Daily Star'' on ...
) and
Magical Company
, also known as Mahō, is a Japanese entertainment company.
History
Established in Kobe in 1983 to design and develop video games, the company was incorporated on May 29, 1985 as Home Data. During the 80's they developed and published various ...
's 2D arcade
fighting game
A fighting game, also known as a versus fighting game, is a video game genre, genre of video game that involves combat between two or more players. Fighting game combat often features mechanics such as Blocking (martial arts), blocking, grappli ...
''Last Apostle Puppet Show'' (to animate digitized
sprites). Motion capture was later notably used to animate the
3D character models in the
Sega Model arcade games
An arcade game or coin-op game is a coin-operated entertainment machine typically installed in public businesses such as restaurants, bars and amusement arcades. Most arcade games are presented as primarily games of skill and include arcade v ...
''
Virtua Fighter
is a series of fighting games created by Sega-AM2 and designer Yu Suzuki. The original ''Virtua Fighter'' was released in October 1993 and has received four main sequels and several spin-offs. The highly influential first ''Virtua Fighter'' game ...
'' (1993)
and ''
Virtua Fighter 2
is a 1994 fighting game, fighting video game developed by Sega. It is the sequel to 1993's ''Virtua Fighter (video game), Virtua Fighter'' and the second game in the ''Virtua Fighter'' series. It was created by Sega's Yu Suzuki-headed Sega AM2, ...
'' (1994). In mid-1995, developer/publisher
Acclaim Entertainment
Acclaim Entertainment, Inc. was an American video game publisher based in Glen Cove, New York. Originally formed by Greg Fischbach, Robert Holmes and Jim Scoroposki out of an Oyster Bay storefront in 1987, the company established a worldwide d ...
had its own in-house motion capture studio built into its headquarters.
Namco
was a Japanese multinational corporation, multinational video game and entertainment company, headquartered in Ōta, Tokyo. It held several international branches, including Namco America in Santa Clara, California, Namco Europe in London, Na ...
's 1995 arcade game ''
Soul Edge
is a fighting game developed by the Namco team Project Soul and published by Namco as the first installment in the ''Soulcalibur'' series of 3D fighting games. Originally released as an arcade game in December 1995, an upgraded and expanded versi ...
'' used passive optical system markers for motion capture. Motion capture also uses athletes in based-off animated games, such as
Naughty Dog
Naughty Dog, LLC (formerly JAM Software, Inc.) is an American first-party video game developer based in Santa Monica, California. Founded by Andy Gavin and Jason Rubin in 1984, the studio was acquired by Sony Computer Entertainment in 2001. Gav ...
's
Crash Bandicoot
''Crash Bandicoot'' is a video game franchise originally developed by Naughty Dog as an exclusive for Sony's PlayStation console. It has seen numerous installments created by various developers and published on multiple platforms. The series c ...
,
Insomniac Games
Insomniac Games, Inc. is an American video game developer based in Burbank, California and a studio of PlayStation Studios. It was founded in 1994 by Ted Price as Xtreme Software, and was renamed Insomniac Games a year later. The company is mo ...
'
Spyro the Dragon
''Spyro the Dragon'' is a platform game developed by Insomniac Games and published by Sony Computer Entertainment for the PlayStation on September 9, 1998. The first game in the ''Spyro'' series, it stars the title character, a young purple dr ...
, and
Rare's
Dinosaur Planet.
Movies use motion capture for CG effects, in some cases replacing traditional cel animation, and for completely
computer-generated creatures, such as
Gollum
Gollum is a fictional Tolkien's monsters, character in J. R. R. Tolkien's Middle-earth legendarium. He was introduced in the 1937 Fantasy (genre), fantasy novel ''The Hobbit'', and became important in its sequel, ''The Lord of the Rings''. Gol ...
,
The Mummy
A mummy is an unusually well preserved corpse.
Mummy or The Mummy may also refer to:
Places
*Mummy Range, a mountain range in the Rocky Mountains of northern Colorado in the United States
*Mummy Cave, a rock shelter and archeological site in Par ...
,
King Kong
King Kong is a fictional giant monster resembling a gorilla, who has appeared in various media since 1933. He has been dubbed The Eighth Wonder of the World, a phrase commonly used within the franchise. His first appearance was in the novelizat ...
,
Davy Jones from ''
Pirates of the Caribbean
''Pirates of the Caribbean'' is a Disney media franchise encompassing numerous theme park rides, a series of films, and spin-off novels, as well as a number of related video games and other media publications. The franchise originated with th ...
'', the
Na'vi
The Pandoran biosphere is a fictional habitat introduced in James Cameron's 2009 science fiction film ''Avatar''. The ecology of the lush moon Pandora, which teems with a biodiversity of bioluminescent species ranging from hexapodal animals to ot ...
from the film
''Avatar'', and Clu from ''
Tron: Legacy''. The Great Goblin, the three
Stone-trolls, many of the orcs and goblins in the 2012 film ''
The Hobbit: An Unexpected Journey'', and
Smaug
Smaug () is a dragon and the main antagonist in J. R. R. Tolkien's 1937 novel ''The Hobbit'', his treasure and the mountain he lives in being the goal of the quest. Powerful and fearsome, he invaded the Dwarf kingdom of Erebor 150 years prior ...
were created using motion capture.
The film ''
Batman Forever
''Batman Forever'' (on-screen title is simply ''Forever'') is a 1995 American superhero film directed by Joel Schumacher and produced by Tim Burton, based on the DC Comics character Batman by Bob Kane and Bill Finger. The third installment of W ...
'' (1995) used some motion capture for certain special effects.
Warner Bros.
Warner Bros. Entertainment Inc. (commonly known as Warner Bros. or abbreviated as WB) is an American film and entertainment studio headquartered at the Warner Bros. Studios complex in Burbank, California, and a subsidiary of Warner Bros. D ...
had acquired motion capture technology from
arcade video game
An arcade video game takes player input from its controls, processes it through electrical or computerized components, and displays output to an electronic monitor or similar display. Most arcade video games are coin-operated, housed in an arca ...
company Acclaim Entertainment for use in the film's production. Acclaim's 1995
video game of the same name also used the same motion capture technology to animate the digitized
sprite graphics.
''
Star Wars: Episode I – The Phantom Menace'' (1999) was the first feature-length film to include a main character created using motion capture (that character being
Jar Jar Binks
Jar Jar Binks is a fictional character of the Gungan race from the ''Star Wars'' saga created by George Lucas. Jar Jar appears throughout the ''Star Wars'' prequel trilogy – as a major character in '' Episode I: The Phantom Menace'', with a ...
, played by
Ahmed Best
Ahmed Best (born August 19, 1973) is an American actor, comedian and musician. He is known for providing motion capture and the voice of the character of Jar Jar Binks in the '' Star Wars'' franchise.
Best likewise collaborated with director G ...
), and
India
India, officially the Republic of India (Hindi: ), is a country in South Asia. It is the seventh-largest country by area, the second-most populous country, and the most populous democracy in the world. Bounded by the Indian Ocean on the so ...
n-
American
American(s) may refer to:
* American, something of, from, or related to the United States of America, commonly known as the "United States" or "America"
** Americans, citizens and nationals of the United States of America
** American ancestry, pe ...
film ''
Sinbad: Beyond the Veil of Mists'' (2000) was the first feature-length film made primarily with motion capture, although many character animators also worked on the film, which had a very limited release. 2001's ''
Final Fantasy: The Spirits Within'' was the first widely released movie to be made primarily with motion capture technology. Despite its poor box-office intake, supporters of motion capture technology took notice. ''
Total Recall'' had already used the technique, in the scene of the x-ray scanner and the skeletons.
''
The Lord of the Rings: The Two Towers'' was the first feature film to utilize a real-time motion capture system. This method streamed the actions of actor
Andy Serkis
Andrew Clement Serkis (born 20 April 1964) is an English actor, director, and producer. He is best known for his performance capture roles comprising motion capture acting, animation, and voice work for computer-generated characters such as Goll ...
into the computer generated skin of Gollum / Smeagol as it was being performed.
Storymind Entertainment, which is an independent
Ukrainian
Ukrainian may refer to:
* Something of, from, or related to Ukraine
* Something relating to Ukrainians, an East Slavic people from Eastern Europe
* Something relating to demographics of Ukraine in terms of demography and population of Ukraine
* So ...
studio, created a
neo-noir
Neo-noir is a revival of film noir, a genre that had originally flourished during the post-World War II era in the United Statesroughly from 1940 to 1960. The French term, ''film noir'', translates literally to English as "black film", indicating ...
third-person / shooter video game called ''
My Eyes On You,'' using motion capture in order to animate its main character, Jordan Adalien, and along with non-playable characters.
Out of the three nominees for the 2006
Academy Award for Best Animated Feature
The Academy Award for Best Animated Feature is given each year for animated films. An animated feature is defined by the Academy as a film with a running time of more than 40 minutes in which characters' performances are created using a frame-by- ...
, two of the nominees (''
Monster House'' and the winner ''
Happy Feet
''Happy Feet'' is a 2006 computer-animated jukebox musical comedy film directed, produced, and co-written by George Miller. It stars the voices of Elijah Wood, Robin Williams, Brittany Murphy, Hugh Jackman, Nicole Kidman, Hugo Weaving, and E.G ...
'') used motion capture, and only
Disney
The Walt Disney Company, commonly known as Disney (), is an American multinational mass media and entertainment conglomerate headquartered at the Walt Disney Studios complex in Burbank, California. Disney was originally founded on October ...
·
Pixar
Pixar Animation Studios (commonly known as Pixar () and stylized as P I X A R) is an American computer animation studio known for its critically and commercially successful computer animated feature films. It is based in Emeryville, Californi ...
's ''
Cars
A car or automobile is a motor vehicle with wheels. Most definitions of ''cars'' say that they run primarily on roads, seat one to eight people, have four wheels, and mainly transport people instead of goods.
The year 1886 is regarded as t ...
'' was animated without motion capture. In the ending credits of
Pixar
Pixar Animation Studios (commonly known as Pixar () and stylized as P I X A R) is an American computer animation studio known for its critically and commercially successful computer animated feature films. It is based in Emeryville, Californi ...
's film ''
Ratatouille
Ratatouille ( , ), oc, ratatolha , is a French Provençal dish of stewed vegetables which originated in Nice, and is sometimes referred to as ''ratatouille niçoise'' (). Recipes and cooking times differ widely, but common ingredients include ...
'', a stamp appears labelling the film as "100% Pure Animation – No Motion Capture!"
Since 2001, motion capture is being used extensively to simulate or approximate the look of live-action cinema, with nearly
photorealistic
Photorealism is a genre of art that encompasses painting, drawing and other graphic media, in which an artist studies a photograph and then attempts to reproduce the image as realistically as possible in another medium. Although the term can be ...
digital character models. ''
The Polar Express
''The Polar Express'' is a children's book written and illustrated by Chris Van Allsburg and published by Houghton Mifflin in 1985. The book is now widely considered to be a classic Christmas story for young children. It was praised for its detai ...
'' used motion capture to allow
Tom Hanks
Thomas Jeffrey Hanks (born July 9, 1956) is an American actor and filmmaker. Known for both his comedic and dramatic roles, he is one of the most popular and recognizable film stars worldwide, and is regarded as an American cultural icon. Ha ...
to perform as several distinct digital characters (in which he also provided the voices). The 2007 adaptation of the saga ''
Beowulf
''Beowulf'' (; ang, Bēowulf ) is an Old English epic poem in the tradition of Germanic heroic legend consisting of 3,182 alliterative lines. It is one of the most important and most often translated works of Old English literature. The ...
'' animated digital characters whose appearances were based in part on the actors who provided their motions and voices. James Cameron's highly popular ''
Avatar
Avatar (, ; ), is a concept within Hinduism that in Sanskrit literally means "descent". It signifies the material appearance or incarnation of a powerful deity, goddess or spirit on Earth. The relative verb to "alight, to make one's appearanc ...
'' used this technique to create the Na'vi that inhabit Pandora.
The Walt Disney Company
The Walt Disney Company, commonly known as Disney (), is an American multinational mass media and entertainment conglomerate headquartered at the Walt Disney Studios complex in Burbank, California. Disney was originally founded on October ...
has produced
Robert Zemeckis
Robert Lee Zemeckis (born May 14, 1952) is an American filmmaker. He first came to public attention as the director of the action-adventure romantic comedy ''Romancing the Stone'' (1984), the science-fiction comedy ''Back to the Future'' film tr ...
's ''
A Christmas Carol
''A Christmas Carol. In Prose. Being a Ghost Story of Christmas'', commonly known as ''A Christmas Carol'', is a novella by Charles Dickens, first published in London by Chapman & Hall in 1843 and illustrated by John Leech. ''A Christmas C ...
'' using this technique. In 2007, Disney acquired Zemeckis'
ImageMovers Digital
ImageMovers (IM), known as South Side Amusement Company until 1997, is an American production company which produces CGI animation, motion-capture, live-action films and television shows. The company is known for producing such films as ''Cast A ...
(that produces motion capture films), but then closed it in 2011, after a
box office failure
A box-office bomb, or box-office disaster, is a film that is unprofitable or considered highly unsuccessful during its theatrical run. Although any film for which the production, marketing, and distribution costs combined exceed the revenue after ...
of
Mars Needs Moms
''Mars Needs Moms'' is a 2011 American 3D computer-animated science fiction film co-written and directed by Simon Wells, produced by ImageMovers Digital and released by Walt Disney Pictures. Based on the Berkeley Breathed book of the same ti ...
.
Television series produced entirely with motion capture animation include ''
Laflaque'' in Canada, ''
Sprookjesboom'' and ' in The Netherlands, and ''
Headcases
''Headcases'' is a British satirical animation show based on current affairs. It employed the same satirical style as ''Spitting Image'', ''2DTV'' and '' Bo' Selecta!'', but using 3D animation created by UK Visual Effects and animation house Red ...
'' in the UK.
Virtual reality
Virtual reality (VR) is a simulated experience that employs pose tracking and 3D near-eye displays to give the user an immersive feel of a virtual world. Applications of virtual reality include entertainment (particularly video games), educ ...
and
Augmented reality
Augmented reality (AR) is an interactive experience that combines the real world and computer-generated content. The content can span multiple sensory modalities, including visual, auditory, haptic, somatosensory and olfactory. AR can be de ...
providers, such as
uSens
uSens, Inc. is a Silicon Valley startup founded in 2014 in San Jose, California. The company's core team includes researchers and developers building interactive and immersive computer-vision tracking solutions. The team has extensive experience ...
and
Gestigon
Gestigon (stylized as gestigon) is a software development company founded in September 2011 by Sascha Klement, Erhardt Barth, and Thomas Martinetz. The company develops software for gesture control and body tracking based on 3D depth data.
, ...
, allow users to interact with digital content in real time by capturing hand motions. This can be useful for training simulations, visual perception tests, or performing a virtual walk-throughs in a 3D environment. Motion capture technology is frequently used in
digital puppetry
Digital puppetry is the manipulation and performance of digitally animated 2D or 3D figures and objects in a virtual environment that are rendered in real time by computers. It is most commonly used in filmmaking and television production, but has ...
systems to drive computer generated characters in real-time.
Gait analysis
Gait analysis is the systematic study of animal locomotion, more specifically the study of human motion, using the eye and the brain of observers, augmented by instrumentation for measuring body movements, body mechanics, and the activity of the ...
is one application of motion capture in
clinical medicine
Medicine is the science and practice of caring for a patient, managing the diagnosis, prognosis, prevention, treatment, palliation of their injury or disease, and promoting their health. Medicine encompasses a variety of health care practice ...
. Techniques allow clinicians to evaluate human motion across several biomechanical factors, often while streaming this information live into analytical software.
Some physical therapy clinics utilize motion capture as an objective way to quantify patient progress.
During the filming of James Cameron's
''Avatar'' all of the scenes involving this process were directed in realtime using
Autodesk MotionBuilder software to render a screen image which allowed the director and the actor to see what they would look like in the movie, making it easier to direct the movie as it would be seen by the viewer. This method allowed views and angles not possible from a pre-rendered animation. Cameron was so proud of his results that he invited
Steven Spielberg
Steven Allan Spielberg (; born December 18, 1946) is an American director, writer, and producer. A major figure of the New Hollywood era and pioneer of the modern blockbuster, he is the most commercially successful director of all time. Spie ...
and
George Lucas
George Walton Lucas Jr. (born May 14, 1944) is an American filmmaker. Lucas is best known for creating the ''Star Wars'' and ''Indiana Jones'' franchises and founding Lucasfilm, LucasArts, Industrial Light & Magic and THX. He served as chairm ...
on set to view the system in action.
In Marvel's ''
The Avengers'', Mark Ruffalo used motion capture so he could play his character
the Hulk
The Hulk is a superhero appearing in American comic books published by Marvel Comics. Created by writer Stan Lee and artist Jack Kirby, the character first appeared in the debut issue of ''The Incredible Hulk'' (May 1962). In his comic book ...
, rather than have him be only CGI as in previous films, making Ruffalo the first actor to play both the human and the Hulk versions of Bruce Banner.
FaceRig software uses facial recognition technology from ULSee.Inc to map a player's facial expressions and the body tracking technology from Perception Neuron to map the body movement onto a 3D or 2D character's motion onscreen.
During ''
Game Developers Conference
The Game Developers Conference (GDC) is an annual conference for video game developers. The event includes an expo, networking events, and awards shows like the Game Developers Choice Awards and Independent Games Festival, and a variety of tutori ...
'' 2016 in San Francisco ''
Epic Games
Epic Games, Inc. is an American video game and software developer and publisher based in Cary, North Carolina. The company was founded by Tim Sweeney as Potomac Computer Systems in 1991, originally located in his parents' house in Potomac, M ...
'' demonstrated full-body motion capture live in Unreal Engine. The whole scene, from the upcoming game ''
Hellblade'' about a woman warrior named Senua, was rendered in real-time. The keynote was a collaboration between ''
Unreal Engine
Unreal Engine (UE) is a 3D computer graphics game engine developed by Epic Games, first showcased in the 1998 first-person shooter game ''Unreal''. Initially developed for PC first-person shooters, it has since been used in a variety of genres ...
'', ''
Ninja Theory
Ninja Theory Limited is a first-party British video game development studio based in Cambridge, England. Notable games it has developed include '' Kung Fu Chaos'', ''Heavenly Sword'', '' Enslaved: Odyssey to the West'', '' DmC: Devil May Cry ...
'', ''
3Lateral'', ''Cubic Motion'', ''IKinema'' and ''
Xsens
Xsens Technologies B.V. (or Xsens) is a supplier of 3D motion capture products, wearable sensors and inertial sensors based upon miniature MEMS inertial sensor technology. The company has created intellectual property in the field of multi-senso ...
''.
Indian film ''
Adipurush'' based on Ramayana. The film is said to be a magnum opus using high-end and real-time technology such as Xsens motion capture and facial capture used by Hollywood to bring the world of Adipurush to life. Adipurush is the story of Lord Ram.
Methods and systems
![Silhouette tracking](https://upload.wikimedia.org/wikipedia/commons/a/ab/Silhouette_tracking.PNG)
Motion tracking or motion capture started as a photogrammetric analysis tool in biomechanics research in the 1970s and 1980s, and expanded into education, training, sports and recently
computer animation
Computer animation is the process used for digitally generating animations. The more general term computer-generated imagery (CGI) encompasses both static scenes (still images) and dynamic images (moving images), while computer animation refe ...
for
television
Television, sometimes shortened to TV, is a telecommunication medium for transmitting moving images and sound. The term can refer to a television set, or the medium of television transmission. Television is a mass medium for advertisin ...
,
cinema
Cinema may refer to:
Film
* Cinematography, the art of motion-picture photography
* Film or movie, a series of still images that create the illusion of a moving image
** Film industry, the technological and commercial institutions of filmmaking
...
, and
video game
Video games, also known as computer games, are electronic games that involves interaction with a user interface or input device such as a joystick, controller, keyboard, or motion sensing device to generate visual feedback. This fee ...
s as the technology matured. Since the 20th century the performer has to wear markers near each joint to identify the motion by the positions or angles between the markers. Acoustic, inertial,
LED
A light-emitting diode (LED) is a semiconductor Electronics, device that Light#Light sources, emits light when Electric current, current flows through it. Electrons in the semiconductor recombine with electron holes, releasing energy i ...
, magnetic or reflective markers, or combinations of any of these, are tracked, optimally at least two times the frequency rate of the desired motion. The resolution of the system is important in both the spatial resolution and temporal resolution as motion blur causes almost the same problems as low resolution. Since the beginning of the 21st century and because of the rapid growth of technology new methods were developed. Most modern systems can extract the silhouette of the performer from the background. Afterwards all joint angles are calculated by fitting in a mathematic model into the silhouette. For movements you can't see a change of the silhouette, there are hybrid Systems available who can do both (marker and silhouette), but with less marker. In robotics, some motion capture systems are based on
simultaneous localization and mapping
Simultaneous localization and mapping (SLAM) is the computational problem of constructing or updating a map of an unknown environment while simultaneously keeping track of an agent's location within it. While this initially appears to be a chick ...
.
Optical systems
''Optical systems'' utilize data captured from image sensors to
triangulate
In trigonometry and geometry, triangulation is the process of determining the location of a point by forming triangles to the point from known points.
Applications
In surveying
Specifically in surveying, triangulation involves only angle me ...
the 3D position of a subject between two or more cameras calibrated to provide overlapping projections. Data acquisition is traditionally implemented using special markers attached to an actor; however, more recent systems are able to generate accurate data by tracking surface features identified dynamically for each particular subject. Tracking a large number of performers or expanding the capture area is accomplished by the addition of more cameras. These systems produce data with three degrees of freedom for each marker, and rotational information must be inferred from the relative orientation of three or more markers; for instance shoulder, elbow and wrist markers providing the angle of the elbow. Newer hybrid systems are combining inertial sensors with optical sensors to reduce occlusion, increase the number of users and improve the ability to track without having to manually clean up data.
Passive markers
![Motion capture facial](https://upload.wikimedia.org/wikipedia/commons/9/95/Motion_capture_facial.jpg)
''Passive optical'' systems use markers coated with a
retroreflective
A retroreflector (sometimes called a retroflector or cataphote) is a device or surface that reflects radiation (usually light) back to its source with minimum scattering. This works at a wide range of angle of incidence, unlike a planar mirror, ...
material to reflect light that is generated near the cameras lens. The camera's threshold can be adjusted so only the bright reflective markers will be sampled, ignoring skin and fabric.
The centroid of the marker is estimated as a position within the two-dimensional image that is captured. The grayscale value of each pixel can be used to provide sub-pixel accuracy by finding the centroid of the
Gaussian
Carl Friedrich Gauss (1777–1855) is the eponym of all of the topics listed below.
There are over 100 topics all named after this German mathematician and scientist, all in the fields of mathematics, physics, and astronomy. The English eponymo ...
.
An object with markers attached at known positions is used to calibrate the cameras and obtain their positions and the lens distortion of each camera is measured. If two calibrated cameras see a marker, a three-dimensional fix can be obtained. Typically a system will consist of around 2 to 48 cameras. Systems of over three hundred cameras exist to try to reduce marker swap. Extra cameras are required for full coverage around the capture subject and multiple subjects.
Vendors have constraint software to reduce the problem of marker swapping since all passive markers appear identical. Unlike active marker systems and magnetic systems, passive systems do not require the user to wear wires or electronic equipment. Instead, hundreds of rubber balls are attached with reflective tape, which needs to be replaced periodically. The markers are usually attached directly to the skin (as in biomechanics), or they are
velcro
Velcro, officially known as Velcro IP Holdings LLC and trading as Velcro Companies, is a British privately held company, founded by Swiss electrical engineer George de Mestral in the 1950s. It is the original manufacturer of hook-and-loop fasten ...
ed to a performer wearing a full-body spandex/lycra
suit designed specifically for motion capture. This type of system can capture large numbers of markers at frame rates usually around 120 to 160 fps although by lowering the resolution and tracking a smaller region of interest they can track as high as 10,000 fps.
Active marker
![Body_Motion_Capture](https://upload.wikimedia.org/wikipedia/commons/5/50/Body_Motion_Capture.jpg)
Active optical systems triangulate positions by illuminating one LED at a time very quickly or multiple LEDs with software to identify them by their relative positions, somewhat akin to celestial navigation. Rather than reflecting light back that is generated externally, the markers themselves are powered to emit their own light. Since inverse square law provides one quarter the power at two times the distance, this can increase the distances and volume for capture. This also enables high signal-to-noise ratio, resulting in very low marker jitter and a resulting high measurement resolution (often down to 0.1 mm within the calibrated volume).
The TV series ''
Stargate SG1'' produced episodes using an active optical system for the VFX allowing the actor to walk around props that would make motion capture difficult for other non-active optical systems.
ILM used active markers in ''
Van Helsing
A van is a type of road vehicle used for transporting goods or people. Depending on the type of van, it can be bigger or smaller than a pickup truck and SUV, and bigger than a common car. There is some varying in the scope of the word across th ...
'' to allow capture of Dracula's flying brides on very large sets similar to Weta's use of active markers in ''
Rise of the Planet of the Apes
''Rise of the Planet of the Apes'' is a 2011 American science fiction film directed by Rupert Wyatt from a screenplay by Rick Jaffa and Amanda Silver. It is a reboot of the ''Planet of the Apes'' film franchise, which is based on the 1963 nov ...
''. The power to each marker can be provided sequentially in phase with the capture system providing a unique identification of each marker for a given capture frame at a cost to the resultant frame rate. The ability to identify each marker in this manner is useful in realtime applications. The alternative method of identifying markers is to do it algorithmically requiring extra processing of the data.
There are also possibilities to find the position by using coloured LED markers. In these systems, each colour is assigned to a specific point of the body.
One of the earliest active marker systems in the 1980s was a hybrid passive-active mocap system with rotating mirrors and colored glass reflective markers and which used masked linear array detectors.
Time modulated active marker
Active marker systems can further be refined by strobing one marker on at a time, or tracking multiple markers over time and modulating the amplitude or pulse width to provide marker ID. 12 megapixel spatial resolution modulated systems show more subtle movements than 4 megapixel optical systems by having both higher spatial and temporal resolution. Directors can see the actors performance in real time, and watch the results on the motion capture driven CG character. The unique marker IDs reduce the turnaround, by eliminating marker swapping and providing much cleaner data than other technologies. LEDs with onboard processing and a radio synchronization allow motion capture outdoors in direct sunlight, while capturing at 120 to 960 frames per second due to a high speed electronic shutter. Computer processing of modulated IDs allows less hand cleanup or filtered results for lower operational costs. This higher accuracy and resolution requires more processing than passive technologies, but the additional processing is done at the camera to improve resolution via a subpixel or centroid processing, providing both high resolution and high speed. These motion capture systems are typically $20,000 for an eight camera, 12 megapixel spatial resolution 120 hertz system with one actor.
Semi-passive imperceptible marker
One can reverse the traditional approach based on high speed cameras. Systems such a
Prakashuse inexpensive multi-LED high speed projectors. The specially built multi-LED IR projectors optically encode the space. Instead of retro-reflective or active light emitting diode (LED) markers, the system uses photosensitive marker tags to decode the optical signals. By attaching tags with photo sensors to scene points, the tags can compute not only their own locations of each point, but also their own orientation, incident illumination, and reflectance.
These tracking tags work in natural lighting conditions and can be imperceptibly embedded in attire or other objects. The system supports an unlimited number of tags in a scene, with each tag uniquely identified to eliminate marker reacquisition issues. Since the system eliminates a high speed camera and the corresponding high-speed image stream, it requires significantly lower data bandwidth. The tags also provide incident illumination data which can be used to match scene lighting when inserting synthetic elements. The technique appears ideal for on-set motion capture or real-time broadcasting of virtual sets but has yet to be proven.
Underwater motion capture system
Motion capture technology has been available for researchers and scientists for a few decades, which has given new insight into many fields.
Underwater cameras
The vital part of the system, the underwater camera, has a waterproof housing. The housing has a finish that withstands corrosion and chlorine which makes it perfect for use in basins and swimming pools. There are two types of cameras. Industrial high-speed-cameras can also be used as infrared cameras. The infrared underwater cameras comes with a cyan light strobe instead of the typical IR light—for minimum falloff under water and the high-speed-cameras cone with an LED light or with the option of using image processing.
=Measurement volume
=
An underwater camera is typically able to measure 15–20 meters depending on the water quality, the camera and the type of marker used. Unsurprisingly, the best range is achieved when the water is clear, and like always, the measurement volume is also dependent on the number of cameras. A range of underwater markers are available for different circumstances.
=Tailored
=
Different pools require different mountings and fixtures. Therefore, all underwater motion capture systems are uniquely tailored to suit each specific pool installment. For cameras placed in the center of the pool, specially designed tripods, using suction cups, are provided.
Markerless
Emerging techniques and research in
computer vision
Computer vision is an interdisciplinary scientific field that deals with how computers can gain high-level understanding from digital images or videos. From the perspective of engineering, it seeks to understand and automate tasks that the hum ...
are leading to the rapid development of the markerless approach to motion capture. Markerless systems such as those developed at
Stanford University
Stanford University, officially Leland Stanford Junior University, is a private research university in Stanford, California. The campus occupies , among the largest in the United States, and enrolls over 17,000 students. Stanford is consider ...
, the
University of Maryland
The University of Maryland, College Park (University of Maryland, UMD, or simply Maryland) is a public land-grant research university in College Park, Maryland. Founded in 1856, UMD is the flagship institution of the University System of M ...
,
MIT
The Massachusetts Institute of Technology (MIT) is a private land-grant research university in Cambridge, Massachusetts. Established in 1861, MIT has played a key role in the development of modern technology and science, and is one of the m ...
, and the
Max Planck Institute
Max or MAX may refer to:
Animals
* Max (dog) (1983–2013), at one time purported to be the world's oldest living dog
* Max (English Springer Spaniel), the first pet dog to win the PDSA Order of Merit (animal equivalent of OBE)
* Max (gorilla) (1 ...
, do not require subjects to wear special equipment for tracking. Special computer algorithms are designed to allow the system to analyze multiple streams of optical input and identify human forms, breaking them down into constituent parts for tracking.
ESC entertainment, a subsidiary of
Warner Brothers Pictures
Warner Bros. Entertainment Inc. (commonly known as Warner Bros. or abbreviated as WB) is an American film and entertainment studio headquartered at the Warner Bros. Studios complex in Burbank, California, and a subsidiary of Warner Bros. D ...
created specially to enable
virtual cinematography
Virtual cinematography is the set of cinematographic techniques performed in a computer graphics environment. It includes a wide variety of subjects like photographing real objects, often with stereo or multi-camera setup, for the purpose of rec ...
, including
photorealistic
Photorealism is a genre of art that encompasses painting, drawing and other graphic media, in which an artist studies a photograph and then attempts to reproduce the image as realistically as possible in another medium. Although the term can be ...
digital look-alike
Human image synthesis is technology that can be applied to make believable and even photorealistic renditions of human-likenesses, moving or still. It has effectively existed since the early 2000s. Many films using computer generated imagery ha ...
s for filming ''
The Matrix Reloaded
''The Matrix Reloaded'' is a 2003 American science-fiction action film written and directed by the Wachowskis. It is a sequel to ''The Matrix'' (1999) and the second installment in the ''Matrix'' film series. The film stars Keanu Reeves, Laure ...
'' and ''
The Matrix Revolutions
''The Matrix Revolutions'' is a 2003 American science fiction action film written and directed by the Wachowskis. It is the third installment in ''The Matrix'' film series, released six months following ''The Matrix Reloaded''. The film stars ...
'' movies, used a technique called Universal Capture that utilized
7 camera setup and the tracking the
optical flow
Optical flow or optic flow is the pattern of apparent motion of objects, surfaces, and edges in a visual scene caused by the relative motion between an observer and a scene. Optical flow can also be defined as the distribution of apparent veloci ...
of all
pixel
In digital imaging, a pixel (abbreviated px), pel, or picture element is the smallest addressable element in a raster image, or the smallest point in an all points addressable display device.
In most digital display devices, pixels are the smal ...
s over all the 2-D planes of the cameras for motion,
gesture
A gesture is a form of non-verbal communication or non-vocal communication in which visible bodily actions communicate particular messages, either in place of, or in conjunction with, speech. Gestures include movement of the hands, face, or ot ...
and
facial expression
A facial expression is one or more motions or positions of the muscles beneath the skin of the face. According to one set of controversial theories, these movements convey the emotional state of an individual to observers. Facial expressions are a ...
capture leading to photorealistic results.
Traditional systems
Traditionally markerless optical motion tracking is used to keep track on various objects, including airplanes, launch vehicles, missiles and satellites. Many of such optical motion tracking applications occur outdoors, requiring differing lens and camera configurations. High resolution images of the target being tracked can thereby provide more information than just motion data. The image obtained from NASA's long-range tracking system on space shuttle Challenger's fatal launch provided crucial evidence about the cause of the accident. Optical tracking systems are also used to identify known spacecraft and space debris despite the fact that it has a disadvantage compared to radar in that the objects must be reflecting or emitting sufficient light.
An optical tracking system typically consists of three subsystems: the optical imaging system, the mechanical tracking platform and the tracking computer.
The optical imaging system is responsible for converting the light from the target area into digital image that the tracking computer can process. Depending on the design of the optical tracking system, the optical imaging system can vary from as simple as a standard digital camera to as specialized as an astronomical telescope on the top of a mountain. The specification of the optical imaging system determines the upper-limit of the effective range of the tracking system.
The mechanical tracking platform holds the optical imaging system and is responsible for manipulating the optical imaging system in such a way that it always points to the target being tracked. The dynamics of the mechanical tracking platform combined with the optical imaging system determines the tracking system's ability to keep the lock on a target that changes speed rapidly.
The tracking computer is responsible for capturing the images from the optical imaging system, analyzing the image to extract target position and controlling the mechanical tracking platform to follow the target. There are several challenges. First the tracking computer has to be able to capture the image at a relatively high frame rate. This posts a requirement on the bandwidth of the image capturing hardware. The second challenge is that the image processing software has to be able to extract the target image from its background and calculate its position. Several textbook image processing algorithms are designed for this task. This problem can be simplified if the tracking system can expect certain characteristics that is common in all the targets it will track. The next problem down the line is to control the tracking platform to follow the target. This is a typical control system design problem rather than a challenge, which involves modeling the system dynamics and designing
controllers to control it. This will however become a challenge if the tracking platform the system has to work with is not designed for real-time.
The software that runs such systems are also customized for the corresponding hardware components. One example of such software is OpticTracker, which controls computerized telescopes to track moving objects at great distances, such as planes and satellites. Another option is the software SimiShape, which can also be used hybrid in combination with markers.
RGB-D Cameras
RGB-D cameras such as
kinect
Kinect is a line of motion sensing input devices produced by Microsoft and first released in 2010. The devices generally contain RGB cameras, and infrared projectors and detectors that map depth through either structured light or time of flig ...
captures both the color and depth images. By fusing the two images, 3D colored
voxel
In 3D computer graphics, a voxel represents a value on a regular grid in three-dimensional space. As with pixels in a 2D bitmap, voxels themselves do not typically have their position (i.e. coordinates) explicitly encoded with their values. Ins ...
can be captured, allowing motion capture of 3D human motion and human surface in real time.
Because of the use of a single-view camera, motions captured are usually noisy. Machine learning techniques have been proposed to automatically reconstruct such noisy motions into higher quality ones, using methods such as
lazy learning In machine learning, lazy learning is a learning method in which generalization of the training data is, in theory, delayed until a query is made to the system, as opposed to eager learning, where the system tries to generalize the training data bef ...
and
Gaussian
Carl Friedrich Gauss (1777–1855) is the eponym of all of the topics listed below.
There are over 100 topics all named after this German mathematician and scientist, all in the fields of mathematics, physics, and astronomy. The English eponymo ...
models. Such method generate accurate enough motion for serious applications like ergonomic assessment.
Non-optical systems
Inertial systems
Inertial motion capture technology is based on miniature inertial sensors, biomechanical models and
sensor fusion
Sensor fusion is the process of combining sensor data or data derived from disparate sources such that the resulting information has less uncertainty than would be possible when these sources were used individually. For instance, one could potentia ...
algorithms. The motion data of the inertial sensors (
inertial guidance system
An inertial navigation system (INS) is a navigation device that uses motion sensors (accelerometers), rotation sensors ( gyroscopes) and a computer to continuously calculate by dead reckoning the position, the orientation, and the velocity (dire ...
) is often transmitted wirelessly to a computer, where the motion is recorded or viewed. Most inertial systems use inertial measurement units (IMUs) containing a combination of gyroscope, magnetometer, and accelerometer, to measure rotational rates. These rotations are translated to a skeleton in the software. Much like optical markers, the more IMU sensors the more natural the data. No external cameras, emitters or markers are needed for relative motions, although they are required to give the absolute position of the user if desired. Inertial motion capture systems capture the full six degrees of freedom body motion of a human in real-time and can give limited direction information if they include a magnetic bearing sensor, although these are much lower resolution and susceptible to electromagnetic noise. Benefits of using Inertial systems include: capturing in a variety of environments including tight spaces, no solving, portability, and large capture areas. Disadvantages include lower positional accuracy and positional drift which can compound over time. These systems are similar to the Wii controllers but are more sensitive and have greater resolution and update rates. They can accurately measure the direction to the ground to within a degree. The popularity of inertial systems is rising amongst game developers,
mainly because of the quick and easy set up resulting in a fast pipeline. A range of suits are now available from various manufacturers and base prices range from $1,000 to US$80,000.
Mechanical motion
Mechanical motion capture systems directly track body joint angles and are often referred to as exoskeleton motion capture systems, due to the way the sensors are attached to the body. A performer attaches the skeletal-like structure to their body and as they move so do the articulated mechanical parts, measuring the performer's relative motion. Mechanical motion capture systems are real-time, relatively low-cost, free from occlusion, and wireless (untethered) systems that have unlimited capture volume. Typically, they are rigid structures of jointed, straight metal or plastic rods linked together with potentiometers that articulate at the joints of the body. These suits tend to be in the $25,000 to $75,000 range plus an external absolute positioning system. Some suits provide limited force feedback or
haptic input.
Magnetic systems
Magnetic systems calculate position and orientation by the relative magnetic flux of three orthogonal coils on both the transmitter and each receiver.
The relative intensity of the voltage or current of the three coils allows these systems to calculate both range and orientation by meticulously mapping the tracking volume. The sensor output is
6DOF
Six degrees of freedom (6DOF) refers to the six mechanical degrees of freedom of movement of a rigid body in three-dimensional space. Specifically, the body is free to change position as forward/backward (surge), up/down (heave), left/right ...
, which provides useful results obtained with two-thirds the number of markers required in optical systems; one on upper arm and one on lower arm for elbow position and angle. The markers are not occluded by nonmetallic objects but are susceptible to magnetic and electrical interference from metal objects in the environment, like rebar (steel reinforcing bars in concrete) or wiring, which affect the magnetic field, and electrical sources such as monitors, lights, cables and computers. The sensor response is nonlinear, especially toward edges of the capture area. The wiring from the sensors tends to preclude extreme performance movements.
With magnetic systems, it is possible to monitor the results of a motion capture session in real time.
The capture volumes for magnetic systems are dramatically smaller than they are for optical systems. With the magnetic systems, there is a distinction between
alternating-current
Alternating current (AC) is an electric current which periodically reverses direction and changes its magnitude continuously with time in contrast to direct current (DC) which flows only in one direction. Alternating current is the form in which ...
(AC) and
direct-current
Direct current (DC) is one-directional flow of electric charge. An electrochemical cell is a prime example of DC power. Direct current may flow through a conductor such as a wire, but can also flow through semiconductors, insulators, or even ...
(DC) systems: DC system uses square pulses, AC systems uses sine wave pulse.
Stretch sensors
Stretch sensors are flexible parallel plate capacitors that measure either stretch, bend, shear, or pressure and are typically produced from silicone. When the sensor stretches or squeezes its capacitance value changes. This data can be transmitted via Bluetooth or direct input and used to detect minute changes in body motion. Stretch sensors are unaffected by magnetic interference and are free from occlusion. The stretchable nature of the sensors also means they do not suffer from positional drift, which is common with inertial systems. Stretchable sensors, on the other hands, due to the material properties of their substrates and conducting materials, suffer from relatively high
signal-to-noise ratio
Signal-to-noise ratio (SNR or S/N) is a measure used in science and engineering that compares the level of a desired signal to the level of background noise. SNR is defined as the ratio of signal power to the noise power, often expressed in deci ...
, requiring
filtering
Filter, filtering or filters may refer to:
Science and technology
Computing
* Filter (higher-order function), in functional programming
* Filter (software), a computer program to process a data stream
* Filter (video), a software component tha ...
or
machine learning
Machine learning (ML) is a field of inquiry devoted to understanding and building methods that 'learn', that is, methods that leverage data to improve performance on some set of tasks. It is seen as a part of artificial intelligence.
Machine ...
to make them usable for motion capture. These solutions result in higher
latency when compared to alternative sensors.
Related techniques
Facial motion capture
Most traditional motion capture hardware vendors provide for some type of low resolution facial capture utilizing anywhere from 32 to 300 markers with either an active or passive marker system. All of these solutions are limited by the time it takes to apply the markers, calibrate the positions and process the data. Ultimately the technology also limits their resolution and raw output quality levels.
High fidelity facial motion capture, also known as performance capture, is the next generation of fidelity and is utilized to record the more complex movements in a human face in order to capture higher degrees of emotion. Facial capture is currently arranging itself in several distinct camps, including traditional motion capture data, blend shaped based solutions, capturing the actual topology of an actor's face, and proprietary systems.
The two main techniques are stationary systems with an array of cameras capturing the facial expressions from multiple angles and using software such as the stereo mesh solver from OpenCV to create a 3D surface mesh, or to use light arrays as well to calculate the surface normals from the variance in brightness as the light source, camera position or both are changed. These techniques tend to be only limited in feature resolution by the camera resolution, apparent object size and number of cameras. If the users face is 50 percent of the working area of the camera and a camera has megapixel resolution, then sub millimeter facial motions can be detected by comparing frames. Recent work is focusing on increasing the frame rates and doing optical flow to allow the motions to be retargeted to other computer generated faces, rather than just making a 3D Mesh of the actor and their expressions.
RF positioning
RF (radio frequency) positioning systems are becoming more viable as higher frequency RF devices allow greater precision than older RF technologies such as traditional
radar
Radar is a detection system that uses radio waves to determine the distance (''ranging''), angle, and radial velocity of objects relative to the site. It can be used to detect aircraft, ships, spacecraft, guided missiles, motor vehicles, w ...
. The speed of light is 30 centimeters per nanosecond (billionth of a second), so a 10 gigahertz (billion cycles per second) RF signal enables an accuracy of about 3 centimeters. By measuring amplitude to a quarter wavelength, it is possible to improve the resolution down to about 8 mm. To achieve the resolution of optical systems, frequencies of 50 gigahertz or higher are needed, which are almost as dependant on line of sight and as easy to block as optical systems. Multipath and reradiation of the signal are likely to cause additional problems, but these technologies will be ideal for tracking larger volumes with reasonable accuracy, since the required resolution at 100 meter distances is not likely to be as high. Many RF scientists believe that radio frequency will never produce the accuracy required for motion capture.
Researchers at Massachusetts Institute of Technology researchers said in 2015 that they had made a system that tracks motion by RF signals, called RF Tracking.
Non-traditional systems
An alternative approach was developed where the actor is given an unlimited walking area through the use of a rotating sphere, similar to a
hamster ball
Hamster balls are hollow spheres made of clear plastic into which hamsters, gerbils, degus and other small rodent pets are placed, allowing them to run around outside their cages without the risk of running away or getting lost under furniture. ...
, which contains internal sensors recording the angular movements, removing the need for external cameras and other equipment. Even though this technology could potentially lead to much lower costs for motion capture, the basic sphere is only capable of recording a single continuous direction. Additional sensors worn on the person would be needed to record anything more.
Another alternative is using a 6DOF (Degrees of freedom) motion platform with an integrated omni-directional treadmill with high resolution optical motion capture to achieve the same effect. The captured person can walk in an unlimited area, negotiating different uneven terrains. Applications include medical rehabilitation for balance training, bio-mechanical research and virtual reality.
3D pose estimation
In
3D pose estimation, an actor's pose can be reconstructed from an image or
depth map
In 3D computer graphics and computer vision, a depth map is an image or image channel that contains information relating to the distance of the surfaces of scene objects from a viewpoint. The term is related (and may be analogous) to ''depth ...
.
[Ye, Mao, et al.]
Accurate 3d pose estimation from a single depth image
" 2011 International Conference on Computer Vision. IEEE, 2011.
See also
*
Animation database
An animation database is a database which stores fragments of animations or human movements and which can be accessed, analyzed and queried to develop and assemble new animations.S. Kuroki, "Walkthrough using Animation database MOVE" in ''Database ...
*
Gesture recognition
Gesture recognition is a topic in computer science and language technology with the goal of interpreting human gestures via mathematical algorithms. It is a subdiscipline of computer vision. Gestures can originate from any bodily motion or sta ...
*
Finger tracking
In the field of gesture recognition and image processing, finger tracking is a high-resolution technique developed in 1969 that is employed to know the consecutive position of the fingers of the user and hence represent objects in 3D.
In additio ...
*
Inverse kinematics
In computer animation and robotics, inverse kinematics is the mathematical process of calculating the variable joint parameters needed to place the end of a kinematic chain, such as a robot manipulator or animation character's skeleton, in a g ...
(a different way of making CGI effects realistic)
*
Kinect
Kinect is a line of motion sensing input devices produced by Microsoft and first released in 2010. The devices generally contain RGB cameras, and infrared projectors and detectors that map depth through either structured light or time of flig ...
(created by Microsoft Corporation)
*
List of motion and gesture file formats
The question of gesture and Motion (physics), motion takes more and more importance with the development of gesture controllers, haptic technology, haptic systems, motion capture systems, etc., on the one hand, and with the need of allowing virtual ...
*
Motion capture acting
Motion-capture acting, also called performance-capture acting and often abbreviated as mo-cap or P-cap, is a type of acting in which an actor wears markers or sensors on a skintight bodysuit or directly on the skin. Hugh Hart, January 24, 2012, W ...
*
Video tracking
Video tracking is the process of locating a moving object (or multiple objects) over time using a camera. It has a variety of uses, some of which are: human-computer interaction, security and surveillance, video communication and compression, aug ...
*
VR positional tracking
In virtual reality (VR) and augmented reality (AR), a pose tracking system detects the precise pose of head-mounted displays, controllers, other objects or body parts within Euclidean space. Pose tracking is often referred to as 6DOF tracking, for ...
References
External links
The fascination for motion capture an introduction to the history of motion capture technology
{{DEFAULTSORT:Motion Capture
Computer-related introductions in 1994
Audiovisual introductions in 2000
Computer animation
3D computer graphics
Computing input devices
Articles containing video clips
Motion control photography