Since the early 2000’s, motion capture technology has significantly advanced in capabilities. These advancements enabled ground-breaking visual effects in major Hollywood movies such as Spider-man: No Way Home, and AAA games like Dead Space. 

Mocap has embedded itself as an integral and transformative element of the entertainment industry over the past few decades. From films to gaming, this boundary-pushing technology has brought digital characters and creatures to life, taking productions to a whole new level.

Mocap and the impact on film production 

Mocap integration on a film set is typically split into three main categories: object/body tracking, virtual camera, also known as complete CGI, and a hybrid combination. Along with turning the industry on its head, these new technologies have revolutionized the way cinematographers work. 

Virtual cameras, or complete CGI, might not actually be a camera at all. Rather, could be an object with markers on and a cinematographer with a monitor. The monitor is then used like it was the viewfinder of a camera and the cinematographer sees the CGI image. 

The beauty of the third category, using both of these methods collaboratively, in real-time is that it enables complete oversight, instant evaluation and then, implementation of adjustments. This empowers cinematographers with full control over what they’re shooting. They’re able to control not only framing, in terms of reactions and movements to and from animated objects or characters, but they can capture, or even design and create, the ‘perfect shot’. 

Mocap has also enabled far more interdepartmental interaction. Being able to work in real-time fosters an environment where decisions are made with the big picture in mind at all levels, not by siloed teams with limited understanding of others processes. Even better, this can be done remotely, removing any geographical restrictions.  

For example, prior to the rise of virtual production, the cinematography and post-production teams would have had very little need to interact, despite their inter-dependence, which led to a siloed mentality in production. The cinematographer needs to capture the shots suitable for VFX and virtual production, and the virtual production team needs to ensure the cinematographer is fully aware of their capabilities and can provide as much insight and context as possible on the desired outcome. 

Evolving workflows and new challenges

As transformational as mocap has been however, like all new technologies, there are challenges along with unrealised opportunities. For truly exceptional results there are certain other pieces of equipment and technology that are key, such as a greenscreen, or an LED wall. Also, although the reliability and accessibility of cameras has increased, high-end models and additions can be quite costly. 

Today’s motion capture technology brings a lot of creative freedom to the production process which opens up creative possibilities, but adjustments are needed to workflows. More control over different elements of productions means more variables can be accounted for, and more opportunity to realize creative visions. However, this takes planning and preparation; the devil is in the details. In order for this increased control to be an advantage to production, rather than an obstacle, there needs to be a shift in focus from on-set and in-action, to pre-planning. 

The ground-breaking capabilities of mocap has enabled the industry to meet the growing demand for new content. However, with the growing demand for volume is the growing demand for urgency, and this remains a key challenge. By using a hybrid approach, the time spent on-set and in post production has been significantly shortened, however, this time and effort is now focused on pre-production and planning.  

Cinematographers still need to contend with traditional challenges like choosing the correct lenses and frame rates to shoot in, but there are also new ones to consider and overcome. Spending more time on pre-planning by, for example,  plotting camera routes, means the cinematographer can ensure their shooting is exact. This is a collaborative effort between cinematography and virtual production and is needed to avoid a collision–both virtually and in real life–between the actors, objects, and background sets and landscapes, VFX and so on. Plotting camera routes and making exact markings beforehand is crucial to a smooth day on-set. 

Another challenge for cinematographers to overcome is understanding and adjusting to an LED wall’s capabilities. Some would use the phrase “limitations of the technology”. I don’t think that’s correct as all the cinematographers I’ve spoken to and work with see it as freedom, with little to no boundaries–just requirements, or challenges. 

One of the biggest game changers when it comes to this technology is lighting, while virtual production and mocap is truly unrivaled with the ability to amend the majority of lighting in real-time, achieving bright sunlight, or very dark shadows isn’t quite possible due to the intensity of light required.  

There are many advantages and benefits to virtual production and LED walls. Productions have no need to build sets and cinematographers can get the shots they need outdoors as easily as they can in the studio. A cinematographer working in-hand with the virtual production team can immerse themselves in the world they’re building and produce the perfect end result, regardless of location and without being dependent on the weather.

Mocap and game engines

In terms of being able to work in real-time, using big game engines such as the Unreal Engine has a huge impact on productions when combined with motion capture, moving outside the boundaries of the most obvious use cases in gaming and making their way into Hollywood productions. 

Big game engines are more popular than they’ve ever been, and with good reason. Initially, there were challenges around finding people who were able to use game engines professionally and from a production perspective. Now however, there’s far more education and free, open source software available which opens up the floor for a new generation of talent to come in and make the most out of the capabilities these engines offer. In some cases game engines are even being enveloped into teaching curriculums. 

The Matrix Awakens: An Unreal Engine 5 Experience demo, is a brilliant example of the convergence of both elements of mocap. It proves just how much can be achieved: cinematic quality content from a game engine. There are still limitations around using a game engine in a production environment, but with their popularity, the expansive user community is regularly providing suggestions and solutions which only leads to a better product and results. Motion capture technology is increasingly integrated into technological applications beyond the world of cinema production: from gaming to live performance and more. These broader use cases are therefore pushing the evolution of motion capture, which in turn inspires more sophisticated mocap application within film production. 

Cinematographers are now involved with their productions in ways not previously possible. There is an opportunity to develop and grow not only new skills, but new ways of looking at a project and finding better methods and techniques to deliver results. As this technology evolves and continues to advance, so will its challenges. But these challenges provide opportunities, and the technology overall helps remove restrictions and industry entry requirements such as budgets and resources.