The virtual world is a fascinating, fast-paced, and valuable space for filmmakers to explore for the right production. However, it is vital to ensure cinematographers’ voices are heard throughout the decision-making process and that they have access to the education and flexible, appropriate spaces necessary to succeed, says Greig Fraser ASC ACS.

Having enjoyed the benefits virtual production offers, Greig Fraser ASC ACS believes it to be a powerful filmmaking tool in a cinematographer’s arsenal, but he also notes a common mistake being made when working in the virtual realm. Fellow cinematographers’ experiences of shooting on volumes have taught him that a production’s chances of triumph can be threatened if the DP and production designer are not involved in the process at the earliest stage. This results in virtual stages being built without adequate consultation with the people who will need to work in them.

Fraser’s most successful experiences of virtual production correlate with being brought into the process at the script stage, giving him the power to make small revisions to the script, a scene’s setting, or the time of day it takes place to push in a direction that takes full advantage of a virtual stage.

This early involvement in the decision- making – in conjunction, with post-production, producers, and production designers – is essential because virtual production is not an all-or-nothing solution. It requires decisions to be made on a case-by-case basis as a script is read and analysed.

“There’s a benefit to coming in and out of the volume in a production,” says Fraser. “You might read a script and realise a scene is perfectly suited to being set entirely on a volume or maybe you build a room, just outside of it, so everything through the windows is volume, and the actors can walk onto a balcony in front of it.”

The Mandalorian season one (2019) – lensed by Fraser and Barry “Baz” Idoine ASC – is an early example of LED screens being effectively used to craft virtual environments that react in real time and immerse the actors. When filming the first series, the production designer and Fraser were primarily responsible for guaranteeing the sets would work within the space at each stage as the environment was built by Industrial Light & Magic (ILM).

But the number of recent productions shot on virtual stages where the concepts are taken through into builds without the cinematographer or designer being consulted concerns Fraser: “Decisions are often made by technical people who know how to put the screens together – which is still an important part of the process – but many of them have never put a scene together. It’s a very different set of skills. A cinematographer knows what you need to make a scene work, and the designer is responsible for building what’s practical and linking it with what you can’t touch in the virtual world.

“When working in a volume, every department thinks of it as something they should own. Post-production wants control because they’re building the loads [files prepared for virtual shooting including virtual sets and characters] and production designers want to own what’s on the screen because they’re designing it. But for cinematographers, it’s a lighting volume more than anything as the amount of LED screen on camera is actually quite small.”

Therefore, cinematographers should avoid relying too much on other team members to build loads which could diminish their input in the project. “Often cinematographers are given this as a tool, after everything else has been decided. ‘Here’s the load you’re using, now go and light it.’ I think that’s unfair to cinematographers because it’s hard to light the volume, particularly when working with a fixed ceiling. So, we need a voice really early in the process.”

BUILDING WORLDS

Fraser first explored virtual production techniques and technologies when shooting Rogue One: A Star Wars Story (2016), designing a virtual stage with the team at ILM that later morphed into the volume used for The Mandalorian.

“In 2016, LED technology was not at a point where you could put it on camera yet, but you could use it to light. ILM built the worlds in 3D and projected them onto the screens, so the reflections and lighting were correct, but it wasn’t an interactive screen. All the content on the screen had to be replaced because we weren’t working with very fine pixel pitched LED screens.”

By the time the first season of The Mandalorian had incorporated virtual production techniques and made use of ILM’s evolved StageCraft technology, LED screen pixel pitch had advanced enough for it to be photographed, meaning 3D content could change perspective in real time. Fast forward to 2022 and ever- progressing innovations have further extended filmmakers’ creative opportunities.

This year’s release of version 5 of Epic Games’ Unreal Engine – the 3D game engine and one of the main drivers behind virtual production – has elevated techniques to new heights while LED panels now offer improved options for off-axis colour, making it possible to create volumes outside of the traditional shape.

“By “traditional”, I mean what we used for The Mandalorian which relied on a fixed ceiling. But not every project needs to follow suit,” says Fraser. “I haven’t really seen a volume that is much different to that, when the reality is, we could be making volumes that are significantly different – straight walls, ceilings that are in set, at an angle, and float.”

While each project varies in its requirements, broadly speaking, Fraser believes it is necessary for a volume to feature ceiling pieces that can raise up or down and tilt, depending on the angle of the camera. “Dynamic ceilings are the way forward as they offer the ability to adjust the basic shape of the volume. If you can create a more modular system, you can move walls and bring sets in and out. Essentially, LED panels are designed to be taken apart, but most volumes seem to be fixed and I don’t quite understand why.”

The lack of standardisation and differing – and often broad – definitions of what virtual production is can also be problematic, making now an important time to discuss the topic and share opinions with an aim to establish a universal description. For Fraser, the technique is “projecting images onto a wall either using projection or LED, so the camera sees the final image, whilst creating a volume of light in which to stage a piece of action.”

But while no description is incorrect, the lack of clarity over what constitutes a volume can impact cinematographers’ experiences when shooting and the final results. “Since we shot The Mandalorian using virtual production, some cinematographers have told me they’ve been sold a volume by a company, but it doesn’t work. When they’ve shown me photos, it isn’t what I would consider a volume.

“Some stages built now are created on the premise of the way The Mandalorian stage was constructed. But that was built for a specific purpose – for that character and that world. The only reason the walls were curved was because of off- axis colour. Now the possibilities around colour have improved and you can create flat walls in a volume.”

Fraser also offers insight into LED panel selection which challenges the general consensus regarding optimum pixel pitch. When designing the system for The Mandalorian season one, he believed a smaller pixel pitch was best because it allowed the camera to be placed closer to the wall without creating moiré. Having shot other productions on volumes since, including The Batman (2022), and hearing other DPs’ experiences, he advises filmmakers not to be preoccupied by pixel pitch and to even consider incorporating screens with a larger pixel pitch.

“What’s more important is off-axis colour, so you can be close to the walls, and experience no colour shift,” he says. “I believe there are many other things more crucial than pixel pitch – colour rendition, skin tone, or being able to shoot in the elements which a larger pixel pitch would allow.”

NEW SKILLS

Whether it’s a virtual or traditional shoot, the rules remain the same – “It’s still got to look right, feel good, and the blocking has to work. But in virtual production there’s a new set of tools to understand and some new roles on set such as a virtual gaffer. They need to know film terminology as well as Unreal because the way you light is different, and lights are moved around in the game engine.”

As well as learning valuable new skills, understanding what Unreal is good for and what it is not suited to is vital. “It’s a very impressive tool. In the same way I think a photographer needs to have some knowledge of Photoshop, the same is true of DPs and Unreal. I’ve made sure my assistant on Dune 2 knows Unreal because I believe all cinematographers going forward should at least learn the basics of the game engine.”

And while there is scope for advancement in terms of the technology at the core of a volume, Fraser is confident it will continue to advance rapidly. “Where virtual production is falling short is the number of people who think they understand it, but who don’t, in fact, have the experience of working with it. In some areas of virtual production, we’re still in cowboy territory where people are claiming to be knowledgeable in certain areas when they are not,” says Fraser, advising cinematographers to make sure their information comes from multiple educated sources.

“Very few know the full story at this point; everybody is coming to it from their own perspective. Not many cinematographers have spent multiple days working in a volume and I think the skills gap will only improve when more of them are shooting in volumes, going through the process, and learning how to work in that environment.”

One filmmaker whose experience and knowledge of shooting in the virtual world has impressed Fraser is Nikolaus Summerer, the cinematographer behind Netflix series 1899. “I visited him at Studio Babelsberg in Berlin and was blown away by what he had been doing. It looked like it had been shot by someone who understood the volume and lighting. It wasn’t used as a fashion piece; it was just used very subtly and beautifully.”

AHEAD OF THE CURVE

Despite being optimistic and excited about virtual production’s future, Fraser admits he has not seen it progress as much as he would like. “There needs to be more thinking outside the box. Look at some of the art projects using LED walls or the huge billboards wrapping around corners. That’s super interesting and I believe it’s time to get more creative with our use of LED screens.”

Urging cinematographers not to accept the traditional virtual stage as standard, he encourages them “to think far further ahead – using modular panels, adding openings and closings in a space so you can create corridors that are real set pieces that open out into a volume of light.”

Fraser used virtual production on The Mandalorian because he knew it was the way of the future, and he could either be behind or ahead of the curve. “After shooting that, I effectively had enough of a voice and enough knowledge to be able to overrule bad ideas. It’s really important that cinematographers keep their voice, too. The sky’s the limit and we, as cinematographers, need to be ahead of this curve and lead the way. If not, other departments will, and this won’t be good for us.”

While confident virtual production is here to stay, Fraser is also aware that, unfortunately, the spike in popularity has brought with it a lack of common knowledge about when the technique is suitable. “Many people are designating shoots to be on the volume when maybe they shouldn’t. Virtual production is very powerful and has plenty of potential to further develop filmmaking, but it must be used on the right productions, in the right way, and by the right people.”

–

Words by Zoe Mutter