UNTAPPED MIXED REALITY

NorthHouse and Helen Dulay for Warner Music Entertainment created a promo for a special performance by Coldplay, featuring songs from “Music of the Spheres”, using ARRI Stage London and hybrid mixed reality techniques to place the band into a fantasy version of Dubai’s Al Wasl dome.

Coldplay’s gig was part of Dubai Expo2020, taking place in February 2022, but the promo had to be shot in London two months earlier, and was achieved in a very efficient 90 minutes.

“The brief was to make them appear as if they were in this environment which hadn’t yet been fully finished and fully created,” says DP James Medcraft. “The client initially wanted a green-screen shoot, which I felt wasn’t going to provide quality of lighting on the band. You’d spend a lot of time closing down your set with black flags to hide green-screen spill.”

Instead, the production turned to ARRI Stage London, amongst the largest permanent facilities of its kind.

For the Coldplay promo, the team took a hybrid approach, blending the benefits of virtual production with green screen. Vicon camera tracking, delivered by Audiomotion, and FIZ (focus/iris/zoom) data relayed by ARRI UMC-4s allowed the system to drive a frustum for each camera that was rendered green – the area of the LED screens seen on camera at any given moment. The remaining screen area rendered the virtual environment of the Al Wasl dome in real-time, providing highly authentic interactive light on the foreground talent.

“We used a hybrid virtual production process, which I think is a slightly untapped use of this technology,” Medcraft explains. “Basically, we projected a green window wherever the camera was facing but used the low-resolution content as an ambient lighting plate, giving the best of both worlds.”

The emissive green provided by the LED walls was easy to key out in post-production, and because it filled only the minimum area needed to cover the frame, there was no green spill on the talent. Instead, the band were lit realistically by the rough virtual backgrounds displayed on the rest of the volume. “It meant we could shoot earlier, light perfectly for the post-production Unreal scene, and then drop the final virtual scene in at the last minute,” says Medcraft, “so it helped every stage of the process.”

The production shot on two ARRI AMIRAs outfitted with Cooke S5 lenses. “We had our ‘A’ camera on a Technocrane, and an extra camera for close-ups, picking up detail.”

“It was important to create a great sense of scale with sweeping, wide camera shots,” mentions Tom Bairstow, creative director at NorthHouse, “but we also wanted to get in close to the band to really feel their performance.”

As well as driving the green frustums, tracking data from the Vicon crowns and UMC-4s was fed to separate servers providing “simulcam” – live images which were chromakeyed in real- time with the virtual backgrounds, showing the crew an approximation of the final shots. Camera position, FIZ data and timecode were also recorded in Unreal from both cameras, making for a more efficient post-production.

“I think selling reality is all about subtle details that you don’t concentrate on in the frame,” says Medcraft. “It’s all about subtle reflections in the eyes and reflective objects like the cymbals, and also for the band – it gives a better experience of being within an environment and therefore a better performance.”

The ARRI Stage London, a collaboration between ARRI and Creative Technology, also boasts a ring of 50 ARRI Orbiter LED fixtures, which can be controlled by Unreal Engine to match the light from the video walls. “We used some of those to bring out some animated highlights in the skin,” Medcraft notes. “It’s really visible in Chris’s eyes.”

Medcraft and the crew spent a day and a half rehearsing in the volume before Coldplay arrived. The band performed on a turntable and this, combined with the ease of flipping the virtual backgrounds on the screens, meant that reverse shots were achievable within the tight timeframe.

The DP enjoyed his experience at ARRI Stage London. “You’ve got all the kit available and expertise in all the different departments. It does feel like a completely welcoming, supported experience.”

–

Words by Neil Oseman