Colour grading meets machine learning

Martin Tlaskal, head of development at FilmLight, shares how the company has been using machine learning to empower colourists and help them stay on the cutting edge of the latest trends and technology in this field.

In the ever-evolving landscape of film and TV post-production, machine learning (ML) has become a pivotal force with a magnitude of ML models for image processing released every single day.

At FilmLight, we’re continuously monitoring the market and our customers’ requirements; we work hard to update and improve our technology so it’s always moving forward and providing a platform that fulfils the colourists’ needs.

This is demonstrated in the latest version of our grading software, Baselight 6.0, which we launched at the end of last year. The platform benefits from several years of development which bring colourists major gains in productivity and creativity, including an improved and modernised timeline, a new primary grading tool, X Grade, and a unique new look development tool, Chromogen.

For the first time, the latest Baselight release also includes ground-breaking ML-based tools and features. Our main goal was to come up with a platform to allow us to do two things.

Firstly, where we felt it would be beneficial to our customers, we wanted to be able to train our own ML models and integrate them deeply inside Baselight. But we also wanted to create a platform which allows us to easily and quickly integrate external models into Baselight – allowing ourselves and our customers to stay on the cutting edge of the latest trends and technology in this field.

Face Track

 Face Track is the first example of our own ML-based model which, following years of research and development (led by Patrice Lacour, Software Developer), has been deeply integrated inside Baselight 6.0.

Face Track was born out of a recognition of the significant amount of time colourists spend on tasks like facial match grading and digital beauty work. Our goal was to create a tool that streamlines and enhances this process, saving the colourist time and increasing productivity.

Previous methods of face tracking involved using trackers, such as area or planar trackers, which, while effective in basic scenarios, faltered when faced with complex facial structures or occlusions.

Face Track works by first detecting all faces in a scene. It will then track each face through time, determining its pose. It will essentially unwrap the skin of the face into a constant coordinate space, or UV space. This space becomes the canvas for colourists to make corrections or enhancements, and the beauty of it is that these corrections are automatically applied to 3D geometry. This means that colour corrections are applied consistently to the face, regardless of how the face moves.

But the benefits don’t stop there. Because the system knows about faces and poses, the colourist can copy and paste their enhancements across multiple scenes, sequences or entire episodes. It will make a ‘best guess’ at matching faces and while it may not be perfect, it will get the colourist 90-95% of the way there, allowing them to focus on finessing the creative aspects rather than time consuming manual tasks.

The feedback so far has been phenomenal, and many customers have already used the beta version of Baselight 6.0 to complete productions – because, once they saw the tools and the efficiencies it provided, they didn’t want to return to previous versions of the software.

Flexi

 Recognising that we can’t create all models ourselves, we have also introduced Flexi architecture into Baselight 6.0.

Many new ML-based models are being released every day with licenses allowing commercial use. However, they’re all written in Python, which means it’s not easy or fast to integrate them into most apps.

Flexi is a new architecture in Baselight 6.0, which allows us and our customers to integrate external ML models written in Python into the Baselight environment. This flexibility enables users to adapt and run various open-source models within Baselight, providing endless possibilities for experimentation and creative exploration.

For example, using our Flexi architecture we’ve been able to integrate the popular MiDaS depth mapper into Baselight. MiDaS has been a world regarded open-source depth mapper and, although perhaps not state of the art today, it is a good example of how an external ML model can be integrated. An accurate depth mapper produces an estimation of the per-pixel depth of an image and allows the user to apply bokeh – a cinematic blur of the background – to produce beautifully natural looking shots with a more filmic depth of field. By taking the current version of MiDaS, including a Flexi Python script, and slightly altering the wrapper script, we were able to get it running inside Baselight in under a day.

For some models, we may choose to integrate them more deeply into Baselight, but for others we may publish the API and allow our customers – many of whom have their own internal scripting teams – to integrate models of interest directly into their Baselight system as required. These models are computationally expensive, requiring a lot of GPU and memory, so doing it this way allows them to use Baselight’s caching and rendering system and allow real-time interaction.

 Our vision with Flexi is to keep up with the rapid advancements in machine learning. We want to make it easy for our users to incorporate the latest and most innovative models into Baselight. Whether it’s image segmentation, video in-painting, or other revolutionary or creative video-centric applications, Flexi aims to be the bridge between open-source ML models and the high-end post-production environment.

Empowering colourists

The film and television industry is constantly evolving, with filmmakers seeking tools that enhance both creativity and efficiency.

As we move forward, we anticipate further improvements and integrations, but we believe Baselight 6.0, with Face Track and Flexi, aligns with these trends by leveraging machine learning to automate and streamline the colour grading process and empowering colourists to be able to achieve consistently exceptional results in less time.