Unity unveils Metacast real-time 3D platform for sports broadcasts

Unity sports exec Peter Moore unveiled Unity Metacast, a new technology for capturing and analyzing sports moves in real time.

In an interview with GamesBeat, Moore said the company hopes to capture athletes in real-time so that broadcasters can use the imagery for a new era of live sports. The maker of the Unity game engine collaborated with UFC and its mixed-martial arts fighters to show off the technology, which can scan an athlete’s body and capture the action from all views.

It’s a bit like the Intel tech, dubbed True View, from 2019. But Moore pointed out that the Unity Metacast tech can capture athletes in real-time, as they’re performing in live events, in contrast to the delayed time frames when Intel could deliver its imagery. (Moore and Nifty Games CEO Jon Middleton will be talking a lot about sports games at our GamesBeat Summit Next event on November 9-10).

“When you go into first-person view, you can look at the level of detail,” Moore said as he was showing the online demo. “What we’re looking at here is volumetric capture about five million voxels per second — a voxel being that three dimensional pixel pushing down the pipe. That’s a first-person view as if I’m Georges looking at Kevin. I can change the camera around and be Kevin looking at George.”

You may notice the tech looks a little rough around the edges, and that’s one of the sacrifices Unity has to make in getting the tech to produce the images in real time. But over time, Moore expects that the quality will just keep on improving.

How it works

Above: Unity processes 5 million voxels to flesh out UFC fighters.

Image Credit: Unity

As you can see in the video, Unity and its partners scanned used cameras to capture real UFC athletes — Kevin Lee and Georges St. Pierre — as they squared off against each other in the ring.

“Sports fans love to analyze the action at every level of detail,” said Moore, senior vice president for sports & live entertainment at Unity. “This is the first time you’re seeing this level of resolution in three dimensional visuals. This is about five million voxels that we’re pushing down. We do the rendering and the editing, and then being able to compute that in real time, push it up in the cloud, and then send it to the broadcasters. We see the first opportunity with UFC, which is perfect for this as it’s a controlled environment in the octagon.”

Unity Metacast uses volumetric technology, which encompasses the process of capturing, viewing, and interacting with the real-world, from moving people to static objects, in 3D. This content can then be viewed from any angle, at any moment in time, giving audiences the ability to see every bead of sweat, blow, takedown and submission, as if they were going toe-to-toe on the famous Octagon canvas themselves.

Metacast doesn’t get into the actual work of hardware or capture systems. It works with partners such as Microsoft, Canon, and other volumetric capture providers who provide the raw volumetric data. Unity then takes over from there by providing an entire pipeline – from ingestion to Unity, content development, delivery, rendering, to interactive experience consumption on various devices.

Above: The real Kevin Lee (left) and Georges St. Pierre.

Image Credit: Unity

This allows content creators to use Unity’s tooling and workflows that are optimized for volumetric data, regardless of the capture provider or the distribution mechanism. Unity’s value in the volumetric pipeline is by providing tooling and workflows that enable creation and consumption of volumetric content. Unity focuses on making volumetric data look good, into compelling experiences, easily distribute them worldwide, and perform optimally on the 20+ platforms/devices Unity supports.

The demos cover MMA, rugby, and basketball, with each one showcasing a different use case of volumetric in the sports world. The MMA example shows a consumer second-screen experience with livestreams, replays viewed from any angle, fan engagement, and fan-produced content. You can freeze the action and view it by spinning around the point of view.

The rugby demo shows volumetric production tools that integrate with existing broadcast stack. It shows 3D digital overlays, cuts and shots, camera path generation, and commentary. The basketball demo shows coaching and analysis that can help coaches explain mistakes or call plays.

The business opportunity

Above: You can analzye and rotate 3D screens to see images from any angle.

Image Credit: Unity

The ideas is to use Unity Metacast as a real-time 3D sports platform for creating and delivering interactive content, direct to the consumer. UFC will collaborate on research and development of potential applications for Unity Metacast within UFC content.

Unity Metacast is part of Unity’s new Sports & Live Entertainment Division led by game industry veteran Moore, who effectively came out of retirement (he had previously been Liverpool Football Club CEO) to run the division after seeing what it could do. This division’s mission is to redefine the way the world interacts with sports and live entertainment. As such, it’s one of the ways that Unity is reaching beyond gaming to other kinds

Moore said there are dynamic advertising opportunities as the green screens can swap out content and patch in ads in real time.

The live cameras will be in place in Las Vegas and the broadcasts will probably go live next year, Moore said.

As for better resolution, Moore said, that will come in time. But he noted that for many purposes such as coaching students, the resolution of the cameras is more than adequate, as coaches often want to freeze the action and show players what is right or wrong about their positions and actions. Moore said he believes broadcasters will be fascinated with the tech because it opens so many avenues for analytical commentary.

Unity Metacast will power an interactive second screen experience for fans, enriched storytelling tools for broadcasters, and data analytics for teams, coaches and athletes along with the chance to create more meaningful and impressionable marketing campaigns.

Unity has more than 1,800 employees and apps developed by Unity creators were downloaded more than five billion times per month in 2020. UFC has more than 625 million fans.

Above: A polygonal view of the UFC fighters.

Image Credit: Unity

Moore noted that when he left Liverpool after three years, he had no plans to go back to full-time work. But he met with Unity CEO John Riccitiello, who showed him a demo. And Moore said he was hooked, as it took him back to his 22 years in video games and decades in sports.

He said, “We’re trying to drive thought leadership and first mover advantage on this, and my sports experience and my network, both here in North America, as well in Europe, have been able to call upon the big leagues and show this technology and the broadcasters and give them a little taste of what the future looks like is going to be,” Moore said. “I’ve absorbed so much information in the last nine months. It keeps me young.”