Key Points When Calling WinRT from .NET Core

These past two days I wanted to write a simple video-compositing tool. If it weren't for distributing it to others, I could have finished it directly with an ffmpeg script or Python. But now this need has arisen, so I considered finishing it quickly with .NET + WinForms.

Video editing in WinRT looked quite simple, so I planned to write a small tool with WinUI 3 — only to discover this thing has murky ties to UMP. For personal use, deployment is super cumbersome; the easiest way is to publish to the Microsoft Store and download from there. What kind of joke is that?

So I switched approaches: I adopted .NET Core 6 and created a WinForms application. Since video compositing needs an overlay layer, and green screens and the like must be made transparent automatically, Chroma Key technology is inevitably needed to create transparent channels — this part has to be done by hand.

Through WinRT's Media, you can use DirectX directly in CompositeFrame, which amounts to GPU-accelerated compositing. Based on experience with big-data algorithms, this kind of efficiency is bound to be much higher.

But then it insisted that the Win2D library must be used. Once I referenced this library, things immediately became disastrous — because runtimes above .NET 5.0 simply don't support it. So I went digging through Microsoft's documentation again, only to find it written unclearly.

The key issue: after referencing Win2D, a flood of missing interface errors appeared. Clearly these interfaces need to be defined somewhere. Being completely unfamiliar with this area made it truly troublesome.

The solutions offered on MSDN are written in a way that easily misleads. For example, the docs say:

.NET Core is the focus of the .NET platform, and .NET 5 is the latest major version. It is an open-source, cross-platform runtime for building device, cloud, and IoT applications.

Previous versions of the .NET Framework and .NET Core both included built-in knowledge of WinRT (a Windows-specific technology). To support .NET 5's goals of portability and efficiency, we removed WinRT projection support from the .NET compiler and runtime and moved it to the C#/WinRT toolkit. C#/WinRT aims for parity with the built-in WinRT support provided by the legacy C# compiler and .NET runtime. For details, see .NET mappings for Windows Runtime types.

Following the docs, it then tells you to install: Install-Package Microsoft.Windows.CsWinRT -Version 1.4.1

After going around in circles, you finally discover this is teaching you how to generate a WinMD file after creating an ordinary class library — whereas in practice, simply creating a Windows Runtime Project from a template will generate it for you. So all this verbiage has no practical significance.

If you create a new Windows Forms application and then try to reference a runtime component, you immediately get something like:

Severity Code Description Project File Line Suppression Status
Error NU1201 Project Effects is incompatible with net6.0-windows10.0.17763 (.NETCoreApp,Version=v6.0). Project Effects supports: uap10.0.17763 (UAP,Version=v10.0.17763)

In the end I really wanted to give up and just switch to ffmpeg — so much simpler. But ffmpeg uses the CPU for computation; if you want to call the GPU you need to install CUDA and configure things, which is quite troublesome for distribution and deployment.

Finally, after repeated trial and error, I figured out how to call it directly from WinForms — and frankly the method is very simple.

First, for the .NET Core WinForms template, bring in Microsoft.VCRTForwarders.140 and Microsoft.Windows.SDK.Contracts via NuGet, then add Win2D. But that's not the end: if you're using the .NET 6.0 runtime, you'll see a message like:

Severity Code Description Project File Line Suppression Status
Error NETSDK1130 Cannot reference Windows.AI.MachineLearning.MachineLearningContract.winmd. Direct use of Windows metadata components is not supported when targeting .NET 5 or higher. See https://aka.ms/netsdk1130 for more information. WinFormsApp1 C:\Program Files\dotnet\sdk\6.0.100\Sdks\Microsoft.NET.Sdk\targets\Microsoft.NET.Sdk.targets 1003

Clearly you need to lower the .NET Core version. In practice, downgrading to .NET Core 3.1 works best.

I have to say, Microsoft's documentation remains unclear. One always feels Microsoft hasn't abandoned its petty mentality — on the one hand striving for openness and wide coverage, on the other remaining ambiguous, wanting to preserve the Windows platform's advantages through a degree of closedness. The documentation is never clear and detailed enough, and often doesn't offer a simple, straightforward implementation path.

Compared with development on macOS or Linux — where the ecosystem may look closed, but solutions for many feature implementations are extremely thorough and come with lots of ready-to-use components —

In this case, although interaction with the WinRT runtime has performance overhead, experiments showed that processing one video by calling ffmpeg took 25 seconds, while using the features provided by Windows 10 took only 8 seconds. An optimization of this magnitude is very easy to achieve.

Finally, let me record the detailed project configuration for future reference:

<Project Sdk="Microsoft.NET.Sdk"><PropertyGroup><OutputType>WinExe</OutputType><TargetFramework>net6.0-windows10.0.17763.0</TargetFramework><TargetPlatformMinVersion>10.0.17763.0</TargetPlatformMinVersion><Nullable>enable</Nullable><UseWindowsForms>true</UseWindowsForms><PlatformTarget>x86</PlatformTarget><LangVersion>10.0</LangVersion></PropertyGroup><ItemGroup><PackageReference Include="Microsoft.VCRTForwarders.140" Version="1.0.7" /><PackageReference Include="Microsoft.Windows.SDK.Contracts" Version="10.0.22000.196" /></ItemGroup></Project>