Awesome ML Kit (background removal and depth)

Awesome ML Kit gives Unity developers easy way to employ optimized real-time ML models centered around computer vision.

Documentation

Forum thread

Background matting demo video

(new) Try also Awesome VFX Kit video / webcam to 3d point cloud — A toolkit for creative video processing (you get Awesome ML Kit for free)

The current set of models includes

1. High-quality background removal/background matting based on RobustVideoMatting. Optimized for Unity, both horizontal and vertical/portrait video aspect ratio supported. Like chromakey but with no chromakey needed

2. Depth estimation by a single image

Features

1. Sample scene for each model. Specially made visualizations with different post effects. Additional 3d scene, background blur and glitch scenes

2. Select solid color / image / video as background replacement; put Unity content between foreground & background layers

3. Different model versions for any quality/speed tradeoff

4. Barracuda / Sentis / Inference Engine inference backends supported

5. Unity WebCamTexture and VideoPlayer support

6. AVPro Live Camera integration (recommended instead of WebCamTexture)

7. AVPro Video 3 integraion

Ideal for

❤️ Video processing

❤️ Media installations

❤️ Apps with video streaming (e.g. can be used with Agora.io)

❤️ Kiosks

Performance (GPU inference)

NVidia RTX3070

🚀 Background removal: 300 FPS

🚀 Depth estimation: 400 FPS

NVidia GTX1050Ti (mobile)

📈 Background removal: 50 FPS

📈 Depth estimation: 80 FPS

Note: only desktop platforms currently supported (tested on Windows DX11 and Linux Vulkan)

Unity versions: 2021 and above (Unity 6 now supported)