Note: Not a live demo, server configuration is just for example. For live working example try the Dolby.io Publisher and Subscriber example. Or AWS Kinesis Publisher and Subscriber example.
This example demonstrates dynamic virtual background removal using Google Mediapipe Segmentation. Rendering is applied using efficient graphics rendering. CPU usage is incurred obtaining the virtual stream for publishing due to software encoding in Chrome therefore the resolution needs to be restricted to 720p.
Backgrounds can be applied using the bgImage or backgroundColor config on the publisher.
With screen toggling enabled, wen switching to screenshare the screenshare video will be rendered as a GPU background texture, the video will be rendered in the foreground resized.
An api is provided to toggle enabling the virtual rendering and updating the background image, color texture or chromakey keycolor. The render type can be toggled between 1 for mediapipe and 2 for chromakey or empty argument for off.
Virtual background is supported in webrtc publishing or web conferencing features.
The maxHeight is to restrict the canvas rendering stream dimensions. Canvas capture streams are software encoding and increases cpu on top of the Mediapipe segmenetation and webgl rendering solution. In 1080p the cpu usage doubles, 720p seems more efficient.
Virtual Screen Sharing
With share screen enabled with toggledScreen in webrtc publishing and virtual enabled. A share screen virtual mixer with camera input will be enabled.
In web conferencing. Screen publishing is seperate and will not be mixed and only the stream of the virtual rendering will be switched to.
The scaledWidth configuration is a percentage of scaling the video texture to the screen share texture background.
Note: Not a live demo, server configuration is just for example. For live working example try the Dolby.io Publisher and Subscriber example. Or AWS Kinesis Publisher and Subscriber example.
Configuring a background image for the video stream is possible by setting the player poster image.
.flowplayer {
background-color: #000000 !important;
background-image: url(../../images/virtualbg.jpg) !important;
background-repeat: 'no-repeat !important';
background-position: 'center center !important';
}
<div class="flex flex-col w-full">
<div id="virtual-background" class=""></div>
</div>
<script type="text/javascript">
var player = flowplayer("#virtual-background", {
"peakmeter": {},
"poster": "../../images/virtualbg.jpg",
"rtc": {
"applicationName": "webrtc",
"autoStartDevice": true,
"publishToken": "eyJhbGciOiJSUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJ3ZWJydGMifQ.MjehxjweF5tPPqUJQjHEHdHLz4sjaXkTkJh0dkM8w9_M5PMKoyzUJZPYyYtgT1qn17eDZlUOUeIeyr37z-KhN1u66L2ScVCwvs0dBjf6s2ZSIw0shvKmCdq7u5bd5llWTY0FDbbtFA1l60CkfOsTd0_dQpeyKm3Y94XgIaPyJB_PCCezO8V1xmcyMT1aqPfwr99AmM8s_P_8nuDL6A1HHppImwZL550AnTjuPQaAMRSVSNuzlLrwFXBA1SRaKOa2AVkIzP0tYkqWCYd03Gvn_CpxZ5dhk5s4UYSoYeK2FX4nz4khn_k8loFO-vDu2M-1r7dvFXnt8iNYWGTzDxPNuQ",
"publisher": true,
"server": "wowza",
"serverURL": "rtc.electroteque.org",
"toggleScreen": true,
"userData": {
"param1": "value1"
},
"virtual": {
"backgroundColor": "000000",
"bgImage": "../../images/virtualbg.jpg",
"maxHeight": 720,
"renderType": 1,
"scaledWidth": 0.25
}
},
"src": [
{
"publisher": true,
"src": "myStream",
"type": "application/webrtc"
}
],
"token": "eyJraWQiOiIyeHRpc0Q5NHZzTjIiLCJ0eXAiOiJKV1QiLCJhbGciOiJFUzI1NiJ9.eyJjIjoie1wiYWNsXCI6NixcImlkXCI6XCIyeHRpc0Q5NHZzTjJcIixcImRvbWFpblwiOltcImVsZWN0cm90ZXF1ZS5vcmdcIl19IiwiaXNzIjoiRmxvd3BsYXllciJ9.WLUkZHpDNoaXWDaFO2V5UfXm7SnDvE1pFAM0e7ppnFovOSyCCZM-b8gQNBSElB5yirTP__x76Qyo8pMWh6lVrw"
});
</script>
