Introduction

Infobip RTC extensions is a JavaScript library which provides extended functionality to Infobip RTC SDK.

Currently available functionalities are:

• audio filter implementations
• video filter implementations

Here you will find an overview, and a quick guide on how to include and use these extensions in your application. There is also in-depth reference documentation available.

Prerequisites

Infobip RTC Extensions requires ES6.

Getting the library

There are a few ways in which you can get our library. We publish it as an NPM package and as a standalone JS file hosted on a CDN.

If you want to add it as an NPM dependency, run the following:

npm install infobip-rtc-extensions --save

After which you would use it in your project like this:

let infobipRtcExtensions = require('infobip-rtc-extensions');

or as ES6 import:

import {RTCVideoFilter} from "infobip-rtc-extensions"

You can include our distribution file in your JavaScript from our CDN:

<script src="//rtc.cdn.infobip.com/1.1.0/infobip.rtc.extensions.js"></script>

The latest tag is also available:

<script src="//rtc.cdn.infobip.com/latest/infobip.rtc.extensions.js"></script>

Audio filters

The Infobip RTC supports user-defined audio filters which manipulate outgoing audio streams during a call. This library implements several audio filters which are easy to configure and use.

Filters which are currently available are:

• BackgroundMusicAudioFilter
• NoiseSuppressionFilter

BackgroundMusicAudioFilter

BackgroundMusicAudioFilter allows the user to specify audio to be played alongside their outgoing stream. This audio is heard by other participants of the call, but not by the user.

To use this filter, an instance of BackgroundMusicAudioFilter needs to be created. The only parameter of the class constructor is a string containing the URL of the audio file which is to be combined with the user's outgoing stream. The audio file loops until the filter is turned off or the call is terminated.

const musicURL = "path/to/desired/audio.mp3";
const backgroundMusicFilter = new BackgroundMusicAudioFilter(musicURL);

NoiseSuppressionFilter

The NoiseSuppressionFilter enhances speech by removing several types of background noise. This filter works in real-time. Currently, it is focused on removing background noises commonly encountered in call centers, such as babble, noise produced by different devices (e.g. air conditioner) and keyboard typing sounds. However, it performs well on a wider range of noise types. It is also independent of the language spoken.

To use the noise suppression filter, an instance of the class NoiseSuppressionFilter needs to be created. This can be done using the NoiseSuppressionFilter.create() function.

Real-time noise suppression is a hardware-intensive process, so not all hardware will be able to sustain it. In case of insufficient performance, the filter is automatically disabled in order to not interfere with the call.

const noiseSupressionFilter = await NoiseSuppressionFilter.create();

During initialisation, the created audio filter instance allocates resources which should be manually released once you are certain you won't need this filter instance anymore.

// load the filter
const noiseSupressionFilter = await NoiseSuppressionFilter.create();

// start using it
activeCall.setAudioFilter(noiseSupressionFilter);

// at the end of the call
await noiseSupressionFilter.release();

Video filters

The Infobip RTC supports user-defined video filters capable of manipulating outgoing video streams during calls. The library provides an extensive implementation of commonly used video filters, making configuration easier and enabling seamless integration.

Currently available implementations are:

• RTCVideoFilter

RTCVideoFilter

This filter allows users to modify their background during video calls.

Supported video filter modes include:

• Virtual background (RTCVideoFilterMode.VIRTUAL_BACKGROUND) - Users can set a custom image to be displayed as their background
• Background blur (RTCVideoFilterMode.BACKGROUND_BLUR) - Users can blur their background.
• Face track (RTCVideoFilterMode.FACE_TRACK) - Automatically adjusts the video to keep the user's face centered and properly framed within the view.
• None (RTCVideoFilterMode.NONE) - No video filtering is applied; video frames are passed through unchanged. This option is recommended over repeatedly reallocating video filter resources to avoid visible disruptions.

To utilize this feature, begin by creating an instance of the RTCVideoFilter object. The constructor accepts optional RTCVideoFilterOptions for customization.

const options = {
    mode: RTCVideoFilterMode.VIRTUAL_BACKGROUND,
    image: sourceImage // can be an instance of ImageBitmap, ImageData, HTMLImageElement, …
};
const videoFilter = new RTCVideoFilter(options);

For optimal performance, it's recommended to avoid reallocating video filter instances solely for mode changes. Instead, pass the new options directly to the existing video filter instance. This approach minimizes resource overhead and enhances overall efficiency.

const options = {
    mode: RTCVideoFilterMode.NONE
};
await videoFilter.setOptions(options);

Applying the video filter

Once you've created the video filter, you can utilize it during calls.

You can set it beforehand when initiating a new ApplicationCall using VideoOptions object within the ApplicationCallOptions object:

let token = obtainToken();
let infobipRTC = new InfobipRTC(token);
infobipRTC.connect();

let videoOptions = VideoOptions.builder().setVideoFilter(videoFilter).build();
let applicationCallOptions = ApplicationCallOptions.builder().setVideo(true).setVideoOptions(videoOptions).build();
let applicationCall = infobipRTC.callApplication('45g2gql9ay4a2blu55uk1628', applicationCallOptions);

Alternatively, you can apply the filter to the existing ApplicationCall using the setVideoFilter method:

await applicationCall.setVideoFilter(videoFilter);