Acoustic echo cancellation (AEC) and beam forming with mic arrays

[fquirin]

I've been experimenting with acoustic-echo-cancellation (AEC) and beam forming lately using Pulseaudio.
The Pulseaudio module-echo-cancel looks very interesting and it certainly does remove the output audio very aggressively, but what doesn't seem to work at all is the aec_args parameter. Also the recording quality during audio output strongly depends on your signal-to-noise ration. If its tool low your recording is scrambled and unrecognizable :-/.

There are some demo scripts in the DIY client repository: https://github.com/SEPIA-Framework/sepia-installation-and-setup/tree/dev/sepia-client-installation/rpi/pulseaudio

Some resources:

• module-echo-cancel on Freedesktop
• module-echo-cancel on ArchLinux
• module-echo-cancel Pulseaudio source code

@StuartIanNaylor I've seen you've done a lot of experiments with this module for Mycroft (A, B) and Rhasspy, have you tried the module recently with Raspberry Pi OS Bullseye? Could you maybe share any of your experiences about the aec_args parameters and their effects? For me 'noise_suppression', 'analog_gain_control', 'digital_gain_control' and 'high_pass_filter' seem to have no effect at all and I'm not sure if the effect of 'beamforming' is just imagination 😅 🙈 .

@synesthesiam I think you've mentioned the new MyCroft mic HAT will use AEC. Is this an integrated codec or any Pulseaudio version?

I've tested several microphones including Respeaker 6mic and 4mic linear, IQAudio Zero and Joy-IT Talking Pi HAT. Next on the list is Respeaker/Waveshare 2mic HAT.

I've also played with RNNoise on the Raspberry Pi 4 and I got it working for a minute ... and then it never worked again O_o. Noise reduction was pretty good but the cut-off was super aggressive and it only worked when you were very close to the microphone.

I'd be very happy if anyone can share their ideas, experiment results and thoughts about it 🙂

[fquirin]

Update: It turns out AEC is broken on Raspberry Pi OS Bullseye 32bit !
I've downgraded to Buster and suddenly every parameter has a large effect! 🤨

[Marcusbjol]

The webrtc dev team depreciated beamforming in the latest versions of pulse audio and the pulse audio devs left the configurations in but nonfunctioning. The pulse audio version on buster (ver 12) is right about when the webrtc folks removed it.

[StuartIanNaylor]

It was never great anyway as it was true echo cancelation as opposed to speex that was echo attenuation, but over a certain quite low SNR it would fail whilst speex would continue attenuating.
I created this realtime beamformer by melding some existing code https://github.com/StuartIanNaylor/2ch_delay_sum and output to loopback so you can down sample to 16Khz.
Due to the speed of sound a single sample equates to 21.5mm of travel whilst 48Khz is 7.1mm so to get some resolution you have to use a higher frequency sampling rate but just output it through a asnd_loop and use standard alsa plug resampling.
Delay sum is extemely low load as it purely sums based on a TDOA delay and its just a hacky 2 channel and all the load is TDOA calc.
Really the FFT should be upgraded to neon as prob at least 2x faster but the above works.
Increasing the mic distance also has problems as higher frequencies hit multiple times creating lower frequency harmonics and geometry is very important.
Luckily the 2 mic based on zero dimentions is pretty reasonable whilst the 6/4 mic respeaker hats are just fubar.
Beamforming's main purpose is to remove reverberation as its a killer of spectrogram recognition but we are hardwired to do the same so the results are subtle but for further far field its effect can be quite large.
For some reason the majority of people use the 2 mic with the microphones pointing to the ceiling and either use the pi vertically (wall mount) or get a 2$ 90 degree adapter that is a tight fit but does the job and your mics are aligned more correctly and the physical pcb will give an element of rear rejection.
I now have every 2 mic avail respeaker ok, RaspiAudio just a weird layout, Adafruit good but expensive and finally Keyes studio clones horrid noisey things.

[StuartIanNaylor]

The method I think is superior BSS https://github.com/fakufaku/torchiva on a rk3588 x2.3 realtime but the pytorch python generally is far to heavy for many frameworks preffered platform.
Likely it could be quantised to tflite / onnx as a C++ ladspa or like above to a loopback, but sadly we seem to be totally lacking compentent DSP C++ devs in a low level DSP heavy field.

[synesthesiam]

@fquirin Mycroft is using the XMOS XVF3510 chip in its SJ201 board. This chip includes AEC as well as VAD, noise suppression, and AGC. I honestly think they could sell these boards alone and people would buy them.

[fquirin]

Yeah that sounds pretty nice. There seem to be some Dev-Kits available with the 'Amazon AVS' branding for over 400$ ... absurd 😆 . Seeedstudio has the ReSpeaker_Mic_Array_v2.0 that uses a XMOS’s XVF-3000 for about 65$ 🤔 ... maybe I should get one for my collection.

I actually realized that the SEPIA client has integrated AEC via Chromium in addition to PulseAudio 😅 which seems to work pretty well. Obviously it creates more CPU load compared to hardware codecs but not too much. For some reason I always thought the "echo-cancel" setting of the audio context was broken ... but I think I never really understood the purpose before ^^. On mobile it only creates distortions because I think Android cannot really handle duplex input-output but on RPi4 it works.

[fquirin]

@synesthesiam so I could not resist and ordered a Respeaker Mic Array v2.0 ^^. At first I was pretty enthusiastic about it, it has DAO and adaptive beamforming and the AEC is pretty awesome with almost perfect cancellation of very loud sources and hardly any distortion 😎 ... but then there is one major drawback :-( Audio output samplerate is limited to 16kHz. There are experimental 48kHz firmwares but I didn't dare to install them since AEC quality seems to be much worse and downgrading (v3.0.0 to v2.0.0) might be irreversibel (I didn't really get the comments about that).
Does Mycroft's SJ201 board support higher samplerate for the output? Do you handle AEC in 48kHz? As far as I understand the AEC math audio-in and audio-out have to be the same samplerate to work.

[StuartIanNaylor]

Didn't know it was broken but I gave up with webrtc aec as it works really well for low SNR in terms of echo but then fails as SNR hits a certain level.
Also it has no method apart from loading config to set the beam angle so its effectively set.
I did think about doing some sort of script of unloading and loading with different beam angles but never bothered.
Also upstream apparently that part of the code has been dropped which was another reason.

If using software I have a preference for Speex AEC as it doesn't cancel like webrtc but merely attenuates but at much higher levels of SNR.

Seems to be quite a few things broken on bullseye but likely if you had compiled the libs yourself then likely they will work as prob there is a mismatch somewhere.
Speex ALSA plugins are like that on buster.

16Khz is sort of the norm for speech as increases in sample rate just increase model parameters and load whilst very little speech of need is above 16Khz so I wouldn't worry too much about that.
Its Nyquist sampling that for accuracy you need double max possible hearing sampling rate so for accurate hi-fidelity ADC/DAC 44/48khz are used but half that really is all we can hear in young ears.

[fquirin]

Didn't know it was broken

I'm still somewhat uncertain what actually works and what not, but for now I gave up on experimenting with OS and AEC at the same time so I've been doing all tests on Buster for now :-/

I gave up with webrtc aec as it works really well for low SNR in terms of echo but then fails as SNR hits a certain level

This is my impression as well and since there are basically no parameters to tweak I guess I'm going to abandon it as well. The noise-supression is somewhat useful though. Since you mentioned Speex I'll definitely give it another try, just to see the difference but I'm not expecting too much ^^.

16Khz is sort of the norm for speech as increases in sample rate

16Khz for the input is totally fine, actually even preferred since all the ASR systems need it anyway, but my problem with the Respeaker 4mic USB is the output. It plays music in 16Khz as well and that sounds like a radio from the 1920s =) :-(

[fquirin]

If using software I have a preference for Speex AEC as it doesn't cancel like webrtc but merely attenuates but at much higher levels of SNR

I'm not entirely sure what just happened, some kind of singularity of sorts 😲 , but after several tries of using the Speex algo via Pulseaudio and seeing no big changes I just now reached a state where I have full attenuation of music at almost highest volume (a small 10W speaker driven by the Respeaker 4mic linear array) WITH clear microphone input 😱 and high chance of wake-word recognition!
Current settings:

aec_method='speex'
aec_args='"filter_size_ms=200 frame_size_ms=20 agc=0 denoise=1 dereverb=1 echo_suppress=1 echo_suppress_attenuation=0 echo_suppress_attenuation_active=0"'

pactl load-module module-echo-cancel \
  source_name=$source_name \
  source_master=$source_master \
  source_properties=$source_properties \
  sink_name=$sink_name sink_master=$sink_master \
  sink_properties=$sink_properties \
  rate=32000 \
  aec_method=$aec_method \
  aec_args=$aec_args \
  use_master_format=1 \
  channels=1

Investigating further tomorrow ... ^^

[StuartIanNaylor]

PS Speechbrain contains various tutorials on beamforming and DOA also pytorch itself https://pytorch.org/audio/master/tutorials/mvdr_tutorial.html

I always had a problem getting torchaudio running on Arm64 so never tested load but I think that might of recently changed and people have got it running so those examples should work.

[synesthesiam]

@fquirin The SJ201 does everything at 48Khz (AEC, AGC, etc.) since this is what the XMOS XVF3510 processes natively. In fact, you get two output channels that can be configured with different settings for noise suppression, AEC, AGC, etc. I'm seeing if (on the Mark II) we can get better performance by using one channel for wake word detection, and the other for speech to text.

[fquirin]

I'd really love to experiment with the SJ201 😁
Do you expect any improvements when splitting WW and ASR? Actually I was wondering why Mycroft decided to put 2Mics on the HAT since beamforming is probably useless with the geometry (direction detection might still work though).

[StuartIanNaylor]

Python is just slow for realtime DSP and even though there are examples for Torch and Speechbrain even with working Torchaudio on arm64 I am expecting the load to be horrendous.
Also FFT DSP on an application OS because of schedulers and frequency switching you bring in frequency drift and inaccuracies into the binning so a fixed cycle realtime OS of the microcontroller is actually preferential for DSP
I have searched compressively for beamforming, trying to find a alsa plugin or C lib but there are very few.
There is the ODAS tools that supposedly can I should have another look as it seemed more DOA and could not figure out how to use and call individual routines.

ODAS stands for Open embeddeD Audition System. This is a library dedicated to perform sound source localization, tracking, separation and post-filtering. ODAS is coded entirely in C, for more portability, and is optimized to run easily on low-cost embedded hardware. ODAS is free and open source.

Then there is an C toolkit with a python2 wrapper but its file based and I am just lost with C and swig to make it streaming https://github.com/kkumatani/distant_speech_recognition

Beamforming is not everything though as if you get a directional mic then perpendicular focus still always bleeds to some extent from the sides and generally this is set by the number of mics you have.
Amazon are heavy on beamforming alone I think and have up to 6 mics in there beamformer whilst Google use only 2 but then feed that into something called semi blind source separation that uses the near and far to make a cleaner near signal.
Its my rough understanding that method is chosen as its quite low load and not as sensitive to non linear echo and likely why I think the Google Nest Audio seems to have a touch better recognition that the echo 4.

Espressif did a limited run demonstration for there ESP32-S3 the ESP32-S3-Box that is a Amazon certified front end of 2 mic -> AEC -> BSS and it works amazingly well but its does contain Espressif blobs and even on that they have a 4 channel ADC for 2mics and 1 channel is the dac fed back as a ref channel to sync on a microcontroller.

Audio is just pressure waves in air and its much better to visualise as ripples in water as really its not directional, its going all directions and rebounding off different density surfaces.
There is a sort of max optimum distance you can beamform but after a certain distance so much becomes reverb and noise that I often think a distributed array microphone system to central ASR/TTS is a much better method.
The softmax probability of a microcontroller KWS sets the array element of use for the current intent sentence where its a matter of cost and choice to how many cover a room.
Its a shame the ESP32-S3-Box was a limited run and so far nobody does a esp32-s3-audio-dev-kit-mini as the microcontroller is $2.50 whilst say the xmos audio chips start at about $12.50 from memory.

So there is either no software and expensive hardware or software and no current hardware and the initial audio processing is essential to providing the expectations that users get with commercial offerings and its a big problem and always has been.

Then there is post processing and full DNN blind source separation where maybe a model for Spleeter rather than music based audio sets could be fed with voice based for 2 stem separation and github contains many DNN BSS variations.
RNNoise isn't great and prob works the wrong way round as it tries to detect noise and subtract it whilst via BSS its the opposite and you try to detect voice and extract it, which I think is more like RTXVoice.

There is so many parameters where the big commercial guys have a huge advantage to the bring your own mic and hardware party. Every mic and hardware setup has a spectral signature that can change significantly on what is used.
They provide specific hardware that their datasets are recorded through and are instantly more accurate by purely having hardware dictate.

I had to let my memory come back to me but ODAS might be worth another try as my tests where on a Pi3 which its a little load heavy but a Pi4 might be a much better platform.
Also I think I struggled to configure a conf file for the 2 mic which I was concentrating on whilst the 4 mic already has an example and run without AEC as Speex must run on hardware input + output that shares a clock so the ref signal doesn't drift.
Also the separation to a file might not be a big problem to setup some sort of alsa file source but I would have a play with ODAS 1st to see if 2mic is possible and how much load a 4 mic might make on a Pi4.
Also this time a 64bit build might be worthwhile to see if it is neon optimised as might run much better that way which is probably something I never checked.

[StuartIanNaylor]

OK Had a look at ODAS again and some memories coming back.
Load is actually nothing on a Pi4 64bit with respeaker 4 mic hat.

https://imgur.com/gallery/TFbcmNG

You need to import raw into audacity the config file I used was 1600khz and it seems that it exports the same channels as import so 4 channels.

This was the config file I used

# Configuration file for ReSpeaker 4 Mic Array (a hat for Raspberry Pi)

version = "2.1";

# Raw

raw: 
{

    fS = 16000;
    hopSize = 128;
    nBits = 32;
    nChannels = 4; 

    # Input with raw signal from microphones
    interface: {
        type = "soundcard";
        card = 2;
        device = 0;
    }

}

# Mapping

mapping:
{

    map: (1, 2, 3, 4);

}

# General

general:
{
    
    epsilon = 1E-20;

    size: 
    {
        hopSize = 128;
        frameSize = 256;
    };
    
    samplerate:
    {
        mu = 16000;
        sigma2 = 0.01;
    };

    speedofsound:
    {
        mu = 343.0;
        sigma2 = 25.0;
    };

    mics = (
        
        # Microphone 1
        { 
            mu = ( -0.0405, +0.0000, +0.0000 ); 
            sigma2 = ( +0.000, +0.000, +0.000, +0.000, +0.000, +0.000, +0.000, +0.000, +0.000 );
            direction = ( +0.000, +0.000, +1.000 );
            angle = ( 80.0, 90.0 );
        },

        # Microphone 2
        { 
            mu = ( +0.0000, +0.0405, +0.0000 ); 
            sigma2 = ( +0.000, +0.000, +0.000, +0.000, +0.000, +0.000, +0.000, +0.000, +0.000 );
            direction = ( +0.000, +0.000, +1.000 );
            angle = ( 80.0, 90.0 );
        },

        # Microphone 3
        { 
            mu = ( +0.0405, +0.0000, +0.0000 ); 
            sigma2 = ( +0.000, +0.000, +0.000, +0.000, +0.000, +0.000, +0.000, +0.000, +0.000 );
            direction = ( +0.000, +0.000, +1.000 );
            angle = ( 80.0, 90.0 );
        },

        # Microphone 4
        { 
            mu = ( +0.0000, -0.0405, +0.0000 ); 
            sigma2 = ( +0.000, +0.000, +0.000, +0.000, +0.000, +0.000, +0.000, +0.000, +0.000 );
            direction = ( +0.000, +0.000, +1.000 );
            angle = ( 80.0, 90.0 );
        }

    );

    # Spatial filters to include only a range of direction if required
    # (may be useful to remove false detections from the floor, or
    # limit the space search to a restricted region)
    spatialfilters = (

        {

            direction = ( +0.000, +0.000, +1.000 );
            angle = (80.0, 90.0);

        }

    );  

    nThetas = 181;
    gainMin = 0.25;

};

# Stationnary noise estimation

sne:
{
    
    b = 3;
    alphaS = 0.1;
    L = 150;
    delta = 3.0;
    alphaD = 0.1;

}

# Sound Source Localization

ssl:
{

    nPots = 4;
    nMatches = 10;
    probMin = 0.5;
    nRefinedLevels = 1;
    interpRate = 4;

    # Number of scans: level is the resolution of the sphere
    # and delta is the size of the maximum sliding window
    # (delta = -1 means the size is automatically computed)
    scans = (
        { level = 2; delta = -1; },
        { level = 4; delta = -1; }
    );

    # Output to export potential sources
    potential: {

        # format = "undefined";
        format = "json";

        interface: {
            # type = "blackhole";
            type = "terminal";
        };
    };

};

# Sound Source Tracking

sst:
{  

    # Mode is either "kalman" or "particle"

    mode = "kalman";

    # Add is either "static" or "dynamic"

    add = "dynamic";

    # Parameters used by both the Kalman and particle filter

    active = (
        { weight = 1.0; mu = 0.3; sigma2 = 0.0025 }
    );

    inactive = (
        { weight = 1.0; mu = 0.15; sigma2 = 0.0025 }
    );

    sigmaR2_prob = 0.0025;
    sigmaR2_active = 0.0225;
    sigmaR2_target = 0.0025;
    Pfalse = 0.1;
    Pnew = 0.1;
    Ptrack = 0.8;

    theta_new = 0.9;
    N_prob = 5;
    theta_prob = 0.8;
    N_inactive = ( 150, 200, 250, 250 );
    theta_inactive = 0.9;

    # Parameters used by the Kalman filter only

    kalman: {

        sigmaQ = 0.001;
        
    };
   
    # Parameters used by the particle filter only

    particle: {

        nParticles = 1000;
        st_alpha = 2.0;
        st_beta = 0.04;
        st_ratio = 0.5;
        ve_alpha = 0.05;
        ve_beta = 0.2;
        ve_ratio = 0.3;
        ac_alpha = 0.5;
        ac_beta = 0.2;
        ac_ratio = 0.2;
        Nmin = 0.7;

    };

    target: ();

    # Output to export tracked sources
    tracked: {

        # format = "undefined";
        format = "json";

        interface: {
            # type = "blackhole";
            type = "terminal";
        };

    };

}

sss:
{
    
    # Mode is either "dds", "dgss" or "dmvdr"

    mode_sep = "dds";
    mode_pf = "ms";

    gain_sep = 1.0;
    gain_pf = 10.0;

    dds: {

    };

    dgss: {

        mu = 0.01;
        lambda = 0.5;

    };

    dmvdr: {

    };

    ms: {

        alphaPmin = 0.07;
        eta = 0.5;
        alphaZ = 0.8;        
        thetaWin = 0.3;
        alphaWin = 0.3;
        maxAbsenceProb = 0.9;
        Gmin = 0.01;
        winSizeLocal = 3;
        winSizeGlobal = 23;
        winSizeFrame = 256;

    };

    ss: {

        Gmin = 0.01;
        Gmid = 0.9;
        Gslope = 10.0;

    }

    separated: {

        fS = 16000;
        hopSize = 512;
        nBits = 16;        

        interface: {
            type = "file";
            path = "separated.raw";
        }        

    };

    postfiltered: {

        fS = 16000;
        hopSize = 512;
        nBits = 16;        

        interface: {
            type = "file";
            path = "postfiltered.raw";
        }        

    };

}

classify:
{
    
    frameSize = 1024;
    winSize = 3;
    tauMin = 32;
    tauMax = 200;
    deltaTauMax = 7;
    alpha = 0.3;
    gamma = 0.05;
    phiMin = 0.15;
    r0 = 0.2;    

    category: {

        format = "undefined";

        interface: {
            type = "blackhole";
        }

    }

}

./odaslive -c respeaker_4_mic_array.cfg

You will get 2x raw files processed and seperated and prob this is as far as I got wondering what is exactly happening.
I think its going to take some playing as not exactly sure what co-ordinates the cfg creates for the 4mic and could even have this thing the wrong way round.

I think it must be the server app that creates the load as actually odaslive is extremely lite but to be honest puzzled by all what this means.
As I get the DOA but confused to beamforming and separation.

[StuartIanNaylor]

@fquirin

I had a look at the speechbrain doa and beamforming examples and checked for speed and if torchaudio seems to work now.
It does and much faster than I expected torch 1.10.2, torchaudio 0.10.2

from speechbrain.dataio.dataio import read_audio

from speechbrain.processing.features import STFT, ISTFT
from speechbrain.processing.multi_mic import Covariance
from speechbrain.processing.multi_mic import GccPhat
from speechbrain.processing.multi_mic import Mvdr

import torch
import time

xs_speech = read_audio('speech_-0.82918_0.55279_-0.082918.flac') # [time, channels]
xs_speech = xs_speech.unsqueeze(0) # [batch, time, channels]
xs_noise_diff = read_audio('noise_diffuse.flac') # [time, channels]
xs_noise_diff = xs_noise_diff.unsqueeze(0) # [batch, time, channels]
xs_noise_loc = read_audio('noise_0.70225_-0.70225_0.11704.flac') # [time, channels]
xs_noise_loc =  xs_noise_loc.unsqueeze(0) # [batch, time, channels]
fs = 16000 # sampling rate


ss = xs_speech
nn_diff = 0.05 * xs_noise_diff
nn_loc = 0.05 * xs_noise_loc
xs_diffused_noise = ss + nn_diff
xs_localized_noise = ss + nn_loc

stft = STFT(sample_rate=fs)
cov = Covariance()
gccphat = GccPhat()
mvdr = Mvdr()
istft = ISTFT(sample_rate=fs)
x = 0


while x < 10:
   starttime = time.time()
   Xs = stft(xs_diffused_noise)
   Nn = stft(nn_diff)
   NNs = cov(Nn)
   XXs = cov(Xs)
   tdoas = gccphat(XXs)
   Ys_mvdr = mvdr(Xs, NNs, tdoas)
   ys_mvdr = istft(Ys_mvdr)
   print(time.time() - starttime)
   x += 1

(venv) pi@raspberrypi:~/speechbrain $ python3 beamform.py
0.8916776180267334
0.8347172737121582
0.8364434242248535
0.832434892654419
0.8356015682220459
0.8487157821655273
0.851046085357666
0.8272495269775391
0.8370933532714844
0.8483691215515137

So approx x12 realtime as sample is 10 seconds with 4 channels on a pi400 Arm64 so maybe when chunked and streamed the load and performance is possible, likely less with 2 mic but again less accurate.
I would have to have a look at the code and see if and what the best chunk size is, greater the size = greater latency.
But torchaudio seems to work didn't actually check the results but remember I had to play with the best DOA for directive noise and beamforming.
In a domestic situation its always directive noise usually other media as diffuse noise are often low to silent.
Some work better directive some diffuse and forgot what my results where last time.

https://colab.research.google.com/drive/1UVoYDUiIrwMpBTghQPbA6rC1mc9IBzi6?usp=sharing
https://speechbrain.github.io/tutorial_processing.html

Might even reduce load by using ODAS for DOA as can connect to a socket and then pytorch speechbrain for the beamform alg.

The Generalized Eigenvalue Beamforming I think is a type of semi blind source separation where the near and far of a beamformer can be further cleaned which can improve speech recognition performance.
I am not so sure with the SJ201 board you can actually get near and far signals from it which is weird as getting 2 stages of different pipelines seems pointless unless on one you can turn all the effects off as presume the closer to far unprocessed has a better effect when passing through Generalized Eigenvalue Beamforming.
Its also quite lite running at x15 realtime so prob works well with the SJ201 as it doesn't have the overhead of doa beamforming to feed near / far, if you can get unprocessed far (just the mic sources)

[StuartIanNaylor]

Its always a mare working out orientation and mic order of the hardware and think this might be right for DOA

from speechbrain.dataio.dataio import read_audio
from speechbrain.processing.features import STFT
from speechbrain.processing.multi_mic import Covariance
from speechbrain.processing.multi_mic import SrpPhat
import sounddevice as sd
import threading
import torch
import time


def sd_callback(rec, frames, time, status):
   waveform = torch.from_numpy(rec).unsqueeze(0)
   Xs = stft(waveform)
   XXs = cov(Xs)
   doas = srpphat(XXs)
   print(doas)

rec_duration = 0.200
fs = 16000
num_channels = 4
bs = int(fs * rec_duration)
mics = torch.zeros((4,3), dtype=torch.float)
mics[0,:] = torch.FloatTensor([+0.027, -0.027, +0.00])
mics[1,:] = torch.FloatTensor([-0.027, -0.027, +0.00])
mics[2,:] = torch.FloatTensor([-0.027, +0.027, +0.00])
mics[3,:] = torch.FloatTensor([+0.027, +0.027, +0.00])

stft = STFT(sample_rate=fs)
cov = Covariance()
srpphat = SrpPhat(mics=mics)


# Start streaming from microphone
with sd.InputStream(channels=num_channels,
                    samplerate=fs,
                    blocksize=bs,
                    callback=sd_callback):
    threading.Event().wait()

You could prob try that with different chunk lengths so you get the best consistency and load I guess
Then just add the delay+sum beamforming to it if you think DOA is correct but load as expected with python is pretty horrendous

[StuartIanNaylor]

PS I had a look at Spleeter and the results are amazing but once again cpu wise the load is huge.

PS @synesthesiam the big data guys don't start turning off speech enhancement effects to fit there datasets they use datasets recorded with those speech enhancement effects.
Prob only effects to be omitted like Google probably do is to use a non beamforming input and beamforming as a near/far for a semi blind source separation to provide further speech enhancement but again the dataset of use will like be recorded through those speech enhancements.
In fact they probably don't even record but just have the DSP and engineering skills to add the hardware spectral signature to the training routine.

For Mycroft if the SJ201 board is going to be 'their' thing that all datasets should be processed through or at least have considerable portions of the dataset via the SJ201 board or create general large models and use transfer learning as the spectral signature of hardware can be thought of as a certain type of accent.

[StuartIanNaylor]

There is this and not exactly sure what this is but looks promising maybe might try it
https://hark.jp/

HARK is open-sourced robot audition software consisting of sound source localization modules, sound source separation modules and automatic speech recognition modules of separated speech signals that work on any robot with any microphone configuration.

Since a robot with ears may be deployed to various auditory environments, the robot audition system should provide an easy way to adapt to them. HARK provides a set of modules to cope with various auditory environments by using an open-sourced middleware, FlowDesigner and reduces the overheads of data transfer between modules.

HARK has been open-sourced since April 2008. The resulting implementation of HARK with MUSIC-based sound source localization, GHDSS-based sound source separation and Missing-Feature-Theory-based automatic speech recognition (ASR) on several robots like HRP-2, SIG, SIG2, and Robovie R2 attains recognizing three simultaneous utterances in real time.

[fquirin]

Hey @StuartIanNaylor there is a lot of info to digest thanks for writing it all down. I will need some time to work through all this but I'll definitely check it out ;-)

Just two quick comments. I've been experimenting with RNNoise as well. In Bullseye I failed to make it work but I tried a few days ago on Buster again ... and it's actually working insanely well on Rpi4! O_O. The only issue I have is that the VAD filter seems to be too agressive cutting a few samples at start. Another thing is that I'm still not sure if the resulting audio data is actually better for ASR systems, to the human ear it seems very clear but I fear there are subtle artifacts that the ASR engines are not trained on :-(. Besides that there is obviously a problem with any noise that is actual speech data since it is ment to let it through and cannot decide if its your voice or noise.

Another thing I was wondering is if beamforming makes any sense at all with the given hardware. I've tried to maximize audio quality by hand using the 4 channels of the Respeaker 4mic linear array and to be honest even with the outer distance of about 15cm there is not much of an improvement happening. I'm wondering if Amazon etc. even doing actual beamforming or if they just use this to do source separation or some fancy ML stuff on their massive cloud servers 🤔

[StuartIanNaylor]

I think amazon are beamforming as they have 6 mic devices so would be sort of pointless to invest in so much mic technology if but beamforming, otherwise what Google do not exactly sure as its a 2mic beamformer into semi blind source separation or some other voodoo that they keep to themselves but they do publish quite openly but never refer to which product.
There are so many types of beamforming though from time domain to frequency, hybrid to neural and any single beamformer has little in common with another other than the term beamform as how and what you beamform will give radically different results.

Stuff above basically too much load there was some Honda opensource that may have a fast BSS alg though.

RNNoise for me isn't that great as the artifacts are huge if your dataset is clean, if you record your dataset through RNNoise then they are not artifacts but features.
That is what the big guys do as there algs are not great but they do create great datasets whilst for some reason many persist with clean unprocessed datasets to put a processed audio stream through.
If I remember rightly RNNoise has relatively long latency and a few odd quirks for ASR but much is the strange relationship of trying to fit hardware to models whilst commercially its the other way round.
Its like spec augment which is often used for KWS & ASR that completely warp waveform and likely kill human recognition but when analysing audio as images it augments further accuracy.
https://arxiv.org/abs/1904.08779 pretty sure Google use spec augment think most do.

Both Amazon and Google seem to work far better than just beamforming but still do fail as the SNR of 3rd party gets above a threshold.
Put the unit between you and the TV and try recognition when the ads come on.

PS anyone any good at C as have a fairly simple low load Delay+Sum beamformer that I am trying to convert from file base to Alsa and currently screaming on my 1st ever C project.
The respeaker usb beamformer underperforms as its constantly doing TDOA on anything that is prevalent and doesn't lock to a source. It will beamform to the TV as happily as it will to you.

The esp32-s3-box feeds 2 channel aec into some form BSS to create its audio pipeline and not up to Google / Amazon standard but far better than many of us have managed to provide.
Prob is the libs are blobs so what exactly is extremely hard to know.

Its likely Amazon and google extract voice and don't try to cancel noise maybe some sort of fasnet that is on silicon https://github.com/yluo42/TAC

https://github.com/asteroid-team/asteroid have a ton of BSS models to try and play with that most can be quantised down with huge jumps in performance.

[StuartIanNaylor]

PS the write up was no worry as it was just me going through and checking for anything new and to refresh my memory as its been a while.

It is very possible to do beamforming wih many form of algs but with streaming DSP python just sucks big style. Its strange we don't have anything other than hobbyist or research submissions and nothing optimised as other audio elements in Linux such as ALSA or Pulseaudio as presume they would suck mightily if written in python.
Python has its uses but fast constant dsp is not one of them and there are loads of examples to be converted from research non realtime to a language that can and it amazes no one in the community has not done so.
Likely it will be a neural network utilising its internal C libs with a smattering of python marshalling input data but still haven't found anything.

Your post got me looking again but there is very little out there and why I am edging my bets of the esp32-s3 as its very cheap and at least has some ready libs for this kind of thing that work OK.

[StuartIanNaylor]

PS https://github.com/sekiguchi92/SoundSourceSeparation if you feed it with a 4 channel wav once more being python the speed is very slow but the results are extremely good.

python3 FastMNMF1.py mix-dual.wav

This was one of the earlier algs they where using I think Google has moved on and uses VoiceFilter-Lite as with there newer smart speakers you set up profiles which is very much how VoiceFilter-Lite works.
There are github examples for VoiceFilter but not VoiceFilter-Lite but you will not run them on a Pi

The example wav I used is here for a quickstart https://drive.google.com/file/d/1KfNXlDYsQU_EI9l88GlI-DDcJlQV5i4r/view?usp=sharing

The 3 track separation is here you will need to mute channels so you dont hear all at once
https://drive.google.com/file/d/1ETzc7skR_wYtKa-GvosG7FIEeJ7Kl7_v/view?usp=sharing

Speaker extraction seems to be where its at and a github serach for Speaker extraction again many are not optimised, streaming or quantised though.

I think VoiceFilter is not much more apart from it has a small attention model of a profiled voice so it only runs extraction on sections indicating that voice.

There is another alg I have found and its more straight semi blind source separation I don't know how it sounds as it just uses to random numpy arrays in the example and never got round applying to a near/far 2 track wav.
https://github.com/Okrio/bss-naec
Still runs really slow as loops in python are loops in python but as an alg goes quite simple just a shame none of us can use C

[fquirin]

That is indeed pretty impressive though I wonder if the result actually still works with ASR systems (too many subtle artifacts maybe?).

In general I try to stay away from everything that doesn't run on Raspberry Pi at the moment, but sometimes with enough optimization it is surprising how fast even the complex things can get.

[StuartIanNaylor]

Its python @fquirin as its a research or hobbyist language that is easy to write for research level workstations. The examples we get are research stuff but for production no real commercial entity would entertain using python for any type of dsp especially on an embedded platform.
For anything production no commercial entity would ever entertain running anything let alone audio dsp through python.
There is a huge rake of algs out there of very advanced cutting edge stuff where papers will present research algs in python but they hardly ever give away all the crown jewels in a productin optimised language lib or binary of some sort.
You can not optimise python in reality you can make it a lot faster but relative to an optimised language we are in a different class.
I have just been going on about it in the rhasspy forum as I don't know C or C++ but its pure crazy to try and do audio dsp with python as its ridiculously bad at it and unfortunately cython, numba whatever you through at it in comparison it sucks.

https://community.rhasspy.org/t/c-vs-python-and-then-some-thoughts-to-audio-dsp/3526

I find it so strange as have been banging on about this for over a year and that in this community arena we lack a single person with a modicum of c / c++ as no innovation is needed as much source code exists it just needs a bit of mangling to create streaming versions and release as common shared utils.

[fquirin]

There is a huge rake of algs out there of very advanced cutting edge stuff where papers will present research algs in python but they hardly ever give away all the crown jewels in a productin optimised language lib or binary of some sort

that is exactly the same problem I see :-/.
I that sense Python is probably a blessing and a curse at the same time and as you've mentioned C is hard and experts are rare :-(.
I've been using a lot of WebAssembly code lately in SEPIA HTML client to do VAD (WebRTC) and resampling (Speex). With a bit of C knowledge I'm pretty sure one could add the Speex AEC as well and get more control over all the parameters.
Of cause I'm not saying that hardware DSP wouldn't be the better solution but it's so much easier to work with software ... 🙈

[StuartIanNaylor]

@fquirin Doh! apols I forgot about https://github.com/athena-team/athena-signal which actually works quite well as does have beamforming and other goodies is C based but has a python wrapper and is pretty good with load.

Also there is https://github.com/breizhn/DTLN where always thought it was a bit load heavy but sanebow did a great job here https://github.com/SaneBow/PiDTLN

But just noticed someone has combined both https://github.com/avcodecs/DTLNtfliteC sort of name says it all but misses that it also athena and C.
Haven't checked how it is to setup or if needs to be converted to streaming sanebow does

I am trying to remember why I didn't initially like athena signal maybe its non streaming but will have a look.

[fquirin]

Ok that looks very interesting :-). At the bottom they mention WebRTC, Speex, I wonder if they imrpved on the code? Or maybe they have some more parameters to work with at least ^^.

[StuartIanNaylor]

Sanebow converted DTLN to a pretty good streaming version with PiDTLN

Athena signal seems much improved and the algs are super lite and great but yeah never did get my head round exactly how to get it to stream prob hackable.
I wrote my 1st C program yesterday 4 channel recorder with portaudio and it may be months before I ever go near C again :)
Its actually not that bad but I am such a noob its hard going, I have MS and short term memory problems and by the time I have took a necessary break to refresh I am back at square one again almost.
The owner of https://github.com/avcodecs/DTLNtfliteC prob would have a much better chance than I :)

[StuartIanNaylor]

Speex could work far better than it does as it occurred to me AEC in the context of use is less important than the use of a line canceller. AEC tries the very hard task of non linear echo cancelation and dereverberation whilst a line canceller is just a much simpler way top stop double talk (feedback of your output).
https://github.com/voice-engine/ec for me has always worked great and never tried but a line canceller uses just a smaller frame and window and probably the level of attenuation could be improved by dropping the hard code frame value and making sure you have got system latency defined pretty well in '-d 200' arg of its input.

'int frame_size = config.rate * 10 / 1000; // 10 ms' the frame is hardcode to the sample rate to give 10ms and prob should be modified to be part of the input parameters but for now a quick edit of ec.c does the same.
A line canceller prob would go as low as 16 maybe even 8 as powers of 2^ are preferable but always get confused if this should be 1,3,7,15,31 with all binary 1's or 1,2,4,8,16 with just the MSB being 1 and oh to the joy of Math.
Prob a line canceller (double talk) would operate window/frame as low as 1024/16 (maybe even lower) whilst the AEC parameters are aiming at 4096/160 at a guesstimate but those are the parameters to look for and how they are matching system latency as the smaller the window the more accurately you have to match latency as you could end up totally out of the echo window if the latency mismatch is greater.

Athena signal look like they only use speex as a reference as from looking its a total rewrite haven't really played with it just know load on a file input is very good.

Description: some codes of the residual echo supression processing refers to
Speex (https://www.speex.org/) which is an open source. It is devided into two
stages, the output of the 1st stage is the input of the double-talk detection,
and the output of the 2nd stage is the really nonlinear output.
Function framework:

1. the output of the linear result is used as input to estimate the residual echo
2. calculate prior SNR and posteriori SNR
3. calculate residual echo processing gain cofficient based on a posteriori SNR
   and prior SNR
4. output the final result

[StuartIanNaylor]

@fquirin Also I forgot to mention about hardware as hats are just about the worse as its near impossible to isolate the mics from vibration.
Never been sure why mics are not breakout or at least there are jumpers to allow so, but hey.

What you can do as telephones did for decade is also use uni-directional microphones that has a rear port that cancels sound that hits the front.
This provides close range unidirectional mic but more importantly it removes local vibration and noise.
Lavalier mics for a couple of $ can be picked up and you can tell as they have vented ports you can see at the back/base of the mic that on a omni front facing mic are not needed.
I posted this one as have no idea but from the pics the one for mobile is uni-dir and one for pc is omni-dir.
https://www.ebay.co.uk/itm/194867590415

Also the capsules themselves are approx $1

This can really help with your AEC as the AEC of the Mic is additional to your digital AEC and together make a great AEC combo.
The directional effect and AEC with a unidirectional is quite short ranged but perfect to reduce its own media as at quite short distances sound becomes reverberent and the difference at the front and back of the diaphragm becomes minimal.

[fquirin]

Also I forgot to mention about hardware as hats are just about the worse as its near impossible to isolate the mics from vibration.
Never been sure why mics are not breakout or at least there are jumpers to allow so, but hey

In that sense the 4mic linear and 6mic array from Seeedstudio are pretty cool because they have the HAT and mic-board separated :-).
I've also been somewhat successful with using GPIO extension cables.

[StuartIanNaylor]

Replying to wrong thread :)

Yeah you can have mics off board but for speex aec the dac & adc have to be on the same clock or clock drift will kill the AEC.
But yeah onboard mics are a crock as it cause all sorts of design choice but they are near impossible to provide any sound insulation.

https://www.scan.co.uk/products/enermax-ap001e-dreambass-usb-soundcard-plus-earphones-genie-with-integrated-80-hz-plus6-db-bass-boos
£5.99 stereo ADC soundcard as why people by expensive hats purely because they don't know what a mic is?!
Only downside is it looks like some sort of weird buttplug but other than that amazing.
https://github.com/rrbluke/CDEC
PS this is a type of semi blind source non linear aec where a 4 or 6 mic beamformer gives more accuracy and a narrower beam to act as the near signal whilst a single mic channel acts as far.
Its a research project in python and prob has not a chance of running unless on a fast machine.

[fquirin]

Replying to wrong thread :)

uch, true O_o, I find this "discussions" design so horrible somehow, like Microsoft Teams ... a stream of posts each with its own answers ... chaos! =)

£5.99 stereo ADC soundcard

It looks pretty fancy ^^. I have a simple USB sound card here for testing but haven't had much luck using it with the Raspberry. I mean it works but it didn't really improve AEC. My hope was the input-output sync. is more stable.

https://github.com/rrbluke/CDEC
... Its a research project in python and prob has not a chance of running unless on a fast machine

The paper seems to be an interesting read at least ^^.

[StuartIanNaylor]

@fquirin Just a thought as not used but pipewire is becoming the 'new' thing for linux its still webrtc but a new implementation have you tried the AEC of pipewire?

[fquirin]

To be honest I never heard of it 😅. Gotta do some research I guess 😁.
Is this going to be a Pulseaudio plugin or rather something entirely new?

[Edit] quickly looked it up, they definitely have a great 8-bit animation 😎 https://pipewire.org/

I remember now that I've seen it before. Does this already work on Raspberry Pi OS?

[StuartIanNaylor]

Its going to replace pulseaudio or aims to its the redhat fedora guys and what they are saying seems right Debian uses pulsewire not sure in the new raspios as raspberry seem to be making an ass of what they do OS wise but I run Debian GNU/Linux 11 (bullseye) and its pipewire by the look of things. I haven't done much as recently switched from Ubuntu as 20.04 looks great but the file manager is a dog actually I prefer bullseye and wil be sticking with it but haven't had a look at sound under the hood.
Its a complete new implementation and even though webrtc it could be a complete new rewrite the repo date is new.

Wim Taymans, Principal Engineer at Red Hat and co-creator of the GStreamer multimedia framework

[fquirin]

The audio system in Linux is so badly fragmented :-( I hope a new project doesn't mean we get another broken interface and shift resources from improving PulseAudio 😬
Let me know if you managed to do some tests 🙂

[StuartIanNaylor]

I think that is the answer they have come up with is to not bother with Pulseaudio and start again think it co-insides currently dunno both pulseaudio and pipewire are installed on Debian 11 that is about as far as I have got.