DNN-based frequency component prediction for frequency-domain audio source separation

Editor's Notes

• #2: Hi everyone, I’m Rui Watanabe / from National Institute of Technology（テクナーロジィ）, / Kagawa College, / Japan. I’m gonna talk about / DNN-based frequency component prediction / for frequency-domain audio source separation.
• #3: Audio source separation / is a technique（テクニーク）to separate audio sources / such as speech,↑ / singing voice,↑ / musical instruments↑, and so on↓. This technology（テクナーロジィ）can be used for many products / including an intelligent speakers,↑/ hearing-aid systems,↑ / and music editing by users↓.
• #4: In particular, / multichannel（マーチチャネル）audio source separation, / MASS（エムエイエスエス）in short, / estimates a separation system W / using multichannel（マーチチャネル） observation / without knowing the mixing system A.（WとAは指し示しながら） This technique（テクニーク）can be divided into two categories（キャテゴリーズ）, / for underdetermined / and overdetermined situations（スィテュエイションズ）. The underdetermined situation（スィテュエイション） is that / the number of microphones / is less than the number of sources in the mixture. For this case, multichannel（マーチチャネル）nonnegative matrix（メイトリクス）factorization, / MNMF in short, / is a popular（パピュラー）algorithm. Also, / many DNN-based approaches / have been proposed so far in this case. On the other hand, / in the overdetermined situation（スィテュエイション）, / the number of microphones is equal to / or larger than the number of sources. In this case, / frequency-domain independent component analysis / and independent low-rank matrix（メイトリクス）analysis / are the most reliable approaches.
• #5: In this presentation,↑/ we only treat “frequency-domain MASS”. In this algorithm↑, / we perform / a short-time Fourier transform to the observed time-domain signal / and obtain the multichannel（マーチチャネル）spectrograms.（図の紫部分を指しながら） Then, / we estimate a frequency-wise separation filter, / which is applied to each frequency like this（図の中央を指しながら）/ to estimate the separated source signals.
• #6: Let me introduce the conventional frequency-domain MASS called “multichannel（マーチチャネル）nonnegative matrix（メイトリクス）factorization,” / MNMF in short. This is an unsupervised source separation algorithm / and does not require any prior（プライォア）information or training（テュレイニン）. As an unsupervised technique（テクニーク）, MNMF tends to provide high quality separation performance. In MNMF, / the observed multichannel signal / is represented by the time-frequency-wise channel correlation matrices / denoted by X. Since X is a frequency-by-time matrix whose element is a channel-by-channel matrix↑, / this is a matrix of matrices, / which is a fourth-order tensor（テンサー）. （frequency-by-timeのところは指し示しながら） MNMF decomposes X / into the source-wise spatial model（マドー）/ and the low-rank spectral model（マドー）of all the sources. Thus, / by clustering the spectral model（マドー）into each source using the estimated spatial model（マドー）, the source separation is achieved. However, it requires a huge computational cost / for estimating the parameters（パラミターズ）/ because there are so many parameters（パラミターズ）in this model（マドー）.
• #7: In this presentation, / our motivation（モーティベイシュン） is that / we want to achieve a high-quality MASS / with a low computational cost. And we propose / a new source separation framework / combining frequency-domain MASS / and deep neural networks. In this framework, / as an initial process, / the mixture signal in specific frequencies are separated by MNMF, / and we obtain the separated source components in that frequencies. In this figure, / since only the low-frequency band of the mixture is input to MNMF, we can get the separated components in the low-frequency band. Of course, / the high-frequency bands of the separated sources / are missing. （しっかり間を開ける） As a post process, / we apply DNN-based frequency component prediction, / namely, / the missing high-frequency bands of the separated sources are predicted by DNN, / where we input not only the separated low-frequency bands / but also the mixture of the high-frequency band.（inputの矢印をそれぞれ指し示しながら） Since the DNN prediction process is much faster than MNMF process, / we can reduce / the total computational cost in this framework. For example, if we divide the frequency bands in half like figure,↑/ we can reduce the computational time / almost half.
• #8: In our framework, / the post DNN process can be interpreted in two ways. First, / the DNN is an audio source separation of specific frequencies, / high-frequency band in this figure. Please note that / the low-frequency bands can be used for predicting high-frequency separated components in our DNN model（マドー）. Second, / the DNN seems to be a bandwidth expansion of each source / because the high-frequency bands are predicted. In general, a bandwidth expansion is a hard task / even for DNN. However, / in our model,（マドー）/ the high-frequency band of the mixture / becomes a strong cue / to achieve the bandwidth expansion.
• #9: The details of the proposed method is as follows. First, / the observed multichannel spectrograms / M1 and M2 / are divided into low- and high-frequency bands. Then, / we apply MNMF to only the low-frequency band / M1(L) and M2(L) / to obtain the separated source components / Y1(L) and Y2(L). The high-frequency band / M1(H) and M2(H) / are not separated in this step.
• #10: Next, / we input the high-frequency band of the mixture / and the low-frequency bands of the separated sources / like this figure. The DNN / outputs two soft-masks / W1 and W2 / such that the high-frequency bands of the separated sources are calculated from M1(H) / by multiplying them. Of course, / the masks are the matrices with the elements between zero（ジロー）and one, / and the sum of each element in W1 and W2 is always unity.
• #11: The DNN prediction is performed for each time frame j, / which is / each column of spectrograms. To utilize the information along time in the prediction, / the input vector for DNN is a concatenation of several time frames around j in the mixture and the separated sources. Also, / before we input the vector to the DNN, we normalize it to stabilize the model（マドー）training（テュレイニン）, / where the normalization coefficient is added / to keep the information of the signal volume.
• #12: The DNN model（マドー）in the proposed method is very simple. We have full-connected four hidden layers, / and we apply Swish function / to each hidden layer. Just before the output, / we apply frequency-wise Softmax function, / to ensure（インシュァー）that / the sum of the masks equals unity in each frequency. The mean squared error / between the separated source vector and the label（レイボーゥ）vector↑/ is used as a loss function of the DNN training（テュレイニン）.
• #13: To confirm the validity of the proposed method, / we have done two experiments（イクスペリメンツ）. In the first experiment, / we evaluate the performance of the DNN model（マドー）/ as the bandwidth expansion. That is, / the DNN restores the high-frequency band from the low-frequency band of the completely separated sources, / where we confirm whether the high-frequency band of the mixture is effective / by comparing these two models（マドーズ）.（図を指しながら） Therefore, / we can confirm the validity of the proposed framework / that utilizes mixture components / for predicting the separated sources. As an evaluation score, / we use sources-to-artifact ratio（レイシオ）, / SAR, / which shows the absence of artificial distortions / in the estimated audio signals.
• #14: This slide shows experimental conditions. For the training of DNN, / we used 100 songs with drums and vocals / in the SiSEC2016 database.（トゥエンティシックスティーンと発音） The boundary frequency between low- and high-frequency bands / was set to 4kHz, / which is a half of Nyquist frequency. As the test dataset, / we used four songs included in the SiSEC2011 database（トゥエンティイレブンと発音）, / where these songs are the mixture of drums and vocals.
• #15: This is the result of bandwidth expansion. For each song, / we showed the SAR values of Drums and Vocals. Higher SAR indicates better audio quality. Two columns show the results of DNN without mixture / and DNN with mixture. In almost all results, / the DNN with mixture outperforms the DNN without mixture. From this result, / we can confirm that / the mixture components help to predict the high-frequency band of the separated sources. Thus, / we can expect that / the proposed framework will perform effectively / in a source separation task.
• #16: Next, / we conducted the MASS experiment. We compare the conventional MNMF and the proposed framework. The conventional method separates fullband mixture by MNMF / whereas the proposed framework separates only the low-frequency band by MNMF, / and the high-frequency band is predicted by DNN post process. We expect that / the computational time is reduced by skipping half number of frequencies in the MNMF process / while the separation performance is almost the same. As a source separation score, / we used source-to-distortion ratio（レイシオ）, / SDR, / which represents the total performance of source separation including both “degree of separation” and the “quality of separated signals.”
• #17: The other conditions are shown in this slide.（ここのtheは重要，otherとthe otherは違う） DNN is trained using the same dataset in the previous experiment. For the MASS test data, / we produced two-channel observed mixtures / by convoluting E2A impulse responses to the Drums and Vocals sources of the test dataset, / where the recording condition of E2A impulse response is depicted here.（図を指しながら） The reverberation time of E2A is 300 ms. The number of bases in MNMF was set to 13, / which provides the best result for both the conventional and proposed methods.
• #18: This is the result for each song. The vertical axis indicates SDR score / averaged over 10 random initial values.（指しながら） The horizontal axis shows the average elapsed time.（指しながら） The black line is the conventional method, / fullband MNMF, / and the red circles are the results of the proposed framework. Since the elapsed time depends on the number of iterations of parameter update in MNMF,↑/ for the proposed framework,↑/ we plot the results with every 10 iterations in the MNMF process. Of course, / the computational time for the DNN prediction process / is included in each red circle, / although the DNN process requires less than 0.1 s（ジロポイントワンセカンズ）. In all the results, / we can confirm the efficacy of the proposed method. In particular, / Song ID 4 shows the result just as we expected, / so let me explain the result of Song ID 4.
• #19: In the case of Song ID 4, / the proposed method achieves 13 dB / in less than 50 s, / whereas fullband MNMF converged to 13 dB in 120 s. This is because / the number of frequencies in MNMF is reduced by half.
• #20: In addition, / the proposed method outperforms fullband MNMF in Song IDs 1, 2, and 4. In particular, / the improvement in Song ID 1 is very large. The reason of these improvements might be that / the proposed method performed more accurate estimation of high-frequency band sources / based on the training with 100 songs. Also, / in the case of Song ID 1, / fullband MNMF might be trapped into a bad local minimum during the iterative optimization.