Black-box Optimization of DNN-based Source Enhancement for Increasing Objective Sound Quality

Copyright©2017 NTT corp. All Rights Reserved.
Black-box Optimization of
DNN-based Source Enhancement
for Increasing Objective Sound Quality
Oct., 26, 2017 @ MERL visiting
1
Yuma Koizumi
NTT Media Intelligence Laboratories

Copyright©2017 NTT corp. All Rights Reserved. 2
Self introduction
 Name: Yuma Koizumi
 Age: 27 (born in 1990 @ Tokyo, Japan)
 Background
 2008-2014: Hosei University
 B.S. and M.S. (received in 2012 and 2014)
 2016-2017: University of Electro-Communications
 Ph.D. degree (received in 2017)
 2014-Now: Researcher at NTT
 Research topics at NTT
 Source enhancement, machine learning, anomaly
detection, etc.

 Introduction
 Conventional method
 Proposed method
1. Basic idea
Black-box optimization to increase OSQA score
2. T-F mask selection based approach
presented in [Koizumi+, ICASSP-2017]
3. T-F mask estimation based approach
in peer review, [IEEE Trans. ASLP]
Table of contents

Introduction
Purpose: improve perceptual sound quality
of DNN source enhancement output
…
…
…
Source enhancement
𝑆 𝜔,𝜏
𝑁 𝜔,𝜏
𝑆 𝜔,𝜏
Automatic
speech recognition
…
Immersive audio
Telecommunication
Anomaly
detection
DNN source enhancement has been widely used
in various practical applications

…
…
…
Source enhancement
𝑆 𝜔,𝜏
𝑁 𝜔,𝜏
𝑆 𝜔,𝜏
Automatic
speech recognition
…
Immersive audio
Telecommunication
Anomaly
detection
How to train DNN source enhancement
to improve “perceptual” quality?
Introduction
Purpose: improve perceptual sound quality
of DNN source enhancement output

 Introduction
 Proposed method
1. Basic idea
Table of contents

Time-frequency (T-F) masking
𝑆 𝜔,𝜏
𝑋 𝜔,𝜏 𝑆 𝜔,𝜏
𝑁 𝜔,𝜏
𝐺 𝜔,𝜏
Filtering
𝑋 𝜔,𝜏
𝑋 𝜔,𝜏 𝐺 𝜔,𝜏
 T-F mask 0 ≤ 𝐺 𝜔,𝜏 ≤ 1 is multiplied to observation 𝑋 𝜔,𝜏
T-F mask
estimation
e.g. 𝑆 𝜔,𝜏 / 𝑋 𝜔,𝜏

Time-frequency (T-F) masking (cont’d)
𝑆 𝜔,𝜏
𝑋 𝜔,𝜏
T-F mask
estimation
e.g. 𝑆 𝜔,𝜏 / 𝑋 𝜔,𝜏
𝑆 𝜔,𝜏
𝑁 𝜔,𝜏
𝐺 𝜔,𝜏
Filtering
𝑋 𝜔,𝜏
𝑆 𝜔,𝜏
How to estimate 𝐺 𝜔,𝜏 ?
 T-F mask 0 ≤ 𝐺 𝜔,𝜏 ≤ 1 is multiplied to observation 𝑋 𝜔,𝜏

𝑆 𝜔,𝜏
𝑋 𝜔,𝜏
DNN-based
T-F mask
estimator
𝑆 𝜔,𝜏
𝑁 𝜔,𝜏
𝐺 𝜔,𝜏
𝑋 𝜔,𝜏
 Deep neural networks are used as T-F mask estimator
DNN-based source enhancement
…
……
……
……
……
…
……𝐗 𝐆
ℳ 𝐱|Θ
Filtering
Obs.
Sigmoid activation to
satisfy 0 ≤ 𝐺 𝜔,𝜏 ≤ 1
T-F mask

Cost
function
𝓙 Θ
Obs.𝐒
Target
𝐍
Noise
Training
data
𝑆 𝜔,𝜏
𝑋 𝜔,𝜏
DNN-based
T-F mask
estimator
𝑆 𝜔,𝜏
𝑁 𝜔,𝜏
𝐺 𝜔,𝜏
𝑋 𝜔,𝜏
DNN-based source enhancement
…
……
……
……
……
…
……𝐗 𝐆
ℳ 𝐱|Θ
T-F mask
Filtering
Obs.
 Deep neural networks are used as T-F mask estimator

 Gradient decent/assent based training
Θ ← Θ − 𝜆𝛻Θℐ Θ
 Mean-squared-error (MSE) is widely used
 Phase sensitive spectrum approximation (PSA) [Erdogan+, 2015]
 Complex ideal ratio mask (cIRM) 𝐺 𝜔,𝜏 ∈ ℂ [Williamson+ 2016]
 predicting real and imaginary parts of cIRM
 minimize MSE between 𝑆 𝜔,𝜏 and 𝐺 𝜔,𝜏 𝑋 𝜔,𝜏 on complex plane
Conventional DNN training
ℐ Θ = 𝔼 𝜔,𝜏 𝔑 cIRM − 𝔑 𝐺 𝜔,𝜏
2
+ ℑ cIRM − ℑ 𝐺 𝜔,𝜏
2
ℐ Θ = 𝔼 𝜔,𝜏 𝑆 𝜔,𝜏 − 𝐺 𝜔,𝜏 𝑋 𝜔,𝜏
2
11

 Minimize MSE ≠ maximize perceptual quality
Pros and cons of MMSE cost
Perceptually motivated cost
 Pros
 Simple implementation
 Easy to adopt backpropagation (analytical gradient)
 A lot of “know-how” for parameter-tuning
 Cons
12

 Introduction
 Proposed method
1. Basic idea
Table of contents

 Subjective evaluation would be best, however…
Perceptual motivated cost function
 What measurement should be used as ℐ Θ ?
𝑆 𝜔,𝜏 𝑋 𝜔,𝜏
𝑆 𝜔,𝜏
𝑁 𝜔,𝜏
Filtering
𝐺 𝜔,𝜏
Target source
…
…
……
…
MOS
Noise
 listening test without sleeping/resting night and day...
That is not realistic!!
14

Perceptual motivated cost function (cont’d)
𝑆 𝜔,𝜏 𝑋 𝜔,𝜏
𝑆 𝜔,𝜏
𝑁 𝜔,𝜏
Filtering
𝐺 𝜔,𝜏
Target source
…
…
……
…
Noise
 Use of Objective Sound Quality Assessment (OSQA)
 Subjective evaluation would be best, however…
 PESQ [ITU-U P.862]：Quality
 STOI [Taal+, 2011]：Intelligibility
OSQA
 What measurement should be used as ℐ Θ ?
 listening test without sleeping/resting night and day...
15

Black-box optimization
for DNN-source enhancement
 Most of OSQAs are black-box function
 Gradient 𝛻Θℐ Θ can not be calculated analytically
 DNN training: gradient assent
Θ ← Θ + 𝜆𝛻Θℐ Θ
 Need to calculate the gradient 𝛻Θℐ Θ
Applying black-box optimization
to train DNN source enhancement
To increase perceptual (black-box) motivated cost
16

 Introduction
 Proposed method
1. Basic idea
Table of contents

Reinforcement learning
 A technique of black-box optimization
 e.g. Atari [Minh+,2015] & Alpha-Go [Silver+, 2016]
Game
score
Reward func.Action selector
Action
candi-
dates
Action
…
…
…
… DNN is used as action “selector”
 Good/bad reward increase/decrease selection probability

Reinforcement learning (cont’d)
 A technique of black-box optimization
 e.g. Atari [Minh+,2015] & Alpha-Go [Silver+, 2016]
Reward func.
T-F
masking
Action
…
…
…
…
T-F mask
template
OSQA score
Mask selector
 DNN is used as T-F mask “selector”
 Good/bad OSQA score increase/decrease selection probability
Investigation weather DNN source enhancement
can be trained by black-box optimization scheme

Implementation
𝑮 𝜏 ← 𝓖 𝑎 𝜏
𝑎 𝜏 ← argmax 𝑎ℳ 𝒙 𝜏, 𝑎|Θ
 DNN ℳ 𝒙 𝜏, 𝑎|Θ selects a T-F mask template
 T-F mask templates are calculated by K-means algorithm
Reward
T-F
masking
…
…
…
…
Action
𝑁 𝜔,𝑘
𝑆 𝜔,𝑘
OSQA score
e.g. PESQ
T-F mask
templates
T-F mask
selector
Mask selector
20
𝓖 𝑎 ≔ 𝐺1,𝑎, 𝐺2,𝑎, … , 𝐺Ω,𝑎
⊤
where ℳ 𝒙 𝜏, 𝑎|Θ𝑎 = 1

21
Evaluation: setting
 Target source: ATR Japanese speech database (3316 utterances)
 Noise source: CHiME-3 noise dataset (cafes, street junctions, public
transport (buses), and pedestrian areas) [Barker+, 2015]
 SNR: 0, 3 and 6 dB
 Training data
 OSQA
 PESQ (P.862.2) and PEASS [Emiya+, 2011]
 DNN architecture
 Input: 64 × 11 units (mel-FB × frame concat.)
 Output: 32 actions (softmax activation)
 Hidden: 2 hidden layers, 64 units, and sigmoid activation
 Comparison method
 Fully-connected DNN estimates log-amplitude-spectrum,
(64 × 11 units )→ 128 sigmoid → 128 sigmoid → 64(linear)

Verification experiment
22
 Both OSQAs were increased as number of update increased
⇒ DNN could be trained by black-box optimization scheme!

Verification experiment
OSQA scores were still significantly lower than SOTA… 
PSA’s score
0dB: 2.36
6dB: 2.72
23
 Both OSQAs were increased as number of update increased
⇒ DNN could be trained by black-box optimization scheme!

 Introduction
 Proposed method
1. Basic idea
Table of contents

Motivation and summary
 Problem: low flexibility
OSQA-based cost Flexibility
MMSE-based ✓
RL-based ✓
This study ✓ ✓
 Key points
1. Gradient is calculated using sampling algorithm
2. DNN outputs PDF parameters of T-F-masked output signal
 Policy gradient method [Williams+,1992] is applied
 Finite-number of (only 32) T-F mask template is not enough
 Selection ⇒ Regression

ℐ Θ = 𝔼 𝐒,𝐗 ℬ 𝐒, 𝐗 = 𝔼 𝐗 𝔼 𝐒|𝐗 ℬ 𝐒, 𝐗
= 𝑝 𝐗 𝑝 𝐒|𝐗 ℬ 𝐒, 𝐗 𝑑𝐒 𝑑𝐗
= 𝑝 𝐗 𝑝 𝐒|𝐗, 𝚯 ℬ 𝐒, 𝐗 𝑑𝐒 𝑑𝐗
Complex Gaussian distribution
…
…
…
…
……
𝑮 𝒙 𝜏
𝝈 𝒙 𝜏
𝑿 𝜔,𝜏
T-F mask and variance
Real
Imag 𝑋 𝜔,𝜏
𝐺 𝜔,𝜏 𝑋 𝜔,𝜏
𝑝 𝑆 𝜔,𝜏|𝑋 𝜔,𝜏, Θ
 Cost: Expectation of OSQA ℬ 𝐒, 𝐗
 DNN estimates PDF of output signal 𝑝 𝐒|𝐗, 𝚯
Cost function and DNN architecture
26

 To calculate gradient, policy gradient method is applied
Gradient estimation
𝛻Θℐ Θ = 𝑝 𝐗 𝛻Θ 𝑝 𝐒|𝐗, 𝚯 ℬ 𝐒, 𝐗 𝑑𝐒 𝑑𝐗
= 𝑝 𝐗 𝑝 𝐒|𝐗, 𝚯 ℬ 𝐒, 𝐗 𝛻Θ ln 𝑝 𝐒|𝐗, 𝚯 𝑑𝐒 𝑑𝐗
Policygradient
[Williams+,1992]
OSQA Likelihood of
output signal
= 𝔼 𝐗 𝔼 𝐒|𝐗 ℬ 𝐒, 𝐗 𝛻Θ ln 𝑝 𝐒|𝐗, 𝚯

 To calculate gradient, policy gradient method is applied
Gradient estimation
𝛻Θℐ Θ = 𝑝 𝐗 𝛻Θ 𝑝 𝐒|𝐗, 𝚯 ℬ 𝐒, 𝐗 𝑑𝐒 𝑑𝐗
≈
1
𝐼
1
𝐾𝑇 𝑖
𝑖
ℬ 𝐒 𝑖,𝑘 , 𝐗(𝑖)
𝑘𝑡
𝛻Θ ln 𝑝 𝐒 𝑡
𝑖,𝑘
|𝐗 𝑡
𝑖
, 𝚯
Expectation is approximately calculated
by sampling algorithm
= 𝑝 𝐗 𝑝 𝐒|𝐗, 𝚯 ℬ 𝐒, 𝐗 𝛻Θ ln 𝑝 𝐒|𝐗, 𝚯 𝑑𝐒 𝑑𝐗
Policygradient
[Williams+,1992]

STEP-1: Estimate T-F mask and variance form 𝑖-th obs. 𝐗 𝑖
+Target
Training data
𝐒(𝑖)
𝐍(𝑖)
𝐗(𝑖)
…
…
𝑮 𝒙 𝜏
(𝑖)
𝝈 𝒙 𝜏
(𝑖)
Noise
……
…
|𝑿, Θ
Random
select
T-F mask sampling
T-F masking
Calc. ℬ 𝐒(𝑖,𝑘)
, 𝐗(𝑖)
𝐆(𝑖,𝑘)
𝐒(𝑖,𝑘)
Monte Carlo sampling (𝐾 times)
Calculate
𝛻Θ Θ
Back-
propagation
Repeat times
𝑋 𝜔,𝜏
𝑝 𝑆 𝜔,𝜏|𝑋 𝜔,𝜏, Θ
Training algorithm
Real
Imag
29

+Target
Training data
𝐒(𝑖)
𝐍(𝑖)
𝐗(𝑖)
…
…
𝑮 𝒙 𝜏
(𝑖)
𝝈 𝒙 𝜏
(𝑖)
Noise
……
…
|𝑿, Θ
Random
select
T-F mask sampling
T-F masking
, 𝐗(𝑖)
𝐆(𝑖,𝑘)
𝐒(𝑖,𝑘)
Calculate
𝛻Θ Θ
Back-
propagation
Repeat times
Training algorithm (cont’d)
STEP-2: Sample output signal 𝐾-times 𝑋 𝜔,𝜏
𝑆′ 𝜔,𝜏
𝑆 𝜔,𝜏 = 𝐺 𝜔,𝜏 𝑋 𝜔,𝜏𝑝 𝑆 𝜔,𝜏|𝑋 𝜔,𝜏, Θ
Real
Imag
30

+Target
Training data
𝐒(𝑖)
𝐍(𝑖)
𝐗(𝑖)
…
…
𝑮 𝒙 𝜏
(𝑖)
𝝈 𝒙 𝜏
(𝑖)
Noise
……
…
|𝑿, Θ
Random
select
T-F mask sampling
T-F masking
, 𝐗(𝑖)
𝐆(𝑖,𝑘)
𝐒(𝑖,𝑘)
Calculate
𝛻Θ Θ
Back-
propagation
Repeat times
STEP-2: Sample output signal 𝐾-times
STEP-3: Calculate 𝑘-th OSQA
𝛻Θℐ Θ ≈
1
𝐼
1
𝐾𝑇 𝑖
𝑖
𝑘𝜏
𝛻Θ ln 𝐒 𝜏
𝑖,𝑘
|𝐗 𝜏
𝑖
, 𝚯
OSQA
31

+Target
Training data
𝐒(𝑖)
𝐍(𝑖)
𝐗(𝑖)
…
…
𝑮 𝒙 𝜏
(𝑖)
𝝈 𝒙 𝜏
(𝑖)
Noise
……
…
|𝑿, Θ
Random
select
T-F mask sampling
T-F masking
, 𝐗(𝑖)
𝐆(𝑖,𝑘)
𝐒(𝑖,𝑘)
Calculate
𝛻Θ Θ
Back-
propagation
Repeat times
STEP-2: Sample output signal 𝐾-times
STEP-3: Calculate 𝑘-th OSQA
STEP-4: Calculate gradient as follow:
𝛻Θℐ Θ ≈
1
𝐼
1
𝐾𝑇 𝑖
𝑖
𝑘𝜏
𝛻Θ ln 𝑝 𝐒 𝜏
𝑖,𝑘
|𝐗 𝜏
𝑖
, 𝚯
OSQA
i.e. weight
Likelihood of
simulated output signal
32

Experimental conditions
 Training data
 Target: JPN speech from ATR speech database (6632 utterances)
 Noise: Noise data of CHiME-3
 SNR: random (-6 to 12 dB)
 Test data
 Target: JPN speech from NEW-Japan speech database (300 utterances)
 Noise: Environmental noise database (4 types of ambient noise)
 SNR: -6, 0, 6, 12 dB
 DNN architecture (fully-connected DNN)
 Input: log-mel-Fbank with 64 mel-filter × 11 frames (5 prev. & 5 future)
 Hidden: 3 hidden layers, 1024 hidden units, and ReLU activation
 Output: T-F mask with 64 mel-filter
 Comparison method
 PSA [Erdogan+, 2015] estimated by fully-connected DNN
33

Input SNR: -6 dB Input SNR: 0 dB Input SNR: 6 dB Input SNR: 12 dB
STOI
Improvement[%]
# of update # of update # of update # of update
PESQ
improvement
Results (PESQ & STOI)
 OSQAs were increased as number of updates increased
⇒ proposed method could increase black-box score
34

Experimental results
Cost func. SDR (dB) STOI (%) PESQ
PSA 𝟏𝟎. 𝟕 ± 𝟏. 𝟖𝟔 86.3 ± 3.81 2.36 ± 0.19
STOI 10.4 ± 2.01 𝟖𝟗. 𝟏 ± 𝟑. 𝟓𝟐 2.36 ± 0.19
PESQ 𝟏𝟎. 𝟕 ± 𝟏. 𝟖𝟒 86.7 ± 3.92 𝟐. 𝟒𝟖 ± 𝟎. 𝟏𝟗
Input SNR: 0 dB
Input SNR: 6 dB
Cost func. SDR (dB) STOI (%) PESQ
PSA 15.1 ± 1.67 92.8 ± 2.67 2.72 ± 0.17
STOI 𝟏𝟓. 𝟓 ± 𝟏. 𝟖𝟑 𝟗𝟒. 𝟔 ± 𝟐. 𝟓𝟏 2.65 ± 0.16
PESQ 14.8 ± 1.57 92.5 ± 2.91 𝟐. 𝟖𝟒 ± 𝟎. 𝟏𝟕
35
 PSA obtained averagely high-score for all measurements
 Proposed method obtained highest score for target OSQA
⇒ specialized to target OSQA (black-box score) !!
Orange: best, Purple: worst

Freq.[kHz]
Time [s]
Time [s]
Observed signal 𝑋 𝜔,𝜏
(SNR: 0dB)
Target source 𝑆 𝜔,𝜏
Freq.[kHz]
Freq.[kHz]Freq.[kHz]Freq.[kHz]
MMSE(PSA)PESQSTOI
T-F mask 𝐺 𝜔,𝜏 Output signal 𝑆 𝜔,𝜏
Time [s] Time [s]
SDR: 11.9 dB
PESQ: 2.44
STOI: 88.4 %
SDR: 11.3 dB
PESQ: 2.62
STOI: 87.4 %
SDR: 10.2 dB
PESQ: 2.25
STOI: 89.4 %
Result examples (T-F mask)
36

Result examples (sound)
37
 Male (SNR: 6dB)
 Female (SNR: 0dB)
Observation PSA STOI PESQ
Observation PSA STOI PESQ
Perhaps, we need a headphone to clearly interpret the differences.
After this talk, please listen to these files using a headphone.

 Improve perceptual quality of DNN-source enhancement
⇒ Black-box optimization of DNN-source enhancement
 Two approaches were introduced
 T-F mask selection based on RL-like approach
 T-F mask estimation based on policy gradient method
 OSQAs were increased as number of updates increased
 DNN could be specialized for target OSQA
⇒ Succeeded to train DNN using black-box cost!
Future work
 Other black-box cost
 Human-in-the-loop audio system [Niwa+, 2017]
 Fixed ASR system (word accuracy) [Watanabe+, 2014]
Conclusion
38

Q & A

Black-box Optimization of DNN-based Source Enhancement for Increasing Objective Sound Quality

More Related Content

Similar to Black-box Optimization of DNN-based Source Enhancement for Increasing Objective Sound Quality (20)

More from Yuma Koizumi (9)

Recently uploaded (20)

Black-box Optimization of DNN-based Source Enhancement for Increasing Objective Sound Quality