Robust Sound Field Reproduction against Listener’s Movement Utilizing Image Sensor
1 like12,317 views
This document proposes a sound field reproduction system that utilizes an image sensor to estimate a listener's position and orientation. It summarizes challenges with existing spectral division methods that rely on a fixed reference listening position. The proposed system applies an "equiangular filter" to the spectral division method to locally synthesize sound fields around the listener by limiting the spatial bandwidth, as estimated from the image sensor. Simulation experiments and subjective assessments show the proposed system can accurately reproduce sound fields at the listener's position regardless of frequency, with improved directional perception over conventional methods. However, it did not clearly outperform alternatives in terms of sound quality. Overall, the system aims to address limitations of existing approaches by adapting sound field synthesis based on estimated listener parameters
![Research background (2/3)
Spectral Division Method (SDM) [J. Ahrens, S. Spors., 2008]
One of the SFR methods that reproduces the sound field by synthesizing a
number of wavefronts.
This method can be realized with a simple system like linear loudspeaker
array.
However, SDM has two problems.
Problem 1: A sound pressure error is occurred by mismatching the
reference listening line.
Problem 2: A disturbance of wavefront is occurred by a spatial aliasing.
Reproduction accuracy: Low
Reproduction region: Wide
High
We aim to reproduce the sound field with high
accuracy by solving these problems in SDM.](https://image.slidesharecdn.com/microsoftpowerpoint-ncsppresentation-140310002737-phpapp02/85/Robust-Sound-Field-Reproduction-against-Listener-s-Movement-Utilizing-Image-Sensor-4-320.jpg)
![Spectral Division Method (SDM) [J. Ahrens, S. Spors., 2008]
Primary source
Primary source
nth secondary
source
nth secondary
source
Reference
listening line
Reference
listening line
Spatial domain
IDFT
Fourier transform
Wavenumber domain
The driving function in the wavenumber domain
The driving function in the spatial domain
： angular frequency
： wavenumber in
： speed of sound
-direction
： imaginary unit
： reference listening distance
： zero-th order modified Bessel function of the second kind
： zero-th order Hankel function of the second kind](https://image.slidesharecdn.com/microsoftpowerpoint-ncsppresentation-140310002737-phpapp02/85/Robust-Sound-Field-Reproduction-against-Listener-s-Movement-Utilizing-Image-Sensor-6-320.jpg)
![Spectral Division Method (SDM) [J. Ahrens, S. Spors., 2008]
Primary source
Primary source
nth secondary
source
nth secondary
source
Reference
listening line
Reference
listening line
Spatial domain
IDFT
Fourier transform
Wavenumber domain
The driving function in the wavenumber domain
The driving function in the spatial domain
：reference listening distance
Problems in SDM
A sound pressure error is occurred by mismatching the reference listening line.
A disturbance of wavefront is occurred by a spatial aliasing.](https://image.slidesharecdn.com/microsoftpowerpoint-ncsppresentation-140310002737-phpapp02/85/Robust-Sound-Field-Reproduction-against-Listener-s-Movement-Utilizing-Image-Sensor-7-320.jpg)
![Problem 2: spatial aliasing (1/2)
0
10
-24
0
-48
0
R参
加
-30
30
20
0
10
-24
0
-48
-30
0
30
Spectral overlap occurs
Discretization of the secondary source
Magnitude[dB]
20
Magnitude [dB]
In SDM, a spectral overlap of the driving function is occurred by
discretization of secondary source, and filter power at high frequency
becomes larger like in the right figure.](https://image.slidesharecdn.com/microsoftpowerpoint-ncsppresentation-140310002737-phpapp02/85/Robust-Sound-Field-Reproduction-against-Listener-s-Movement-Utilizing-Image-Sensor-9-320.jpg)
![Local sound field synthesis (1/2) [J. Ahrens, S. Spors., 2011]
0
10
-24
0
-48
-30
0
30
Spectral overlap occurs
20
0
10
-24
0
-48
-30
0
30
Spectral overlap is suppressed
Rectangular window for the spectrum of the driving function
By applying a rectangular window to a spectrum in the left figure, we
enable to suppress a spectral overlap like in the right figure.
Magnitude[dB]
20
Magnitude[dB]
Local sound field synthesis: the method enables to suppress a spatial
aliasing by limiting spatial bandwidth in the wavenumber domain.](https://image.slidesharecdn.com/microsoftpowerpoint-ncsppresentation-140310002737-phpapp02/85/Robust-Sound-Field-Reproduction-against-Listener-s-Movement-Utilizing-Image-Sensor-11-320.jpg)
![Local sound field synthesis (2/2) [J. Ahrens, S. Spors., 2011]
By applying a rectangular window, we enable to suppresses a
disturbance of wavefront and enable to increase the maximum
frequency in which the sound field can be correctly reproduced.
Synthesized wavefront (unfiltered)
Synthesized wavefront (filtered)
0.0
-1.5
0.0
1.5
-0.10
Spatial aliasing occurs
1.5
0.00
0.0
-1.5
0.0
1.5
Amplitude
0.00
Amplitude
1.5
0.10
3.0
0.10
3.0
-0.10
Disturbance of wavefront is suppressed
Reproduction area is localized
Therefore, It is necessary to design a filter to precisely control the
reproduced direction in order to take advantage of this method.](https://image.slidesharecdn.com/microsoftpowerpoint-ncsppresentation-140310002737-phpapp02/85/Robust-Sound-Field-Reproduction-against-Listener-s-Movement-Utilizing-Image-Sensor-12-320.jpg)
![Result of applying the equiangular filter (1/2)
An example when we applied a designed filter to a spectrum
0
10
-24
0
-48
-30
0
30
Spectral overlap occurs
and the angular width
is
.
20
0
10
-24
0
-48
-30
0
30
Spectral overlap is suppressed
Equiangular filter for the spectrum of the driving function
Equiangular filter used in this presentation is cut by applying a low-pass
filter with respect to the frequency that exceeds the maximum
frequency
, and we do not reproduce the sound field.
Magnitude[dB]
20
is
Magnitude[dB]
This case that the angular](https://image.slidesharecdn.com/microsoftpowerpoint-ncsppresentation-140310002737-phpapp02/85/Robust-Sound-Field-Reproduction-against-Listener-s-Movement-Utilizing-Image-Sensor-14-320.jpg)
Robust Sound Field Reproduction against Listener’s Movement Utilizing Image Sensor
- 1. Robust Sound Field Reproduction against Listener’s Movement Utilizing Image Sensor Toshihide Aketo，Hiroshi Saruwatari，Satoshi Nakamura (Nara Institute of Science and Technology, Japan)
- 2. Outline Research background Conventional method Spectral Division Method Local sound field synthesis Proposed method Equiangular filter Sound field reproduction system utilizing image sensor Simulation experiment Subjective assessment on directional perception on sound quality
- 3. Research background (1/3) Objective of sound field reproduction (SFR) system To reproduce the primary sound field to another space with wide range and high accuracy. However, it is difficult to realize such a system because the system size becomes larger and the system configuration becomes complex. Therefore, the recent research is focused on reproducing sound field with wide range and high accuracy using small and simple system. Surrounded (large and complex) Circular or spherical (a little complex) Linear or planer (simple) Boundary surface control (BoSC) Ambisonics Stereo or surround system Wave field synthesis (WFS) Focused Complex Simple
- 4. Research background (2/3) Spectral Division Method (SDM) [J. Ahrens, S. Spors., 2008] One of the SFR methods that reproduces the sound field by synthesizing a number of wavefronts. This method can be realized with a simple system like linear loudspeaker array. However, SDM has two problems. Problem 1: A sound pressure error is occurred by mismatching the reference listening line. Problem 2: A disturbance of wavefront is occurred by a spatial aliasing. Reproduction accuracy: Low Reproduction region: Wide High We aim to reproduce the sound field with high accuracy by solving these problems in SDM.
- 5. Research background (3/3) To cope with these problems, we propose the novel SFR system with linear loudspeaker array, which combines listener’s position estimation by Kinect and SDM with local sound field synthesis. Image sensor Kinect Local sound field synthesis Reproduction accuracy Low Reproduction region: Wide Reproduction accuracy: High Reproduction region: localized around listener
- 6. Spectral Division Method (SDM) [J. Ahrens, S. Spors., 2008] Primary source Primary source nth secondary source nth secondary source Reference listening line Reference listening line Spatial domain IDFT Fourier transform Wavenumber domain The driving function in the wavenumber domain The driving function in the spatial domain ： angular frequency ： wavenumber in ： speed of sound -direction ： imaginary unit ： reference listening distance ： zero-th order modified Bessel function of the second kind ： zero-th order Hankel function of the second kind
- 7. Spectral Division Method (SDM) [J. Ahrens, S. Spors., 2008] Primary source Primary source nth secondary source nth secondary source Reference listening line Reference listening line Spatial domain IDFT Fourier transform Wavenumber domain The driving function in the wavenumber domain The driving function in the spatial domain ：reference listening distance Problems in SDM A sound pressure error is occurred by mismatching the reference listening line. A disturbance of wavefront is occurred by a spatial aliasing.
- 8. Problem 1 : sound pressure error A sound pressure is correctly reproduced only on the reference listening line under 2.5-dimensional synthesis condition. Sound pressure is correctly reproduced on the reference listening line. 2.0 2.0 1.0 1.0 0.0 0.0 -1.0 0.0 1.0 Primary sound field -1.0 0.0 Sound pressure error 1.0 occurs outside the reference listening line. Reproduced sound field Therefore, to correctly reproduce the sound field to listener's position, we must set the reference listening distance equal to listener's distance.
- 9. Problem 2: spatial aliasing (1/2) 0 10 -24 0 -48 0 R参 加 -30 30 20 0 10 -24 0 -48 -30 0 30 Spectral overlap occurs Discretization of the secondary source Magnitude[dB] 20 Magnitude [dB] In SDM, a spectral overlap of the driving function is occurred by discretization of secondary source, and filter power at high frequency becomes larger like in the right figure.
- 10. Problem 2: spatial aliasing (2/2) The effect of spectral overlap in the wavenumber domain appears as a spatial aliasing in the spatial domain. 1.5 0.00 0.0 -1.5 0.0 1.5 -0.10 3.0 Synthesized wavefront (discrete array) 0.10 1.5 0.00 0.0 -1.5 0.0 1.5 Amplitude 0.10 Amplitude 3.0 Synthesized wavefront (continuous array) -0.10 Disturbance of wavefront occurs Discretization of the secondary source
- 11. Local sound field synthesis (1/2) [J. Ahrens, S. Spors., 2011] 0 10 -24 0 -48 -30 0 30 Spectral overlap occurs 20 0 10 -24 0 -48 -30 0 30 Spectral overlap is suppressed Rectangular window for the spectrum of the driving function By applying a rectangular window to a spectrum in the left figure, we enable to suppress a spectral overlap like in the right figure. Magnitude[dB] 20 Magnitude[dB] Local sound field synthesis: the method enables to suppress a spatial aliasing by limiting spatial bandwidth in the wavenumber domain.
- 12. Local sound field synthesis (2/2) [J. Ahrens, S. Spors., 2011] By applying a rectangular window, we enable to suppresses a disturbance of wavefront and enable to increase the maximum frequency in which the sound field can be correctly reproduced. Synthesized wavefront (unfiltered) Synthesized wavefront (filtered) 0.0 -1.5 0.0 1.5 -0.10 Spatial aliasing occurs 1.5 0.00 0.0 -1.5 0.0 1.5 Amplitude 0.00 Amplitude 1.5 0.10 3.0 0.10 3.0 -0.10 Disturbance of wavefront is suppressed Reproduction area is localized Therefore, It is necessary to design a filter to precisely control the reproduced direction in order to take advantage of this method.
- 13. Equiangular filter In order to design a filter to accurately control the reproduced direction, we derive the relation equation between reproduced direction , wavenumber in -direction and frequency . constant proportional ： wavenumber in -direction ： speed of sound ：reproduced direction ： frequency If reproduced direction is constant, since it is found that proportional to , we design a new filter as follows ： angular frequency ： angular width ： wavenumber ： equiangular filter is
- 14. Result of applying the equiangular filter (1/2) An example when we applied a designed filter to a spectrum 0 10 -24 0 -48 -30 0 30 Spectral overlap occurs and the angular width is . 20 0 10 -24 0 -48 -30 0 30 Spectral overlap is suppressed Equiangular filter for the spectrum of the driving function Equiangular filter used in this presentation is cut by applying a low-pass filter with respect to the frequency that exceeds the maximum frequency , and we do not reproduce the sound field. Magnitude[dB] 20 is Magnitude[dB] This case that the angular
- 15. Result of applying the equiangular filter (2/2) By applying the equiangular filter, we enable to suppress a disturbance of wavefront and enable to reproduce the sound field to the specific direction. Synthesized wavefront (unfiltered) Synthesized wavefront (filtered) 0.0 -1.5 0.0 1.5 -0.10 Spatial aliasing occurs 1.5 0.00 0.0 -1.5 0.0 1.5 Amplitude 0.00 Amplitude 1.5 0.10 3.0 0.10 3.0 -0.10 Disturbance of wavefront is suppressed However, there is a problem that it is impossible to match the sweet spot to the listener’s position if listener’s direction is unknown in advance.
- 16. Summary of problems Problems in SDM A sound pressure error occurs in the case that the reference listening distance does not match listener's distance. A spatial aliasing is occurred by discretization of secondary sources. Second problem can be solved by applying an equiangular filter Problems in equiangular filter It is impossible to match the sweet spot to the listener’s position if listener’s direction is unknown in advance. These problems can be solved if we know the listener’s position, therefore, introduction of the image sensor enables to solve these problems.
- 17. Condition of simulation experiment Primary source (monopole source) 34 ch linear secondary source array (monopole source) Parameter name measurement plane aliasing frequency Parameter value W4.0 D4.0 approximately 2019 Hz angular width reproduced direction Reference listening line synthesis frequency 3, 5 kHz Evaluation score ： radiation characteristic of primary sound field ： radiation characteristic of secondary sound field It is assumed that listener’s position is obtained by the image sensor, we calculate the reproduced direction from sound source position and listener's position.
- 18. Results of simulation experiment 0.10 0.10 2.0 1.0 0.00 0.0 -1.0 -1.5 0.0 1.5 -0.10 2.0 1.0 0.00 0.0 -1.0 -1.5 0.0 Amplitude Synthesized wavefront (5 kHz) Amplitude Synthesized wavefront (3 kHz) 1.5 -0.10 Evaluated value (3 kHz) Evaluated value (5 kHz) 0 0 2.0 2.0 -24 0.0 -1.0 1.0 -24 -48 1.0 0.0 -48 -1.0 -1.5 0.0 1.5 -1.5 0.0 1.5 ： Listener ： Primary source
- 19. Results of simulation experiment 0.10 0.10 2.0 1.0 0.00 0.0 -1.0 -1.5 0.0 1.5 -0.10 2.0 1.0 0.00 0.0 -1.0 -1.5 0.0 Amplitude Synthesized wavefront (5 kHz) Amplitude Synthesized wavefront (3 kHz) 1.5 -0.10 Evaluated value (3 kHz) Evaluated value (5 kHz) 0 0 2.0 2.0 -24 0.0 -1.0 1.0 -24 -48 1.0 0.0 -48 -1.0 -1.5 0.0 1.5 -1.5 0.0 1.5 The sound field is correctly reproduced at listener’s direction regardless of the frequency. ： Listener ： Primary source
- 20. Condition of subjective assessment on directional perception parameter name Acoustic transparent curtain ： Primary source ： Answer number card parameter value sampling frequency 48 kHz quantization bit rate 16 bit test sound white Gaussian noise with 3 seconds aliasing frequency 34 ch linear loudspeaker array angular width approximately 2019 Hz sound source direction number of evaluator type of sound source Loudspeaker distance Reference listening line 7 ・sound source without bandwidth limitation (Conventional1) ・sound source with bandwidth limitation in frequencies under 2 kHz (Conventional2) ・sound source in which we applied the equiangular filter(Proposed) Evaluation score Pos 1 Pos 2 Pos 3 ： number of evaluator ： answered direction ： true source direction We asked evaluators to answer which card position you perceive the sound source exists as an evaluation procedure.
- 21. Results of subjective assessment on directional perception Conventional1 (without bandwidth limitation) Conventional2 (with bandwidth limitation in frequencies under 2 kHz) Proposed (in which we applied the equiangular filter) Bad (a) In Pos1 (b) In Pos2 (c) In Pos3 Good Proposed is superior to Conventional1 and Conventional2 in Pos1 and Pos2. However, Proposed is almost the same as Conventional2 in Pos3. This is because in equiangular filter, as the angle of reproduced direction becomes larger, the maximum frequency becomes low. As the user moves to right (from Pos1 to Pos3), directional perception error of Conventional1 becomes larger owing to the effect of a spatial aliasing. The superiority of the proposed method is shown on directional perception.
- 22. Condition of subjective assessment on sound quality Acoustic transparent curtain ： Primary source ： Reference loudspeaker parameter name parameter value sampling frequency 34 ch linear loudspeaker array 48 kHz quantization bit rate 16 bit test sound aliasing frequency White Gaussian noise with 3 seconds approximately 2019 Hz angular width Loudspeaker distance sound source direction number of evaluator type of sound source Reference listening line Pos 1 Pos 2 Pos 3 7 ・sound source without bandwidth limitation (Conventional1) ・sound source with bandwidth limitation in frequencies under 2 kHz (Conventional2) sound source in which we applied the equiangular filter(Proposed) We sounded two synthesized sound after reference sound radiated by reference loudspeaker, and asked evaluators to answer which synthesized sound you perceive closer to the reference sound as an evaluation procedure.
- 23. Results of subjective assessment on sound quality Conventional1 (without bandwidth limitation) Conventional2 (with bandwidth limitation in frequencies under 2 kHz) Proposed (in which we applied the equiangular filter) Good (a) In Pos1 (b) In Pos2 (c) In Pos3 ꥰꥰ Bad In all results, evaluators chose Conventional1 or Proposed, and didn’t choose Conventional2. In all listener’s position, more evaluator chose Conventional1 than Proposed. It was suggested that the effect in which high frequency region of sound is cut is larger than the effect of spatial aliasing on sound quality.
- 24. Conclusion The objective of SFR system is to reproduce the primary sound field to another space with wide range and high accuracy as much as possible. Since it is difficult to reproduce the sound field with a complex system, the SFR method utilizing simple system has been desired. SDM can be realized with a simple system like linear loudspeaker array. However, to reproduce the sound field with high accuracy utilizing this method is impossible. ꥰꥰ We proposed the SFR system which reproduce the sound field with high accuracy to listener's position by estimating the listener's direction. As results of subjective assessment, the superiority of proposed method is shown on directional perception. However, since the superiority failed to show on sound quality, it is necessary to improve the equiangular filter that we do not apply the lowpass filter. Thank you for your attention!