SlideShare a Scribd company logo

Harmonic Analysis and Deep Learning

S
Sungbin Lim

The document discusses the intersection of harmonic analysis and deep learning, focusing on mathematical theories such as filter, activation, and pooling in deep convolutional neural networks (DCNN). It highlights the importance of constructs like Lipschitz continuity, polynomial approximations, and the significance of Hilbert spaces in efficiently representing functions across various fields like neuroscience and finance. Additionally, it covers concepts such as feature extraction and invariance in neural networks, with references to established mathematical theorems and recent developments in scattering networks.

Science
Related topics:
Deep Learning
1 of 58
Downloaded 124 times
1
2
Most read
3
4
5
6
7
8
Most read
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
Harmonic Analysis & Deep Learning Sungbin Lim
In this talk… Mathematical theory about filter, activation, pooling through multi-layers based on DCNN Encompass general ingredients Lipschitz continuity & Deformation sensitivity WARNING : Very tough mathematics …without non-Euclidean geometry (e.g. Geometric DL)
What is Harmonic Analysis? f(x)= X n2N an n(x), an := hf, niH How to represent a function efficiently in the sense of Hilbert space? Number theory Signal processing Quantum mechanics Neuroscience, Statistics, Finance, etc… Includes PDE theory, Stochastic Analysis
What is Harmonic Analysis? f(x)= X n2N an n(x), an := hf, niH How to represent a function efficiently in the sense of Hilbert space? Number theory Signal processing Quantum mechanics Neuroscience, Statistics, Finance, etc… Includes PDE theory, Stochastic Analysis
Hilbert space & Inner product Banach space : Hilbert space : © Kyung-Min Rho
Hilbert space & Inner product © Kyung-Min Rho Banach space : Normed space + Completeness Hilbert space :
Banach space : Normed space + Completeness Hilbert space : Banach space + Inner product Hilbert space & Inner product © Kyung-Min Rho
Banach space : Normed space + Completeness Hilbert space : Banach space + Inner product Rd , L2, Wn 2 , · · · Hilbert space & Inner product Cn , Lp, Wn p · · · © Kyung-Min Rho
Banach space : Normed space + Completeness Hilbert space : Banach space + Inner product Rd , L2, Wn 2 , · · · hu, vi = dX k=1 ukvk hf, giL2 = Z f(x)g(x)dx hf, giW n 2 = hf, giL2 + nX k=1 h@k xf, @k xgiL2 Hilbert space & Inner product Cn , Lp, Wn p · · · © Kyung-Min Rho
Why Harmonic Analysis? Pn(x) = anxn + an 1xn 1 + · · · + a1x + a0
Why Harmonic Analysis? Pn(x) = anxn + an 1xn 1 + · · · + a1x + a0 (an, an 1, . . . , a1 , a0) Encoding
Why Harmonic Analysis? Pn(x) = anxn + an 1xn 1 + · · · + a1x + a0 (an, an 1, . . . , a1 , a0) Encoding Pn(x) = anxn + an 1xn 1 + · · · + a1x + a0 Decoding
Why Harmonic Analysis? Pn(x) = anxn + an 1xn 1 + · · · + a1x + a0 (an, an 1, . . . , a1 , a0) Encoding Pn(x) = anxn + an 1xn 1 + · · · + a1x + a0 Decoding Why we prefer polynomial?
Stone-Weierstrass theorem Polynomial is Universal approximation! 8f 2 C(X), 8" > 0, 9Pn s.t. max x2X |f(x) Pn(x)| < " © Wikipedia
8f 2 C(X), 9Pn s.t. lim n!1 kf Pnk1 = 0 Stone-Weierstrass theorem Polynomial is Universal approximation! © Wikipedia
Stone-Weierstrass theorem Even we can approximate derivatives! 9Pn s.t. lim n!1 kf PnkCn ! 0 Polynomial is Universal approximation! 8f 2 Ck (X), © Wikipedia
Stone-Weierstrass theorem Even we can approximate derivatives! Universal approximation = {DL, polynomials, Tree,…} Polynomial is Universal approximation! 9Pn s.t. lim n!1 kf PnkCn ! 0 8f 2 Ck (X), © Wikipedia
Stone-Weierstrass theorem Even we can approximate derivatives! Universal approximation = {DL, polynomials, Tree,…} But why we do not use polynomial? Polynomial is Universal approximation! 9Pn s.t. lim n!1 kf PnkCn ! 0 8f 2 Ck (X), © Wikipedia
Local interpolation works well for low dimension © S. Mallat
Local interpolation works well for low dimension Need " d points to cover [0, 1]d at a distance " © S. Mallat
Local interpolation works well for low dimension Need " d points to cover [0, 1]d at a distance " High dimension ⇢ Curse of dimension! © H. Bölcskei
Universal approximator = Good feature extractor ?
Universal approximator = Good feature extractor …in HIGH dimension!
Nonlinear Feature Extraction © S. Mallat, © H. Bölcskei
Dimension Reduction ⇢ Invariants © S. Mallat
Dimension Reduction ⇢ Invariants How? © S. Mallat
Main Topic in Harmonic Analysis Linear operator ⇢ Convolution + Multiplier Invariance vs Discriminability L[f](x) = hTx[K], fi () dL[f](!) = bK(!) bf(!)
Main Topic in Harmonic Analysis L[f](x) = hTx[K], fi () dL[f](!) = bK(!) bf(!) Linear operator ⇢ Convolution + Multiplier Invariance vs Discriminability
Main Topic in Harmonic Analysis L[f](x) = hTx[K], fi () dL[f](!) = bK(!) bf(!) Linear operator ⇢ Convolution + Multiplier Discriminability vs Invariance Littlewood-Paley Condition ⇢ Semi-discrete Frame AkfkH  kL[f]kH  BkfkH
Main Topic in Harmonic Analysis L[f](x) = hTx[K], fi () dL[f](!) = bK(!) bf(!) AkfkH  kL[f]kH  BkfkH Linear operator ⇢ Convolution + Multiplier Discriminability vs Invariance Littlewood-Paley Condition ⇢ Semi-discrete Frame kL[f1] L[f2]kH = kL[f1 f2]kH Akf1 f2kH i.e. f1 6= f2 ) L[f1] 6= L[f2]
Main Topic in Harmonic Analysis L[f](x) = hTx[K], fi () dL[f](!) = bK(!) bf(!) AkfkH  kL[f]kH  BkfkH Linear operator ⇢ Convolution + Multiplier Discriminability vs Invariance Littlewood-Paley Condition ⇢ Semi-discrete Frame k L · · · L| {z } n-fold [f]kH  Bk L · · · L| {z } (n-1)-fold [f]kH  · · ·  Bn kfkH
Main Topic in Harmonic Analysis L[f](x) = hTx[K], fi () dL[f](!) = bK(!) bf(!) AkfkH  kL[f]kH  BkfkH Linear operator ⇢ Convolution + Multiplier Discriminability vs Invariance Littlewood-Paley Condition ⇢ Semi-discrete Frame k L · · · L| {z } n-fold [f]kH  Bk L · · · L| {z } (n-1)-fold [f]kH  · · ·  Bn kfkH Banach fixed-point theorem
Main Tasks in Deep CNN Representation learning Feature Extraction Nonlinear transform
Main Tasks in Deep CNN Representation learning Feature Extraction Nonlinear transform
Main Tasks in Deep CNN Representation learning Feature Extraction Nonlinear transform Lipschitz continuity ex) ReLU, tanh, sigmoid … |f(x) f(y)|  Ckx yk () krf(x)k  C
How to control Lipschitz ? k⇢(L[f])kH  N(B, C)kfkH Theorem No change in Invariance!
k⇢(L[f])kH  N(B, C)kfkH Proof) No change in Invariance! Let ⇢ = ReLU, H = W1 2 . Then Theorem How to control Lipschitz ?
k⇢(L[f])kH  N(B, C)kfkH Proof) No change in Invariance! Let ⇢ = ReLU, H = W1 2 . Then Theorem k⇢(L[f])kW 1 2 = k max{L[f], 0}kL2 + kr⇢(L[f])kL2  kL[f]kL2 + k ⇢0 (L[f]) | {z } =1 or 0 r(L[f])kL2  kL[f]kL2 + kr(L[f])kL2 = kL[f]kW 1 2  BkfkW 1 2 How to control Lipschitz ?
k⇢(L[f])kH  N(B, C)kfkH Proof) No change in Invariance! Let ⇢ = ReLU, H = W1 2 . Then Theorem k⇢(L[f])kW 1 2 = k max{L[f], 0}kL2 + kr⇢(L[f])kL2  kL[f]kL2 + k ⇢0 (L[f]) | {z } =1 or 0 r(L[f])kL2  kL[f]kL2 + kr(L[f])kL2 = kL[f]kW 1 2  BkfkW 1 2 How to control Lipschitz ?
k⇢(L[f])kH  N(B, C)kfkH Proof) No change in Invariance! Let ⇢ = ReLU, H = W1 2 . Then Theorem k⇢(L[f])kW 1 2 = k max{L[f], 0}kL2 + kr⇢(L[f])kL2  kL[f]kL2 + k ⇢0 (L[f]) | {z } =1 or 0 r(L[f])kL2  kL[f]kL2 + kr(L[f])kL2 = kL[f]kW 1 2  BkfkW 1 2 How to control Lipschitz ?
k⇢(L[f])kH  N(B, C)kfkH Proof) No change in Invariance! Let ⇢ = ReLU, H = W1 2 . Then Theorem k⇢(L[f])kW 1 2 = k max{L[f], 0}kL2 + kr⇢(L[f])kL2  kL[f]kL2 + k ⇢0 (L[f]) | {z } =1 or 0 r(L[f])kL2  kL[f]kL2 + kr(L[f])kL2 = kL[f]kW 1 2  BkfkW 1 2 How to control Lipschitz ?
k⇢(L[f])kH  N(B, C)kfkH Proof) No change in Invariance! Let ⇢ = ReLU, H = W1 2 . Then Theorem k⇢(L[f])kW 1 2 = k max{L[f], 0}kL2 + kr⇢(L[f])kL2  kL[f]kL2 + k ⇢0 (L[f]) | {z } =1 or 0 r(L[f])kL2  kL[f]kL2 + kr(L[f])kL2 = kL[f]kW 1 2  BkfkW 1 2 How to control Lipschitz ?
k⇢(L[f])kH  N(B, C)kfkH Proof) No change in Invariance! Let ⇢ = ReLU, H = W1 2 . Then Theorem k⇢(L[f])kW 1 2 = k max{L[f], 0}kL2 + kr⇢(L[f])kL2  kL[f]kL2 + k ⇢0 (L[f]) | {z } =1 or 0 r(L[f])kL2  kL[f]kL2 + kr(L[f])kL2 = kL[f]kW 1 2  BkfkW 1 2 How to control Lipschitz ? What about Discriminability?
Scale Invariant Feature Translation Invariant Stable at Deformation © S. Mallat
Scale Invariant Feature Translation Invariant Stable at Deformation
Scattering Network (Mallat, 2012) (f) = [ n ( · · · |f ⇤ g (j) | ⇤ g (k) · · · ⇤ g (p) | {z } n-fold convolution ⇤ n ) (j),··· , (p) © H. Bölcskei
Generalized Scattering Network (Wiatowski, 2015) (f) = [ n ( · · · |f ⇤ g (j) | ⇤ g (k) · · · ⇤ g (p) | {z } n-fold convolution ⇤ n ) (j),··· , (p) Gabor frame Tensor wavelet Directional wavelet Ridgelet frame Curvelet frame © H. Bölcskei
Generalized Scattering Network (Wiatowski, 2015) (f) = [ n ( · · · |f ⇤ g (j) | ⇤ g (k) · · · ⇤ g (p) | {z } n-fold convolution ⇤ n ) (j),··· , (p) © S. Mallat
Generalized Scattering Network (Wiatowski, 2015) (f) = [ n ( · · · |f ⇤ g (j) | ⇤ g (k) · · · ⇤ g (p) | {z } n-fold convolution ⇤ n ) (j),··· , (p) Linearize symmetries © S. Mallat
Generalized Scattering Network (Wiatowski, 2015) (f) = [ n ( · · · |f ⇤ g (j) | ⇤ g (k) · · · ⇤ g (p) | {z } n-fold convolution ⇤ n ) (j),··· , (p) Linearize symmetries “Space folding”, Cho (2014) © S. Mallat
(f) = [ n ( · · · |f ⇤ g (j) | ⇤ g (k) · · · ⇤ g (p) | {z } n-fold convolution ⇤ n ) (j),··· , (p) f 7! Sd/2 n Pn(f)(Sn·) |k n(Ttf) n(f)|k = O ktk Qn j=1 Sj ! Theorem Generalized Scattering Network (Wiatowski, 2015)
f 7! Sd/2 n Pn(f)(Sn·) |k n(Ttf) n(f)|k = O ktk Qn j=1 Sj ! Theorem Features become more translation invariant with increasing network depth Generalized Scattering Network (Wiatowski, 2015)
Generalized Scattering Network (Wiatowski, 2015) © Philip Scott Johnson (f) = [ n ( · · · |f ⇤ g (j) | ⇤ g (k) · · · ⇤ g (p) | {z } n-fold convolution ⇤ n ) (j),··· , (p) Theorem F⌧,! = e2⇡i!(x) f(x ⌧(x)) |k (F⌧,![f]) (f)k|  C(k⌧k1 + k!k1)kfkL2
Generalized Scattering Network (Wiatowski, 2015) © Philip Scott Johnson Theorem F⌧,! = e2⇡i!(x) f(x ⌧(x)) |k (F⌧,![f]) (f)k|  C(k⌧k1 + k!k1)kfkL2 Multi-layer convolution linearize Features i.e. stable to deformations
Generalized Scattering Network (Wiatowski, 2015) © Philip Scott Johnson
Ergodic Reconstructions © Philip Scott Johnson © S. Mallat
David Hilbert Wir müssen wissen. Wir werden wissen.
Q.A

More Related Content

PDF
Matrix calculus
Sungbin Lim
 
PDF
Wasserstein GAN 수학 이해하기 I
Sungbin Lim
 
PDF
Linear algebra
Sungbin Lim
 
PPTX
Metric space
beenishbeenish
 
PPTX
십분딥러닝_16_WGAN (Wasserstein GANs)
HyunKyu Jeon
 
PPT
Fractional Calculus PP
VRRITC
 
PDF
Gmm to vgmm
수철 박
 
PPTX
topology definitions
RAMYA vetrivel
 
Matrix calculus
Sungbin Lim
 
Wasserstein GAN 수학 이해하기 I
Sungbin Lim
 
Linear algebra
Sungbin Lim
 
Metric space
beenishbeenish
 
십분딥러닝_16_WGAN (Wasserstein GANs)
HyunKyu Jeon
 
Fractional Calculus PP
VRRITC
 
Gmm to vgmm
수철 박
 
topology definitions
RAMYA vetrivel
 

What's hot (20)

PDF
Natural Policy Gradient 직관적 접근
Sooyoung Moon
 
PDF
Group Theory
shinojmadhu
 
PDF
가깝고도 먼 Trpo
Woong won Lee
 
PPTX
Convolutional neural networks 이론과 응용
홍배 김
 
PDF
강화학습 알고리즘의 흐름도 Part 2
Dongmin Lee
 
PDF
Fractional calculus and applications
PlusOrMinusZero
 
PPTX
Section 8: Symmetric Groups
Kevin Johnson
 
PDF
안.전.제.일. 강화학습!
Dongmin Lee
 
PDF
강화학습의 흐름도 Part 1
Dongmin Lee
 
PDF
Flow based generative models
수철 박
 
PDF
Contraction mapping
Hancheol Choi
 
PDF
metric spaces
HamKarimRUPP
 
PPTX
圏論とHaskellは仲良し
ohmori
 
PDF
素数の分解法則（フロベニウスやばい） #math_cafe
Junpei Tsuji
 
PDF
차원축소 훑어보기 (PCA, SVD, NMF)
beom kyun choi
 
PDF
Lecture 2 predicates quantifiers and rules of inference
asimnawaz54
 
PDF
Lesson 11: Limits and Continuity
Matthew Leingang
 
PDF
Deep Learning: Recurrent Neural Network (Chapter 10)
Larry Guo
 
PPT
Partial derivative1
Nidhu Sharma
 
PDF
Relation matrix & graphs in relations
Rachana Pathak
 
Natural Policy Gradient 직관적 접근
Sooyoung Moon
 
Group Theory
shinojmadhu
 
가깝고도 먼 Trpo
Woong won Lee
 
Convolutional neural networks 이론과 응용
홍배 김
 
강화학습 알고리즘의 흐름도 Part 2
Dongmin Lee
 
Fractional calculus and applications
PlusOrMinusZero
 
Section 8: Symmetric Groups
Kevin Johnson
 
안.전.제.일. 강화학습!
Dongmin Lee
 
강화학습의 흐름도 Part 1
Dongmin Lee
 
Flow based generative models
수철 박
 
Contraction mapping
Hancheol Choi
 
metric spaces
HamKarimRUPP
 
圏論とHaskellは仲良し
ohmori
 
素数の分解法則（フロベニウスやばい） #math_cafe
Junpei Tsuji
 
차원축소 훑어보기 (PCA, SVD, NMF)
beom kyun choi
 
Lecture 2 predicates quantifiers and rules of inference
asimnawaz54
 
Lesson 11: Limits and Continuity
Matthew Leingang
 
Deep Learning: Recurrent Neural Network (Chapter 10)
Larry Guo
 
Partial derivative1
Nidhu Sharma
 
Relation matrix & graphs in relations
Rachana Pathak
 
Ad

Similar to Harmonic Analysis and Deep Learning (20)

PDF
IVR - Chapter 1 - Introduction
Charles Deledalle
 
PDF
Some Thoughts on Sampling
Don Sheehy
 
PDF
Wavelets and Other Adaptive Methods
Kamrul Hasan
 
PDF
QMC: Operator Splitting Workshop, A New (More Intuitive?) Interpretation of I...
The Statistical and Applied Mathematical Sciences Institute
 
PDF
Introduction to Fourier transform and signal analysis
宗翰 謝
 
PDF
Signal lexture
Zong-han Xie
 
PDF
QMC: Transition Workshop - Probabilistic Integrators for Deterministic Differ...
The Statistical and Applied Mathematical Sciences Institute
 
PDF
Nodal Domain Theorem for the p-Laplacian on Graphs and the Related Multiway C...
Francesco Tudisco
 
PDF
Can we estimate a constant?
Christian Robert
 
PDF
Testing for mixtures by seeking components
Christian Robert
 
PDF
The dual geometry of Shannon information
Frank Nielsen
 
PDF
Divergence clustering
Frank Nielsen
 
PDF
A sharp nonlinear Hausdorff-Young inequality for small potentials
VjekoslavKovac1
 
PDF
Optimization Methods for Machine Learning and Engineering: Optimization in Ve...
FadiAkil2
 
PDF
QMC: Operator Splitting Workshop, Composite Infimal Convolutions - Zev Woodst...
The Statistical and Applied Mathematical Sciences Institute
 
PDF
Divergence center-based clustering and their applications
Frank Nielsen
 
PDF
Approximation Methods Of Solutions For Equilibrium Problem In Hilbert Spaces
Lisa Garcia
 
PDF
2018 MUMS Fall Course - Statistical Representation of Model Input (EDITED) - ...
The Statistical and Applied Mathematical Sciences Institute
 
PDF
Building Compatible Bases on Graphs, Images, and Manifolds
Davide Eynard
 
PDF
Stochastic Control and Information Theoretic Dualities (Complete Version)
Haruki Nishimura
 
IVR - Chapter 1 - Introduction
Charles Deledalle
 
Some Thoughts on Sampling
Don Sheehy
 
Wavelets and Other Adaptive Methods
Kamrul Hasan
 
QMC: Operator Splitting Workshop, A New (More Intuitive?) Interpretation of I...
The Statistical and Applied Mathematical Sciences Institute
 
Introduction to Fourier transform and signal analysis
宗翰 謝
 
Signal lexture
Zong-han Xie
 
QMC: Transition Workshop - Probabilistic Integrators for Deterministic Differ...
The Statistical and Applied Mathematical Sciences Institute
 
Nodal Domain Theorem for the p-Laplacian on Graphs and the Related Multiway C...
Francesco Tudisco
 
Can we estimate a constant?
Christian Robert
 
Testing for mixtures by seeking components
Christian Robert
 
The dual geometry of Shannon information
Frank Nielsen
 
Divergence clustering
Frank Nielsen
 
A sharp nonlinear Hausdorff-Young inequality for small potentials
VjekoslavKovac1
 
Optimization Methods for Machine Learning and Engineering: Optimization in Ve...
FadiAkil2
 
QMC: Operator Splitting Workshop, Composite Infimal Convolutions - Zev Woodst...
The Statistical and Applied Mathematical Sciences Institute
 
Divergence center-based clustering and their applications
Frank Nielsen
 
Approximation Methods Of Solutions For Equilibrium Problem In Hilbert Spaces
Lisa Garcia
 
2018 MUMS Fall Course - Statistical Representation of Model Input (EDITED) - ...
The Statistical and Applied Mathematical Sciences Institute
 
Building Compatible Bases on Graphs, Images, and Manifolds
Davide Eynard
 
Stochastic Control and Information Theoretic Dualities (Complete Version)
Haruki Nishimura
 
Ad

Recently uploaded (20)

PPTX
CORDINATION COMPOUND AND ITS APPLICATIONS
documentsfile2024
 
PPT
6.1 High Risk New Born. Padetric health ppt
pickyblinder789
 
PDF
Warm, water-depleted rocky exoplanets with surfaceionic liquids: A proposed c...
Sérgio Sacani
 
PDF
CHAPTER 3 Cell Structures and Their Functions Lecture Outline.pdf
DaniseNobrera
 
PDF
An interstellar mission to test astrophysical black holes
Sérgio Sacani
 
PPTX
ognitive-behavioral therapy, mindfulness-based approaches, coping skills trai...
OliveJolly1
 
PDF
Biophysics 2.pdffffffffffffffffffffffffff
MiraMusleh
 
PPTX
Science Quipper for lesson in grade 8 Matatag Curriculum
AnneLoquenario
 
PDF
Worlds Next Door: A Candidate Giant Planet Imaged in the Habitable Zone of ↵ ...
Sérgio Sacani
 
PDF
Sciences of Europe No 170 (2025)
Sciences of Europe
 
PDF
Looking into the jet cone of the neutrino-associated very high-energy blazar ...
Sérgio Sacani
 
PPTX
Introcution to Microbes Burton's Biology for the Health
fliwertyfw154
 
PPT
veterinary parasitology ````````````.ppt
midolyon1990gmailcom
 
PDF
. Radiology Case Scenariosssssssssssssss
MiraMusleh
 
PPTX
BODY FLUIDS AND CIRCULATION class 11 .pptx
chinmayikalokhe
 
PPTX
Microbes in human welfare class 12 .pptx
chinmayikalokhe
 
PPTX
Application of enzymes in medicine (2).pptx
safaparvin28
 
PPTX
Hypertension_Training_materials_English_2024[1] (1).pptx
johnsalaoudine
 
PPTX
Fluid dynamics vivavoce presentation of prakash
ssuser9878d0
 
PDF
The Land of Punt — A research by Dhani Irwanto
Dhani Irwanto
 
CORDINATION COMPOUND AND ITS APPLICATIONS
documentsfile2024
 
6.1 High Risk New Born. Padetric health ppt
pickyblinder789
 
Warm, water-depleted rocky exoplanets with surfaceionic liquids: A proposed c...
Sérgio Sacani
 
CHAPTER 3 Cell Structures and Their Functions Lecture Outline.pdf
DaniseNobrera
 
An interstellar mission to test astrophysical black holes
Sérgio Sacani
 
ognitive-behavioral therapy, mindfulness-based approaches, coping skills trai...
OliveJolly1
 
Biophysics 2.pdffffffffffffffffffffffffff
MiraMusleh
 
Science Quipper for lesson in grade 8 Matatag Curriculum
AnneLoquenario
 
Worlds Next Door: A Candidate Giant Planet Imaged in the Habitable Zone of ↵ ...
Sérgio Sacani
 
Sciences of Europe No 170 (2025)
Sciences of Europe
 
Looking into the jet cone of the neutrino-associated very high-energy blazar ...
Sérgio Sacani
 
Introcution to Microbes Burton's Biology for the Health
fliwertyfw154
 
veterinary parasitology ````````````.ppt
midolyon1990gmailcom
 
. Radiology Case Scenariosssssssssssssss
MiraMusleh
 
BODY FLUIDS AND CIRCULATION class 11 .pptx
chinmayikalokhe
 
Microbes in human welfare class 12 .pptx
chinmayikalokhe
 
Application of enzymes in medicine (2).pptx
safaparvin28
 
Hypertension_Training_materials_English_2024[1] (1).pptx
johnsalaoudine
 
Fluid dynamics vivavoce presentation of prakash
ssuser9878d0
 
The Land of Punt — A research by Dhani Irwanto
Dhani Irwanto
 

Harmonic Analysis and Deep Learning

  • 2. In this talk… Mathematical theory about filter, activation, pooling through multi-layers based on DCNN Encompass general ingredients Lipschitz continuity & Deformation sensitivity WARNING : Very tough mathematics …without non-Euclidean geometry (e.g. Geometric DL)
  • 3. What is Harmonic Analysis? f(x)= X n2N an n(x), an := hf, niH How to represent a function efficiently in the sense of Hilbert space? Number theory Signal processing Quantum mechanics Neuroscience, Statistics, Finance, etc… Includes PDE theory, Stochastic Analysis
  • 4. What is Harmonic Analysis? f(x)= X n2N an n(x), an := hf, niH How to represent a function efficiently in the sense of Hilbert space? Number theory Signal processing Quantum mechanics Neuroscience, Statistics, Finance, etc… Includes PDE theory, Stochastic Analysis
  • 5. Hilbert space & Inner product Banach space : Hilbert space : © Kyung-Min Rho
  • 6. Hilbert space & Inner product © Kyung-Min Rho Banach space : Normed space + Completeness Hilbert space :
  • 7. Banach space : Normed space + Completeness Hilbert space : Banach space + Inner product Hilbert space & Inner product © Kyung-Min Rho
  • 8. Banach space : Normed space + Completeness Hilbert space : Banach space + Inner product Rd , L2, Wn 2 , · · · Hilbert space & Inner product Cn , Lp, Wn p · · · © Kyung-Min Rho
  • 9. Banach space : Normed space + Completeness Hilbert space : Banach space + Inner product Rd , L2, Wn 2 , · · · hu, vi = dX k=1 ukvk hf, giL2 = Z f(x)g(x)dx hf, giW n 2 = hf, giL2 + nX k=1 h@k xf, @k xgiL2 Hilbert space & Inner product Cn , Lp, Wn p · · · © Kyung-Min Rho
  • 10. Why Harmonic Analysis? Pn(x) = anxn + an 1xn 1 + · · · + a1x + a0
  • 11. Why Harmonic Analysis? Pn(x) = anxn + an 1xn 1 + · · · + a1x + a0 (an, an 1, . . . , a1 , a0) Encoding
  • 12. Why Harmonic Analysis? Pn(x) = anxn + an 1xn 1 + · · · + a1x + a0 (an, an 1, . . . , a1 , a0) Encoding Pn(x) = anxn + an 1xn 1 + · · · + a1x + a0 Decoding
  • 13. Why Harmonic Analysis? Pn(x) = anxn + an 1xn 1 + · · · + a1x + a0 (an, an 1, . . . , a1 , a0) Encoding Pn(x) = anxn + an 1xn 1 + · · · + a1x + a0 Decoding Why we prefer polynomial?
  • 14. Stone-Weierstrass theorem Polynomial is Universal approximation! 8f 2 C(X), 8" > 0, 9Pn s.t. max x2X |f(x) Pn(x)| < " © Wikipedia
  • 15. 8f 2 C(X), 9Pn s.t. lim n!1 kf Pnk1 = 0 Stone-Weierstrass theorem Polynomial is Universal approximation! © Wikipedia
  • 16. Stone-Weierstrass theorem Even we can approximate derivatives! 9Pn s.t. lim n!1 kf PnkCn ! 0 Polynomial is Universal approximation! 8f 2 Ck (X), © Wikipedia
  • 17. Stone-Weierstrass theorem Even we can approximate derivatives! Universal approximation = {DL, polynomials, Tree,…} Polynomial is Universal approximation! 9Pn s.t. lim n!1 kf PnkCn ! 0 8f 2 Ck (X), © Wikipedia
  • 18. Stone-Weierstrass theorem Even we can approximate derivatives! Universal approximation = {DL, polynomials, Tree,…} But why we do not use polynomial? Polynomial is Universal approximation! 9Pn s.t. lim n!1 kf PnkCn ! 0 8f 2 Ck (X), © Wikipedia
  • 19. Local interpolation works well for low dimension © S. Mallat
  • 20. Local interpolation works well for low dimension Need " d points to cover [0, 1]d at a distance " © S. Mallat
  • 21. Local interpolation works well for low dimension Need " d points to cover [0, 1]d at a distance " High dimension ⇢ Curse of dimension! © H. Bölcskei
  • 22. Universal approximator = Good feature extractor ?
  • 23. Universal approximator = Good feature extractor …in HIGH dimension!
  • 24. Nonlinear Feature Extraction © S. Mallat, © H. Bölcskei
  • 25. Dimension Reduction ⇢ Invariants © S. Mallat
  • 26. Dimension Reduction ⇢ Invariants How? © S. Mallat
  • 27. Main Topic in Harmonic Analysis Linear operator ⇢ Convolution + Multiplier Invariance vs Discriminability L[f](x) = hTx[K], fi () dL[f](!) = bK(!) bf(!)
  • 28. Main Topic in Harmonic Analysis L[f](x) = hTx[K], fi () dL[f](!) = bK(!) bf(!) Linear operator ⇢ Convolution + Multiplier Invariance vs Discriminability
  • 29. Main Topic in Harmonic Analysis L[f](x) = hTx[K], fi () dL[f](!) = bK(!) bf(!) Linear operator ⇢ Convolution + Multiplier Discriminability vs Invariance Littlewood-Paley Condition ⇢ Semi-discrete Frame AkfkH  kL[f]kH  BkfkH
  • 30. Main Topic in Harmonic Analysis L[f](x) = hTx[K], fi () dL[f](!) = bK(!) bf(!) AkfkH  kL[f]kH  BkfkH Linear operator ⇢ Convolution + Multiplier Discriminability vs Invariance Littlewood-Paley Condition ⇢ Semi-discrete Frame kL[f1] L[f2]kH = kL[f1 f2]kH Akf1 f2kH i.e. f1 6= f2 ) L[f1] 6= L[f2]
  • 31. Main Topic in Harmonic Analysis L[f](x) = hTx[K], fi () dL[f](!) = bK(!) bf(!) AkfkH  kL[f]kH  BkfkH Linear operator ⇢ Convolution + Multiplier Discriminability vs Invariance Littlewood-Paley Condition ⇢ Semi-discrete Frame k L · · · L| {z } n-fold [f]kH  Bk L · · · L| {z } (n-1)-fold [f]kH  · · ·  Bn kfkH
  • 32. Main Topic in Harmonic Analysis L[f](x) = hTx[K], fi () dL[f](!) = bK(!) bf(!) AkfkH  kL[f]kH  BkfkH Linear operator ⇢ Convolution + Multiplier Discriminability vs Invariance Littlewood-Paley Condition ⇢ Semi-discrete Frame k L · · · L| {z } n-fold [f]kH  Bk L · · · L| {z } (n-1)-fold [f]kH  · · ·  Bn kfkH Banach fixed-point theorem
  • 33. Main Tasks in Deep CNN Representation learning Feature Extraction Nonlinear transform
  • 34. Main Tasks in Deep CNN Representation learning Feature Extraction Nonlinear transform
  • 35. Main Tasks in Deep CNN Representation learning Feature Extraction Nonlinear transform Lipschitz continuity ex) ReLU, tanh, sigmoid … |f(x) f(y)|  Ckx yk () krf(x)k  C
  • 36. How to control Lipschitz ? k⇢(L[f])kH  N(B, C)kfkH Theorem No change in Invariance!
  • 37. k⇢(L[f])kH  N(B, C)kfkH Proof) No change in Invariance! Let ⇢ = ReLU, H = W1 2 . Then Theorem How to control Lipschitz ?
  • 38. k⇢(L[f])kH  N(B, C)kfkH Proof) No change in Invariance! Let ⇢ = ReLU, H = W1 2 . Then Theorem k⇢(L[f])kW 1 2 = k max{L[f], 0}kL2 + kr⇢(L[f])kL2  kL[f]kL2 + k ⇢0 (L[f]) | {z } =1 or 0 r(L[f])kL2  kL[f]kL2 + kr(L[f])kL2 = kL[f]kW 1 2  BkfkW 1 2 How to control Lipschitz ?
  • 39. k⇢(L[f])kH  N(B, C)kfkH Proof) No change in Invariance! Let ⇢ = ReLU, H = W1 2 . Then Theorem k⇢(L[f])kW 1 2 = k max{L[f], 0}kL2 + kr⇢(L[f])kL2  kL[f]kL2 + k ⇢0 (L[f]) | {z } =1 or 0 r(L[f])kL2  kL[f]kL2 + kr(L[f])kL2 = kL[f]kW 1 2  BkfkW 1 2 How to control Lipschitz ?
  • 40. k⇢(L[f])kH  N(B, C)kfkH Proof) No change in Invariance! Let ⇢ = ReLU, H = W1 2 . Then Theorem k⇢(L[f])kW 1 2 = k max{L[f], 0}kL2 + kr⇢(L[f])kL2  kL[f]kL2 + k ⇢0 (L[f]) | {z } =1 or 0 r(L[f])kL2  kL[f]kL2 + kr(L[f])kL2 = kL[f]kW 1 2  BkfkW 1 2 How to control Lipschitz ?
  • 41. k⇢(L[f])kH  N(B, C)kfkH Proof) No change in Invariance! Let ⇢ = ReLU, H = W1 2 . Then Theorem k⇢(L[f])kW 1 2 = k max{L[f], 0}kL2 + kr⇢(L[f])kL2  kL[f]kL2 + k ⇢0 (L[f]) | {z } =1 or 0 r(L[f])kL2  kL[f]kL2 + kr(L[f])kL2 = kL[f]kW 1 2  BkfkW 1 2 How to control Lipschitz ?
  • 42. k⇢(L[f])kH  N(B, C)kfkH Proof) No change in Invariance! Let ⇢ = ReLU, H = W1 2 . Then Theorem k⇢(L[f])kW 1 2 = k max{L[f], 0}kL2 + kr⇢(L[f])kL2  kL[f]kL2 + k ⇢0 (L[f]) | {z } =1 or 0 r(L[f])kL2  kL[f]kL2 + kr(L[f])kL2 = kL[f]kW 1 2  BkfkW 1 2 How to control Lipschitz ?
  • 43. k⇢(L[f])kH  N(B, C)kfkH Proof) No change in Invariance! Let ⇢ = ReLU, H = W1 2 . Then Theorem k⇢(L[f])kW 1 2 = k max{L[f], 0}kL2 + kr⇢(L[f])kL2  kL[f]kL2 + k ⇢0 (L[f]) | {z } =1 or 0 r(L[f])kL2  kL[f]kL2 + kr(L[f])kL2 = kL[f]kW 1 2  BkfkW 1 2 How to control Lipschitz ? What about Discriminability?
  • 44. Scale Invariant Feature Translation Invariant Stable at Deformation © S. Mallat
  • 45. Scale Invariant Feature Translation Invariant Stable at Deformation
  • 46. Scattering Network (Mallat, 2012) (f) = [ n ( · · · |f ⇤ g (j) | ⇤ g (k) · · · ⇤ g (p) | {z } n-fold convolution ⇤ n ) (j),··· , (p) © H. Bölcskei
  • 47. Generalized Scattering Network (Wiatowski, 2015) (f) = [ n ( · · · |f ⇤ g (j) | ⇤ g (k) · · · ⇤ g (p) | {z } n-fold convolution ⇤ n ) (j),··· , (p) Gabor frame Tensor wavelet Directional wavelet Ridgelet frame Curvelet frame © H. Bölcskei
  • 48. Generalized Scattering Network (Wiatowski, 2015) (f) = [ n ( · · · |f ⇤ g (j) | ⇤ g (k) · · · ⇤ g (p) | {z } n-fold convolution ⇤ n ) (j),··· , (p) © S. Mallat
  • 49. Generalized Scattering Network (Wiatowski, 2015) (f) = [ n ( · · · |f ⇤ g (j) | ⇤ g (k) · · · ⇤ g (p) | {z } n-fold convolution ⇤ n ) (j),··· , (p) Linearize symmetries © S. Mallat
  • 50. Generalized Scattering Network (Wiatowski, 2015) (f) = [ n ( · · · |f ⇤ g (j) | ⇤ g (k) · · · ⇤ g (p) | {z } n-fold convolution ⇤ n ) (j),··· , (p) Linearize symmetries “Space folding”, Cho (2014) © S. Mallat
  • 51. (f) = [ n ( · · · |f ⇤ g (j) | ⇤ g (k) · · · ⇤ g (p) | {z } n-fold convolution ⇤ n ) (j),··· , (p) f 7! Sd/2 n Pn(f)(Sn·) |k n(Ttf) n(f)|k = O ktk Qn j=1 Sj ! Theorem Generalized Scattering Network (Wiatowski, 2015)
  • 52. f 7! Sd/2 n Pn(f)(Sn·) |k n(Ttf) n(f)|k = O ktk Qn j=1 Sj ! Theorem Features become more translation invariant with increasing network depth Generalized Scattering Network (Wiatowski, 2015)
  • 53. Generalized Scattering Network (Wiatowski, 2015) © Philip Scott Johnson (f) = [ n ( · · · |f ⇤ g (j) | ⇤ g (k) · · · ⇤ g (p) | {z } n-fold convolution ⇤ n ) (j),··· , (p) Theorem F⌧,! = e2⇡i!(x) f(x ⌧(x)) |k (F⌧,![f]) (f)k|  C(k⌧k1 + k!k1)kfkL2
  • 54. Generalized Scattering Network (Wiatowski, 2015) © Philip Scott Johnson Theorem F⌧,! = e2⇡i!(x) f(x ⌧(x)) |k (F⌧,![f]) (f)k|  C(k⌧k1 + k!k1)kfkL2 Multi-layer convolution linearize Features i.e. stable to deformations
  • 55. Generalized Scattering Network (Wiatowski, 2015) © Philip Scott Johnson
  • 56. Ergodic Reconstructions © Philip Scott Johnson © S. Mallat
  • 57. David Hilbert Wir müssen wissen. Wir werden wissen.
  • 58. Q.A