On the limitations of representing functions on sets
0 likes240 views
1. The document discusses limitations of representing functions on sets using neural networks like "Deep Sets" that assume permutation invariance or equivariance. 2. It notes that continuity in a countable domain does not guarantee continuity in an uncountable domain, which is needed for universal function approximation. 3. For a neural network to represent all continuous permutation-invariant functions on sets of size ≤ M, the latent space dimension must be at least M. Experiments show increased error when the latent dimension is smaller than the maximum set size.
![On the Limitations
of
Representing Functions on Sets
Kaushalya Madhawa
Murata Group
[Wagstaff+, ICML 2019]](https://image.slidesharecdn.com/icml2019-functionsonsets-190709083852/85/On-the-limitations-of-representing-functions-on-sets-1-320.jpg)
![Sum-decomposition
• Proposed in “Deep Sets” architecture [Zaheer+, 2017]
• is sum-decomposable via Z for some
• and can be modeled with NNs
!4
f(X) = ρ (Σx∈Xϕ(x))
+ ρϕ
X ∈ ℝM
ℝNxM
ℝN ℝ
Input Output
x1
xM
Z
f(x1, …, xM)
ϕ(x1)
ϕ(xM)
ϕ : ℝ → Z
ρ : Z → ℝ
ϕ : 𝔛 → Zf
* X is a set (no repeated elements)
ϕ ρ](https://image.slidesharecdn.com/icml2019-functionsonsets-190709083852/85/On-the-limitations-of-representing-functions-on-sets-4-320.jpg)
![Continuity in a countable domain does not
guarantee continuity in an uncountable domain!
A theoretical guarantee of
continuity on is weak, and does
not imply continuity on
• The universal approximation
theorem relies on continuity on
• Example: is continuous at
every rational point in [0, ln4].
Discontinuous in
!7
ℝ
ℚ
ℝ
ℝ
ψ](https://image.slidesharecdn.com/icml2019-functionsonsets-190709083852/85/On-the-limitations-of-representing-functions-on-sets-7-320.jpg)
On the limitations of representing functions on sets
- 1. On the Limitations of Representing Functions on Sets Kaushalya Madhawa Murata Group [Wagstaff+, ICML 2019]
- 2. Contributions “We point out that some of the previous work is therefore of limited practical relevance, but regard it as mathematically interesting.” !2
- 3. Functions on sets? •Permutation-invariance •Permutation-equivariance f ({ }), , f ({ }), ,= f ({ }), , { f ( }),( ),(= ) f ({ }), , { f ( )}),( ),(= !3
- 4. Sum-decomposition • Proposed in “Deep Sets” architecture [Zaheer+, 2017] • is sum-decomposable via Z for some • and can be modeled with NNs !4 f(X) = ρ (Σx∈Xϕ(x)) + ρϕ X ∈ ℝM ℝNxM ℝN ℝ Input Output x1 xM Z f(x1, …, xM) ϕ(x1) ϕ(xM) ϕ : ℝ → Z ρ : Z → ℝ ϕ : 𝔛 → Zf * X is a set (no repeated elements) ϕ ρ
- 5. Countable vs Uncountable domains is either • Countable: number of elements is smaller or equal to the number of elements in • e.g. • Uncountable: number of elements is greater than the number of elements in • e.g. !5 ℕ ℕ ℕ, ℚ 𝔛 ℝ
- 6. What’s missing in “Deep Sets”? • “Deep Sets” considers functions on countable domains • Universal approximation theorem: • “Any continuous function can be approximated by a neural network.” • The universal approximation theorem for neural networks relies on continuity on !6 ℝ
- 7. Continuity in a countable domain does not guarantee continuity in an uncountable domain! A theoretical guarantee of continuity on is weak, and does not imply continuity on • The universal approximation theorem relies on continuity on • Example: is continuous at every rational point in [0, ln4]. Discontinuous in !7 ℝ ℚ ℝ ℝ ψ
- 8. Practical Implications Necessary and sufficient conditions 1. A latent dimensionality of M is sufficient for representing all continuous permutation-invariant functions on sets of size ≤ M. 2. To guarantee that all continuous permutation-invariant functions can be represented for sets of size ≤ M, a latent dimensionality of at least M is necessary. The latent space in which the summation happens must be chosen to have dimension at least M. !8
- 9. Experiments • Given a set of M elements, predict its median • Input dimension M and latent dimension N varied • and modeled with MLPs • The mapping doesn’t need to be injective. (solvable with N<M) !9 100 101 102 103 N (latent dim) 10 2 10 1 100 RMSE set size 15 30 60 100 200 300 400 500 0 100 200 300 400 500 600 set size M 0 20 40 60 80 100 criticallatentdimNc ϕ ρ
- 10. Conclusion • Importance of the continuity requirements on uncountable domains • Necessary and sufficient condition to model universal function representations: Latent space dimension should be at least as large as the maximum input set size • Permutation-equivariance is yet to be addressed !10
- 11. References • Wagstaff, Edward, et al. "On the Limitations of Representing Functions on Sets." arXiv preprint arXiv: 1901.09006 (2019). • Zaheer, Manzil, et al. "Deep sets." Advances in neural information processing systems. 2017. • DeepSets: Modeling Permutation Invariance https:// www.inference.vc/deepsets-modeling-permutation- invariance/ !11