Activation distribution in a neural network
Download as PPTX, PDF0 likes223 views
This document discusses neural network activation distribution and three algorithms (A, B, C) for modeling the process. It analyzes the common behavioral characteristics and provides visual representations. The core characteristics of related software discussed are accuracy, efficiency, and platform independency. Visual results are shown for the algorithms under different parameter settings on Erdos-Renyi graphs.
Activation distribution in a neural network
- 1. Martin Ayvazyan Activation distribution in a neural network 20 August 2018
- 2. Restricted © 2017 Mentor Graphics Corporation Contents Why are neural networks so important Basic network characteristics Basic description of the process, formulation of algorithms, comparison of behavioral characteristics Common behavioral characteristics of the process A visual representation of the process Core characteristics of the software Activation distribution in a neural network, 20182
- 3. Restricted © 2017 Mentor Graphics Corporation Cell Division and Cancer Activation distribution in a neural network, 20183
- 4. Restricted © 2017 Mentor Graphics Corporation The global economic crisis prevention / elimination Activation distribution in a neural network, 20184
- 5. Restricted © 2017 Mentor Graphics Corporation Traffic control / jams prevention Activation distribution in a neural network, 20185
- 6. Restricted © 2017 Mentor Graphics Corporation Deep learning Activation distribution in a neural network, 20186
- 7. Restricted © 2017 Mentor Graphics Corporation Astronomy Activation distribution in a neural network, 20187
- 8. Restricted © 2017 Mentor Graphics Corporation Basic definitions ER model G(N, p) N – nodes count in the graph p – component of connectivity P(G) – the probability that the graph is connected Activation distribution in a neural network, 20188 • p = 𝑐∙𝑙𝑛𝑁 𝑁 && c>1 => graph is connected (c < 1 => graph is disconnected) • p = 𝑐∙𝑙𝑛𝑁 𝑁 && (c>3 && N > 100) => 𝑃 𝑁,𝑝 𝐺 ≥ 1 − 1 𝑁 𝑛 𝑁∗ 𝑁−1 2 𝑃(𝐺0) = 𝑝 𝑛 ∙ (1 − 𝑝) 𝑁∙(𝑁−1) 2 −𝑛 𝑃(𝐺′) = 𝑛 𝑁∙(𝑁−1) 2 ∙ 𝑝 𝑛 ∙ (1 − 𝑝)( 𝑁∙ 𝑁−1 2 − 𝑛)
- 9. Restricted © 2017 Mentor Graphics Corporation Small world Activation distribution in a neural network, 20189
- 10. Restricted © 2017 Mentor Graphics Corporation Clusterization Activation distribution in a neural network, 201810 𝐶𝑗 𝑞 𝑗 = 𝑡𝑗 𝑞 𝑗 ∙ (𝑞 𝑗 − 1)/2 𝐶 = 1 𝑁 𝑗 𝐶𝑗
- 11. Restricted © 2017 Mentor Graphics Corporation Degree distribution Activation distribution in a neural network, 201811
- 12. Restricted © 2017 Mentor Graphics Corporation Algorithm A Input: a network, list of active nodes in the network, μ, λ Step 1: If there is no an active node in the network – exit Step 2: Randomly is selected a node Step 3: If the selected node doesn’t active – go to Step 2 Step 4: Randomly is selected an adjacent node and in case if the neighbor doesn’t active – activate it with 𝛌 (𝛌+ 𝛍) probability Step 5: The temporary selected node is deactivated with 𝜇 (𝜆+ 𝜇) probability Step 6: Go to Step 1 Activation distribution in a neural network, 201812
- 13. Restricted © 2017 Mentor Graphics Corporation Algorithm A 𝑁=1024, 𝑝=0.5, μ=0.0 Activation distribution in a neural network, 201813
- 14. Restricted © 2017 Mentor Graphics Corporation Algorithm A Activation distribution in a neural network, 201814 P={0.5, 0.1, 0.05} λ = 0.005 P={0.5, 0.1, 0.05} λ = 0.05
- 15. Restricted © 2017 Mentor Graphics Corporation Algorithm B Input: a network, list of active nodes in the network, μ, λ Step 1: If there is no an active node in the network – exit Step 2: Randomly is selected a node Step 3: If the selected node doesn’t active – go to Step 2 Step 4: Each adjacent inactive node is activated with 𝛌 (𝛌+ 𝛍) probability Step 5: The temporary selected node is deactivated with 𝜇 (𝜆+ 𝜇) probability Step 6: Go to Step 1 Activation distribution in a neural network, 201815
- 16. Restricted © 2017 Mentor Graphics Corporation Algorithm B Activation distribution in a neural network, 201816 N=1024, p=0.5, μ=0.0
- 17. Restricted © 2017 Mentor Graphics Corporation Algorithm B Activation distribution in a neural network, 201817 N=1024, P = {0.5, 0.1, 0.05}, λ = 0.05 N=1024, P = {0.5, 0.1, 0.05}, λ = 0.0005
- 18. Restricted © 2017 Mentor Graphics Corporation Algorithm C Input: a network, list of active nodes in the network, μ, λ Step 1: If there is no an active node in the network – exit Step 2: For each active note of the current stage: — Step 2.1: Randomly is selected an adjacent node and in case if the neighbor doesn’t active – activate it with 𝛌 (𝛌+ 𝛍) probability — Step 2.2: The node is deactivated with 𝜇 (𝜆+ 𝜇) probability Step 3: Go to Step 1 Activation distribution in a neural network, 201818
- 19. Restricted © 2017 Mentor Graphics Corporation Algorithm C Activation distribution in a neural network, 201819 N=1024, p=0.5, μ=0.0
- 20. Restricted © 2017 Mentor Graphics Corporation Algorithm C Activation distribution in a neural network, 201820 N=1024, P = {0.5, 0.1, 0.05}, λ = 0.0005 N=1024, P = {0.5, 0.1, 0.05}, λ = 0.0005
- 21. Restricted © 2017 Mentor Graphics Corporation Common behavioral characteristics of the process ER(n,p), active_nodes_count = 1 The probability that in the current step will be activated at least one node: Activation distribution in a neural network, 201821 ‘A’ ‘B’ ‘C’ 1 𝑛 ∙ 𝜆 1 𝑛 ∙ (1 − 1 − 𝜆 𝑝∙ 𝑛−1 ) 𝜆
- 22. Restricted © 2017 Mentor Graphics Corporation Common behavioral characteristics of the process The probability that in the current step will be activated m / s.t {1 < m <= p ∙ (n − 1)} nodes: The probability that after the current step in the network there will be no active nodes: Activation distribution in a neural network, 201822 ‘A’ ‘B’ ‘C’ 0 1 𝑛 ∙ ( 𝑚 𝑝 ∙ 𝑛 − 1 ∙ 𝜆 𝑚 ∙ 1 − 𝜆 𝑝∙ 𝑛−1 −𝑚 ) 0 ‘A’ ‘B’ ‘C’ μ∙( 1 𝑛 ∙ (1 − 𝜆)) μ∙( 1 𝑛 ∙ 1 − 𝜆 𝑝∙(𝑛−1)) μ ∙ 1 − 𝜆
- 23. Restricted © 2017 Mentor Graphics Corporation A visual representation of the process ER(n, p) / n=65, p = 005, active_nodes_count=65, 𝜆 = 0.314159265, μ=1.0 Activation distribution in a neural network, 201823 0 20 30 50 500 1500
- 24. Restricted © 2017 Mentor Graphics Corporation Core characteristics of the software Accuracy Efficiency Platform independency Activation distribution in a neural network, 201824
- 25. Restricted © 2017 Mentor Graphics Corporation www.mentor.com