UNET: Massive Scale DNN on Spark
- 1. UNET: Massive Scale DNN on Spark
- 2. Deep Neural Net Input Layer Hidden Layer 1 Hidden Layer 2 Hidden Layer 3
- 4. Overview Components: Solver, Parameter Server, Model Splits. Massive Scale: Data Parallel & Model Parallel. Train Method: Async and Sync Algorithms: RBM, DA, SGD, CNN, LSTM, AdaGrad, L1/L2, L-BFGS. CG, etc. Extensibility: Can be extended to any algorithm that can be modeled as data flow. Highly optimized with lock free implementation, and software pipeline maximizing the performance. Highly flexible and modulized to support arbitrary network.
- 5. Architecture: Data / Model Parallel Solver Model1_3 Model1_2 Model1_1 Q PS_2 Q PS_3 Q PS_1 One Solver RDD (1 partition) One Parameter Server RDD (3 Partitions) Three Replicated Model RDD (3 Partitions Each)
- 6. Data Parallel Component: Models & Parameter server Multiple models trained independently Each model fits one splits of training data, and calculates the sub-gradient Asynchronously, each model update/retrieve parameters to/from parameter server
- 7. Data Parallel (2 replicated Models with 1 Parameter Server) Parameter Server Q ModelYModelX Parameter Sync
- 8. Model Parallel Model is huge, and cannot be hold in one machine. Training is computational heavy Model partitioned into multiple splits. Each split may located in different physical machines.
- 9. Model Parallel (3 Partitions) Data Communication: • node-level • group-level Control RPC traffic Netty based Data Traffic Master Executor Executor Executor
- 10. Data / Model Parallel Solver Model1_3 Model1_2 Model1_1 Q PS_2 Q PS_3 Q PS_1 One Solver RDD (1 partition) One Parameter Server RDD (3 Partitions) Three Replicated Model RDD (3 Partitions Each)
- 11. A Simple Network Convolutional Fully Mesh Softmax Facility Master
- 12. Parameter Management ParamMgr.Node for fully meshed layer Managed by individual node. ParamMgr.Group for convolutional layer Shared by all nodes in the group, and managed by the group. The group gather/scatter the parameters from its members, which may locate in different executors. ParamMgr.Const for softmax master layer The parameters are constant.
- 13. qi,1 qi,2 qi,3 qi,4 Node Params Parameter Type (Link vs. Node) q1,I l q2,I l q3,I l Left-link Params qi,1 l+1 qi,2 l+1 qi,3 l+1 Right-link Params 1. Each parameter is associated with either a link or a node. 2. Each node/link may have multiple parameters associated. 3. Link parameters are managed by upstream. 4. Each category of parameters may be managed by either the node or the group.
- 14. Network Partitioning • The DNN network is organized by layers • Each layer is defined as three-dimension cube by (x, y, z). • Each dimension can be arbitrarily partitioned, defined as (sx, sy, sz), s specifies the number of partitions of one dimension. • One layer can be in multiple executors, and one partition is the basic unit to be distributed in executors. x(sx=3) z(sz=3) y (sy=2)
- 15. Software Components Layer: logical group in deep neuron net. Group: logical unit having similar input/output topology and functionality. A group can further have subgroups. Node: the basic computation unit provide neuron functionality. Connection: define the network topology between layers, such as fully meshed, convolutional, tiled convolutional, etc. Adaptors: mapping the remote upstream/down stream neuron to local neuron in the topology defined by connections. Function: define the activation of each neuron. Master: provide central aggregation and scatter for softmax neuron. Solver: central place to drive the model training and monitoring. Parameter Server: the server used by neuron to update/retrieve parameters.
- 16. Memory Overhead Neuron does not need to keep the inputs from upstream, but only keeps the aggregation record. The calculation is associative in both forward/backward path (through function split trick). The link gradient is calculated and updated in the upstream Memory overhead is O(N + M), N is the neuron size and M is the parameter size.
- 17. Network Overhead Neuron forwards same output to its upstream/downstream neurons. Receiving neurons compute the input or update the gradient. Neuron forwards its output to the executors only if it hosts neurons requesting it. Neuron forwards its output to an executor only once regardless of the number of neurons requesting it.
- 18. Complexity Memory: O(M+N) independent of network partition mechanism. M: the number of parameters N: The number of nodes. Communication: O(N) Realized by Each node managing its outgoing link parameter instead of incoming link parameter The trick to split the function across the layers
- 19. Distributed Pipeline MicroBatch: The number of training examples in one pipeline stage max_buf: the length of the pipleline. Batch algorithms: Significantly improve the performance when the training data set is big enough to fully populate the pipeline. SGD: the improvement is limited, because the pipeline cannot be fully populated if the miniBatch size is not big enough. Executor 4 Executor 3 Executor 2 Executor 1 Micro Batch i +4 Micro Batch i +3 Micro Batch i +2 Micro Batch i +1 Micro Batch i +1 Micro Batch i +1 Micro Batch i +1 Micro Batch i +2 Micro Batch i +2 Micro Batch i +3 T1 T2 T3 T4
- 20. Connections Easy extensible through Adaptors. Adaptor is used to mapping global status to its local status. Fully Meshed (Tiled) Convolutional NonShared Convolutional