Case Study: Large Scale Deployment for Machine Learning with Highspeed Storage
1 like270 views
This document outlines the corevo computing infrastructure by NTT Corp, detailing the components such as GPU servers, AI frameworks, storage solutions, and monitoring systems. It highlights the hardware specifications, network configurations, and software stacks used in their infrastructure. Overall, the document serves as a technical reference for understanding their computational resources and capabilities.
Case Study: Large Scale Deployment for Machine Learning with Highspeed Storage
- 1. Copyright©2018 NTT Corp. All Rights Reserved. : :
- 2. Copyright©2018 NTT Corp. All Rights Reserved. NTT Confidential ) 4 424 4. ) 1 431 1 ) . (11 1. 4 3 (1 1 . 1 . 1. , 1 1 . 4 4 1
- 3. Copyright©2018 NTT Corp. All Rights Reserved. NTT Confidential • 1 . : . • . . - .
- 4. Copyright©2018 NTT Corp. All Rights Reserved. NTT Confidential • . 1 / , / , ()) : , , 1 , , 1 . , . , , . , , A , : , . , / . :, : . : . .
- 5. Copyright©2018 NTT Corp. All Rights Reserved. NTT Confidential • & : - : • 1 • - - / - 5 - 5 / 5 - - 5 - - 5 / 5 • - - - 5 5 5 5 corevo Computing Infrastructure (On-Premise) : . - Compute (GPU Servers) Storage Dashboard Monitoring High Speed Network AI Framework Stacks datamodels - - 5 5 - 5 - - -
- 6. Copyright©2018 NTT Corp. All Rights Reserved. NTT Confidential • . 1 1 AI Researchers, Framework Design Facilities, Network Design Scale-Out Storage Design Monitoring/ and Operation Design C D 6 6 2 2A2 2 2 2 2 22 2 2 2 1 2 2 6 22 6 6 2A2 2 2 2 2 2 6 2 2 6 22
- 7. Copyright©2018 NTT Corp. All Rights Reserved. NTT Confidential . 1 .1 GPUs Millanox 100G EDR Infini Band Backend AI Framework Stacks APPAPP Bare-Metal HW Stack Storage 7 - 7 Managements Tesla K80 GTX1080 GTX1080 Tesla K80 7 - 7 7 7 TCP 10G Ethernet Service TCP 1G Ethernet Management, IPMI Software Stack Ubuntu - - Docker KVM Guest OSGuest OSGuest OS Dashboard Monitoring Ansible Swift - 7 NFS Chainer MN Chainer SwiftFS
- 8. Copyright©2018 NTT Corp. All Rights Reserved. NTT Confidential
- 9. Copyright©2018 NTT Corp. All Rights Reserved. NTT Confidential • : . : A . • 9 , • . A: : / : • 9 99 9 : • : C . 9 C ,D • . / :/ • , 99 9 : D : 9 : • . ?. : • - 9 B A9 • : 9 B B D . / : ?
- 10. Copyright©2018 NTT Corp. All Rights Reserved. NTT Confidential . 1 : 70: 6 2-2 7 70 1 : -2//.9.6 . 9 2/2 2 26 . 20.6 . 126. . 96260 -.. . 96260 2 1 . 20 . /97 9 -2 276 126. . 96260 7 -.. . 96260 9. 20 - 6- 7 260 9.:7 9 .:
- 11. Copyright©2018 NTT Corp. All Rights Reserved. NTT Confidential . 7- 0 0: 7- 2 -7 :7 1 7- - 2 / 71 0 1 22 - 0- 2: - . 7 1 1 / 0 1 / /2- 2 / 7- 1 1 1 / : 7- 2- -..- 1 - - 0 2
- 12. Copyright©2018 NTT Corp. All Rights Reserved. NTT Confidential . • : A :A : 2 A A A : 2 • A D : D A - ,. :: AA A 1 A : A A 2A A A 1 A A : AD • :: 2 A A : D A A A A : 2 • A 2 A A : A A : , :: : A , :: : / 2 A :: : 2 A A A : : F
- 13. Copyright©2018 NTT Corp. All Rights Reserved. NTT Confidential A G C C ( G - D : - : A D : . . CD D D: K 2G D K 2G D K 2G D K 2G D K : I D OF ( OF OF ( OF OF OF OF ( OF . D C F R R P P R D D ) 2 / 3))1 AGIGMG 2 / . - ( 3 2 / 3))1 ) DGG 2 / H F2 / L IF K , I ). D 2( 2( 2 / 2( 2 / 2( 2 / 2( 2 / CD D D A HIG I G I E NGIC M IO A A )A F I23 LHHGIK K H I DD D E N KA 2 / K F IDO K GM I A A H FK I GFF K F F F F 3))1
- 14. Copyright©2018 NTT Corp. All Rights Reserved. NTT Confidential • . : . . . . . A • 4 14 341 • 3 1. . 4 4 14 341 34 • . : . : : : . : - . 41 33 41 4- . -. 1 . 1- . 3 2 1 4 -. 131 . . 2 . 1 1 . - 1414/ C . . . .
- 15. (Copyright©2018 NTT Corp. All Rights Reserved. NTT Confidential . / / / / 3 31 / 13 / 13 / 13 / 13 / 3 1 A 1/ 3/2 3 2/ / 2 0A 3 3 13 3 31 2/ / 31 2/ / 31 2/ / 3A / 3 3A / 3 3A / 3 3A / 3 . / . / . / . / A A / 3 5 23 5 A 5 A 5 A 5 A ) 3 /5 5 3 5 / 32A13 /4 3 0/1 / 2 0 / 31 2/ / . /
- 16. Copyright©2018 NTT Corp. All Rights Reserved. NTT Confidential • . : . • -M> M.66 • 7I 2 991 C KC> .H D H L C • 6--4 • 46: 70/. • . 9L L EE H >H D H L C K 3C D L L C C K CIL L F KL H> • KL H> , NH D H> K 9LHIK EE H H L C K N L C C C CK >
- 17. *Copyright©2018 NTT Corp. All Rights Reserved. NTT Confidential . . - . ( ) * , ( ) * , 5 = D G 9 G D 6 D 47 D9 G DI 8 G D 4D=G 23056 G 8 1 D G
- 18. ,Copyright©2018 NTT Corp. All Rights Reserved. NTT Confidential - . ( ) * ( ) * 78A8DI8 0A8EG 5 B G PN D 3D D 8 D 8 8 D .8 B 8A D I8 2B8 3 * GA8 , ( DG G (146G
- 19. Copyright©2018 NTT Corp. All Rights Reserved. NTT Confidential • . . . . . . . : . : : • 1 /1/ 1 /1 • . . : : . . . : . • . . ./ : : - • : : : . :/ / : . . : : A D . .
- 20. Copyright©2018 NTT Corp. All Rights Reserved. NTT Confidential • 3: 3 .3 . 3 • 2 0 • • 0 0 • , . 3: 3 .3 . 3 • 2 0 • 0 • , . . .3 . 3 3.
- 21. Copyright©2018 NTT Corp. All Rights Reserved. NTT Confidential • - : D . :: 3 3 ) ( 3C 3 3 C D C • ( . : C3 C 3 C . ) . : 3 D 3 . :
- 22. Copyright©2018 NTT Corp. All Rights Reserved. NTT Confidential . . . IB SW IB SW IB SW IB SW IB SW IB SW 2 - 2 - CPU CPU () HCA () HCA GPU GPU GPU GPU GPU GPU - -
- 23. Copyright©2018 NTT Corp. All Rights Reserved. NTT Confidential • . . • -366078 8 1 2 ) ( 3 . - . .
- 24. Copyright©2018 NTT Corp. All Rights Reserved. NTT Confidential • -- • / 2 2 / B / 1 B /44 1 1/ • B: 4 2 9 8 B/9 B: 4 B 9 8 9 18 8 - 3 3 3. 3 3 - ( 9/ /1 / -. IB SW IB SW IB SW IB SW IB SW IB SW 9/ /9 ) 19B 2 4/
- 25. )Copyright©2018 NTT Corp. All Rights Reserved. NTT Confidential 3 : 3 ? ?3 3 3 3 • 3 3 3 3 3 3 • 4ECD 37: AH EM - 6 H 7 5A GIM A A A • 3 3 3 : • 1 G 9I AM 5HMAG AIH 7: 2) , • 3AHA MEIH G HA ) ) 3/ 7 5A Q ( GIM 7: • 75 , 3/ IHHA MEIH AMPAAH 7: • 3 : : : 3 3 3 C :A . 3 3 C :A 3 3 CPU CPU 75 7 5A 9 7 5A 9 GPU GPU GPU GPU GPU GPU IHCA IHCA 54 .- 5HBEHE H 4I M D HHAG . MI
- 26. Copyright©2018 NTT Corp. All Rights Reserved. NTT Confidential • 1 3 1 3 D 1 3 . 1 1 ::A 1 C A CPU CPU IHCA IHCA GPU GPU GPU GPU GPU GPU
- 27. Copyright©2018 NTT Corp. All Rights Reserved. NTT Confidential • 3: ) : ) • 2AA ( 2 2B2 B 2 A7 G • ( A CAB B B A2 B D • 3: ) • 2AA ( 2 2B2 B 2 A7 G 2B +C B A • + 2 D ) ) AB 2 2 B ( C ( : . ( 3 GPU GPU GPU IHCA GPU IHCA CPU CPU CPU
- 28. Copyright©2018 NTT Corp. All Rights Reserved. NTT Confidential • 1 3 1 3 D 1 3 . 1 1 ::A 1 C A CPU CPU IHCA IHCA GPU GPU GPU GPU GPU GPU
- 29. ,Copyright©2018 NTT Corp. All Rights Reserved. NTT Confidential . 3 ( ) * , ( ) * , 7 D G= KI K = G= 8 EGI 059 G K D GM I = E DK G= 5IGKE 34178 K D 2 G K 6 56148 IGE *
- 30. Copyright©2018 NTT Corp. All Rights Reserved. NTT Confidential .3 . . . GPUs Tesla K80Tesla K80 - 0 Prometheus https://prometheus.io/ GPU monitoring script 3 B : 3 / B Grafana (https://grafana.com/, UI) : 3 0: Managements : . : B : :
- 31. Copyright©2018 NTT Corp. All Rights Reserved. NTT Confidential - . . 3
- 32. Copyright©2018 NTT Corp. All Rights Reserved. NTT Confidential - . . 3 GPU Clocks GPU Temperature
- 33. Copyright©2018 NTT Corp. All Rights Reserved. NTT Confidential - . . 3 GPU Clocks GPU Temperature
- 34. Copyright©2018 NTT Corp. All Rights Reserved. NTT Confidential • : : : : • . : . • . : :3 • A 4 A ?4 4 ? ? - ( : : Storage ? 3 ) ? 4 4 Storage ? 3 ) ? Storage ? 3 ) ? 4 4 4 AI Framework Stacks Chainer MN Chainer Other AI Framework e.g. TensorFlow) 4 D - ? C ( 4 4 A 43
- 35. Copyright©2018 NTT Corp. All Rights Reserved. NTT Confidential • 3 . 3 • 3 3 3 3 5 3 35 5 3 3 3 (( ) (( ( ( ( (
- 36. Copyright©2018 NTT Corp. All Rights Reserved. NTT Confidential • . 2 223 . 1 F D ( ) .3- F 1 D 1 D A /A 1D 1 D D D 6 D 11) .3- 11) 1(1 )) 1(1 2 - 3 3 . 2 11) .3- 11) 1(1 )) 1(1 H 2 6 D ( ) 6 . 1 D D 1 D 6 1 D 1 1 D 3
- 37. Copyright©2018 NTT Corp. All Rights Reserved. NTT Confidential 3 . . 3 - / -C7 (7 7 3 3 - / 3 -C7 )- ) 3 / 3 7:3 D 3 -C7 :: - . 3 D 3 / 7 7:3 / 3 -C7 : 3 7 7 C7 Swift FS (File-System-in-User-Space, FUSE) 3 / C7 :/ 3
- 38. Copyright©2018 NTT Corp. All Rights Reserved. NTT Confidential . 3 ) 1 A 1 ) 1 : 31 OS Swift FS (File-System-in-User-Space, FUSE) AI Framework Stacks Chainer MN Chainer ) 1 : 31 ( A 1 18 ( : ) ( ( Swift Client SDKs
- 39. Copyright©2018 NTT Corp. All Rights Reserved. NTT Confidential • + : . : • 3 3 • 9 33 3 • 3 : • 3 • 9 . 3 3 9 33 . 3 3 : . . -
- 40. Copyright©2018 NTT Corp. All Rights Reserved. NTT Confidential • • 4 / 0 / / • 4 4 / / • 0 4 • • 4 / 4 / • 4 / 4 0 / ,3 .- .-
- 41. Copyright©2018 NTT Corp. All Rights Reserved. NTT Confidential • • .4 8 4 : ? F A F 4 8 64?8 F ?8 F8 A? A: C8A F46 I F • A 8D4F A I F 64?8 F 8 D84 A FD F8 46 A8 ?84DA A: • 64A ?84DA 4F FFC : !:? D • : : , : 3 : • /88 F ?84DA D8!!! • 8 4A ? F A 4 8 4? 4668 8FI88A ?8 F8 4A 86F 1- , 3 - , 3. 1?84 8 4D8 6 F 8 : ? F A
- 42. Copyright©2018 NTT Corp. All Rights Reserved. NTT Confidential • 4--, : / - - - . - . - - -4: 4 - • /- - 4 .. :/- /- . -:- -4 4 -4: - . 4 / , - :- -: • 4 .4 2 4 • 2 4 4 .4 4 • 4 - - 44 : - : - 4,- . - • - 4 - 4-,
- 43. Copyright©2018 NTT Corp. All Rights Reserved. NTT Confidential > < @ > > < > C > C > . >> 4 < > 4 < . >> 3 > > 3 . >>