![Dodai = dodai-deploy + dodai-compute
Dynamically and securely separated clusters
PaaS Hadoop MPI Hadoop MPI Hadoop MPI
…
SunGrid
Hadoop
Engine
IaaS OpenStack-2
OpenStack-1 Eucalyptus
Upper layer: Deploying software [dodai-deploy]
CaaS
Lower layer: Preparation of nodes [dodai-compute]
20](https://image.slidesharecdn.com/openstackdesignsummitcolonysession-120419093627-phpapp02/85/OpenStack-design-summit-colony-session-21-320.jpg)
OpenStack design summit (colony session)
- 1. Intercloud object storage: colony - Trial to use swift 2012.4.18 @jxta in wide area network - Shigetoshi Yokoyama National Institute of Informatics
- 2. Cloud projects @ NII 1
- 3. Goal: academic community cloud NII Cloud Academic Community Cloud Education Cloud Univ.-X Cloud Univ. -A Cloud Univ.-B Cloud Research Cloud ・・・ Intercloud services Science Information Network 2
- 4. Colony and dodai in a scenario VMI: Virtual Machine Image VM: Virtual Machines VMI colony VMI Launch machine images Virtual Machines VM VM VM VM VM VM VM VM IaaS IaaS IaaS IaaS Cluster Cluster Cluster Cluster dodai Physical Machines SINET (L2VPN) 3
- 5. Colony 4
- 6. Intercloud object storage service Colony is a project which Nova federates cloud object storage services, like swift Glance Swift for intercloud Swift to realize intercloud object storage service. use Swift for intercloud use Swift for Nova Swift for Swift local use intercloud Glance use Glance Swift Nova Swift for intercloud use 5
- 7. Users’ points of view Cloud Services Cloud-B Object B1-1 Cloud-A Container B1 Container B2 Container B3 Object B1-2 Object B1-3 Object A1-1 Swift-B Object I1-1 Container A1 Object A1-2 Inter-cloud Container I1 Object I1-2 Container A2 Object A1-3 Inter-cloud Container I8 Object I1-3 Swift-A Container A3 Inter-cloud Container I10 Inter-cloud Container I1 Object I4-1 Inter-cloud Container I4 Object I4-2 Inter-cloud Container I13 Object I4-3 Geographically Inter-cloud Container Object I1-1 Object I1-2 Distributed I1 Inter-cloud Container I2 Object I1-3 Swift-I Inter-cloud Container I3 Inter-cloud Container I4 Object I4-1 Object I4-2 Object I4-3 Inter-cloud object storage service : colony 6
- 8. Colony realizes the federation Shibboleth IdP Authenticate with Shibboleth IdP Cloud-A User Colony Apache mod_wsgi mod_shib Colony-horizon Colony-keystone Colony-dispatcher Squid Provide seamless access to Slapd Ubuntu multiple swifts Swift Swift Swift-I Colony-Keystone Colony-Keystone Swift-A Slapd Slapd 7
- 9. Developed software components in colony • Colony-Horizon – based on diablo/stable Horizon with some enhancements • Multi-region support – Users can choose which swift is used to store/retrieve objects • Swift Container’s ACL ,metadata support • Swift Object’s metadata support • >5G segment upload support … • Colony-Keystone – based on diablo/stable Keystone with some enhancements • Authenticate with Shibboleth • %{tanant_name} can be used for endpointTemplates in addition to %{tenant_id} for federating cloud services • Colony-Dispatcher - new • Relay requests to multiple object services (and merge response for clients) • Relay requests to a specific object service indicated by URI • Choose the “nearest” swift-proxy server to relay requests • Copy objects among different swifts • Utilities - new • Tools to make admin task easier to provide the federation of object storage services 8
- 10. Colony-horizon Users can choose swift Swift -I Swift -A 9
- 11. Colony – keystone Shibboleth IdP Modifications to keystone • Add ePPN field to keystone schema • ADD rest api services to create token by ePPN ('/token_by/eppn') and email address('/token_by/email') ('/ /email') • Add a rest api service to register/update ePPN ('/users/{user_id}/eppn') ('/users/{ 1. ID/passwd 2. Attribute: ePPN, mail_addr 0-1. User registration by mail_addr 0-2 . Associate ePPN to mail_addr by initial access Shibboleth SP Colony- Colony-Horizon 3. Attribute: ePPN Colony- 4. auth_token Keystone 10
- 12. Colony-dispatcher 1. Swift client can send requests to Swift-A and Swift-I through Swift Dispatcher 2. Swift Dispatcher merges the response from each Swift and send it to Swift Client Swift Client Requests modified for merging responses. A:container1 •Account Info A:container2 •Container List •X-Copy-from/to I:container1 I:container2 Colony Dispatcher Response merged by Swift Proxy Swift Proxy Swift Proxy Colony Dispatcher has a prefix to indicate which Swift is used to store. Swift-A (local) Swift-I (intercloud ) 11
- 13. Caching Swift Dispatcher can use cache proxy (like squid) per swift proxy to retrieve objects from remote swifts. A:container1 A:container2 I:container1 Colony Dispatcher I:container2 Cache(Proxy) Swift Client Swift Proxy Swift Proxy Swift Proxy Swift-A (local) Swift-I (intercloud ) 12
- 14. Download performance with cache Download performance (5times, average) 300,000,000 250,000,000 200,000,000 Cache is bytes/sec 150,000,000 effective, 100,000,000 especially for small size 50,000,000 0 objects. 1K(bytes) 1M(bytes) 10M(bytes) 100M(bytes) 1G(bytes) 5G(bytes) Normal 13,201 11843208 77113446 152263382 163115324 154252275 Cache 382,004 135938185 226800796 256492362 261189612 260956927 ・AMD Opteron(tm) Processor 6128 2000Mhz 16core Ubuntu Server 11.04 x86_64 Client ・32GB Memory Colony-Dispatcher ・10000baseT 10000Mb/s Full Duplex Squid Swift-Proxy ・AMD Opteron(tm) Processor 6128 2000Mhz 16core Ubuntu Server 11.04 x86_64 Swift-Storage(account, container, object)x 3 ・32GB Memory ・10000baseT 10000Mb/s Full Duplex 13
- 15. Performance : original swift Swift cannot perform well when confgirure it in inter-region. Swift @Sapporo •Upload – Always effected by worst case 1 2 3 4 5 1K 1,755 2,165 395 3,178 1.375 1M 431,840 446,824 537,768 412,542 43,307 10M 710,179 741,350 737,565 723,446 731,824 100M 780,548 775,131 788,620 782,503 786,224 Swift •Download – worse case is ramdomly choosed @Tokyo Swift@ (about 1/3) Chiba Client 1 2 3 4 5 6.15Mbits/sec 1K 11,608 13,293 12,916 14,691 12,126 1G 6.29Mbits/sec 7.80Mbits/sec 1M 3,427,827 3,427,402 3,455,788 673,614 3,386,009 7.73Mbits/sec netperf 10M 958,018 959,098 7,188,901 974,072 949,389 941.23Mbits/sec 10G 100M 1,109,440 98,400,555 99,641,184 15,411,221 92,400,774 929.15Mbits/sec 14
- 16. Performance : with cache Swift Cache works @Sapporo •Upload – Always effected by worst case 1 2 3 4 5 1K 2,388 2,027 1,629 2,718 2,423 1M 521,456 479,606 488,142 455,359 541,622 10M 727,240 678,511 724,562 730,127 699,518 100M 786,893 777,227 787,707 782,546 784,544 Swift @Tokyo Swift@ Chiba •Download (through Cache) 1 2 3 4 5 Client 6.15Mbits/sec 1K 398,133 415,753 397,979 446,576 406,510 1G 6.29Mbits/sec 1M 9,238,473 202,193,598 177,694,628 161,992,275 134,450,057 7.80Mbits/sec 7.73Mbits/sec netperf 10M 55,186,177 288,776,403 319,892,614 331,691,392 307,500,293 941.23Mbits/sec 100M 473,135,008 471,484,134 467,654,981 412,153,419 451,464,516 10G 929.15Mbits/sec 15
- 17. Performance : with zone-awareness Swift should be zone-aware for geographically distributed use. Swift Modifications for zone-awareness (just a hack) @Sapporo •Uploading • Calculate the number of unfinished tasks in the send queue for each area and when one area are much worth than others, quit the uploading job for its area. •Downloading • Check connection performance of object servers and try to retrieve an object from the fastest one. Swift •Upload @Tokyo Swift@ Chiba 1 2 3 4 5 1K 11,356 13,157 13,074 12,758 12,680 Client 6.15Mbits/sec 1M 9,824,750 11,205,249 7,599,312 10,931,206 11,199,982 1G 6.29Mbits/sec 7.80Mbits/sec 10M 52,294,403 51,437,092 51,050,686 52,641,471 52,300,141 7.73Mbits/sec netperf 100M 97,937,987 101,847,002 102,385,002 102,413,801 101,462,855 941.23Mbits/sec 10G 929.15Mbits/sec 16
- 18. Project status of colony • Prototyping is done in NII research cloud https://github.com/nii-cloud/colony/ • Future plan - Performance enhancements - Real deployment and evaluation 17
- 19. Dodai 18
- 20. CaaS : Cluster as a Service Clusters for each private cloud Elastic Private cloud - A Private cloud - B Private cloud - C Private Cloud Release servers to server resource pool Allocate servers from server resource pool Server resource pool 19
- 21. Dodai = dodai-deploy + dodai-compute Dynamically and securely separated clusters PaaS Hadoop MPI Hadoop MPI Hadoop MPI … SunGrid Hadoop Engine IaaS OpenStack-2 OpenStack-1 Eucalyptus Upper layer: Deploying software [dodai-deploy] CaaS Lower layer: Preparation of nodes [dodai-compute] 20
- 22. Dodai-deploy • It is a framework for deployment tool – New software can be added by anyone – Now supports • Openstack Essex • Openstack Diablo • Hadoop • SGE • It is fast – Do installation parallel on different machines. • It is easy to use – Not just Web UI – CLI for maintainer – Rest API for integration 21
- 23. How to use dodai-deploy • Install – Just one command to install all-in-one • curl https://raw.github.com/nii-cloud/dodai-deploy/master/setup-env/setup- all-in-one.sh | sudo sh • Use – Web UI • Learn step-by-step guidance at http://$dodai_deploy_server:3000/ – REST APIs • Try at http://$dodai_deploy_server:3000/rest_apis/index.html – CLI • script/cli.rb list SERVER 22
- 24. What’s inside in dodai-deploy MCollective dodai-deploy server puppet Ruby on Rails dodai-deploy nodes 23
- 25. dodai-compute • In a word –Bare metal version nova • Futures –Provide the same interface as nova to operate on physical machines –Separate network segments with Open Flow Controller 24
- 26. What’s inside in dodai-compute OpenFlow modified nova controller cobbler OpenFlow switch 25
- 27. Project status of dodai 1. Dodai-deploy - One of the OpenStack compute automated installations tools http://docs.openstack.org/trunk/openstack-compute/admin/content/openstack-compute-deployment-tool-with-puppet.html https://github.com/nii-cloud/dodai-deploy/ - Used in NII research cloud development 2. Dodai-compute https://github.com/nii-cloud/dodai-compute/ - Used in NII research cloud development 3. Future plan - Enahance based on Feedbacks from NII researchers - Experiments on intercloud compute service 26
- 28. NII research cloud gunnii 27
- 29. NII academic cloud : gunnii “Elastic Private Cloud” on on on demand demand demand Cluster as a Service Based on Deployment Tool dodai-deploy generalized Compute dodai-compute bare metal supported 28
- 30. gunnii prototype system NII shibboleth IdP Researchers L3 switch Object storage (Swift) GUI server OpenFlow Switch/controller Storage nodes X 8 (initial) NFS Deployed Hadoop, OpenStack, Sun Grid Engine Compute nodes×40 (initial) 29
- 31. Demo ? gunnii is shibboleth ready. 30
- 32. https://github.com/nii-cloud/ Thank you 31