An introduction and evaluations of a wide area distributed storage system
3 likes936 views
The document discusses the development and evaluation of a wide area distributed storage system, focusing on its fast random I/O performance and scalability. It highlights features such as live migration between distributed areas and the storage architecture's ability to ensure data consistency and redundancy. Additionally, the system is assessed in the context of disaster recovery scenarios and international data transfer performance.
![Eurasian
plate
North
American
Plate
Pacific
Ocean
Plate
Philippine Sea
Plate
epicenter
of 3.11
Nankai
(South Sea)
Trough
[NEXT]](https://image.slidesharecdn.com/21q8smkntrczmf8gd6uq-signature-6d788d9796c834efb4b09d764dcb172182f300de6f54cdd59af88b97bba3dea3-poli-140916134706-phpapp02/85/An-introduction-and-evaluations-of-a-wide-area-distributed-storage-system-18-320.jpg)
![write read re-readre-write
random read backwords read records rewrite
strided read
random write
fwrite
file size [Bytes] file size [Bytes] file size [Bytes]
recordsize[KB]
recordsize[KB]recordsize[KB]
4
16
64
256
1024
4096
16384
4
16
64
256
1024
4096
16384
0
20
40
60
80
100
120
64KB 256 4 16 64 256 1GB 4 161MB 64KB 256 4 16 64 256 1GB 4 161MB
64KB 256 4 16 64 256 1GB 4 161MB 64KB 256 4 16 64 256 1GB 4 161MB
64KB 256 4 16 64 256 1GB 4 161MB 64KB 256 4 16 64 256 1GB 4 161MB 64KB 256 4 16 64 256 1GB 4 161MB 64KB 256 4 16 64 256 1GB 4 161MB
64KB 256 4 16 64 256 1GB 4 161MB 64KB 256 4 16 64 256 1GB 4 161MB
64KB 256 4 16 64 256 1GB 4 161MB
MB/sec
4
16
64
256
1024
4096
16384
frewrite](https://image.slidesharecdn.com/21q8smkntrczmf8gd6uq-signature-6d788d9796c834efb4b09d764dcb172182f300de6f54cdd59af88b97bba3dea3-poli-140916134706-phpapp02/85/An-introduction-and-evaluations-of-a-wide-area-distributed-storage-system-60-320.jpg)
An introduction and evaluations of a wide area distributed storage system
- 1. An introduction and evaluations of a wide area distributed storage system
- 4. 1978
- 7. 80s
- 10. 90s
- 11. the Internet
- 14. 2003.8.14 Northeast blackout of 2003
- 15. in Japan
- 16. 2011.3.11The aftermath of the 2011 Tohoku earthquake and tsunami
- 18. Eurasian plate North American Plate Pacific Ocean Plate Philippine Sea Plate epicenter of 3.11 Nankai (South Sea) Trough [NEXT]
- 19. Gunnma
- 20. Gunnma
- 21. Gunnma
- 22. Ishikari
- 23. Ishikari
- 24. Is Two enough ?
- 25. cost
- 29. Kitami Institute of Technology University of the Ryukyus SINET the longest path
- 30. Cybermedia Center Osaka University Kitami Institute of Technology University of the Ryukyus XenServer 6.0.2 CloudStack 4.0.0 XenServer 6.0.2 CloudStack 4.0.0
- 31. problems
- 32. shared storage
- 33. ≒50ms
- 34. RTT > 200ms
- 35. Storage XenMotion Live Migration without shared storage > XenServer 6.1
- 37. VMware VSAN
- 40. requirement
- 41. 64 256 1024 409616384655362621441.04858e+064.1943e+061.67772e+076.71089e+074 16 64 256 1024 4096 16384 0 20000 40000 60000 80000 100000 120000 Kbytes/sec File size in 2^n KBytes Record size in 2^n Kbytes 0 20000 40000 60000 80000 100000 120000 High Random R/W Performance
- 42. POSIX file system interface protocl NFS, CIFS, iSCSI
- 45. Con$idential Global VM migration is also available by sharing "storage space" by VM host machines. Real time availability makes it possible. Actual data copy follows. (VM operator need virtually common Ethernet segment and fat pipe for memory copy) TOYAMA site OSAKA site TOKYO site before Migration Copy to DR-sites Copy to DR-sites live migration of VM between distributed areas real time and active-active features seem to be just a simple "shared storage". Live migration is also possible between DR sites (it requires common subnet and fat pipe for memory copy, of course) after Migration Copy to DR-sites
- 46. Con$idential Front-end servers aggregate client requests (READ / WRITE) so that, lots of back-end servers can handle user data in parallel & distributed manner. Both of performance & storage space are scalable, depends on # of servers. front-end (access server) Access Gateway (via NFS, CIFS or similar) clients back-end (core server) WRITE req. write blocks read blocks READ req. scalable performance & scalable storage size by parallel & distributing processing technology
- 47. File block block block block block block block block block Meta Data consistent hash backend (core servers)
- 49. Con$idential 1. assign a new unique ID for any updated block (to ensure consistency). 2. make replication in local site (for quick ACK) and update meta data. 3. make replication in global distributed environment (for actual data copies). back-end (multi-sites) a file, consisted from many blocks multiplicity in multi-location, makes each user data, redundant in local, at first, 3 distributed copies, at last. (2) create 2 copies in local for each user data, write META data, ant returns ACK (1) (1') (3-a) (3-a) (3-a) make a copy in different location right after ACK. (3-b) remove one of 2 local blocks, in a future. (3-b) (1) assign a new unique ID for any updated block, so that, ID ensures the consistency Most important ! the key for "distributed replication"
- 50. NFS CIFS iSCSI
- 51. redundancy = 3 r = 2 ACK r = 1 r = 0 write
- 52. dundancy = 3 ACK r = 2 e = 0 r = 1 e = 0 r = 0 e = 1 r = -1 e = 2 external
- 55. 316 km 440 km 690 km Hiroshima Univ. Kanazawa Univ.
- 56. Hiroshima Univ. Kanazawa Univ. NII VMM: virtual machine monitor CS: core servers HS: hint servers AS: access servers AS AS VMM VMM CS CS CS CS CS CSHS HS CS CS CSHS L3VPN L3VPN L2VPN L2VPN L2VPN L2VPN L3VPN EXAGE-LAN EXAGE-LAN admin LAN admin LANMIGRATION-LAN EXAGE-LAN MIGRATION-LAN
- 58. iozone -aceI a: full automatic mode c: Include close() in the timing calculations e: Include flush (fsync,fflush) in the timing calculations I: Use DIRECT_IO if possible for all file operations.
- 59. write 64 256 1024 409616384655362621441.04858e+064.1943e+061.67772e+076.71089e+074 16 64 256 1024 4096 16384 0 20000 40000 60000 80000 100000 120000 Kbytes/sec File size in 2^n KBytes Record size in 2^n Kbytes 0 20000 40000 60000 80000 100000 120000 64 256 1024 4096 16384655362621441.04858e+064.1943e+061.67772e+076.71089e+07 4 16 64 256 1024 4096 16384 File size in 2^n KBytes Recordsizein2^nKbytes
- 60. write read re-readre-write random read backwords read records rewrite strided read random write fwrite file size [Bytes] file size [Bytes] file size [Bytes] recordsize[KB] recordsize[KB]recordsize[KB] 4 16 64 256 1024 4096 16384 4 16 64 256 1024 4096 16384 0 20 40 60 80 100 120 64KB 256 4 16 64 256 1GB 4 161MB 64KB 256 4 16 64 256 1GB 4 161MB 64KB 256 4 16 64 256 1GB 4 161MB 64KB 256 4 16 64 256 1GB 4 161MB 64KB 256 4 16 64 256 1GB 4 161MB 64KB 256 4 16 64 256 1GB 4 161MB 64KB 256 4 16 64 256 1GB 4 161MB 64KB 256 4 16 64 256 1GB 4 161MB 64KB 256 4 16 64 256 1GB 4 161MB 64KB 256 4 16 64 256 1GB 4 161MB 64KB 256 4 16 64 256 1GB 4 161MB MB/sec 4 16 64 256 1024 4096 16384 frewrite
- 61. 0 20 40 60 80 100 120 10MB 100MB 1GB 10GB Throughput(MB/sec) File size write rewrite read reread random read random write bkwd read stride read fwrite fread legend record rewrite 0 20 40 60 80 100 120 10MB 100MB 1GB 10GB Throughput(MB/sec) File size 0 20 40 60 80 100 120 10MB 100MB 1GB 10GB Throughput(MB/sec) File size 0 20 40 60 80 100 120 10MB 100MB 1GB 10GB Throughput(MB/sec) File size 0 20 40 60 80 100 120 10MB 100MB 1GB 10GB Throughput(MB/sec) File size 0 20 40 60 80 100 120 10MB 100MB 1GB 10GB Throughput(MB/sec) File size 0 20 40 60 80 100 120 10MB 100MB 1GB 10GB Throughput(MB/sec) File size 0 20 40 60 80 100 120 10MB 100MB 1GB 10GB Throughput(MB/sec) File size 0 20 40 60 80 100 120 10MB 100MB 1GB 10GB Throughput(MB/sec) File size 0 20 40 60 80 100 120 10MB 100MB 1GB 10GB Throughput(MB/sec) File size 0 20 40 60 80 100 120 10MB 100MB 1GB 10GB Throughput(MB/sec) File size 従来方式 Exage/Storage 広域対応 Exage/Storage
- 62. SINET4 Hiroshima University EXAGE L3VPN SINET4 Kanazawa University EXAGE L3VPN
- 65. proposedmethod shared NFS Read (before migration) Read (after migration) Write (before migration) Write (after migration) Throughput(MB/sec)
- 68. Ikuo Nakagawa INTEC Inc. / Osaka University
- 69. Kohei Ichikawa Nara Institute of Science and Technology
- 70. We have been developing a widely distributed cluster storage system and evaluating the storage along with various applications. The main advantage of our storage is its very fast random I/O performance, even though it provides a POSIX compatible file system interface on the top of distributed cluster storage.
- 71. an initial plan
- 72. s Shinji Shimojo Director of JGN-X, NICT
- 73. It s not so fun.
- 74. real stage
- 76. Blocks (chunks) are located on the nearest
- 77. consistent hash Meta data is not suitable for wide area
- 78. type of line load condition required time (sec) domestic no load 17.9 international no load 201.6 read load 175.4 write load 400.6 required time to migration IO performance type of access pattern load condition domestic (read) 64.6 domestic (write) 58.7 international (read) 25.4 international write) 20.9 average throughput (MB/s) of dd
- 79. Live migration demo on an international line
- 80. Evaluations of distcloud on an international line
- 82. U.S. region will be build soon
- 83. Future Works
- 85. SC142014/11/16∼21 @Ernest N. Morial Convention Center
- 89. mobile devices sensor devices personal data aggregation service high latency power consumption
- 92. the Internet distcloud storage region A region B region C live migration optimize routes with remaining independence of each region users from the Internet can access the VM after live migration
- 93. Layer method outline features L3 routing update routing table by each migrations ○ routing per region cannot routing per VM routing operation cost routing + L2 extension VPLS, IEEE802.1ad PB(Q in Q) IEEE802.1ah (Mac-in-Mac) ○ stability, operation cost poor scalability L2 over L3 VXLAN, OTV, NVGRE ○ stability overhead of tunneling IP multicast SDN OpenFlow ○ programable operation cost of equipment ID/locator separation LISP ○ scalability, routing per VM cost, immediacy IP mobility MAT, NEMO, MIP (Kagemusha) ○ scalability load of router L4 mSCTP SCTP multipath ○ independent from L2/L3 limited in SCTP L7 DNS + reverseNAT Dynamic DNS ○ independent from L2/L3 altering IP addr. closing connection
- 94. 2011.3.11The aftermath of the 2011 Tohoku earthquake and tsunami
- 97. 2014 Storage Developer Conference. © Osaka University All Rights Reserved. go on to the next stage