Link Samba to Cloud Storage
- 1. Link Samba to Cloud Storage! May 2011! Fabrizio Manfredi Furuholmen!
- 2. Agenda Link Cloud Storage! Console! Samba Management Goals! Update! Solution ! Design! Cloud Overview! New Functions! Architecture ! Demo! Components! Examples! Limitations! * One Session two talks ! 2! 5/11/11!
- 3. Zeropiu ZEROPIU SpA, founded in 1994 by a group of professionals from leading multinational Information Technology companies.! ! We operate at international level and our Headquarters are in Italy. We have subsidiaries in Scandinavia called ZEROPIU Nordic, and in Libya with ZEROPIU MEA.! • Competence Center, able to • Identity Access Management support customers in meeting (IAM), Authentication and the standard regulation Authorization, Single Sign-On requirements (D.Lgs. and Provisioning! 196/2003), Basilea II, • Video on Demand solutions ! Sarbanes Oxley Act, BS7799/ Iso and 17799/Iso 27001. ! • Intranet, Extranet and Consulting! Solutions! Enterprise Portal integrated vertical solutions and applications! Application Services! Management! • Software infrastructure • Application and Facility design, management and management! maintenance! • Service Desk and technical • Application upgrade and Call Center.! evolution! • Testing and Quality Assurance! 3! 5/11/11!
- 4. Project Goals Disaster Recovery, Easy Reduce cost! Restore from Any Snapshots! Share information through many sites distributed Size without limit! on different locations! New Storage Without Changing IT Infrastructure! 4! 5/11/11!
- 5. Storage “Cloud storage is a model of networked online storage where data are stored on multiple virtual servers, generally hosted by third parties, rather Cloud Storage! NAS or SAN ?! than being hosted on dedicated servers…” (wikipedia)! Add unlimited, on- demand space! Pay for the storage Pro ! actually used! Availability, Reliability, load balancing! Security of stored data! Performance! Dependencies from used network Cons! connection! 5! 5/11/11!
- 6. Who does the Work ? Cloud! Network! • Your Internet Connection! • Fuse implementation! Fuse ! • Python-fuse, python interface to fuse lib! • Export Local drive via CIFS! Samba! • Policy, Permission, Auth! • rsync! Backup! • lvm! • Amanda/Bacula! 6! 5/11/11!
- 7. Architecture Cloud storage gateways! Linux System Windows SMB! Cloud Clients Connector 1 SAMBA! ?! Fuse Cloud! Windows SMB! cache Storage ! Servers Cloud NFS! Connector n Unix Servers NFS Fuse cache Local hard drive 7! 5/11/11!
- 8. HDFS Namenode • An HDFS cluster consists of a single NameNode • It is a master server that manages the file system namespace and regulates access to files by clients. Datanodes • Datanode manage storage attached to the system it run on • Applay the map rule of MapReduce Blocks • File is split into one or more blocks and these blocks are stored in a set of DataNodes 8!
- 9. Amazon Simple Storage Service Amazon S3 provides a simple web services interface that can be used to store and retrieve any amount of data, at any time, from anywhere on the web. It gives any developer access to the same highly scalable, reliable, secure, fast, inexpensive infrastructure…! ! Designed to provide 99.999999999% durability and 99.99% availability of objects over a given year.! (Amazon disclaimer)! http://s3.amazonaws.com/bucket/object! ! S3 stores arbitrary objects up to 5 terabytes in size! Objects are organized into buckets, and identified unique name! Bucket! Buckets and objects can be created, listed, and retrieved using either a REST-style HTTP interface or a SOAP interface. ! Objects! Objects can be downloaded using the HTTP GET interface and the BitTorrent protocol.! Objects Versioning ! Info! Metadata! ACL! Md5! S3 storage service now hosts more than 100 billion objects !! 9! 5/11/11!
- 10. Fuse FUSE , filesystem in user space! ! Simple library API ! Simple installation (no need to patch or recompile the kernel)! Secure implementation ! Userspace – kernel interface is very efficient! Usable by non privileged users ! Runs on Linux kernels 2.4.X and 2.6.X! Lot of Language bindings! Lot of file systems supported (> 50 )! Mount as a local filesystem! 10! 5/11/11!
- 11. s3fs: Cloud Translator Cloud! Unix! Bucket! Mount point! Cloud storage! Storage! • more application programming Objects! File! • based on block- or file-based! interface! Objects empty! Directory! Amz-meta- File Attributes! custom fields! Md5 Dirty cache! comparison! Remote copy! Rename/move! 11! 5/11/11!
- 12. Samba CIFS Translator! Permission Control! Samba! Locks! Authentication! 12! 5/11/11!
- 13. Demo 13! 5/11/11!
- 14. Configuration S3fs! use_cache! retries! connect_timeout! readwrite_timeout! max_stat_cache_size! prefix ! ! Fuse ! allow_other! kernel_cache ! ! Samba! It is simple share! Locks handle by samba ! ! 14! 5/11/11!
- 15. Limitations The buckets sharing is not safe due to the Amazons S3 consistency data model. ! Only module supported by Fuse ! Extended attributes and locks not supported! On change, the entire file is transferred! Network bandwidth! ! 15! 5/11/11!
- 16. Plans Short-term! • Better user and acl handling! • More cache triggers! • Encryption! • Compression! Middle-term! • Move s3fs to samba VFS plugin! • Locks! • Write only differences! • Split file in block (rsync model)! In the future! • Mixed local disk with cloud in a more intelligent way! • Software appliance ! • Distributed across different cloud storages! 16! 5/11/11!
- 17. Plans Short-term! • Better user and acl handling! • More cache triggers! • Encryption! • Compression! Middle-term! • Move s3fs to samba VFS plugin! • Locks! • Write only differences! • Split file in block (rsync model)! In the future! • Mixed local disk with cloud in a more intelligent way! • Software appliance ! • Distributed across different cloud storages! 17! 5/11/11!
- 18. Dream: myS3 Service! myS3! myS3! Application! Object replication! object! object! Filesystem! data! data! 18! 5/11/11!
- 19. Advantages Move from local server to cloud storage! No Change! • CIFS/NFS! Share! • WEB/Internet Disk! Disaster Recover! • Backup! 19! 5/11/11!
- 20. Conclusion S3 Advantage! • Pay for the storage used.! • No maintenance contracts, administration .. ! • Very high levels of availability.! • Very good for Backups and Disaster recover! • You can test for Free! • Good for share (web site)! • Many reads, low write rate! Disadvantage! • Edit large file! • Network performance! • Concurrence! • Changing one bit is the same of all file! • Not Posix! 20! 5/11/11!
- 21. Conclusion HDFS Advantage! • Low hw cost! • Very high levels of availability.! • You can test for Free! • Many reads, low write rate! • Good performance in read! Disadvantage! • Small file! • Not Posix! • Administration! 21! 5/11/11!