BHL hardware architecture - storage and clusters
0 likes999 views
The document discusses various aspects of open-source distributed storage solutions and their applications in cloud computing, particularly highlighting systems like GlusterFS for managing large-scale data. It emphasizes the importance of redundancy, fault tolerance, and data management strategies to mitigate hardware risks, and presents a proof of concept for a clustered filesystem demonstrating distributed processing capabilities. Additionally, it outlines the collaborative efforts of the Biodiversity Heritage Library to digitize and share biodiversity literature through a global commons initiative.
BHL hardware architecture - storage and clusters
- 1. January 2010 Beijing, China Phil Cryer open source systems architect and accidental tourist Tuesday, January 12, 2010
- 2. > BHL hardware architecture • storage issues and solutions • hardware risks and configuration • cloud computing possibilities • code and communication Tuesday, January 12, 2010
- 3. > Storage issues • more data being created and saved • expanding SANs can be expensive • storage has not kept up with Moore’s Law • backups are usually for disaster recovery, not redundancy or failover Tuesday, January 12, 2010
- 4. > Storage solutions: distributed storage • write once, read anywhere • replication and fault tolerance • error correction • automatic redundancy • scalable horizontally Tuesday, January 12, 2010
- 5. > Distributed storage - Options • hosted storage (cloud, Amazon S3) • self hosted with proprietary hardware and software (Sun Thumper) • self hosted with commodity hardware and open source distributed filesystem (GlusterFS, Lustre) Tuesday, January 12, 2010
- 6. > Distributed storage - GlusterFS • GlusterFS: a cluster file-system capable of scaling to several petabytes* • open source software on commodity hardware • simple to install and manage • very customizable • offers seamless expansion *(1,024 terabytes = 1 petabyte) Tuesday, January 12, 2010
- 7. > Distributed storage - Archival • Fedora-commons is an open source repository (not a Linux distribution) • accounts for all changes, so it provides built-in version control • provides disaster recover • open standards to mesh with future file formats • provides open sharing services such as OAI-PMH Tuesday, January 12, 2010
- 8. Tuesday, January 12, 2010
- 9. > Distributed storage - Mirrored data • now we have redundancy • in fact, multiple redundant copies • provides fault tolerance • offers load balancing • gives us future geographical distribution Tuesday, January 12, 2010
- 10. > Distributed processing • more abilities available than just storing data • with distributed storage comes distributed processing • distributed processing means faster answers • faster answers mean new questions • lather, rinse, repeat Tuesday, January 12, 2010
- 11. > Distributed processing • make existing data more useful • image and OCR re-processing • distributed web services • identifier resolution pools • map/reduce frameworks • generate new visualizations, text mining, NLP and ??? Tuesday, January 12, 2010
- 12. Finder TaxonFinder TaxonFinder TaxonFinder TaxonFinder Taxon WebService WebService Load Balancer Cluster Node Cluster Site Request Tuesday, January 12, 2010
- 13. > Hardware configuration: risks • computers crash • hard drives die • networks fail • natural disasters occur but... Tuesday, January 12, 2010
- 14. ...so plan for it. Tuesday, January 12, 2010
- 15. > Hardware configurations • six 5U sized servers hosted at MBL in Woods Hole • 8 Gigs RAM in each server • 24 / 1.5 TB drives in each server • 100 TB storage overall • fast connection, far greater bandwidth than ever before • mirror sets (3 and 3) will be stored in different locations but... Tuesday, January 12, 2010
- 16. > Some assembly required (optional) • while our example uses new, faster commodity hardware... • it could run on any hardware that can run Linux • you could chain together old "out dated" computers • build your own cluster for next to nothing (host it in your basement) • solves some infrastructure funding issues and provides a working proof of concept • hardware vendor neutrality Tuesday, January 12, 2010
- 17. > Our proof of concept • we ran a six box cluster to demonstrate GlusterFS clustered filesystem • ran Debian/GNU Linux • simulated hardware failures • synced data with a remote cluster (STL to MBL/WH) • ran map/reduce jobs (distributed computing) • defined procedures, configurations and build scripts Tuesday, January 12, 2010
- 18. > Distributed storage - Projected costs $246,000 Graph from Backblaze (http://www.backblaze.com) Tuesday, January 12, 2010
- 19. > Cloud computing • BHL is participating in a pilot for Duraspace with The New York Public Library • Duraspace would provide a link to cloud providers • pilot to show feasibility of hosting • testing use of image server, other services in the cloud • cloud could seed new clusters Tuesday, January 12, 2010
- 20. > Code (63 6f 64 65) • our code and configurations are open source, hosted on Google Code http://code.google.com/p/bhl-bits/ phil.cryer@gmail.com | My favorites | Profile | Sign out bhl-bits Search projects Open Source code from the various Biodiversity Heritage Library (BHL) projects Project Home Downloads Wiki Issues Source Administer Summary | Updates | People This is the repository for code from various Starred (view starred projects) Biodiversity Heritage Library (BHL) projects, all of which are released under various Open Source licenses. Our current portal website runs on .NET, Code license: New BSD License connects to a MSSQL backend database, and utilizes an Apache Tomcat webapp fronted image Labels: bhl, linux, djatoka, imageviewer, viewer. Our new citation web site runs under Linux java, asp.net, c-sharp (Debian), Drupal, Apache, PHP and MySQL with help from HAProxy, Solr/Lucene, OpenVZ, KVM and other Links: Biodiversity Heritage Library software. We'll be releasing code, as well as the all important configuration files for applications we use Blogs: BHL Blog and support on our servers. We're open to any and BHL updates on Twitter all suggestions from the community, and of course accept any patches that improve our code. We Feeds: Project feeds appreciate your interest in our projects. Latest updates People details Project owners: Portal - portal is the .NET based code that Tuesday, January 12, 2010 our current portal site, Biodiversity phil.cryer, mlichtenberg@hotmail.com, powers
- 21. > Communication • our Projects server is a portal for various projects and ideas http://projects.biodiversitylibrary.org/ Home Ten major natural history museum libraries, botanical libraries, and Latest projects research institutions have joined to form the Biodiversity Heritage Library Project. The group is developing a strategy and operational Data Hosting Centre Task Group (12/02/2009 02:24 PM) plan to digitize the published literature of biodiversity held in their Whilst an unprecedented volume of biodiversity data is respective collections. This literature will be available through a global “biodiversity commons.” currently being generated worldwide, it is perceived that significant amounts of data get lost or will be lost after The participating libraries have over two million volumes of project closure. To investigate the causes of such loss, and biodiversity literature collected over 200 years to support the work of recommend strategies as to how biodiversity data can be scientists, researchers, and students in their home institutions and throughout the world. rescued and archived, through ‘data hosting centres’ a Task Group on ‘Data Hosting Centres’ will be commissioned. The BHL will provide basic, important content for Consistant, updated Identity (09/21/2009 03:50 AM) immediate research and for An ongoing project to update BHL's identity, theme and logo multiple bioinformatics across BHL Portal and Citebank initiatives. For the first time in history, the core of our Clustered storage (08/25/2009 05:28 AM) natural history and herbaria library collections will be The project to build a reproducible clustered file-system available to a truly global distributed across multiple servers, utilizing GlusterFS, to audience. Web-based house all of BHL Global content. access to these collections will provide a substantial benefit to people living and working in the developing world -- whether scientists or Website feedback (06/15/2009 11:11 AM) policymakers. The Biodiversity Heritage Library Project strives to A project to accept feedback from users to cite.bhl.org establish a major corpus of digitized publications on the Web drawn from the historical biodiversity literature. This material will be MOBOT djatoka implementation (04/13/2009 04:30 PM) available for open access and responsible use as a part of a global Biodiversity Commons. We will work with the global taxonomic community, rights holders, and other interested parties to ensure that this legacy literature is available to all. The Biodiversity Heritage Library Project must be a multi-institutional project because no single natural history museum or botanical garden library holds the complete corpus of legacy literature, even within the individual sub-domains of taxonomy. However, taken together, the proposed consortium of collections Tuesday, January 12, 2010 represents a uniquely comprehensive assemblage of literature.
- 22. > Communication • each project has its own wiki, allowing for detailed, how-to instructions http://projects.biodiversitylibrary.org/wiki/clusteredfilesystem/ INSTALL the following is a continuously updated doc and howto for configuring a BHL cluster node on a current Debian GNU/Linux installation. A breakdown of times involved in the physical building of nodes is available at the build times page System configuration 1) upgrade system to Squeeze (Lenny uses kernel 2.26, which doesn't fully support ext4) and enable contrib and non-free repos Copy the next 4 lines, and paste them in as 1: echo "deb http://ftp.us.debian.org/debian/ squeeze main contrib non-free deb-src http://ftp.us.debian.org/debian/ squeeze main deb http://security.debian.org/ squeeze/updates main deb-src http://security.debian.org/ squeeze/updates main" > /etc/apt/sources.list and update the system apt-get update aptitude safe-upgrade and now is a good a time as any to make sure the timezone data is all setup correctly (mine wasn't) dpkg-reconfigure tzdata 2) reboot to use latest kernel (if needed) /sbin/shutdown -r now 3) install needed utiltities and applications apt-get install build-essential flex bison screen nginx vim-nox git-core subversion saidar Tuesday, January 12, 2010
- 23. > Code (63 6f 64 65) • our code and configurations are open source and hosted on Google Code • our projects server shares detailed instructions to get you get started • get involved • join our mailing-lists (bhl-tech, bhl-bits) • ask us questions • you can do it, and we will help Tuesday, January 12, 2010
- 24. code http://code.google.com/p/bhl-bits projects http://projects.biodiversitylibrary.org questions phil.cryer@mobot.org Tuesday, January 12, 2010