A Technical Overview of DuraCloud
Download as PPTX, PDF1 like2,098 views
The document provides a technical overview of Duracloud, an open-source, cloud-based service platform that integrates various storage systems and administrative interfaces. It describes the functionalities related to storage management, service deployment, and reporting tools, as well as the security measures in place. Additionally, it outlines both near-term and long-term roadmap features for enhancing the Duracloud services, including improvements to access control and multimedia processing.
A Technical Overview of DuraCloud
- 1. A Technical Overview Bill Branan DuraCloud Technical Lead
- 2. Agenda • Introduction • Storage • Services • Reporting • UI • Security • Tools • Roadmap
- 3. Introduction • DuraCloud is: – Hosted service • Runs on cloud-based compute systems • Connects to cloud-based storage systems • Provides a service platform – Open source software suite • Webinar Focus: – Components of software platform – Interacting with hosted DuraCloud
- 4. DuraCloud in Context User Data Center Data File System Repository DuraCloud Management Console • Account Management • Instance Management
- 5. DuraCloud Storage Administrative User Interface (DurAdmin) Storage Management Service Management Report Management (DuraStore) (DuraService) (DuraReport) REST API REST API REST API Service Container DuraCloud Instance
- 6. DuraCloud Storage User Data Center Amazon Storage Amazon S3 Adapter REST API Rackspace Rackspace Storage Storage Adapter Cloud Files Provider Storage Interface Mediation Azure Storage Microsoft Azure Adapter Storage SDSC Storage Storage Management Adapter SDSC Cloud (DuraStore) Storage
- 7. Storage REST Interface Space Actions Content Actions • Add Space • Add Content • Get/Set Space Properties • Get/Set Content Properties • Get Spaces List • Get Content • Get Space Content List • Copy Content • Get/Set Space Access • Delete Content • Delete Space Storage Provider Interface Other Actions • Get Stores • Get Tasks List • Perform Task
- 8. Storage Adapters Transform calls from DuraCloud interface into calls understood by a specific storage provider • Determine how DuraCloud “spaces” translate into top level provider containers (buckets) • Define how space and content properties are stored and retrieved • Manage eventual consistency concerns
- 9. DuraCloud Services Administrative User Interface (DurAdmin) Storage Management Service Management Report Management (DuraStore) (DuraService) (DuraReport) REST API REST API REST API Service Container DuraCloud Instance
- 10. DuraCloud Services Media Streamer Duplicate Duplicate on on Demand Change Bit Bit Bit Integrity Integrity Integrity Checker Checker Checker Tools Bulk Image Image Transformer Transformer Bulk Image Server … so how are they deployed?
- 11. DuraCloud Service Deployment Services Service Management Available? (DuraService) 1 REST API 2 Service List Service Manager Services In Plan? 4 Service Registry Service List 3 Service Bundle Part 1: Service Config Ask which services are available Service Plan
- 12. DuraCloud Service Deployment Deploy Service Management Service (DuraService) Service 1 Config REST API 2 Service Manager Part 2: Retrieve Deploy Service Service Bundle Deploy Service 4 Service Service Bundle Config Service Service Registry Bundle Service Container Service Bundle 3 Service Bundle Service Configurator Deployed Deployed Service Config Service Service Config Service Plan
- 13. Service REST Interface • Get Services • Deploy Service • Get (Deployed) Service • Get Deployed Service Properties • Update Service Configuration • UnDeploy Service
- 14. DuraCloud DuraCloud Instance Distributed Services Services Media Streamer Duplicate Duplicate on on Demand Change Bit Bit Bit Integrity Integrity Integrity Checker Checker Checker Tools Bulk Image Image Transformer Transformer Bulk Image Server • Runs primarily outside of • Runs on DuraCloud Instance DuraCloud Instance • Connects to DuraStore • Connects to DuraStore • Can be direct Java service • Makes use of cloud network • Can be deployed web app or computation features
- 15. DuraCloud Bulk Service Deployment Service Management 1 (DuraService) 2 REST API Task Management Service Manager (DuraStore) REST API Run Service Registry Service 4 Run Task 3 6 Hadoop 5 Bundle Job Service Container Config Hadoop Deployed Service Plan Service Config
- 16. DuraCloud Reporting Administrative User Interface (DurAdmin) Storage Management Service Management Report Management (DuraStore) (DuraService) (DuraReport) REST API REST API REST API Service Container DuraCloud Instance
- 17. DuraCloud Report Generation Report Management (DuraReport) Storage Service Report Report Builder Builder Deposit Reports Message Broker Storage Data REST API REST API Service Storage Management Event Service Management Messages (DuraStore) (DuraService)
- 18. Report REST Interface Storage Report Service Report • Start Storage Report • Get Deployed Services • Get Latest Storage Report Report • Get Storage Report List • Get Latest Completed Services Report • Get Storage Report • Get Completed Services • Get Storage Report Info Report List • Cancel Storage Report • Get Services Report • Schedule Storage Report • Cancel Storage Report Schedule
- 19. DuraCloud UI Administrative User Interface (DurAdmin) Storage Management Service Management Report Management (DuraStore) (DuraService) (DuraReport) REST API REST API REST API Service Container DuraCloud Instance
- 20. DuraCloud UI Interaction Administrative User Interface (DurAdmin) Store Client Service Client Report Client REST API REST API REST API Storage Management Service Management Report Management (DuraStore) (DuraService) (DuraReport)
- 21. DuraCloud Security DurAdmin Storage Provider DuraStore DuraService DuraReport Application Security Storage Provider Access Control Instance Firewall Transport Security User Data Center
- 22. DuraCloud Tools Transfer to DuraCloud Transfer from DuraCloud • Upload Tool • Retrieval Tool – GUI to select files and – Retrieve files from folders one, selected, or all – Visual transfer progress spaces • Sync Tool • Stitch Tool – Monitors for changes – Retrieve large files – Run one time or – Acts on a single file continually • Chunk Tool – Transfer large files – Acts on a single file
- 23. DuraCloud Tools User Data Center Store Client Store Client Chunk Tool Stitch Tool Sync Tool Retrieval Tool Upload Tool REST API Storage Management (DuraStore)
- 24. Near Term Road Map New features • Access control at the space level • Improved format identification • Automated bit integrity checking • Email notification on service completion • Multi-Account DuraCloud instances • Content indexing and Search – Improved discovery – Improved content browsing • Folder-based navigation • Improved handling of very large files • Multi-Item updates/deletes Integrations • SDSC storage provider (production version) • Fedora-in-the-cloud prototype • DSpace-in-the-cloud prototype
- 25. Long Term Road Map New Features • Video transcoding • Document format transformation • Improved image support services • Media streaming access controls • Metadata extraction services Integrations • Azure storage provider (production version) • Eucalyptus Walrus storage provider • Rackspace compute instances • Managed Fedora-in-the-cloud • Managed DSpace-in-the-cloud
- 26. Questions?
- 27. Upcoming DuraCloud Webinars DSpace and DuraCloud November 30, 2011 at 1pm ET Fedora and DuraCloud January 11, 2012 at 1pm ET
- 28. Thank You! Further Questions? info@duracloud.org