Emc vi pr data services

Editor's Notes

• #5: The Advanced Software Division (ASD) was formed in January 2012 as a result of the merger of the Cloud infrastructure Group (CIG) with the Infrastructure Management Group (IMG). This brought together EMC’s object and cloud storage platforms with all of EMC’s management software. ASD now develops cloud storage and management software for private and public cloud and optimizing storage management and automation. Amitabh Srivastava is the President of the Advanced Software Division at EMC. Before joining EMC in April 2011, Srivastava was the Senior Vice President of Server and Cloud Division at Microsoft Corporation, where he was responsible for Windows Azure and Windows Server products.
• #7: The intelligent economy produces a constant stream of data that is being monitored and analyzed. IDC estimates that the digital universe will be 40ZB by 2020. That’s a 40 followed by 21 zeroes. Social interactions, mobile devices, facilities, equipment, R&D, simulations, and physical infrastructure all contribute to the flow of information. In aggregate, this is what is called Big Data. The Big Data economy, is characterized by: More Sources of data Communities Mobile Devices Sensors Imaging Equipment Richer Content Pictures Videos Data Streams Longer utility Durable value – information and information about information (metadata) has value for a long time after its creation. All this data can have business value. Regulatory burdens – always a contributor to the need to retain data for longer and longer periods of time, often indefinitely.
• #8: Traditional applications such a customer relationship management (CRM) and enterprise resource planning (ERP) applications still make up the majority of enterprise workflows and they’re growing 70% per year. But next-gen Web, mobile and cloud applications are growing 700% per year. Even more importantly, these workloads generate tremendous amounts of unstructured data, with expected growth of more than 100x over the next decade.
• #9: Based on the requirements on the previous slide, enterprise IT storage infrastructure needs to adapt. The IT architecture remains silo’ed and many of its processes are manual. The combination results in massive complexity, which makes it difficult to Scale efficiently – Having separate storage siloes results in a lot of unused capacity and higher fixed costs. Management complexity – IT has multiple storage management control points and higher operational costs. Development risk – Legacy infrastructure is simply not built to support next-gen, Web and mobile applications. Development projects take longer, require more code and rely on tight hooks to infrastructure. This limits IT agility and IT’s ability to compete with public clouds
• #10: The era of Big Data places new demands on data storage. Storage must contend with varying data types, all of which need to be stored securely for a long periods of time and be available for analysis. Data Unification: There is an increasing focus on data unification meaning that the storage infrastructure for Big Data has to cater to structured, semi-structured, and unstructured data types. In-Place Analytics: There is a growing emphasis on in-place analytics in which the compute workloads such as Hadoop Map/Reduce operations are run right where the data lives. Data Compliance: This market is fraught with challenges stemming from regulatory and compliance requirements. As the platform that hosts data the instant it is created, storage is not immune to these challenges — and how data gets stored in the long term.
• #11: The premise of “software-defined” is to reduce the tight coupling of software and hardware. It’s simply faster and easier to innovate through software without tight hooks into hardware. This also gives customers choice of persistence layers. It also brings new features to existing hardware, similarly to how virtualization let a Windows serve support multiple applications – even apps not written to windows. It also increases the value of the software itself since it is now available to more platforms. An ISV can write an app once and target multiple platforms, expanding their addressable market.
• #13: SDS characteristics: It’s software – many competing solutions feature a physical appliance or software deployed on a switch. We call this “hardware-defined software-defined storage”. SDS should be software without requiring a truck, 2 wheeler or physical installation. Decouples control plane from data plane – this is the essence of software-defined. Separating the control and data paths means you can centrally manage heterogeneous physical devices, centralize all data provisioning and data management tasks, and leverage storage data services but not necessitate overhead on the data path. You can continue to use the unique data services embedded in the storage arrays. Support for multiple storage platforms – SDS should support file, block, object from multiple vendors Programmable API’s – SDS should be API driven. This makes storage a programmatic resource of the SDDC SDS must feature: Automation – the value is in reducing the # of management control points and automating repetitive manual tasks Extensibility – SDS should enable anyone to extend their storage environment support additional arrays and create new data services that run on top. Openness – That doesn’t mean “open source”, it means it can’t be proprietary. The platform should feature open, easily accessible APIs. You can’t be extensible if you’re not open. At least not effectively.
• #14: The second data service that we will provide in 2013 is HDFS. HDFS is becoming increasingly popular as a file system layer for distributed applications, beyond Hadoop Scenarios: High aggregate throughput access to data, e.g. MapReduce. In some cases, low latency access. Concerns: Scale, durability, cost, management
• #15: ViPR SDS abstracts the data services – the storage engine and APIs - from physical resources and creates a unified storage platform that has the following characteristics: Heterogeneous The storage engine that can target a variety of storage devices Unified Cloud data types can be layered over traditional storage Flexible Data services are implemented in software, hardware agnostic Scale-out Data services leverage node-based, scale-out architecture for linear scale, efficiency Extensible Partners and customers can extend and customize the platform and develop new data services Enterprise-grade Leverage and extend the capabilities of underlying storage devices – for example, you can store objects on a file-based arrays and make use of snapshots for enterprise-level data protection.
• #16: ViPR aggregates multi-vendor heterogeneous storage into a unified storage platform, that, in turn, can be leveraged as a logical scale-out layer which can serve as the underlying infrastructure for hosting a range of data services to support collecting, managing and utilizing unstructured content at massive scale. ViPR Data Services are implemented in software and feature a simple, lightweight, low-touch, scale-out design. Data services are storage abstractions that reflect the combination of a data type (file, object or block of data), access protocols (iSCSI, NFS, REST, etc.), and durability, availability, and security characteristics (snapshots, replication, etc.) In ViPR, block, file, object, and HDFS are all data services, though ViPR is not in the data path for file and block (these can be thought of us “control services”). Object and HDFS are available in 2013 with more to follow. Data services can be used to provide different semantic views of the same data. You can manipulate a file as a file or as an object without having to move the data to a different platform that features that semantic.
• #17: This depicts the architecture for ViPR and highlights the data services functionality. At the bottom are the physical arrays that ViPR can manage. Above the arrays is the ViPR controller which has features that enable a distributed infrastructure (Cassandra, a distributed DB and Zookeeper to manage status of different nodes in the system) and device drivers to hook into APIs of arrays so the Controller can automate provisioning, management, etc. The ViPR Controller can also offer lifecycle services such as elastic volumes. ViPR can use the control path functionality to spin up data services. A good example are elastic volumes – similar to Amazon EBS. ViPR can provision volumes at scale. On top of that are ViPR data services. The real differentiation and IP we’ve built: Geo-scale index – performant distributed indexing mechanism to find data very quickly. Geo-scale metadata – store metadata associated with customer data separately from data. It’s stored in an efficient way so we can efficiently run operations against the metadata. Transactions – executing a large # of user transactions concurrently with very little latency Key bottleneck for object stores are the metadata database and the index. They’re bottlenecks since they are usually stored in a database and databases can’t scale. ViPR does not store metadata or the index in a separate database. The geo-scale index and geo-scale metadata are stored on the actual arrays. The improves performance, particularly at large, distributed scale. ViPR data services provides primitives like read/write. ViPR provides a RESTful API and EMC has also build different semantic heads on top of that API. For example, the Object data service will support the Atmos, OpenStack Swift and Amazon S3 APIs. We’ll also add, in the future, file heads, Key Value (tables) and expose the ViPR API to allow others to build new data services.
• #18: ViPR and ViPR data services address the new requirements for Big Data mentioned previously. Data Unification: ViPR abstracts storage into a software layer and manages with intelligent, policy-driven software. ViPR data services are software that span arrays and support different semantic views of the same data. ViPR creates a unified storage platform that includes block, file, object, HDFS and more data services. In-Place Analytics: Again, data services provide a way to manipulate data by different mechanisms in place. Using the HDFS data service, organizations can bring analytics to the data rather then bringing the data to a separate analytics infrastructure. Data Compliance: ViPR creates a unified platform that combines the features of array-level data protection with policy-driven data management to simplify tiering and compliance.
• #20: Object storage EMC pioneered object storage over a decade ago with the release of EMC Centera, which today remains the leader in compliance archiving. At this early stage, object storage featured a single namespace, a proprietary API and scaled more efficiently that file-based archive solutions. CAS was valuable for storing unstructured content and guaranteeing the authenticity and immutability of content. EMC continued its innovation in object storage with the introduction of EMC Atmos which is an industry leader for cloud storage and next-generation Web, mobile and REST-based applications. Atmos built on the strengths of Centera – linear scale, efficiency, API access - by adding support for a RESTful API, multi-tenancy and adding geo-replication and distribution of content. It also featured a single namespace and multi-tenancy. EMC raised the bar again for object storage with the coming availability of EMC ViPR Object Data Services that extend object capabilities to heterogeneous file-based storage. The strategy foe EMC moving forward is to provide universal API access, multiple access methods and support analytics on unstructured data. It’s more than object storage, it’s providing cloud storage that can offer universal access to object, file, block and HDFS data services on highly-efficient scale-out geo-distributed storage that can support unstructured and semi-structured data. And, by being truly software-defined, gives customers hardware options - supporting scale-out appliances, commodity, and legacy infrastructure.
• #21: Object storage is not one-size-fits all. EMC is investing in purpose-built hardware/software platforms and software-defined solutions that will give customers the freedom to choose the object storage platform that meets their unique requirements. Today, customers can choose a dedicated object platform such as Centera and Atmos, or begin right away with the software-defined approach. If customers are purely focused on object storage, a purpose-built appliance gives them a single system to buy and scale and they don’t have to buy a compute and network infrastructure separately. It’s a homogenous object platform that’s more plug and play. If customers have more general needs beyond object, the software-defined approach of ViPR is the choice. Today, ViPR can support storing objects on file-based arrays such as Isilon, VNX and NetApp and support the Atmos API as well as OpenStack Swift and Amazon S3. Customers would choose the software-defined approach today if they want to use their existing file-based storage to support objects and RESTful APIs , Atmos, OpenStack and amazon S3. The software-defined approach also appeals to customers that see value in applying multiple data services across their arrays. In 2014, ViPR will add support for the CAS API and Centera hardware. Later in the future, ViPR will also support Atmos hardware. Customers can invest with confidence in Centera and Atmos, knowing there is a path to ViPR and a co-existence strategy that leverages their investments and bring new capabilities to existing Centera and Atmos deployments.
• #23: The ViPR object data services is very similar to the Amazon S3 model but ViPR adds extensions such as byte-range updates, atomic append, rich ACLs, etc. On top of the Object data service, ViPR includes several API heads, Amazon S3, OpenStack Swift and our own EMC Atmos. ViPR will add support for the CAS API in the first half of 2014. A key capability of the Object Data Service is its ability to store objects on file based storage. Now, an enterprise or service provider can use their existing file-based infrastructure to store and manipulate objects as files or objects on existing file-based storage.
• #24: The ViPR object data service leverages the same configuration and provisioning process provided by the ViPR Controller. A storage administrator defines Virtual Storage Arrays and configures Virtual Storage Pools. The administrator configures the object Virtual Storage Pool with certain characteristics and defines how the actual objects are stored on the underlying arrays.
• #25: The object data services supports physical segregation of the objects. An object virtual storage pool can write to multiple, physically separate arrays. In this example, one pool is writing to open virtual storage arrays that is writing to multiple commodity arrays. The other object virtual storage pool is writing to a different virtual array that is writing to three distinct physical file arrays.
• #26: In addition to physical segregation, buckets provide logical segregation within the object store. Just like in S3, a user can create buckets which logically segregate applications or sets of data. These buckets can grown and shrink on-demand. The actual data objects are distributed and intermingled across the physical devices that comprise the virtual storage array.
• #27: The actual processing of objects is done by a virtual set of data nodes. The data nodes are simply VMs the process object requests and feature a scale-out architecture to facilitate full failover between nodes. Being VMs, they’re easy to deploy.
• #28: A very unique aspect of the ViPR object data service is that it stores objects in “chunks” or containers. Containers are append-only, once a container is full, it becomes immutable. Another important differentiator that was mentioned previously is the distributed index to track data locations. Rather than maintaining a separate index in a database, which can’t scale well, the index is located on the actual arrays in containers. In this way, the index can scale massively and be geo-distributed. A key bottleneck of existing object storage platforms is the indexing and metadata databases. By storing the index and metadata on the actual arrays, the ViPR object data service can locate objects in a massive scale, geo-distributed object store much faster. The way an write works, is the that write first updates the container, then updates the index. Only after the index is successfully updated will the write be successful. One thing to note about container immutability. Over time, authorized deletes may result in a container which has very few indexed objects in it. There is a garbage collection process that will identify a container with a large % of objects that are no longer in the index. In this case, the ViPR object data service can create a new append-only container, move the objects and update the index. This ensures efficient storage.
• #29: For the first time, organizations, via the ViPR object data service, extend object storage to existing file based storage such as EMC VNX and EMC Isilon and third party file-based storage such as NetApp FAS. A bucket of objects can be dynamically toggled between three different access modes: Object mode: Full Object API. No File access. File mode: Native Read/Write access to Files, directly on the underlying device, via NFS mount. No Object access. Dual mode: Read access as either Object or File. Enterprises can deploy REST-based applications (based on S3, Atmos or OpenStack Swift) on their existing file-based storage. This fits a couple different use cases: This gives them the ability to deploy a private cloud with existing file-based capacity instead of buying a dedicated object storage platform. It allows them to protect objects with existing NAS data protection capabilities such as snapshots, replication. Perhaps there are compliance concerns or a backup process they want to extend to object data. Support mixed-use workflows. They may have a need to manipulate the same data with a REST-based app and a file-based app. Example on the next slide.
• #30: In this example, a next-gen video sharing Web application ingests objects into a higher latency object store, then exposes the uploaded video content. Before exposing that content, however, a video processing application that is written to a lower latency file storage platform such as an Isilon, needs to perform analytics, curation, watermarking, etc. before posting back to the Web. Consequently the organization must copy all the uploaded object data into a file-based storage, perform the curation, then copy back to the object store. This is a long, complex process. With ViPR Object, now the objects can be ingested via REST (S3, Atmos or OpenStack Swift) directly to the Isilon (or VNX or other filer). An administrator can simply toggle to file mode, mount NFS, perform the curation and save the files. Toggle back to object mode and expose the curated objects to the Web. The data never has to be moved or copied and only one storage platform is required.
• #33: Product information covers confidential information Contains future product information Can only be presented under NDA
• #34: Software-defined storage generally, and ViPR data services specifically enables EMC to innovate horizontally across various storage technologies. EMC can introduce new capabilities in software that work across heterogeneous storage. And customers will have the choice of using a vendor-provided storage array or their own storage on their own terms. The array model will continue and customers will continue to buy Atmos and Centera arrays, however, the software be ViPR Software (Controller and Data Services). In building out the roadmap for object storage and ViPR, the mission is to provide choice and flexibility for our customers. EMC understands that, in addition to the software-defined approach of ViPR, there will still be customers that will want to buy an purpose-built appliance, particular Atmos customers, although many Centera customers also will want a purpose-built object appliance. For those Atmos and Centera customers that want to stay with a purpose-built appliance, they can simply stay with their existing investment. In the future, ViPR will add capabilities such as geo-replication, geo-distribution, compliance, etc. an Atmos customer will simply be able to refresh their Atmos investment. The next tech refresh of Atmos (v.Next or “Atmos 3.0”) will feature ViPR Controller and ViPR Data Services (specifically the ViPR Object data service) on Dense 480 hardware. In addition to all the functionality that an Atmos customer uses and expects, they can also expect access to new data services such as HDFS and to take advantage of the new indexing and metadata that will add speed and efficiency. Other customers may choose the purely software-defined approach. This will give them the ability to choose different hardware platforms as well as low-cost commodity. This is for a customer that needs to go beyond homogeneous object and support multiple data types across multiple arrays.
• #35: The following slides provide a high-level preview of roadmap for the Object data service. The four core capabilities in addition to adding support for Centera arrays and the Centera CAS API are: Geo Replication Geo Distribution Compliance Commodity hardware support
• #36: ViPR will support multi-site in the next major release and the ability to replicate objects across multiple sites. More details on replication mechanisms (mirroring, distributed erasure coding) will be available in 2014.
• #37: With multi-site, also comes the ability to distribute and replicate content and the index and metadata across the sites and access everything through a single namespace. This will also deliver the active-active architecture that Atmos customers use.
• #38: Atmos recently added support for SEC compliance. And, in order for ViPR Object to support Centera, it will need compliance features. ViPR Object will add those capabilities at object or bucket level.
• #39: ViPR will add support for storing objects on commodity hardware. Today, ViPR Object and HDFS data services assume an underlying shared file system. Commodity drives like JBODs are block-only so they do not provide this file system. So EMC is creating its own distributed file persistence layer to allow the ViPR software stack to sit directly on top of JBOD. In the future, ViPR will add support for HDFS and Key-Value and native file on commodity.
• #40: Today, there are many object platforms from multiple vendors. The vision for ViPR is to represent platform semantics as bucket attributes. The uniqueness of a Centera or an Atmos platform can be represented in buckets. Buckets can also be used to capture specific application functionality. This is the benefit of designing things in software. The freedom from hardware provides a great deal of flexibility.
• #41: To make commodity support possible, ViPR supported protected writes. Configurable by an administrator, a write will update multiple containers that are stored on different nodes and drives, then updates the index. The index will only update after all the writes have succeeded. The writes are not confirmed/written until the index updates succeeds. This is certainly overkill for existing object platforms or NAS systems (since they have a file system) but is a requirement for supporting commodity disks.
• #42: As mentioned earlier, ViPR efficiently stores objects in append-only containers. This also ensures efficient utilization on commodity. As stated previously, this can be overkill for an existing object or NAS filer that has its own persistent file system, but is important for efficient storage on JBOD. To repeat, only open containers can accept objects. One a container is full, it becomes immutable. One thing to note about container immutability. Over time, authorized deletes may result in a container which has very few indexed objects in it. There is a garbage collection process, a bulk operation that will identify a container with a large % of objects that are no longer in the index. In this case, the ViPR object data service can create a new append-only container, move the objects and update the index. This ensures efficient storage.
• #43: ViPR presents a common index for unified object storage. Regardless of whether the logical bucket represents an Atmos, a Centera, S3 application, etc., ViPR has a unified index extended to all object types. This will enable more flexible metadata search and treats object storage as a first-class citizen. It also makes the different object types available for analytics (eg HDFS on Object).
• #44: Centera: Primary Use Case: On premise compliance archive solution. Meets stringent compliance requirements of various verticals, Six 9’s reliability. Deploy and forget. Best of breed reliability, performance, and compliance. Roadmap: Hardware refreshes. Increased scale. Atmos: Primary Use Case: Active cloud storage infrastructure for next generation geo distributed applications. Best of breed geo replication and efficiency. Roadmap: Hardware refreshes. Compliance. Operational efficiency. ViPR Object Service: Primary: State of art hyper-scale object service on all arrays. Unifies object layers over Centera, Atmos and File. Enables mix and match Roadmap: Geo, Compliance, HDFS and other Data Service