Software-Defined Data Services: Interoperable and Network-Aware Big Data Executions (Best Paper Award: SDS-2018)
1 like789 views
The document discusses software-defined data services (SDDS), which aim to enhance interoperability and scalability in big data executions by leveraging software-defined networking (SDN). It proposes a framework that optimizes the placement of data and services to minimize communication overhead and improve performance. The SDDS is shown to outperform traditional data services in terms of data locality, resource efficiency, and throughput.
Software-Defined Data Services: Interoperable and Network-Aware Big Data Executions (Best Paper Award: SDS-2018)
- 1. Software-Defined Data Services: Interoperable and Network-Aware Big Data Executions Pradeeban Kathiravelu, Peter Van Roy, Luís Veiga 5th IEEE International Conference on Software Defined Systems (SDS 2018). Barcelona, Spain. 24/04/2018.
- 2. Introduction ● Big data with increasing volume and variety. – Volume requires scalability. – Variety requires interoperability. ● Data Services – Services that access and process big data. – Unified web service interface to data → Interoperability! ● Chaining of data services. – Composing chains of numerous data services. – Data Access → Data cleaning → Data Integration.
- 3. Problem Statement ● Data services offer interoperability. ● But when related data and services are distributed far from each other → Bad performance with scale. – How to scale out efficiently? ● How to minimize communication overheads?
- 4. Software-Defined Data Services (SDDS)Software-Defined Data Services (SDDS) 4/20 Motivation ● Software-Defined Networking (SDN). – A unified controller to the data plane devices. – Brings network awareness to the applications. ● To make big data executions – Interoperable. – Network-aware.
- 5. Software-Defined Data Services (SDDS)Software-Defined Data Services (SDDS) 5/20 Our Proposal ● Can we bring SDN to the data services? ● Software-Defined Data Services (SDDS).
- 6. Software-Defined Data Services (SDDS)Software-Defined Data Services (SDDS) 6/20 Contributions ● SDDS as a generic approach for data services. – Extending and leveraging SDN in the data centers. ● A software-defined framework for data services. – Efficient performance and management of data services. – Interoperability and scalability.
- 7. Software-Defined Data Services (SDDS)Software-Defined Data Services (SDDS) 7/20 Solution Architecture ● A bottom-up approach, extending SDN. – Data Plane (SDN OpenFlow Switches) – Storage PlaneStorage Plane (SQL and NoSQL data stores) – Control Plane (SDN Controller, In-Memory Data Grids (IMDGs), ..) – Execution Plane (Orchestrator and Web Service Engines)Execution Plane (Orchestrator and Web Service Engines)
- 8. Software-Defined Data Services (SDDS)Software-Defined Data Services (SDDS) 8/20 Network-Aware Service Executions with SDN
- 9. Software-Defined Data Services (SDDS)Software-Defined Data Services (SDDS) 9/20 SDDS Planes and Layered Architecture
- 10. Software-Defined Data Services (SDDS)Software-Defined Data Services (SDDS) 10/20 SDDS Approach ● Define all the data operations as interoperable services. ● SDN for distributing data and service executions – Inside a data center (e.g. Software-Defined Data Centers). – Beyond data centers (extend SDN with Message-Oriented Middleware). ● Optimal placement of data and service execution. – Minimize communication overhead and data movements. ● Keep the related data and executions closer. ● Send the execution to data, rather than data to execution. – Execute data service on the best-fit server, until interrupted.
- 11. Software-Defined Data Services (SDDS)Software-Defined Data Services (SDDS) 11/20 Efficient Data and Execution Placement
- 12. Software-Defined Data Services (SDDS)Software-Defined Data Services (SDDS) 12/20 Efficient Data and Execution Placement {i, j} – related data objects D – datasets of interest n – execution node Σ – spread of the related data objects
- 13. Software-Defined Data Services (SDDS)Software-Defined Data Services (SDDS) 13/20 Prototype Implementation ● Data services implemented with web service engines. – Apache Axis2 1.7.0 and Apache CXF 3.2.1. ● IMDG clusters – Hazelcast 3.9.2 and Infinispan 9.1.5. ● Persistent storage – MySQL Server and MongoDB. ● Core SDN Controller – OpenDaylight Beryllium.
- 14. Software-Defined Data Services (SDDS)Software-Defined Data Services (SDDS) 14/20 Evaluation Environment ● A cluster of 6 servers. – AMD A10-8700P Radeon R6, 10 Compute Cores 4C+6G × 4. – 8 GB of memory. – Ubuntu 16.04 LTS 64 bit operating system. – 1 TB disk space.
- 15. Software-Defined Data Services (SDDS)Software-Defined Data Services (SDDS) 15/20 Evaluation ● How does SDDS comply as a network-aware big data execution compared to network-agnostic execution? – SDDS vs data services on top of Infinispan IMDG. – A data storage and update service ● with an increasing volume of persistent data across the cluster ● up to a total of 6 TB data. ● Measured the throughput from the service plane – by the total amount of data processed through the data services per unit time.
- 16. Software-Defined Data Services (SDDS)Software-Defined Data Services (SDDS) 16/20 Evaluation ● SDDS outperforms the base. – Better data locality ● by distributing data adhering to network topology. – Better resource efficiency. ● by avoiding scaling out prematurely. – Better throughput with minimal distribution when there is no need to utilize all the 6 servers.
- 17. Software-Defined Data Services (SDDS)Software-Defined Data Services (SDDS) 17/20 Related Work ● Software-Defined Systems. – Software-Defined Service Composition. – Software-Defined Cyber-Physical Systems and SDIoT. ● Industrial SDDS offerings. – Many of them storage focused. ● PureStorage, PrimaryIO, HPE, RedHat, .. – Many focus on specific data services. ● Containers and devops – Atlantix and Portworx. ● Data copying and sharing – IBM Spectrum Copy Data Management and Catalogic ECX. ● We are the first to propose a generic SDDS framework.
- 18. Software-Defined Data Services (SDDS)Software-Defined Data Services (SDDS) 18/20 Conclusion Summary ● Software-Defined Data Services (SDDS) offer both interoperability and scalability to big data executions. ● SDDS leverages SDN in building a software-defined framework for network-aware executions. ● SDDS caters to data services and compositions of data services for an efficient execution.
- 19. Software-Defined Data Services (SDDS)Software-Defined Data Services (SDDS) 19/20 Conclusion Summary ● Software-Defined Data Services (SDDS) offer both interoperability and scalability to big data executions. ● SDDS leverages SDN in building a software-defined framework for network-aware executions. ● SDDS caters to data services and compositions of data services for an efficient execution. Future Work ● Extend SDDS for edge and IoT/CPS environments.
- 20. Software-Defined Data Services (SDDS)Software-Defined Data Services (SDDS) 20/20 Conclusion Summary ● Software-Defined Data Services (SDDS) offer both interoperability and scalability to big data executions. ● SDDS leverages SDN in building a software-defined framework for network-aware executions. ● SDDS caters to data services and compositions of data services for an efficient execution. Future Work ● Extend SDDS for edge and IoT/CPS environments. Thank you! Questions?