Data replication and synchronization tool
0 likes989 views
The document presents a data replication and synchronization tool developed for sharing subsets of hierarchical medical data, particularly focusing on the cancer imaging archive. It describes a system architecture utilizing an in-memory data grid (Infinispan) for storing replica sets and includes a publisher-consumer API for data communication. The solution emphasizes high performance, fault tolerance, and extensibility with integration options for various data sources and technologies.
Data replication and synchronization tool
- 1. Data Replication and Synchronization Tool Ashish Sharma Pradeeban Kathiravelu PPoowweerrppooiinntt TTeemmppllaatteess 1
- 2. Introduction • Data is huge • Consumers often share a sub set of data with others. – Pointers to data, actually. • Medical data is structured in hierarchies. Powerpoint Templates 2
- 3. Motivation • Creating and sharing pointers to interesting sub sets of data. • Data Sharing Synchronization System – Fault-tolerant. – In-Memory. • Generic, while targeting the medical images and meta data. – The Cancer Imaging Archive Powerpoint Templates 3 (TCIA)
- 4. Solution Architecture • Users create, share, and update replica sets from a data source. • Infinispan In-Memory Data Grid (version 6.0.2) to store the replica sets. Fig 1. Deployment Architecture Powerpoint Templates 4
- 5. Execution Flow • Publisher-Consumer API to consume the replica sets and Data Provider API to communicate with the data source. Powerpoint Fig 2. Execution Templates Flow 5
- 6. Design Fig 3. Back-end Class Hierarchy • DataProSpecs API createReplicaSet getReplicaSet updateReplicaSet duplicateReplicaSet deleteReplicaSet getRawData Powerpoint Templates 6
- 7. Extensibility • Not tightly coupled to the technology. – Other data-grids • Hazelcast, Terracotta Big Memory, Oracle Coherence – Persistence • Integration to SQL or NoSQL solutions such as Mongo DB. –Data sources other than TCIA. Powerpoint Templates 7
- 8. What Infinispan offers? • High Performance and Scalability. • Fault-tolerance – Multiple nodes with TCP-IP or Multicast based JGroups clustering configurations. • Distributed Execution. – Optimized for single node as a local cache as well as a multiple-node execution. • MapReduce Framework. Powerpoint Templates 8
- 9. What Infinispan offers? • High Performance and Scalable. • Fault-tolerant – Multiple nodes with TCP-IP or Multicast based JGroups clustering configurations. Thank you! • Distributed Execution. – Optimized for single node as a local cache as well as a multiple-node execution. • MapReduce Framework. Powerpoint Templates 9
- 10. What Infinispan offers? • High Performance and Scalable. • Fault-tolerant – Multiple nodes with TCP-IP or Multicast based JGroups clustering configurations. Thank you! • Distributed Execution. – Optimized for single node as a local cache as well as a multiple-node execution. • MapReduce Framework. Powerpoint Templates 10