A scalable and reliable matching service for content based
- 1. A SCALABLE AND RELIABLE MATCHING SERVICE FOR CONTENT- BASED PUBLISH/SUBSCRIBE SYSTEMS
- 2. ABSTRACT: Characterized by the increasing arrival rate of live content, the emergency applications pose a great challenge: how to disseminate large-scale live content to interested users in a scalable and reliable manner. The publish/subscribe (pub/sub) model is widely used for data dissemination because of its capacity of seamlessly expanding the system to massive size. However, most event matching services of existing pub/sub systems either lead to low matching throughput when matching a large number of skewed subscriptions, or interrupt dissemination when a large number of servers fail. The cloud computing provides great opportunities for the requirements of complex computing and reliable communication. In this paper, we propose SREM, a scalable and reliable event matching service for content-based pub/sub systems in cloud computing environment. To achieve low routing latency and reliable links among servers, we propose a distributed overlay SkipCloud to organize servers of SREM. Through a hybrid space partitioning technique HPartition, large-scale skewed subscriptions are mapped into multiple subspaces, which ensures high matching throughput and provides multiple candidate servers for each event. Moreover, a series of dynamics maintenance mechanisms are extensively studied. To evaluate the performance of SREM, 64 servers are deployed and millions of live content items are tested in a CloudStack testbed. Under various parameter settings, the experimental results demonstrate that the traffic overhead of routing events in SkipCloud is at least 60 percent smaller than in Chord overlay, the matching rate in SREM is at least 3.7 times and at most 40.4 times larger than the single- dimensional partitioning technique of BlueDove. Besides, SREM enables the event loss rate to drop back to 0 in tens of seconds even if a large number of servers fail simultaneously.
- 3. EXISTING SYSTEM: In traditional data dissemination applications, the live content are generated by publishers at a low speed, which makes many pub/subs adopt the multi-hop routing techniques to disseminate events. A large body of broker-based pub/subs forward events and subscriptions through organizing nodes into diverse distributed overlays, such as tree based design, cluster- based design and DHT-based design.
- 4. DISADVANTAGES OF EXISTING SYSTEM: The system cannot scalable to support the large amount of live content. The Multihop routing techniques in these broker-based systems lead to a low matching throughput, which is inadequate to apply to current high arrival rate of live content. Most of them are inappropriate to the matching of live content with high data dimensionality due to the limitation of their subscription space partitioning techniques, which bring either low matching throughput or high memory overhead.
- 5. PROPOSED SYSTEM: Specifically, we mainly focus on two problems: one is how to organize servers in the cloud computing environment to achieve scalable and reliable routing. The other is how to manage subscriptions and events to achieve parallel matching among these servers. We propose a distributed overlay protocol, called SkipCloud, to organize servers in the cloud computing environment. SkipCloud enables subscriptions and events to be forwarded among brokers in a scalable and reliable manner. Also it is easy to implement and maintain. To achieve scalable and reliable event matching among multiple servers, we propose a hybrid multidimensional space partitioning technique, called HPartition. It allows similar subscriptions to be divided into the same server and provides multiple candidate matching servers for each event. Moreover, it adaptively alleviates hot spots and keeps workload balance among all servers.
- 6. ADVANTAGES OF PROPOSED SYSTEM: We propose a scalable and reliable matching service for content- based pub/sub service in cloud computing environments, called SREM. We propose a hybrid multidimensional space partitioning technique, called HPartition SSPartition. To alleviate the hot spots whose subscriptions fall into a narrow space, we propose a subscription set partitioning, Through a hybrid multi-dimensional space partitioning technique, SREM reaches scalable and balanced clustering of high dimensional skewed subscriptions.
- 7. PROBLEM STATEMENT & SCOPE PROBLEM STATEMENT : The proposed event matching service can efficiently filter out irrelevant users from big data volume, there are still a number of problems we need to solve. Firstly, we do not provide elastic resource provisioning strategies in this paper to obtain a good performance price ratio. SCOPE: Scope is to design and implement the elastic strategies of adjusting the scale of servers based on the churn workloads. Secondly, it does not guarantee that the brokers disseminate large live content with various data sizes to the corresponding subscribers in a real-time manner. For the dissemination of bulk content, the upload capacity becomes the main bottleneck. Based on our proposed event matching service, we will consider utilizing a cloud- assisted technique to realize a general and scalable data dissemination service over live content with various data sizes.
- 8. MODULES Number of Modules After careful analysis the system has been identified to have the following modules: 1. Scalable And Reliable Event Matching. 2. Skip Cloud Performance. 3. Hybrid multidimensional partition Technique. 4. Publisher/Subscriber Module.
- 9. 1.Scalable And Reliable Event Matching: All brokers in SREM as the front-end are exposed to the Internet, and any subscriber and publisher can connect to them directly. To achieve reliable connectivity and low routing latency, these brokers are connected through an distributed overlay, called SkipCloud. The entire content space is partitioned into disjoint subspaces, each of which is managed by a number of brokers. Subscriptions and events are dispatched to the subspaces that are overlapping and events falling into the same subspace are matched on the same broker. After the matching process completes, events are broadcasted to the corresponding interested subscribers. 2.Skipcloud Performance: SkipCloud organizes all brokers into levels of clusters. At the top level, brokers are organized into multiple clusters whose topologies are complete graphs. Each cluster at this level is called top cluster. It contains a leader broker which generates a unique b-ary identifier with length using a hash function cluster are responsible for the same content subspaces, which provides multiple matching candidates for each event. Since brokers in the same top cluster generate frequent communication among themselves, such as updating subscriptions and dispatching events, they are organized into a complete graph to reach each other in one hop. After the top clusters have been well organized, the clusters at the rest levels can be generated level by level.. This identifier is called ClusterID.
- 10. 3.Hybrid multidimensional partition Technique: To achieve scalable and reliable event matching among multiple servers, we propose a hybrid multi-dimensional space partitioning technique, called HPartition. It allows similar subscriptions to be divided into the same server and provides multiple candidate matching servers for each event. Moreover, it adaptively alleviates hot spots and keeps workload balance among all servers. HPartition divides the entire content space into disjoint subspaces . Subscriptions and events with overlapping subspaces are dispatched and matched on the same top cluster of SkipCloud. To keep workload balance among servers, HPartition divides the hot spots into multiple cold spots in an adaptive manner . 4. Publisher/Subscriber: each subscriber establishes affinity with a broker (called home broker), and periodically sends its subscription as a heartbeat message to its home broker. The home broker maintains a timer for its every buffered subscription. If the broker has not received a heartbeat message from a subscriber over Tout time, the subscriber is supposed to be offline. Next, the home broker removes this subscription from its buffer and notifies the brokers containing the failed subscription to remove it.
- 12. SYSTEM REQUIREMENTS: HARDWARE REQUIREMENTS: System : Pentium IV 2.4 GHz. Hard Disk : 40 GB. Floppy Drive : 1.44 Mb. Monitor : 15 VGA Colour. Mouse : Logitech. Ram : 512 Mb. SOFTWARE REQUIREMENTS: Operating system : Windows XP/7. Coding Language : JAVA/J2EE IDE : Netbeans 7.4 Database : MYSQL
- 13. CONCLUSION: This paper introduces SREM, a scalable and reliable event matching service for content-based pub/sub systems in cloud computing environment. SREM connects the brokers through a distributed overlay Skip- Cloud, which ensures reliable connectivity among brokers through its multi-level clusters and brings a low routing latency through a prefix routing algorithm. Through a hybrid multi- dimensional space partitioning technique, SREM reaches scalable and balanced clustering of high dimensional skewed subscriptions, and each event is allowed to be matched on any of its candidate servers. Extensive experiments with real deployment based on a CloudStack testbed are conducted, producing results which demonstrate that SREM is effective and practical, and also presents good workload balance, scalability and reliability under various parameter settings.
- 14. REFERENCE: Xingkong Ma, Student Member, IEEE, Yijie Wang, Member, IEEE, and Xiaoqiang Pei, “A Scalable and Reliable Matching Service for Content-Based Publish/Subscribe Systems” IEEE TRANSACTIONS ON CLOUD COMPUTING, VOL. 3, NO. 1, JANUARY-MARCH 2015.