DT2 Eventstore ss_v001.1
- 1. DT2_Eventstore_v1.1 2013-07-29 Information Page 1 of 10 DT2 Eventstore version 001.1 Summary This document presents a high level view of the DT2 Eventstore. It focuses on the implementation options of an eventstore
- 2. DT2_Eventstore_v1.1 2013-07-29 Contents 1. HISTORY OF THE EVENTSTORE....................................................................3 2. APPLICATION SCOPE......................................................................................3 3. PROBLEM DOMAINS........................................................................................3 4. IMPLEMENTATION FLEXIBILITY.....................................................................3 5. CORE ARCHITECTURE....................................................................................4 6. INTEGRATION...................................................................................................4 7. PERFORMANCE................................................................................................4 8. FURTHER INFORMATION................................................................................5 Information Page 2 of 10
- 3. DT2_Eventstore_v1.1 2013-07-29 1. History of the Eventstore The Eventstore was originally created to enable the collection, storage and timely retrieval of event records, specifically CDR records, in high volume, continuous operation situations. The goals were, 1. Start from a 'clean sheet of paper'. 2. Process WORM data. 3. Deliver flexible storage capabilities. 4. Deliver flexible retrieval capabilities. 5. Provide 'Single point of truth' for event data. 6. Focus on data management, not analytics. 7. Deliver low-cost, low-skill solution. Information Page 3 of 10
- 4. DT2_Eventstore_v1.1 2013-07-29 2. Application Scope The system can be used for any event data. Sample application cases are, 1. Data retention with selective retrieval. 2. Fact tables in datawarehouses. Information Page 4 of 10
- 5. DT2_Eventstore_v1.1 2013-07-29 3. Problem Domains Sample problem cases are, 1 Data Retention & Retrieval 1.1 CDR retention and retrieval (also IPDR etc...) 1.2 Logrecord retention and retrieval. 1.3 JSON doc retention and retrieval. 1.4 Probe data retention and retrieval. Superset records can be stored to provide a common store for multiple record types. 2 Data Feeds 2.1 Data feeds to rules engines as filters and triggers. 2.2 Data feeds to analytic systems. 2.3 Data feeds to aggregators. Information Page 5 of 10
- 6. DT2_Eventstore_v1.1 2013-07-29 4. Implementation flexibility. Sample cases are, 1. Single server system, single index. 2. Single server system, multiple index. 3. Single server system, compressed data. 4. Mutliple server systems for MPP scaleability. The servers can be in different locations. 5. Blade server system. 6. Storage configuration is user defined. It can be simple discs, raided discs, SDD, HDFS. Information Page 6 of 10
- 7. DT2_Eventstore_v1.1 2013-07-29 5. Core architecture The DT2 Eventstore took the “All in one” RDBMS model and broke apart the components. This loose coupling provides, 1. A high degree of flexibility in implementation and performance tuning. 2. Easy customisation of solutions as required on a case by case basis. 3. No lock-in. When components of a solution are loosely coupled it is much easier to replace each component. The DT2 Eventstore in its core, provides a multi, secondary indexing engine, a storage engine and a query engine. DT2 can also provide services to process raw data into stored form, and to deliver selected data sets to any analytic system. Information Page 7 of 10
- 8. DT2_Eventstore_v1.1 2013-07-29 6. Integration The DT2 Eventstore can integrate with RDBMS systems by providing cleaned granular data, aggregates etc... The DT2 Eventstore can integrate with Hadoop by providing files with external indexes to speed up selective data retrieval. The DT2 Eventstore can use HDFS directly to provide replication. The DT2 Eventstore can retain data for multiple years and can age out old data with simple file deletions. Information Page 8 of 10
- 9. DT2_Eventstore_v1.1 2013-07-29 7. Performance The best indicator of performance is to try the DT2 Eventstore in context, however, as an indication of what to expect, the following have been taken from live contexts. 1. A configuration of 3 standard rack servers stores 1 billion CDRs per hour with 7 fields indexed, in continuous operation. 2. A cold search (from disk) can retrieve 10s of thousands of records per second in queries. 3. Cached queries can retrieve 100s of thousands of records per second. Information Page 9 of 10
- 10. DT2_Eventstore_v1.1 2013-07-29 8. Further information Contact Alec at alecdt2@gmail.com for further information. DT2 Pte Ltd is a company which specialises in very high volume event data management DT2 maintains a development system which is used for testing and demos. This can be accessed by arrangement, and we can configure a demo eventstore to handle your records. After this stage, we can provide a pre-configured test system to run yourself. Using this sequence of steps is not essential but saves time. Store configuration is simple so you can change your test system easily. Performance tuning requires more time, but we can offer advice forgiven cases. In the context of an operational system, there are many factors to consider to maintain a reliable data flow and we are happy to discuss these. This area is common to any large data solution. Information Page 10 of 10