Using Couchbase and Elasticsearch as data layers
Download as PPTX, PDF1 like512 views
This document summarizes Sizmek's use of Couchbase and Elasticsearch as their data layer technologies. They use Couchbase for its fast reads/writes and support for large, unstructured data volumes. Elasticsearch is used for its full-text search capabilities. Some key points: - Couchbase is used for its JSON support, indexing, querying abilities, and cross data center replication. N1QL is used for queries. - Elasticsearch faces consistency issues in clustered environments that can cause test failures. Solutions include waiting before checking for updates or using refresh to force index updates. - Maintaining uniqueness constraints is done by saving uniqueness documents with entity names as keys and IDs as values. - The Java
Using Couchbase and Elasticsearch as data layers
- 1. SIZMEK MDX NXT USING COUCHBASE AND ELASTICSEARCH AS DATA LAYER TAL MAAYANI YUVAL PERRY
- 3. HIGH LEVEL MISSION Building a scalable , highly available , large and fast ad management platform
- 4. MAIN TECHNOLOGIES USED AWS Deployment Micro services architecture JavaRX Couchbase Elasticsearch RabbitMQ Consul
- 5. WHY NOSQL? NoSQL Document Database Relational Database Unstructured data Structured data Memory first approach Disk first approach No transactions Transactional Scale Horizontally Scale vertically No SQL DB allows • Fast read and writes • Hold variety of data models • Large data volumes • Cloud friendly deployment • No single point of failure Still need to take care of transactionless, eventually consistent data source
- 6. WHY COUCHBASE ? JSON support Indexing and Querying Cross data center replicationIncremental Map Reduce
- 7. OUR DATA LAYER Generic Data Access Layer Query Get(Id) Save / Update XDCR N1QL
- 8. DEMO – SIZMEK COUCHBASE ADMINISTRATOR TOOL • In house development tool that allows to perform ES queries as well as N1QL queries • Usage • Data investigation • Data migration
- 9. HOW WE MAINTAIN ATOMICITY ON TRANSACTION LESS DATA SOURCE Transaction manager service – maintain flow state between multiple entities change Provide atomicity & tracking Example: Save smart version ad flow Dynamic Campaign Optimization Transaction Manager Asset Mgmt Ad Service Create Ad Upload ad assets Create Smart version 1. Assets created 2. Ad Created 3. Smart version created
- 10. ELASTIC SEARCH – CONSISTENCY PROBLEM AND HOW TO OVERCOME THIS IN AUTOMATIC TESTING The problem In a clustered elastic search environment, one document update is not automatically reflected in all notes. This caused an inconsistent results in automatic testing. Example Change campaign name from A to B. Automatic test verifies that the change actually tool place by getting the entity and verify its name. Possible Solutions • Wait few seconds before checking for updated status • Use elastic search refresh to force in memory index update
- 11. NAME UNIQUENESS IMPLEMENTATION HOW TO IMPLEMENT UNIQUE CONSTRAINS USING COUCHBASE Problem Maintain unique entity name Real use case Keep advertisement name unique system wide Possible Solution Save uniqueness document Key: entity name Value : entity id Save succeeded? Save entity Return error Delete uniqueness doc Input: entity to save Still need to take care of orphan uniqueness documents
- 12. N1QL EXAMPLE • Use Query Workbench Developer Preview • Example queries 1. select mvbucket.`key` from mvbucket where payload._type = 'AdSmartVersion' and payload.createdOn is not missing 2. select * from mvbucket where payload._type = 'AdSmartVersion' and payload.masterAdId = 1073741825 and payload.createdOn between 1349057369158 and 1449057369158 3. select payload.masterAdId, count(1) from mvbucket where payload._type = 'AdSmartVersion' and payload.createdOn between 1349057369158 and 1449057369158 4. select payload.masterAdId, count(1) as count from mvbucket where payload._type = 'AdSmartVersion'
- 13. COUCHBASE JAVA CLIENT 2 NOTES ON JAVA CLIENT • Built in support of JSON documents • Support counters • Asynchronous client using java RX • Allow exploit already used reactive business logic • Parallel efficient processing • Inherent error handling – for example retries get document with an exponential backoff Observable .from(docIds) .flatMap(id -> { return bucket .async() .get(id) .retryWhen(RetryBuilder .anyOf(BackpressureException.class) .delay(Delay.exponential(TimeUnit.MILLISECONDS, 100)) .max(10) .build() ); }) .subscribe();
- 14. OUR USE OF ELASTICSEARCH QUERY ENGINE • Free text search – user boolean queries • Data filtering – data grid filtering • Grouping – data grid grouping • Authorization – filter document according to user permissions • batch processing – internal services that use scan and scroll to operate on large data set
- 15. ELASTIC SEARCH – SOME BEST PRACTICES • Carefully maintain index schema • Avoid using Dynamic mapping • Data type collisions • Large data set – do not save data that is not used • Build static schema from data model • Updating data model searchable field trigger build of new index • Some changes in schema required re-indexing, e.g. adding mandatory field, change of enumeration value • Inconsistency – updated data is not immediately appears on query result • System overall design must be aware of this limitation • Throttling – must control number of writes
- 16. COUCHBASE 4.1 OUR USAGE • Use optimistic locking - update operations are done through updater lambda function • N1QL • Do not meet performance for large data set with order by queries • Took more than 5 sec to query 250 entities • Used for business logic where no sorting is required • Used when consistency is important • XDCR • Customize plugin to index required entities • Add support of parent child relationship in elasticsearch
- 18. Thank you
Editor's Notes
- #3: For more details http://www.slideshare.net/erwinsizmek/sizmek-company-profile-2014?next_slideshow=2
- #5: NoSQL is the most critical notion in the architecture
- #7: RDBMS optimization – required DBA team NoSQL is like space ship vs RDBMS can be like rolls roise
- #8: Add N1QL
- #9: Internal tool for couchbase DB querie
- #10: N1QL – fetch documents by Ids
- #13: Do not ensure isolation (ACID) Add title
- #14: A refresh effectively calls a reopen on the lucene index reader, so that the point in time snapshot of the data that you can search on gets updated. This lucene feature is part of the lucene near real-time api. An elasticsearch refresh makes your documents available for search, but it doesn't make sure that they are written to disk to a persistent storage, as it doesn't call fsync, thus doesn't guarantee durability. What makes your data durable is a lucene commit (flush), which is way more expensive. An elasticsearch flush effectively triggers a lucene commit, and empties also the transaction log, since once data is committed on the lucene level, durability can be guaranteed by lucene itself. Flush is exposed as an api too and can be tweaked, although usually that is not necessary. Flush happens automatically depending on how many operations get added to the transaction log, how big they are, and when the last flush happened.
- #18: The following code retries with an exponential backoff (with a 100 millisecond ceiling), but stops after 10 attempts and propagates the error. Observable. For more info see http://developer.couchbase.com/documentation/server/4.0/sdks/java-2.2/documents-bulk.html
- #20: Elastic search usually used for free text and auto complete
- #21: Show rebuild of static index with high availability
- #23: Responsiveness is the cornerstone of usability and utility, but more than that, responsiveness means that problems may be detected quickly and dealt with effectively Resilience is achieved by replication, containment, isolation and delegation Elastic Systems can react to changes in the input rate by increasing or decreasing the resources allocated to service these inputs. Message driven ensures loose coupling, isolation, location transparency, and provides the means to delegate errors as messages