Big Data LDN 2018: FORTUNE 100 LESSONS ON ARCHITECTING DATA LAKES FOR REAL-TIME ANALYTICS
- 1. Sunil Mistry – Solution Architecture - EMEA Fortune 100 Lessons on Architecting Data Lakes for Real-time Analytics & AI ATTUNITY
- 2. 2© 2017 Attunity 2© 2017 Attunity Data science | Advanced analytics | AI Real time processing of data in flight Transactions + new sources (IoT…) Decision Engines Are Changing… Business reporting Batch analysis of data at rest Transactional sources Traditional Analytics Modern Approaches
- 3. 3© 2017 Attunity 3© 2017 Attunity CLOUD STREAMINGDATA LAKES MODERN DATA ARCHITECTURES Self Service ON PREM DATA WAREHOUSE BATCH
- 4. 4© 2017 Attunity 4© 2017 Attunity Analyse a broader set of data structures along with structured data Faster and improved decision making AI, Machine Learning Requires managed Data Lake creation and Big Data processing at scale Requires real-time data from on-premise systems and cloud platforms Next Generation Analytics Separation of Storage and Compute Meet new business requirements Support more advanced analytics Replace traditional ETL with modern self- service capabilities Requires real-time data from on-premise systems and cloud platforms Datawarehouse Modernization Legacy application modernization Faster, easier application development Elasticity and Scalability Infrastructure cost savings Requires real-time data from on-premise systems Cloud Application Development Trends Driving Integration Modernization & Automation IaaS PaaS DB MF EDW FILES SaaS IaaS PaaS DB MF EDW FILES DWaaS BIG DATA LANDING ZONE DATA CONSUMPTION & ANALYTICS DB MF EDW FILES
- 5. 5© 2017 Attunity Modern Analytics Requires Examples / Lessons AI/ML/IoT Analytics Scale Use data from 1000’s of sources with minimal development resources and impact Streaming Analytics Real-time Create real-time streams from database transactions Cloud Analytics Efficiency Transfer large data volumes from multiple data centers over limited network bandwidth Agile Deployment Self Service Enable non-developers to rapidly deploy solutions Diverse Analytics Platforms Flexibility Easily adopt and adapt new platforms and methods …Creating Data Integration Challenges THESE CHALLENGES BREAK TRADITIONAL TOOLS
- 6. 6© 2017 Attunity 6© 2017 Attunity Attunity LEADING provider of Streaming CDC Support most sources with best performance and least impact LEADING cloud database migration technology Already moved over 100,000 databases to public cloud platforms LEADING in agility and platform coverage Pre-packaged automation of complex processes and modern UX to accelerate delivery by “data people” THE LEADING PLATFORM FOR DELIVERING DATA EFFICIENTLY AND IN REAL-TIME TO CLOUDS, DATA LAKES, AND STREAMING ARCHITECTURES
- 7. 7© 2017 Attunity 7© 2017 Attunity Raw Deltas STANDARDIZE MERGE FORMAT Full Change History ENRICH SUBSET HDS ODS Snapshot CAPTURE PARTITION Attunity Platform – Analytics Ready Data GENERATE DELIVER REFINE Metadata Services TRACE Lineage CATALOG Data VALIDATE Transfers ANALYZE Data Usage 3rd PARTY Interoperability Operations Management DESIGN Dataflows MANAGE Platform MONITOR Tasks ANALYZE Trends DESIGN & MANAGE SAP RDBMS DATA WAREHOUSE FILES MAINFRAME Consume ANALYZE MONITOR
- 8. 8© 2017 Attunity 8© 2017 Attunity Fortune Customer Example – Manufacturing Legacy Data Environment Future Data Environment DATA INGESTION § 4,000+ data environments § 200+ data warehouses § Legacy mainframe to modern servers § Corporate data lake § Modern infrastructure § Real-time data visibility REAL-TIME PREDICTIVE AI
- 9. 9© 2017 Attunity 9© 2017 Attunity Copies live transactions without touching production Securely transfers them for client usage on global AWS microservices platform Need to efficiently roll out extensive cloud- based microservices platform Must minimize latency and security risk while synchronizing massive transactional updates globally PROBLEM SOLUTION $1 Trillion ATTUNITY REPLICATEMainframe (on prem) KINESIS DYNAMO DB MICROSERVICE HUB ON RDS DYNAMO DB STREAMS RDS EMEA CUSTOMER APJ CUSTOMER AWS CLOUD CDC Fortune Customer Example – Financial Services Assetsunder Management
- 10. 10© 2017 Attunity 10© 2017 Attunity CASE STUDY: INTERNATIONAL FOOD PROCESSOR CDC to HDP data lake Attunity Replicate feeds HDFS, HBase for timely reporting and product delivery Needed real-time view of production capacity and customer orders Nightly batch loads couldn’t keep up => Fulfilment delays, inaccurate reports PROBLEM SOLUTION ATTUNITY REPLICATE Log based CDC SAP ECC 10 tables (purchase orders, production plans) HDP DATA LAKE
- 11. 11© 2017 Attunity 11© 2017 Attunity CASE STUDY: FORTUNE 100 PHARMACEUTICAL FIRM CDC to Kafka to Lambda Architecture Multi-pronged analysis of clinical data at scale Minimal administrative burden; no PROD impact Needed efficient, scalable delivery of clinical data for analytics Lacked tools for low-impact data capture PROBLEM SOLUTION ATTUNITY REPLICATE Log based CDC Lambda Architecture Clinical Systems Structured Analysis Clinical Systems Kafka Batch Historical Data Real Time Updates Stream Processing Graph Analysis Natural Language Processing Machine Learning
- 12. 12© 2017 Attunity “We load +10 Billion rows/hour doing full load while replicating from Oracle Exadata with Attunity Replicate.” “Attunity handles 460,000 records/sec doing CDC from large and highly active Oracle databases into a data warehouse, with peaks of 100Gb per hour” To Summarise - optimized with CDC and batch bulk-loads optimized with in- memory streaming optimized file transfer & compression optimized for each different target (RDBMS, DW, Hadoop) Extraction Transfer on-premises Transfer to cloud Ingest SECURE PERFORMANCE-OPTIMIZED DATA TRANSFER
- 13. 13© 2017 Attunity FINANCIAL SERVICES MANUFACTURING/ INDUSTRIAL HEALTH CARE GOVERNMENT TECHNOLOGY / TELECOM RETAIL OTHER INDUSTRIES Over 2000 Customers
- 14. 14© 2017 Attunity 14© 2017 Attunity Trusted by Microsoft with 3 OEMs, bundled inside SQL Server Trusted by Amazon (AWS) with strategic partnership for cloud database migration Trusted by IBM and Oracle with respective OEMs of Attunity technology Trusted by Teradata and HP as resellers for data warehouse and analytics Trusted by global system integrators Trusted by over 2000 customers for commitment, flexibility and speed 2000+ Trusted by SAP as certified solution in use with over 200 SAP customers Trusted by big data leaders for data lake solutions Challenger on the Gartner MQ for Data Integration Tools Trusted by Teradata and HP as resellers for data warehouse and analytics Technology Partner of Choice