Global Manufacturer Averts Data Swamp with New Data Lake Architecture
- 1. www.antuit.com 1 The information contained in this document is proprietary. ©2016 Antuit. All rights reserved. Program Highlights GLOBAL MANUFACTURER AVERTS DATA SWAMP WITH NEW DATA LAKE ARCHITECTURE A multi-billion dollar global manufacturer of electronic components, connectors and sensors wanted to enhance the value being derived from their extensive data. The company had launched a strategic initiative to utilize data as a strategic asset to transform the business The company had built a Hadoop data lake consisting of multiple disparate data sources of structured and unstructured data, yet they were unable to effectively leverage the data to create actionable business insights. The Hadoop implementation was also showing signs of performance issues. The organization turned to Antuit’s team of big data architects and engineers to improve the performance of their architecture and create a scalable platform for data consumption. Working collaboratively with the client, the Antuit team audited the existing process, and then re-engineered and implemented a robust scalable architecture. As a result of this process, Antuit uncovered a number of challenges. The client’s existing architecture could not handle the 10+ years of sales and marketing data. The existing systems did not scale, and therefore were not prepared to handle the velocity or volume of data expected in the future. Some power users within the organization were executing overwhelmingly complex queries that exceeded system limitations. Finally, the data systems themselves were housed and managed by disparate business units with minimal integration. Mindful of the significant investment the client had made in its Hadoop architecture, Antuit recommended and then implemented a number of changes. The Antuit team helped the client restructure the data lake and created a better data process by partitioning and compressing data, using split-able file formats, and helping them to identify and use the right data types. To create a seamless experience, Antuit was able to leverage multiple test environments to validate approaches and identify ancillary technologies that, once integrated, would keep their data lake running smoothly and efficiently. While getting the data lake up and running was priority number one, changing internal behaviors and the manner in which queries were written was an equally important challenge. Antuit established a new set of guidelines for internal users, directing them as to how to retrieve desired data from the lake without bringing the entire system to a halt. Program Highlights Challenge A global electronics component manufacturer launched a sales and marketing data hub, only to find that it could not scale to handle the volume or velocity that its one terabyte of data presented. They needed an infrastructure that was easier, faster, and more reliable. Solution Designed and Implemented a new data lake design in Hadoop that was stable and scalable by reformatting data types, partitioning and compressing data, and using new file formats. Outcome New data lake architecture is fast, reliable, and easily accessible to business decision makers.
- 2. We’ve successfully delivered projects to our clients globally. Learn more about our capabilities www.antuit.com follow us on: 2 About Antuit Antuit solves business problems with analytics solutions that deliver measurable and sustainable business value. By combining data, science, technology and industry expertise, Antuit helps companies gain a competitive advantage in their respective industries. Antuit’s global team of dreamers, hackers, and hustlers work across a wide range of industry sectors, developing and deploying analytics solutions that help clients anticipate customer needs, predict business outcomes, and quickly respond and react to business changes. Armed with Antuit’s analytics-powered sales, pricing, marketing and supply chain solutions, organizations are better equipped to achieve exceptional business success in a rapidly changing environment. Founded in 2013 and backed by Goldman Sachs, Antuit has offices in New York, Chicago, Dallas, London, Hong Kong, Singapore, Tokyo, Auckland, Melbourne, Bangalore, and Pune. The information contained in this document is proprietary. ©2016 Antuit. All rights reserved. With the solid data architecture designed by Antuit in place, the organization now has accessible data at their fingertips. With a scalable data lake architecture and sales and marketing data models in place, the company is now able to utilized data as a strategic asset to help transform the way the business is run through advanced analytics. Antuit continues to work with the organization to build new analytical models that solve business problems extract value from their growing data sets. 4 Key Steps to Improve Hadoop Performance 1. Partitioned larger data by effective key. 2. Implemented split-able file format like Sequence, Avro or RC file. 3. Utilized data compress techniques like snappy, bzip2. 4. Used proper data type in hive table. Structured Data ERPs CRMs Enterprise Data Warehouse Semistructured & Unstructured Data Hot Machine Data Real-Time Third-Party Social Media, Geo spatial etc. Warm Enterprise Data Hub (Hadoop) Landing Zone Spark ETL, Data processing and DQ Staging Guided Analytics Standard Reporting Data Discovery / Visualization Predictive Analytics Machine Learning Published • Inventory • Payables • Sales • Purchasing Labs