Architecting for Big Data: Trends, Tips, and Deployment Options
5 likes1,036 views
Joe Caserta outlines the evolution and significance of big data in business intelligence, emphasizing the need for organizations to adapt their data governance and analytics strategies. He highlights the shift from traditional data warehousing to integrating big data solutions and the emerging roles of data scientists and non-technical users. The presentation also discusses the tools, practices, and technologies that facilitate effective data management and analysis in a rapidly changing landscape.
Architecting for Big Data: Trends, Tips, and Deployment Options
- 1. @joe_Caserta@GreatLakesBI Architecting for Big Data: Trends, Tips, and Deployment Options Joe Caserta President Caserta Concepts
- 2. @joe_Caserta@GreatLakesBI Top 20 Big Data Consulting - CIO Review Joe Caserta Timeline Launched Big Data practice Co-author, with Ralph Kimball, The Data Warehouse ETL Toolkit (Wiley) Dedicated to Data Warehousing, Business Intelligence since 1996 Began consulting database programing and data modeling 25+ years hands-on experience building database solutions Founded Caserta Concepts in NYC Web log analytics solution published in Intelligent Enterprise Formalized Alliances / Partnerships – System Integrators Partnered with Big Data vendors Cloudera, Hortonworks, IBM, Cisco, Datameer, Basho more… Launched Training practice, teaching data concepts world-wide Laser focus on extending Data Warehouses with Big Data solutions 1986 2004 1996 2009 2001 2010 2013 Launched Big Data Warehousing Meetup in NYC ~ 1,500 Members 2012 2014 Established best practices for big data ecosystem implementation – Healthcare, Finance, Insurance Top 20 Most Powerful Big Data consulting firms Dedicated to Data Governance Techniques on Big Data (Innovation)
- 3. @joe_Caserta@GreatLakesBI About Caserta Concepts • Technology services company with expertise in data analysis: • Big Data Solutions • Data Warehousing • Business Intelligence • Core focus in the following industries: • eCommerce / Retail / Digital Marketing • Financial Services / Insurance • Healthcare / Higher Education • Established in 2001: • Increased growth year-over-year • Industry recognized work force • Strategy, Implementation, Analytics • Writing, Education, Mentoring • Data Science & Analytics • Cloud Computing • Data Interaction & Visualization
- 4. @joe_Caserta@GreatLakesBI Sales Marketing Finance ETL Data Exploration Horizontally Scalable Environment - Optimized for Analytics Big Data Lake Big Data Analytics NoSQL Databases ETL Ad-Hoc/Canned Reporting Traditional BI Spark MapReduce Pig/Hive N1 N2 N4N3 N5 Hadoop Distributed File System (HDFS) Others… The Evolution of Enterprise Data Data Science Enterprise Data Warehouse ETL
- 5. @joe_Caserta@GreatLakesBI Tools and Technologies Best Practices Data Warehousing/ ETL/Data Integration BI/Visualization/ Analytics Big Data Analytics
- 7. @joe_Caserta@GreatLakesBI The one’s you need to know…. Hadoop Distribution: Cloudera, Hortonworks, MapR, Pivotal-HD, IBM Tools: Hive: Map data to structures and use SQL-like queries Pig: Data transformation language for big data Sqoop: Extracts external sources and loads Hadoop Spark: General-purpose cluster computing framework Storm: Real-time ETL NoSQL: Document: MongoDB, CouchDB Graph: Neo4j, Titan Key Value: Riak, Redis Columnar: Cassandra, Hbase Search: Lucene, Solr, ElasticSearch Languages: Python, SciPy, Java, R, Scala
- 8. @joe_Caserta@GreatLakesBI Advertising Real time interactive queries on massive audience datasets in the cloud Global analytics on the cloud Integrate SAP implementations from across the globe into single cloud solution Why are we Changing? Recommendation Engines “You chose… you might also like…” Real-Time Aggregation, Monitoring & Alerting on events at extremely high message rates… ~1M msgs/sec Big Data Warehouse Extending EDW with Hadoop Governing data from the “lake” to the EDW Personal/Commercial Banking Investment/Trading Bank World-wide beauty company Cable Television Audience-based Advertising
- 9. @joe_Caserta@GreatLakesBI • This is the ‘people’ part. Establishing Enterprise Data Council, Data Stewards, etc.Organization • Definitions, lineage (where does this data come from), business definitions, technical metadataMetadata • Identify and control sensitive data, regulatory compliancePrivacy/Security • Data must be complete and correct. Measure, improve, certify Data Quality and Monitoring • Policies around data frequency, source availability, etc. Business Process Integration • Ensure consistent business critical data i.e. Members, Providers, Agents, etc. Master Data Management • Data retention, purge schedule, storage/archiving Information Lifecycle Management (ILM) Components of Data Governance
- 10. @joe_Caserta@GreatLakesBI • This is the ‘people’ part. Establishing Enterprise Data Council, Data Stewards, etc.Organization • Definitions, lineage (where does this data come from), business definitions, technical metadataMetadata • Identify and control sensitive data, regulatory compliancePrivacy/Security • Data must be complete and correct. Measure, improve, certify Data Quality and Monitoring • Policies around data frequency, source availability, etc. Business Process Integration • Ensure consistent business critical data i.e. Members, Providers, Agents, etc. Master Data Management • Data retention, purge schedule, storage/archiving Information Lifecycle Management (ILM) Components of Data Governance • Add Big Data to overall framework and assign responsibility • Add data scientists to the Stewardship program • Assign stewards to new data sets (twitter, call center logs, etc.) • Graph databases are more flexible than relational • Lower latency service required • Distributed data quality and matching algorithms • Data Quality and Monitoring (probably home grown, drools?) • Quality checks not only SQL: machine learning, Pig and Map Reduce • Acting on large dataset quality checks may require distribution • Larger scale • New datatypes • Integrate with Hive Metastore, HCatalog, home grown tables • Secure and mask multiple data types (not just tabular) • Deletes are more uncommon (unless there is regulatory requirement) • Take advantage of compression and archiving (like AWS Glacier) • Data detection and masking on unstructured data upon ingest • Near-zero latency, DevOps, Core component of business operations For Big Data
- 11. @joe_Caserta@GreatLakesBI The Big Data Pyramid Data has different governance demands at each tier. Only top tier of the is fully governed and ready for Enterprise BI Big Data Warehouse Data Science Workspace Data Lake – Integrated Sandbox Landing Area – Source Data in “Full Fidelity” Metadata Catalog ILM who has access, how long do we “manage it” Raw machine data collection, collect everything Data is ready to be turned into information: organized, well defined, complete. Agile business insight through data-munging, machine learning, blending with external data, development of to-be BDW facts Metadata Catalog ILM who has access, how long do we “manage it” Data Quality and Monitoring Monitor completeness of data Metadata Catalog ILM who has access, how long to “manage it” Data Quality and Monitoring Monitoring of completeness of data Fully Data Governed ( trusted) User community arbitrary queries and reporting
- 12. @joe_Caserta@GreatLakesBI • The Big Data movement breaks the relational database barrier and enables analysis on massive amounts of structured and unstructured data. • NoSQL puts the value of SQL based relational databases into question. This disruption is forging a new road for the progress and advancement of scalable data analytics. • The value of legacy Business Intelligence comes into question. • Rather than forcing data users to become technologists, it must make data analysis available for the masses. BI is About to be Disrupted!
- 13. @joe_Caserta@GreatLakesBI • The role of the ‘Business Analyst’, the primary user of the BI tool, is being replaced or by two types of data users: 1. Highly technical Data Scientists 2. Non-technical Business Persons • New analytics (BI) platforms must be created to accommodate the new users. We see these very discrete users using very different technologies. • Perhaps legacy BI tools will not go away, but the market is absolutely about to be disrupted. Who Does BI Today?
- 14. @joe_Caserta@GreatLakesBI • Data Scientists have deep technical knowledge • They enjoy writing code and mining data • The best way to serve a data scientist is to provide access to raw data and then get out of their way. Empower the Data Scientist
- 15. @joe_Caserta@GreatLakesBI What does a Data Scientist Do, Anyway? Searching for the data they need Making sense of the data Figuring why the data looks the way is does and assessing its validity Cleaning up all the garbage within the data so it represents true business Combining events with Reference data to give it context Correlating event data with other events Finally, they write algorithms to perform mining, clustering and predictive analytics – the sexy stuff. Writes really cool and sophisticated algorithms that impacts the way the business runs. Much of the time of a Data Scientist is spent: NOT
- 16. @joe_Caserta@GreatLakesBI • Business users don’t have, and don’t want to have, technical wherewithal to interact with ‘data’. • “We have a business to run! Programming should be done by people in rooms with no windows.” • “I need information at my fingertips and I should not need a PhD in SQL to get it.” • “It’s a myth that BI tools will solve my problems, I still need IT to get new reports. This is unacceptable.” • Every business professional on the planet knows how to search for needed information via a Google search bar. • Business people want to be able to ‘Google’ their corporate data for the information they need. Empower the Business Person
- 17. @joe_Caserta@GreatLakesBI The Future of BI (if the Business gets its way)…
- 18. @joe_Caserta@GreatLakesBI Facets created automatically based on relevant data Navigating Data in BI…
- 19. @joe_Caserta@GreatLakesBI • During normal BI implementations, much time is spent/wasted on selecting the best way to graphically represent a set of metrics. • We can embed algorithms that are statistically proven to best represent information depending on the type of question being asked. • The user should be able to preview and change from the default infographic as easy as clicking ‘next’ on a Yahoo! Slideshow. Why do we make it so difficult?
- 20. @joe_Caserta@GreatLakesBI Lady gaga sales by state by customer age Go! joe@casertaconcepts.com Region Northeast Midwest South West Product Records Perfume Clothes Performances Dates 2009 to 2013 DOWNLOAD TO EXCEL Imagine the Possibilities….
- 21. @joe_Caserta@GreatLakesBI • Modern web application framework • Developed and supported by Google • Bootstrap used for Mobile Angular • JavaScript library for data visualization. • Exposes full capability CSS3, HTML5 and SVG. Is extremely fast • Support large datasets and dynamic behaviors for interaction D3.js • The “glue” that brings other components together • The ‘engine’ that transforms search strings into queries. • Integrated with the Customer Metadata repository Python • Full-text and faceted-search engine and database • This is the backbone of the applicationSolr • Customer Metadata repository. Stores all business rules (default facets, etc) and user preferences (default graph types, etc) • Cassandra may not be ultimate selection Cassandra • Amazon Web Services • Product is a zero-footprint cloud based solution • User experience is same as Googling info AWS Building the Future of BI (Hint: it’s Big Data)
- 22. @joe_Caserta@GreatLakesBI Innovation is the only sustainable competitive advantage a company can have. Closing Thought Challenge the status quo!
- 23. @joe_Caserta@GreatLakesBI Thank You & Questions Joe Caserta President, Caserta Concepts joe@casertaconcepts.com (914) 261-3648 @joe_Caserta