Limitless Data, Rapid Discovery, Powerful Insight: How to Connect Cloudera to SAP Lumira with Simba

Editor's Notes

• #3: At Cloudera, our mission is to help organizations gain value from all their data. Increasingly, leading organizations view data as among their most important strategic assets, but only if they’re able to leverage that data to meet their business objectives.
• #4: We see a few trends driving the increased importance of having a strategy for data. The Internet has changed everything, and we are more connected than ever before. We all expect to be on the web these days; we rely on it for work, shopping, entertainment, and social interaction. With the simultaneous proliferation of mobile devices and sensors, we now have the ability measure almost everything. As a result, we’re generating data, and moving it, at a rate that’s entirely new. In this new online world, customers and employees expect more personalization, but not at the cost of privacy. Security matters. Ultimately, data enables us to better understand our customers, patients, employees, or students. Innovative organizations embrace experimentation and agile methods. Representative Customer Stories Vivint: Everything that can be measured, will be measured. Challenge: Vivint needed a central repository to gather and analyze data generated from each of the 20-30 sensors -- e.g. thermostats, smart appliances, video cameras, window and door sensors, and smoke and carbon monoxide sensors -- in every one of its 800,000 customers' homes. Solution: Vivint has deployed an enterprise data hub on Cloudera that allows it to look across many data streams simultaneously for behaviors, geo-location, and actionable events. Benefit: With its enterprise data hub that combines sensor data across multiple data streams, Vivint can glean new insights that help the company understand and enrich customers' lives. For example, knowing when a home is occupied or vacant is important to security – but when tied into the heating, ventilation and cooling (HVAC) system, you can add a layer of energy cost savings by cooling or heating a home based on occupancy. Western Union: Employees and customers expect more personal interactions. Challenge: With customers spanning every corner of the globe and all walks of life, Western Union saw an opportunity to personalize the experience for each customer by combining the volumes of information about their transactions -- of which Western Union processed 29 per second in 2013 -- with user behavior data, clickstream data, and mobile usage patterns. Solution: Western Union implemented an enterprise data hub on Cloudera to centralize its data -- both structured and unstructured -- in order to provide a 360-degree customer view, while also supporting use cases for risk management and AML compliance. Benefit: Deeper customer understanding is driving product improvements and enhancements that improve Western Union customers' experience. For example, Western Union learned through its EDH that many customers in key sectors process the same transactions repeatedly, prompting the company to add a "Send Again" button to its mobile app to streamline the processing of repeat transactions. By deploying that capability, the company immediately saw a conversion uptake in those key sectors. Marketing Associates: The most innovative companies embrace experimentation and agility. Challenge: Marketing Associates' Magnify Analytic Solutions division has built expertise executing B2C online marketing contests and product giveaways for large clients such as Chrysler, DuPont, Ford, and Jaguar, which requires intensive data processing, elastic flexibility and scalability, and agility and performance. Ford recently offered Magnify the opportunity to manage its entire CRM system -- which Magnify jumped at, but knew it would need a new big data infrastructure to support. Solution: Without any prior in-house experience with Hadoop, Magnify built an enterprise data hub on Cloudera, leaning heavily on Cloudera Manager, Search, Impala, and integrations with SAS and Tableau to streamline the new platform's adoption. Benefit: The EDH has been a tremendous success, enabling Magnify to deliver a self-service, 360-degree view of consumers to its clients (vs. sending them Excel spreadsheets every 1-2 days which was the case prior). And better yet, the all-inclusive price of Cloudera Enterprise, Data Hub Edition and all resources needed to built its development, production, and QA environment came in well below ongoing costs of the traditional environment.
• #5: Key take away: Analysts are trying to accomplish 1 of 3 things when they embark on data discovery projects. Building reports, models, or rules. Definition: A report is visual representation of static data. Once you have the report you can operationalize it by incorporating it into a dashboard that gets refreshed regular that you want your team to see. A model is a function of variables that have weights given to certain attributes resulting in an output. The output can be served to end users so that they get the relevant info they need. A rule is an attribute that can be inputted into a point solution to change the output (ad platform, fraud detection, etc). Example of difference between model and rule: When marketers target on ad platforms they input rules into the engine (gender, age, location, etc.) They are not in charge of the model that actually determines the optimal person to target the individuals. That model is built into the point solution.
• #6: Key take away: It takes a lot of time and processing frameworks to arrive at value. The process of discovering value from data is an cyclical process that takes multiple processing frameworks, a wide variety of data, and countless iterations through out the process. The analyst must discover the data sets they want to included in analysis and then transform and cleanse this data in preparation for analysis. Depending on what the analyst is looking for, a report, model, or rule, they would use a variety of techniques in order to arrive at the outcome they think to be most effective. Once the report, model, or rule has been developed the Data Discovery process is over. They now must implement this information into a solution in order for this value to reach the masses. Single Data Discovery is extremely important, but shouldn’t be the end goal. Once a single analysts discovers this information they must make sure an entire team, department, organization, or customer base gets this information in a timely manner. If these output doesn’t influence optimal behavior, and the KPIs don’t move, then the analyst must go back to the discovery process and optimize the output. Optimizing means that they will include a different data or change the transformation or analysis technique they used. Let’s take a look at how traditional architectures are set up to handle this process.
• #7: Key take away: It takes a lot of time and processing frameworks to arrive at value. The process of discovering value from data is an cyclical process that takes multiple processing frameworks, a wide variety of data, and countless iterations through out the process. The analyst must discover the data sets they want to included in analysis and then transform and cleanse this data in preparation for analysis. Depending on what the analyst is looking for, a report, model, or rule, they would use a variety of techniques in order to arrive at the outcome they think to be most effective. Once the report, model, or rule has been developed the Data Discovery process is over. They now must implement this information into a solution in order for this value to reach the masses. Single Data Discovery is extremely important, but shouldn’t be the end goal. Once a single analysts discovers this information they must make sure an entire team, department, organization, or customer base gets this information in a timely manner. If these output doesn’t influence optimal behavior, and the KPIs don’t move, then the analyst must go back to the discovery process and optimize the output. Optimizing means that they will include a different data or change the transformation or analysis technique they used. Let’s take a look at how traditional architectures are set up to handle this process.
• #8: Key take away: It takes a lot of time and processing frameworks to arrive at value. The process of discovering value from data is an cyclical process that takes multiple processing frameworks, a wide variety of data, and countless iterations through out the process. The analyst must discover the data sets they want to included in analysis and then transform and cleanse this data in preparation for analysis. Depending on what the analyst is looking for, a report, model, or rule, they would use a variety of techniques in order to arrive at the outcome they think to be most effective. Once the report, model, or rule has been developed the Data Discovery process is over. They now must implement this information into a solution in order for this value to reach the masses. Single Data Discovery is extremely important, but shouldn’t be the end goal. Once a single analysts discovers this information they must make sure an entire team, department, organization, or customer base gets this information in a timely manner. If these output doesn’t influence optimal behavior, and the KPIs don’t move, then the analyst must go back to the discovery process and optimize the output. Optimizing means that they will include a different data or change the transformation or analysis technique they used. Let’s take a look at how traditional architectures are set up to handle this process.
• #9: Key takeaway: It is not just a BI challenge, it is the way that data is managed. Keeping 3 main high level objectives of an architecture built for Data Discovery in mind- accessing data, analyzing data, and experimenting and iterating fast- we can examine a traditional architecture and see where organizations might run into issues. Questions for customer: Does this look like your architecture?
• #10: Key takeaway: Experimentation and iterations take time with traditional architectures making it difficult to fail fast or succeed.
• #11: Key takeaway: Experimentation and iterations take time with traditional architectures making it difficult to fail fast or succeed.
• #12: Key takeaway: Experimentation and iterations take time with traditional architectures making it difficult to fail fast or succeed.
• #13: Key takeaway: An EDH provides the foundation to change the way you collect and manage data in order to provide your analyst what they need in less time. ETL on the fly: Talk to schema-on-write vs schema-on-read (http://www.slideshare.net/awadallah/schemaonread-vs-schemaonwrite).
• #14: Key takeaway: An EDH provides the foundation to change the way you collect and manage data in order to provide your analyst what they need in less time. ETL on the fly: Talk to schema-on-write vs schema-on-read (http://www.slideshare.net/awadallah/schemaonread-vs-schemaonwrite).
• #15: Key takeaway: An EDH provides the foundation to change the way you collect and manage data in order to provide your analyst what they need in less time. ETL on the fly: Talk to schema-on-write vs schema-on-read (http://www.slideshare.net/awadallah/schemaonread-vs-schemaonwrite).
• #16: Link to account record in SFDC (valid for Cloudera employees only): https://na6.salesforce.com/00180000019dZ6D The State of Indiana builds an enterprise data management platform to reduce costs and improve lives of its citizens Background: The state of Indiana has a population of more than 6.5 million people (known as “Hoosiers”) and 36,500 square miles of land area. It ranks 16th in the country based on population. Challenge: One of the state’s goals is “transparency,” providing citizens with comprehensive insight into state operations to confirm that taxpayer dollars are delivering the most efficient and effective services possible. State officials also see great opportunity in using data and analytics to help improve the lives of Indiana citizens. However, as with most state governments, officials found it difficult to integrate data stored in silos across 71 departments quickly or efficiently. Its existing data platforms couldn’t scale (except at great cost) to manage the huge amount of data needed. Additionally, ETL processes to move data into a common platform were extremely time-consuming. In one case, staff found that integrating expense reports from different agencies so they could be analyzed took more than 8 hours, which was unacceptable to users. Solution By implementing a Hadoop-based operational data store with Cloudera Enterprise, Data Hub Edition, the organization is tackling these challenges--reducing the time and cost to mine its data and gaining new insight.   Cloudera will ingest, process, and analyze data from SAP HANA and more than 50 other data sources, including virtual SQL tables, across the organization. Staff will be able to analyze statewide data via Impala + R. SAP Lumira will be used for data visualization. Cloudera Navigator will support data auditing, lineage and discovery. And enterprise architects will use Cloudera Manager to monitor and quickly diagnose cluster issues. Security was a significant concern for state officials given that the state manages sensitive information, including financial and health data. Cloudera was selected over HortonWorks due to the integrated encryption via Sentry. The state will use Sentry to encrypt data columns and Kerberos to encrypt the drives.   Bringing together so much data in a single view can be challenging and vendor support can make a significant difference between success and failure. According to state enterprise data architects, Cloudera is “much easier to work with” than other vendors – enabling staff to focus on their big picture goals.   Benefit: What will be the benefit of this state’s enterprise data platform and work with Cloudera? From an operational perspective, current tests show significant time savings from offloading ETL work to Hadoop, with queries once taking more than eight hours reduced to just four seconds. Additionally, the platform will help reduce costs as IT staff can optimize how and when they use SAP HANA, offloading less critical or even hot workloads to Hadoop.   However, what’s most exciting is the new insight that will be gained to help improve the lives of Indiana’s citizens.   Take, for example, the state’s goal to reduce the infant mortality rate. Indiana currently has one of the highest infant mortality rates in the U.S. One baby dies every 13 hours in Indiana. At the 2nd annual Indiana Infant Mortality Summit, held in 2014, presenters reported that if the state could reduce infant mortality rate to national average, 60 babies would survive each year. But the question for officials is: Which programs are best delivered to which mothers and when? Many factors contribute to infant mortality, including smoking, obesity, prenatal care, unsafe sleep, and early deliveries. By being able to integrate and analyze data across a family’s interaction with state agencies – from social and family services, to health services, to financial aid and food programs – state officials are confident they’ll uncover important insights that help them prevent unnecessary deaths. For example, officials want to understand the relationship between infant mortality and nutrition programs; do moms who receive WIC funds (The Special Supplemental Nutrition Program for Women, Infants, and Children) have healthier babies, and if so, would increasing funding WIC funds in specific areas of the state help save newborns?
• #17: Data Services certification for CDH in progress 2. HANA SP08 + CDH connector validated Deeper certification (PE) planned post HANA SP09 release 3. Lumira ODBC driver for Impala certification –Q4 InfiniteInsights certification under investigation
• #19: Download and install on your desktop in less than 5 minutes Insight from many data sources Combine, manipulate, and enrich data to apply it to your business scenarios Self-service visualizations and analytics to tell your story Optimized for SAP HANA for real time on detailed data Connectivity to Hadoop Extract value from your Hadoop data by performing analysis on the data Simple connections and an easy to use interface mean business users can extract value from Big Data sources Mash together data from big data and traditional sources for better insights Big Data visualizations, heat maps, scatter charts, create your own charts thru the CVOM SDK Extensible…if your needs go beyond desktop analysis you have the SAP stack behind you. interopability with Predictive Analysis means you can go beyond what’s already happened and make predictions on future behavior. Use results from Predictive to create visualizations in Lumira. With HANA means you can leverage Smart Data Access to access data directly in Hadoop and centralize data management. do rapid calculations when needed. BI Suite?? Share stories beyond the data analyst with Lumira Server or Lumira Cloud. Same interface for web, desktop, and cloud. Extensibility of the datasource, create your own data drivers with the open API if you have a customized datasource.