Big data analytics and building intelligent applications
1 like1,380 views
This document discusses big data analytics, intelligent applications, and the future of artificial intelligence. It covers the current state of the art in machine learning, data mining, and data science. It also addresses challenges in scaling artificial intelligence to meet future data generation rates, which will outpace computation and communication speeds. New distributed computation models that move code to data instead of vice versa will be needed. Fundamental limits of physics, networks and computation must also be considered in architecting intelligent systems of tomorrow.
Big data analytics and building intelligent applications
- 1. 1confidentialFlytxt. All rights reserved. 30 June 201530 June 2015© Architecting Intelligence Big Data Analytics and Building Intelligent Applications
- 2. 2confidentialFlytxt. All rights reserved. 30 June 201530 June 2015© Our discussion today Big Data Analytics and Intelligent Applications • The puzzle, the hype, the customer? • Man-machine collaboration State of the Art • Practical AI, Machine Learning, Data Mining • Data Science, Data Games What does the future guarantee? • Physics, Networks and Computation • New computation models?
- 3. 3confidentialFlytxt. All rights reserved. 30 June 201530 June 2015© Big Data, the Meme
- 4. 4confidentialFlytxt. All rights reserved. 30 June 201530 June 2015© Big (Data Analytics) Distraction
- 5. 5confidentialFlytxt. All rights reserved. 30 June 201530 June 2015© Big (Data) Crowd
- 6. 6confidentialFlytxt. All rights reserved. 30 June 201530 June 2015© Start unraveling the complexity What do I want to communicate that currently requires a significant amount of time and energy to analyze, interpret, and share? • Stuart Frankel, “Data Scientists Don’t Scale”, Harvard Business Review, May 2015 What economic value will my customer gain from Big Data?
- 7. 7confidentialFlytxt. All rights reserved. 30 June 201530 June 2015© Flytxt Our vision is to create >10% measurable economic value for Mobile Enterprises through Big Data Analytics Flytxt’s solutions create incremental revenues from new and existing sources, optimize margins and enhance customer experience Dutch company with corporate office in Dubai, global delivery centres in India and regional presence in Mexico City, Johannesburg, Singapore, Dhaka and Nairobi. Sample text Awards and Recognitions
- 8. 8confidentialFlytxt. All rights reserved. 30 June 201530 June 2015© Benefits delivered to customers PartnersOperators Proven across many Countries, Brands and Logos Brands IIT DELHI 4% Increase in Gross Revenue 30% Growth in Mobile Money users 10% Growth in Data Users 105% Increase in Special offer Sales 300% Increase in Store Footfall 25% Drop in Churn
- 9. 9confidentialFlytxt. All rights reserved. 30 June 201530 June 2015© Big Data Technology Architecture – the Flytxt example
- 10. 10confidentialFlytxt. All rights reserved. 30 June 201530 June 2015© Our discussion today (Part 2) Big Data Analytics and Intelligent Applications • The puzzle, the hype, the customer? • Man-machine collaboration State of the Art • AI, Machine Learning, Data Mining • Data Science, Data Games What does the future guarantee? • Physics, Networks and Computation • New computation models?
- 11. 11confidentialFlytxt. All rights reserved. 30 June 201530 June 2015© Practical AI: Personalized Driving Navigator suggest alternative route due to traffic congestion System identifies primary/secondary driver Bluetooth connection to Car Systems Car system connects with database to access unique ID info Infotainment settings are modified (language preference, radio station…) Navigator presents favorite destinations Connected office identifies next meeting happens in 10 min and offers re-scheduling
- 12. 12confidentialFlytxt. All rights reserved. 30 June 201530 June 2015© Practical AI: Pattern Classification in Location Analytics Automatic classification of venues / routes based on their features Each venue/route is represented by a set of features Labeled examples corresponding to various venue types / route types which represent classes Learn a decision boundary that separates the classes & then make predictions
- 13. 13confidentialFlytxt. All rights reserved. 30 June 201530 June 2015© Customer Marketing Program Product Rol Infra- structure Location Descriptive Exploratory Heuristic Predictive Prescriptive Visualization CLV Monitoring Opportunity Identification Behavioral Variations Action Prediction Personalized Recommendation Effectiveness Measurement Program Reach Analysis Business impact Analysis Outcome Forecasting Impact Optimization Product Popularity Monitoring Product Promotion Analysis Product Association Profitability Simulation Product Promotion Recommendation Business Health Monitoring KPI Impact Analysis Business Impact measurement Impact Forecasting Yield Optimization Utilization Monitoring Challenge Identification Cost Benefit Analysis Event Prediction Optimization Recommendation Geo-Spatial Reporting Location Affinity Analysis Location- Behavior Association Location based Forecasting Location based Recommendation Roots of Practical AI: Analytics Models built by Data Scientists
- 14. 14confidentialFlytxt. All rights reserved. 30 June 201530 June 2015© Data Sciences: State of the Art KDD Cup: organized by ACM Special Interest Group on Knowledge Discovery and Data Mining 2010: Predict student performance on mathematical problems from Intelligent Tutoring System logs 2011: Recommending Music Items based on the Yahoo! Music Dataset 2012: Predict which information sources one user might follow in Weibo (Chinese “twitter”) 2013: Determine whether an author has written a given paper 2014: Predict funding requests that deserve an A+ (for DonorsChoose.org) 2015: Predict student dropout on a Massive Open Online Course platform (XuetangX)
- 15. 15confidentialFlytxt. All rights reserved. 30 June 201530 June 2015© Our discussion today (Part 3) Big Data Analytics and Intelligent Applications • The puzzle, the hype, the customer? • Man-machine collaboration State of the Art? • AI, Machine Learning, Data Mining • Data Science, Data Games What does the future guarantee? • Physics, Networks and Computation • New computation models?
- 16. 16confidentialFlytxt. All rights reserved. 30 June 201530 June 2015© Memoirs from the Past: Hilbert’s Program In 1900, David Hilbert, a very influential universal mathematician, announced a grand search for a complete and consistent set of axioms for all mathematics In 1931, Kurt Gödel announced his discovery of the Incompleteness Theorem: There will always be statements about the natural numbers that are true, but that are unprovable within the system Hilbert probably dedicated his life trying to prove his hypothesis, which Gödel proved cannot be true! However, Gödel’s work inspired Alan Turing and Alonzo Church, and in 1936, they mathematically defined “computation”
- 17. 17confidentialFlytxt. All rights reserved. 30 June 201530 June 2015© The future AI platform is a network! Courtesy: Maulik Kamdar, Stanford University
- 18. 18confidentialFlytxt. All rights reserved. 30 June 201530 June 2015© Future AI agents AI agents will compute; with data that gets generated on many devices 2025: 100 billion connected devices, 175 zeta bytes of data per year (Huawei) Data volumes will grow faster than any network or computer can be sized How will you scale the AI of tomorrow?
- 19. 19confidentialFlytxt. All rights reserved. 30 June 201530 June 2015© Practical AI: Moving data, moving code Code must meet data to compute – code moves and/or data does, across a (wireless) network History: All data moved to where the code was Near past: Parallel and distributed computation – partition code & data Now: (approximately) Move code to where the data is (Hadoop etc) Future: Determine the code-data match and optimize movement? • Is there is a computational model for this?
- 20. 20confidentialFlytxt. All rights reserved. 30 June 201530 June 2015© Physics, Networks, Computation – immutable laws Energy dissipation in radiation (Gauss’s / Coulomb’s Laws) • Low energy reception implies higher decoding error (Shannon’s Limit) • How fast can memory-to-memory transfers happen? Capacity of a wireless network is constrained by interference (e.g. see Gupta & Kumar, 2000) • Spectrum (# channels) available will remain finite • Channel allocations will be dynamic, but how fast can two interfering pairs find free channels? Are there limits to local computation? (e.g. see works by Ning Xie, Shai Vardi) • Moving code or data implies “local” processing • How much AI can be computed, and at what cost?
- 21. 21confidentialFlytxt. All rights reserved. 30 June 201530 June 2015© Example: High-dimension Clustering Basic machine learning algorithm to group nodes (users, people, devices) by state (behavior) • Each node produces a vector describing current state • Nodes are clustered together by some measure of vector similarity “Moving code” distributed implementations available today (on Hadoop/Spark) Future: Rate of change of state will outpace speeds of computation and communication Is the solution hierarchical, is the paradigm divide and conquer? How will network & algorithm design and implementation change? • Can all clustering problems be solved “locally”?
- 22. 22confidentialFlytxt. All rights reserved. 30 June 201530 June 2015© Discussion summary Big Data Analytics and Intelligent Applications • Build for customer value, build simple solutions State of the Art • Practical AI and Data Sciences What does the future guarantee? • Need to scale AI compute: Data generation rates faster than compute / communication rates
- 23. 23confidentialFlytxt. All rights reserved. 30 June 201530 June 2015© Thank You www.flytxt.com