EDR-8202 Statistics IIWeek 2 Assignment Worksheet C.docx
- 1. EDR-8202: Statistics II Week 2 Assignment Worksheet: Conduct Hypothesis Testing You will now have the opportunity to become re-acquainted with hypothesis testing concepts, and the use of SPSS to review some of the basic statistical concepts learned and to enhance APA format when presenting statistical results. The following small data set is from a study conducted within a single middle school. Fundamentally, this study is a comparison of the differences between male and female teachers in personal Confidence Scores and was conducted to determine if a relationship exists between the number of Years of Experience and Confidence Scores. The researcher wants to examine if years of experience could be used as a predictor of confidence scores and if gender could also impact confidence scores. Sex Years of Experience Confidence Scores Male 15 110 Male 3 117 Female
- 3. 2 103 Female 21 125 Male 28 115 Female 9 115 Male 5 110 Female 3 110 Using scholarly writing and proper APA format, complete, and then submit the following: I. Please answer the following questions using the readings assigned for this week. 1. What is hypothesis testing? 2. Discuss the following statement: “The sample must be representative of the population.” In your discussion, please include the concepts of generalizability and sampling procedures. 3. In hypothesis testing, there are some errors that could be committed if the wrong decision is made regarding the null hypothesis. These errors are Type I and Type II errors. Please describe each one of them, and provide an example when you are committing a Type I and Type II error.
- 4. 4. Discuss each of the three main assumptions for parametric statistics. Be sure your discussion includes the variables that will be tested for the assumptions, the independent or dependent variable. a. Normality b. Homogeneity of variance (please provide an example of an analysis where this assumption is important) c. Independence II.Please open the file used in assignment 1 (Confidence scores and years of experience), or type the data provided in the table above in SPSS. Once the dataset is open or the data is typed, proceed to answer questions 4 and 5. 5. Conduct the appropriate test (Kolmogorov-Smirnov or Shapiro-Wilk’s) for normality, and provide the results, and a brief discussion of the results below (test used, test results, p- value, etc.). Be sure to discuss if the assumption of normality was met or violated. 6. Conduct the test homogeneity of variance, and provide the results, and a brief discussion of the results below (test used, test results, p-value, etc.). Be sure to discuss if the assumption of normality was met or violated. iii Preface xxv
- 5. About the Authors xxxiv PART I Introduction to Analytics and AI 1 Chapter 1 Overview of Business Intelligence, Analytics, Data Science, and Artificial Intelligence: Systems for Decision Support 2 Chapter 2 Artificial Intelligence: Concepts, Drivers, Major Technologies, and Business Applications 73 Chapter 3 Nature of Data, Statistical Modeling, and Visualization 117 PART II Predictive Analytics/Machine Learning 193 Chapter 4 Data Mining Process, Methods, and Algorithms 194 Chapter 5 Machine-Learning Techniques for Predictive Analytics 251 Chapter 6 Deep Learning and Cognitive Computing 315 Chapter 7 Text Mining, Sentiment Analysis, and Social Analytics 388 PART III Prescriptive Analytics and Big Data 459 Chapter 8 Prescriptive Analytics: Optimization and Simulation 460 Chapter 9 Big Data, Cloud Computing, and Location Analytics: Concepts and Tools 509 PART IV Robotics, Social Networks, AI and IoT 579 Chapter 10 Robotics: Industrial and Consumer Applications 580
- 6. Chapter 11 Group Decision Making, Collaborative Systems, and AI Support 610 Chapter 12 Knowledge Systems: Expert Systems, Recommenders, Chatbots, Virtual Personal Assistants, and Robo Advisors 648 Chapter 13 The Internet of Things as a Platform for Intelligent Applications 687 PART V Caveats of Analytics and AI 725 Chapter 14 Implementation Issues: From Ethics and Privacy to Organizational and Societal Impacts 726 Glossary 770 Index 785 BRIEF CONTENTS A01_SHAR2016_11_SE_FM.indd 3 21/12/18 1:43 PM iv CONTENTS Preface xxv About the Authors xxxiv PART I Introduction to Analytics and AI 1
- 7. Chapter 1 Overview of Business Intelligence, Analytics, Data Science, and Artificial Intelligence: Systems for Decision Support 2 1.1 Opening Vignette: How Intelligent Systems Work for KONE Elevators and Escalators Company 3 1.2 Changing Business Environments and Evolving Needs for Decision Support and Analytics 5 Decision-Making Process 6 The Influence of the External and Internal Environments on the Process 6 Data and Its Analysis in Decision Making 7 Technologies for Data Analysis and Decision Support 7 1.3 Decision-Making Processes and Computerized Decision Support Framework 9 Simon’s Process: Intelligence, Design, and Choice 9 The Intelligence Phase: Problem (or Opportunity) Identification 10 0 APPLICATION CASE 1.1 Making Elevators Go Faster! 11 The Design Phase 12 The Choice Phase 13 The Implementation Phase 13 The Classical Decision Support System Framework 14
- 8. A DSS Application 16 Components of a Decision Support System 18 The Data Management Subsystem 18 The Model Management Subsystem 19 0 APPLICATION CASE 1.2 SNAP DSS Helps OneNet Make Telecommunications Rate Decisions 20 The User Interface Subsystem 20 The Knowledge-Based Management Subsystem 21 1.4 Evolution of Computerized Decision Support to Business Intelligence/Analytics/Data Science 22 A Framework for Business Intelligence 25 The Architecture of BI 25 The Origins and Drivers of BI 26 Data Warehouse as a Foundation for Business Intelligence 27 Transaction Processing versus Analytic Processing 27 A Multimedia Exercise in Business Intelligence 28 A01_SHAR2016_11_SE_FM.indd 4 21/12/18 1:43 PM Contents v
- 9. 1.5 Analytics Overview 30 Descriptive Analytics 32 0 APPLICATION CASE 1.3 Silvaris Increases Business with Visual Analysis and Real-Time Reporting Capabilities 32 0 APPLICATION CASE 1.4 Siemens Reduces Cost with the Use of Data Visualization 33 Predictive Analytics 33 0 APPLICATION CASE 1.5 Analyzing Athletic Injuries 34 Prescriptive Analytics 34 0 APPLICATION CASE 1.6 A Specialty Steel Bar Company Uses Analytics to Determine Available-to-Promise Dates 35 1.6 Analytics Examples in Selected Domains 38 Sports Analytics—An Exciting Frontier for Learning and Understanding Applications of Analytics 38 Analytics Applications in Healthcare—Humana Examples 43 0 APPLICATION CASE 1.7 Image Analysis Helps Estimate Plant Cover 50 1.7 Artificial Intelligence Overview 52 What Is Artificial Intelligence? 52
- 10. The Major Benefits of AI 52 The Landscape of AI 52 0 APPLICATION CASE 1.8 AI Increases Passengers’ Comfort and Security in Airports and Borders 54 The Three Flavors of AI Decisions 55 Autonomous AI 55 Societal Impacts 56 0 APPLICATION CASE 1.9 Robots Took the Job of Camel- Racing Jockeys for Societal Benefits 58 1.8 Convergence of Analytics and AI 59 Major Differences between Analytics and AI 59 Why Combine Intelligent Systems? 60 How Convergence Can Help? 60 Big Data Is Empowering AI Technologies 60 The Convergence of AI and the IoT 61 The Convergence with Blockchain and Other Technologies 62 0 APPLICATION CASE 1.10 Amazon Go Is Open for Business 62 IBM and Microsoft Support for Intelligent Systems Convergence 63
- 11. 1.9 Overview of the Analytics Ecosystem 63 1.10 Plan of the Book 65 1.11 Resources, Links, and the Teradata University Network Connection 66 Resources and Links 66 Vendors, Products, and Demos 66 Periodicals 67 The Teradata University Network Connection 67 A01_SHAR2016_11_SE_FM.indd 5 21/12/18 1:43 PM vi Contents The Book’s Web Site 67 Chapter Highlights 67 • Key Terms 68 Questions for Discussion 68 • Exercises 69 References 70 Chapter 2 Artificial Intelligence: Concepts, Drivers, Major Technologies, and Business Applications 73 2.1 Opening Vignette: INRIX Solves Transportation Problems 74 2.2 Introduction to Artificial Intelligence 76
- 12. Definitions 76 Major Characteristics of AI Machines 77 Major Elements of AI 77 AI Applications 78 Major Goals of AI 78 Drivers of AI 79 Benefits of AI 79 Some Limitations of AI Machines 81 Three Flavors of AI Decisions 81 Artificial Brain 82 2.3 Human and Computer Intelligence 83 What Is Intelligence? 83 How Intelligent Is AI? 84 Measuring AI 85 0 APPLICATION CASE 2.1 How Smart Can a Vacuum Cleaner Be? 86 2.4 Major AI Technologies and Some Derivatives 87 Intelligent Agents 87 Machine Learning 88
- 13. 0 APPLICATION CASE 2.2 How Machine Learning Is Improving Work in Business 89 Machine and Computer Vision 90 Robotic Systems 91 Natural Language Processing 92 Knowledge and Expert Systems and Recommenders 93 Chatbots 94 Emerging AI Technologies 94 2.5 AI Support for Decision Making 95 Some Issues and Factors in Using AI in Decision Making 96 AI Support of the Decision-Making Process 96 Automated Decision Making 97 0 APPLICATION CASE 2.3 How Companies Solve Real- World Problems Using Google’s Machine-Learning Tools 97 Conclusion 98 A01_SHAR2016_11_SE_FM.indd 6 21/12/18 1:43 PM Contents vii
- 14. 2.6 AI Applications in Accounting 99 AI in Accounting: An Overview 99 AI in Big Accounting Companies 100 Accounting Applications in Small Firms 100 0 APPLICATION CASE 2.4 How EY, Deloitte, and PwC Are Using AI 100 Job of Accountants 101 2.7 AI Applications in Financial Services 101 AI Activities in Financial Services 101 AI in Banking: An Overview 101 Illustrative AI Applications in Banking 102 Insurance Services 103 0 APPLICATION CASE 2.5 US Bank Customer Recognition and Services 104 2.8 AI in Human Resource Management (HRM) 105 AI in HRM: An Overview 105 AI in Onboarding 105 0 APPLICATION CASE 2.6 How Alexander Mann
- 15. Solution s (AMS) Is Using AI to Support the Recruiting Process 106 Introducing AI to HRM Operations 106 2.9 AI in Marketing, Advertising, and CRM 107 Overview of Major Applications 107 AI Marketing Assistants in Action 108 Customer Experiences and CRM 108 0 APPLICATION CASE 2.7 Kraft Foods Uses AI for Marketing and CRM 109 Other Uses of AI in Marketing 110 2.10 AI Applications in Production-Operation
- 16. Management (POM) 110 AI in Manufacturing 110 Implementation Model 111 Intelligent Factories 111 Logistics and Transportation 112 Chapter Highlights 112 • Key Terms 113 Questions for Discussion 113 • Exercises 114 References 114 Chapter 3 Nature of Data, Statistical Modeling, and Visualization 117 3.1 Opening Vignette: SiriusXM Attracts and Engages a New Generation of Radio Consumers with Data-Driven Marketing 118 3.2 Nature of Data 121 3.3 Simple Taxonomy of Data 125
- 17. 0 APPLICATION CASE 3.1 Verizon Answers the Call for Innovation: The Nation’s Largest Network Provider uses Advanced Analytics to Bring the Future to its Customers 127 A01_SHAR2016_11_SE_FM.indd 7 21/12/18 1:43 PM viii Contents 3.4 Art and Science of Data Preprocessing 129 0 APPLICATION CASE 3.2 Improving Student Retention with Data-Driven Analytics 133 3.5 Statistical Modeling for Business Analytics 139 Descriptive Statistics for Descriptive Analytics 140 Measures of Centrality Tendency (Also Called Measures of Location or Centrality) 140
- 18. Arithmetic Mean 140 Median 141 Mode 141 Measures of Dispersion (Also Called Measures of Spread or Decentrality) 142 Range 142 Variance 142 Standard Deviation 143 Mean Absolute Deviation 143 Quartiles and Interquartile Range 143 Box-and-Whiskers Plot 143 Shape of a Distribution 145 0 APPLICATION CASE 3.3 Town of Cary Uses Analytics to Analyze Data
- 19. from Sensors, Assess Demand, and Detect Problems 150 3.6 Regression Modeling for Inferential Statistics 151 How Do We Develop the Linear Regression Model? 152 How Do We Know If the Model Is Good Enough? 153 What Are the Most Important Assumptions in Linear Regression? 154 Logistic Regression 155 Time-Series Forecasting 156 0 APPLICATION CASE 3.4 Predicting NCAA Bowl Game Outcomes 157 3.7 Business Reporting 163 0 APPLICATION CASE 3.5 Flood of Paper Ends at FEMA 165 3.8 Data Visualization 166 Brief History of Data Visualization 167 0 APPLICATION CASE 3.6 Macfarlan Smith Improves
- 20. Operational Performance Insight with Tableau Online 169 3.9 Different Types of Charts and Graphs 171 Basic Charts and Graphs 171 Specialized Charts and Graphs 172 Which Chart or Graph Should You Use? 174 3.10 Emergence of Visual Analytics 176 Visual Analytics 178 High-Powered Visual Analytics Environments 180 3.11 Information Dashboards 182 A01_SHAR2016_11_SE_FM.indd 8 21/12/18 1:43 PM Contents ix
- 21. 0 APPLICATION CASE 3.7 Dallas Cowboys Score Big with Tableau and Teknion 184 Dashboard Design 184 0 APPLICATION CASE 3.8 Visual Analytics Helps Energy Supplier Make Better Connections 185 What to Look for in a Dashboard 186 Best Practices in Dashboard Design 187 Benchmark Key Performance Indicators with Industry Standards 187 Wrap the Dashboard Metrics with Contextual Metadata 187 Validate the Dashboard Design by a Usability Specialist 187 Prioritize and Rank Alerts/Exceptions Streamed to the Dashboard 188
- 22. Enrich the Dashboard with Business-User Comments 188 Present Information in Three Different Levels 188 Pick the Right Visual Construct Using Dashboard Design Principles 188 Provide for Guided Analytics 188 Chapter Highlights 188 • Key Terms 189 Questions for Discussion 190 • Exercises 190 References 192 PART II Predictive Analytics/Machine Learning 193 Chapter 4 Data Mining Process, Methods, and Algorithms 194 4.1 Opening Vignette: Miami-Dade Police Department Is Using Predictive Analytics to Foresee and Fight Crime 195 4.2 Data Mining Concepts 198 0 APPLICATION CASE 4.1 Visa Is Enhancing the Customer
- 23. Experience while Reducing Fraud with Predictive Analytics and Data Mining 199 Definitions, Characteristics, and Benefits 201 How Data Mining Works 202 0 APPLICATION CASE 4.2 American Honda Uses Advanced Analytics to Improve Warranty Claims 203 Data Mining Versus Statistics 208 4.3 Data Mining Applications 208 0 APPLICATION CASE 4.3 Predictive Analytic and Data Mining Help Stop Terrorist Funding 210 4.4 Data Mining Process 211 Step 1: Business Understanding 212 Step 2: Data Understanding 212
- 24. Step 3: Data Preparation 213 Step 4: Model Building 214 0 APPLICATION CASE 4.4 Data Mining Helps in Cancer Research 214 Step 5: Testing and Evaluation 217 A01_SHAR2016_11_SE_FM.indd 9 21/12/18 1:43 PM x Contents Step 6: Deployment 217 Other Data Mining Standardized Processes and Methodologies 217 4.5 Data Mining Methods 220 Classification 220 Estimating the True Accuracy of Classification Models 221
- 25. Estimating the Relative Importance of Predictor Variables 224 Cluster Analysis for Data Mining 228 0 APPLICATION CASE 4.5 Influence Health Uses Advanced Predictive Analytics to Focus on the Factors That Really Influence People’s Healthcare Decisions 229 Association Rule Mining 232 4.6 Data Mining Software Tools 236 0 APPLICATION CASE 4.6 Data Mining goes to Hollywood: Predicting Financial Success of Movies 239 4.7 Data Mining Privacy Issues, Myths, and Blunders 242 0 APPLICATION CASE 4.7 Predicting Customer Buying Patterns—The Target Story 243 Data Mining Myths and Blunders 244
- 26. Chapter Highlights 246 • Key Terms 247 Questions for Discussion 247 • Exercises 248 References 250 Chapter 5 Machine-Learning Techniques for Predictive Analytics 251 5.1 Opening Vignette: Predictive Modeling Helps Better Understand and Manage Complex Medical Procedures 252 5.2 Basic Concepts of Neural Networks 255 Biological versus Artificial Neural Networks 256 0 APPLICATION CASE 5.1 Neural Networks are Helping to Save Lives in the Mining Industry 258 5.3 Neural Network Architectures 259 Kohonen’s Self-Organizing Feature Maps 259
- 27. Hopfield Networks 260 0 APPLICATION CASE 5.2 Predictive Modeling Is Powering the Power Generators 261 5.4 Support Vector Machines 263 0 APPLICATION CASE 5.3 Identifying Injury Severity Risk Factors in Vehicle Crashes with Predictive Analytics 264 Mathematical Formulation of SVM 269 Primal Form 269 Dual Form 269 Soft Margin 270 Nonlinear Classification 270 Kernel Trick 271 A01_SHAR2016_11_SE_FM.indd 10 21/12/18 1:43 PM
- 28. Contents xi 5.5 Process-Based Approach to the Use of SVM 271 Support Vector Machines versus Artificial Neural Networks 273 5.6 Nearest Neighbor Method for Prediction 274 Similarity Measure: The Distance Metric 275 Parameter Selection 275 0 APPLICATION CASE 5.4 Efficient Image Recognition and Categorization with knn 277 5.7 Naïve Bayes Method for Classification 278 Bayes Theorem 279 Naïve Bayes Classifier 279 Process of Developing a Naïve Bayes Classifier 280
- 29. Testing Phase 281 0 APPLICATION CASE 5.5 Predicting Disease Progress in Crohn’s Disease Patients: A Comparison of Analytics Methods 282 5.8 Bayesian Networks 287 How Does BN Work? 287 How Can BN Be Constructed? 288 5.9 Ensemble Modeling 293 Motivation—Why Do We Need to Use Ensembles? 293 Different Types of Ensembles 295 Bagging 296 Boosting 298 Variants of Bagging and Boosting 299
- 30. Stacking 300 Information Fusion 300 Summary—Ensembles are not Perfect! 301 0 APPLICATION CASE 5.6 To Imprison or Not to Imprison: A Predictive Analytics-Based Decision Support System for Drug Courts 304 Chapter Highlights 306 • Key Terms 308 Questions for Discussion 308 • Exercises 309 Internet Exercises 312 • References 313 Chapter 6 Deep Learning and Cognitive Computing 315 6.1 Opening Vignette: Fighting Fraud with Deep Learning and Artificial Intelligence 316 6.2 Introduction to Deep Learning 320 0 APPLICATION CASE 6.1 Finding the Next Football Star with
- 31. Artificial Intelligence 323 6.3 Basics of “Shallow” Neural Networks 325 0 APPLICATION CASE 6.2 Gaming Companies Use Data Analytics to Score Points with Players 328 0 APPLICATION CASE 6.3 Artificial Intelligence Helps Protect Animals from Extinction 333 A01_SHAR2016_11_SE_FM.indd 11 21/12/18 1:43 PM xii Contents 6.4 Process of Developing Neural Network–Based Systems 334 Learning Process in ANN 335 Backpropagation for ANN Training 336
- 32. 6.5 Illuminating the Black Box of ANN 340 0 APPLICATION CASE 6.4 Sensitivity Analysis Reveals Injury Severity Factors in Traffic Accidents 341 6.6 Deep Neural Networks 343 Feedforward Multilayer Perceptron (MLP)-Type Deep Networks 343 Impact of Random Weights in Deep MLP 344 More Hidden Layers versus More Neurons? 345 0 APPLICATION CASE 6.5 Georgia DOT Variable Speed Limit Analytics Help Solve Traffic Congestions 346 6.7 Convolutional Neural Networks 349 Convolution Function 349 Pooling 352
- 33. Image Processing Using Convolutional Networks 353 0 APPLICATION CASE 6.6 From Image Recognition to Face Recognition 356 Text Processing Using Convolutional Networks 357 6.8 Recurrent Networks and Long Short-Term Memory Networks 360 0 APPLICATION CASE 6.7 Deliver Innovation by Understanding Customer Sentiments 363 LSTM Networks Applications 365 6.9 Computer Frameworks for Implementation of Deep Learning 368 Torch 368 Caffe 368 TensorFlow 369
- 34. Theano 369 Keras: An Application Programming Interface 370 6.10 Cognitive Computing 370 How Does Cognitive Computing Work? 371 How Does Cognitive Computing Differ from AI? 372 Cognitive Search 374 IBM Watson: Analytics at Its Best 375 0 APPLICATION CASE 6.8 IBM Watson Competes against the Best at Jeopardy! 376 How Does Watson Do It? 377 What Is the Future for Watson? 377 Chapter Highlights 381 • Key Terms 383 Questions for Discussion 383 • Exercises 384
- 35. References 385 A01_SHAR2016_11_SE_FM.indd 12 21/12/18 1:43 PM Contents xiii Chapter 7 Text Mining, Sentiment Analysis, and Social Analytics 388 7.1 Opening Vignette: Amadori Group Converts Consumer Sentiments into Near-Real-Time Sales 389 7.2 Text Analytics and Text Mining Overview 392 0 APPLICATION CASE 7.1 Netflix: Using Big Data to Drive Big Engagement: Unlocking the Power of Analytics to Drive Content and Consumer Insight 395 7.3 Natural Language Processing (NLP) 397 0 APPLICATION CASE 7.2 AMC Networks Is Using Analytics to
- 36. Capture New Viewers, Predict Ratings, and Add Value for Advertisers in a Multichannel World 399 7.4 Text Mining Applications 402 Marketing Applications 403 Security Applications 403 Biomedical Applications 404 0 APPLICATION CASE 7.3 Mining for Lies 404 Academic Applications 407 0 APPLICATION CASE 7.4 The Magic Behind the Magic: Instant Access to Information Helps the Orlando Magic Up their Game and the Fan’s Experience 408 7.5 Text Mining Process 410 Task 1: Establish the Corpus 410
- 37. Task 2: Create the Term–Document Matrix 411 Task 3: Extract the Knowledge 413 0 APPLICATION CASE 7.5 Research Literature Survey with Text Mining 415 7.6 Sentiment Analysis 418 0 APPLICATION CASE 7.6 Creating a Unique Digital Experience to Capture Moments That Matter at Wimbledon 419 Sentiment Analysis Applications 422 Sentiment Analysis Process 424 Methods for Polarity Identification 426 Using a Lexicon 426 Using a Collection of Training Documents 427 Identifying Semantic Orientation of Sentences and Phrases 428
- 38. Identifying Semantic Orientation of Documents 428 7.7 Web Mining Overview 429 Web Content and Web Structure Mining 431 7.8 Search Engines 433 Anatomy of a Search Engine 434 1. Development Cycle 434 2. Response Cycle 435 Search Engine Optimization 436 Methods for Search Engine Optimization 437 A01_SHAR2016_11_SE_FM.indd 13 21/12/18 1:43 PM xiv Contents
- 39. 0 APPLICATION CASE 7.7 Delivering Individualized Content and Driving Digital Engagement: How Barbour Collected More Than 49,000 New Leads in One Month with Teradata Interactive 439 7.9 Web Usage Mining (Web Analytics) 441 Web Analytics Technologies 441 Web Analytics Metrics 442 Web Site Usability 442 Traffic Sources 443 Visitor Profiles 444 Conversion Statistics 444 7.10 Social Analytics 446 Social Network Analysis 446 Social Network Analysis Metrics 447
- 40. 0 APPLICATION CASE 7.8 Tito’s Vodka Establishes Brand Loyalty with an Authentic Social Strategy 447 Connections 450 Distributions 450 Segmentation 451 Social Media Analytics 451 How Do People Use Social Media? 452 Measuring the Social Media Impact 453 Best Practices in Social Media Analytics 453 Chapter Highlights 455 • Key Terms 456 Questions for Discussion 456 • Exercises 456 References 457 PART III Prescriptive Analytics and Big Data 459
- 41. Chapter 8 Prescriptive Analytics: Optimization and Simulation 460 8.1 Opening Vignette: School District of Philadelphia Uses Prescriptive Analytics to Find Optimal