Statistical Inference for development statistical model.pptx
- 2. Statistical Inference Scenario • Data Generation • The world we live in is complex, random, and uncertain. At the same time, it’s one big data-generating machine. • As we commute to work on subways and in cars, as our blood moves through our bodies, as we’re shopping, emailing, procrastinating at work by browsing the Internet and watching the stock market, as we’re building things, eating things, talking to our friends and family about things, while factories are producing products, this all at least potentially produces data.
- 3. Data Collection and Sampling methods • Imagine spending 24 hours looking out the window, and for every minute, counting and recording the number of people who pass by. Or gathering up everyone who lives within a mile of your house and making them tell you how many email messages they receive every day for the next year. Imagine heading over to your local hospital and rummaging around in the blood samples looking for patterns in the DNA. That all sounded creepy, but it wasn’t supposed to. The point here is that the processes in our lives are actually data-generating processes.
- 4. Roles of data scientist • We’d like ways to describe, understand, and make sense of these processes, in part because as scientists we just want to understand the world better, but many times, understanding these processes is part of the solution to problems we’re trying to solve. • Data represents the traces of the real-world processes, and exactly which traces we gather are decided by our data collection or sampling method. • You, the data scientist, the observer, are turning the world into data.
- 5. Challenges in Data interpretation • Once you have all this data, you have somehow captured the world, or certain traces of the world. But you can’t go walking around with a huge Excel spreadsheet or database of millions of transactions and look at it and, with a snap of a finger, understand the world and process that generated it. • Simplification through Mathematical models • So you need a new idea, and that’s to simplify those captured traces into something more comprehensible, to something that somehow captures it all in a much more concise way, and that something could be mathematical models or functions of the data, known as statistical estimators.
- 6. Statistical Inference Def and importance • This overall process of going from the world to the data, and then from the data back to the world, is the field of statistical inference. • More precisely, statistical inference is the discipline that concerns itself with the development of procedures, methods, and theorems that allow us to extract meaning and information from data that has been generated by stochastic (random) processes. Statistical inference is the process of drawing conclusions or making predictions about a population based on data from a sample. In other words, it is a set of methods used to make inferences about an entire population based on a smaller subset of the population (the sample).
- 7. Applications Statistical Inference • Statistical inference is widely used in many fields, including science, engineering, economics, and social sciences, among others. It plays a critical role in making informed decisions and drawing meaningful conclusions from data.
- 8. Population and Sample • In classical statistical literature, a distinction is made between the population and the sample. The word population immediately makes us think of the entire Pakistani population of 220 million people, or the entire world’s population of 7 billion people. • But put that image out of your head, because in statistical inference population isn’t used to simply describe only people. It could be any set of objects or units, such as tweets or photographs or stars. • If we could measure the characteristics or extract characteristics of all those objects, we’d have a complete set of observations, and the convention is to use N to represent the total number of observations in the population.
- 9. Population and Sample • Suppose your population was all emails sent last year by employees at a huge corporation, BigCorp. Then a single observation could be a list of things: the sender’s name, the list of recipients, date sent, text of email, number of characters in the email, number of sentences in the email, number of verbs in the email, and the length of time until first reply. • When we take a sample, we take a subset of the units of size n in order to examine the observations to draw conclusions and make inferences about the population.
- 10. • In the BigCorp email example, you could make a list of all the employees and select 1/10th of those people at random and take all the email they ever sent, and that would be your sample.
- 11. What is a model? • Humans try to understand the world around them by representing it in different ways. • Architects capture attributes of buildings through blueprints and three-dimensional, scaled-down versions. • Molecular biologists capture protein structure with three- dimensional visualizations of the connections between amino acids. • Statisticians and data scientists capture the uncertainty and randomness of data-generating processes with mathematical functions that express the shape and structure of the data itself.
- 12. What is a model? • A model is our attempt to understand and represent the nature of reality through a particular lens, be it architectural, biological, or mathematical. • A model is an artificial construction where all extraneous detail has been removed or abstracted. Attention must always be paid to these abstracted details after a model has been analyzed to see what might have been overlooked.
- 13. Statistical Modeling • Statistical modeling is the use of mathematical models and statistical assumptions to generate sample data and make predictions about the real world. A statistical model is a collection of probability distributions on a set of all possible outcomes of an experiment. • Statistical modeling refers to the data science process of applying statistical analysis to datasets. A statistical model is a mathematical relationship between one or more random variables and other non- random variables. The application of statistical modeling to raw data helps data scientists approach data analysis in a strategic manner, providing intuitive visualizations that aid in identifying relationships between variables and making predictions.
- 14. Statistical Modeling • Common data sets for statistical analysis include Internet of Things (IoT) sensors, census data, public health data, social media data, imagery data, and other public sector data that benefit from real-world predictions.
- 15. Probability • Probability denotes the possibility of something happening. It is a mathematical concept that predicts how likely events are to occur. The probability values are expressed between 0 and 1. The definition of probability is the degree to which something is likely to occur. This fundamental theory of probability is also applied to probability distributions.
- 16. Probability Distribution • Probability distribution yields the possible outcomes for any random event. It is also defined based on the underlying sample space as a set of possible outcomes of any random experiment. These settings could be a set of real numbers or a set of vectors or a set of any entities. It is a part of probability and statistics. • A probability distribution is a table or an equation that links each possible value that a random variable can assume with its probability of occurrence.
- 18. Introduction to R • R is a programming language created by statisticians for statistics, specifically for working with data. • used by business analysts, data analysts, data scientists, and • scientists. • R is unique in that it is not general purpose. • statistical analysis and data visualization. • Academics, scientists, and researchers use it to analyze the results of experiments. • businesses of all sizes and in every industry use it to extract insights from the increasing amount of daily data they generate.