Data science
1 like910 views
Data science involves cleaning and analyzing large amounts of data to gain insights and make predictions. It requires expertise in mathematics, statistics, programming, and domain knowledge. While big data is hyped as providing definitive answers, data scientists know that all models are imperfect and data must be understood in context. Open source tools like R and Python are commonly used to explore, visualize, and draw conclusions from data.
![Expertise
!
Maths &
Stats
Hacking skills
!
!
!Danger
zone!
Machine
Learning
Research
Data
Science
[Drew Conway]](https://image.slidesharecdn.com/datascience-140513125920-phpapp02/85/Data-science-5-320.jpg)
![Maths
&
Stats
!
!
Hacking
skills
!
!
!
!
Evil
Expertise
danger
zone!
!
machine
learning
!
James Bond
villain
data
science
outside
committee
member
identity
thief
thesis
advisor
office
mate
NSA
[Joel Grus]](https://image.slidesharecdn.com/datascience-140513125920-phpapp02/85/Data-science-7-320.jpg)
![THE REALITY
• You’ll clean a lot of data.A LOT
• A lot of mathematics. Get over it
• Some days will be long. Get more coffee
• Not everything is about Big Data
• Most people don’t care about data
• Spend time finding the right questions
[David Coallier]](https://image.slidesharecdn.com/datascience-140513125920-phpapp02/85/Data-science-9-320.jpg)
![Data Scientific Method
[DJ Patil, J Elman]](https://image.slidesharecdn.com/datascience-140513125920-phpapp02/85/Data-science-14-320.jpg)
![[1,54,853,23,4,73,…] Have we seen 73?
min H(1,54,853,…) < H(73) ?
min H2(1,54,853,…) < H2(73) ?
min H3(1,54,853,…) < H3(73) ?](https://image.slidesharecdn.com/datascience-140513125920-phpapp02/85/Data-science-26-320.jpg)
![Nearest bar
Map
∀ , search graph
< ,{ ,distance} >
Input
graph
(node,label)
Shuffle/
Sort
by id
Reduce
< ,[{ ,distance},
{ ,distance}] >
-> min()
Output
< , >
< , >
marked graph](https://image.slidesharecdn.com/datascience-140513125920-phpapp02/85/Data-science-49-320.jpg)
Data science
- 3. WHAT IS DATA SCIENCE?
- 4. Hacking skills Maths & StatsExpertise
- 5. Expertise ! Maths & Stats Hacking skills ! ! !Danger zone! Machine Learning Research Data Science [Drew Conway]
- 7. Maths & Stats ! ! Hacking skills ! ! ! ! Evil Expertise danger zone! ! machine learning ! James Bond villain data science outside committee member identity thief thesis advisor office mate NSA [Joel Grus]
- 8. – David Coallier “Whenever you read about data science or data analysis, it’s about the ability to store petabytes of data, retrieve that data in nanoseconds, then turn it into a rainbow with a unicorn dancing on it.” THE HYPE – Harvard Business Review “Data Scientist: The Sexiest Job of the 21st Century”
- 9. THE REALITY • You’ll clean a lot of data.A LOT • A lot of mathematics. Get over it • Some days will be long. Get more coffee • Not everything is about Big Data • Most people don’t care about data • Spend time finding the right questions [David Coallier]
- 10. BIG DATA THE END OFTHEORY? – Chris Anderson,WIRED Out with every theory of human behavior, from linguistics to sociology.Who knows why people do what they do?The point is they do it… With enough data, the numbers speak for themselves.
- 11. All models are wrong. But some are useful.
- 12. All models are wrong, and increasingly you can succeed without them.
- 13. Big data is a step forward. But our problems are not lack of access to data, but understanding them.
- 14. Data Scientific Method [DJ Patil, J Elman]
- 15. START WITH A QUESTION Based on an observation
- 16. START WITH A QUESTIONWhat (just) happened? Why did it happen? What will/could happen next?
- 17. ANALYSE CURRENT DATA Create an Hypothesis
- 19. ANALYSE RESULTS Won’t be pretty, repeat
- 20. LET DATA FRAME THE CONVERSATION Data gives you the what Humans give you the why
- 21. LET DATA FRAME THE CONVERSATION Let the dataset change your mindset
- 22. CONVERSE • What data is missing? Where can we get it? • Automate data collection • Clean data, then clean it more • Visualize data: the brain sees • Merge various sources of information • Reformulate hypotheses • Reformulate questions
- 24. We can't solve problems at the same level of thinking with which we've created them…
- 25. Probabilistic algorithms When polynomial time is just too slow
- 26. [1,54,853,23,4,73,…] Have we seen 73? min H(1,54,853,…) < H(73) ? min H2(1,54,853,…) < H2(73) ? min H3(1,54,853,…) < H3(73) ?
- 27. MINHASHING Bloom filters Find similar documents, photos, …
- 28. MapReduce
- 29. map map map split reduce reduce reduce shuffle (remote read) read data(HDFS) data(HDFS) write readread worker nodes (local) worker nodes (local) split1split2splitn
- 37. Mapper <a,apple> <a’,slices> Reducer Input file Intermediate file (local) Output file
- 38. <p,pineapple> <p’,slices> <a,apple> <o,orange> <a’,slices> <o’,slices> Input file Output fileIntermediate file
- 39. split 0 split 1 split 2 1 mapper/split 1 reducer/key(set) shufflesplit
- 40. map map map split reduce reduce reduce shuffle (remote read) read data(HDFS) data(HDFS) write readread worker nodes (local) worker nodes (local) split1split2splitn
- 41. map map map split reduce reduce shuffle + parallel sort (remote read) read data(HDFS) data(HDFS) write readread split1split2splitn master node assigns map/reduce jobs reassigns if nodes fail
- 46. Nearest bar ? nearest within distance d? Input graph (node,label)
- 47. Nearest bar Map ∀ , search graph with radius d < ,{ ,distance} > Input graph (node,label)
- 48. Nearest bar Map ∀ , search graph < ,{ ,distance} > Shuffle/ Sort by id Input graph (node,label)
- 49. Nearest bar Map ∀ , search graph < ,{ ,distance} > Input graph (node,label) Shuffle/ Sort by id Reduce < ,[{ ,distance}, { ,distance}] > -> min() Output < , > < , > marked graph
- 50. Sensor data
- 52. NOISE
- 53. MATHS Convolution signal kernel convolution multiplied by 1:5 and delayed by 14 sample intervals. Evidently, we have just described in words the following definition of discrete convolution with a response function of finite duration M: .r s/j Á M=2 X kD M=2C1 sj k rk (13.1.1) If a discrete response function is nonzero only in some range M=2 < k Ä M=2, where M is a sufficiently large even integer, then the response function is called a finite impulse response (FIR), and its duration is M. (Notice that we are defining M as the number of nonzero values of rk; these values span a time interval of M 1 sampling times.) In most practical circumstances the case of finite M is the case of interest, either because the response really has a finite duration, or because we choose to truncate it at some point and approximate it by a finite-duration response function. The discrete convolution theorem is this: If a signal sj is periodic with period N, so that it is completely determined by the N values s0; : : : ; sN 1, then its discrete convolution with a response function of finite duration N is a member of the discrete Fourier transform pair, g h Á Z 1 1 g. /h.t / d (12 e that g h is a function in the time domain and that g h D h g. It turn the function g h is one member of a simple transform pair, g h ” G.f /H.f / convolution theorem (12.0 ther words, the Fourier transform of the convolution is just the product o vidual Fourier transforms. The correlation of two functions, denoted Corr.g; h/, is defined by Corr.g; h/ Á Z 1 1 g. C t/h. / d (12.0 correlation is a function of t, which is called the lag. It therefore lies in the ain, and it turns out to be one member of the transform pair: Corr.g; h/ ” G.f /H .f / correlation theorem (12.0
- 54. COMPLEXITY
- 55. MATHS,AGAIN scale space decomposition signal kernel 1 convolution 2 kernel 2 convolution 1
- 56. SCALE-SPACE
- 57. Baseline (σ64) Traffic Jams (σ16 -σ64) Slowdown (σ4 -σ16) Vehicles (σ0 -σ4) SCALE-SPACE DECOMPOSITION
- 59. Reduce Reduce Reduce Reduce Reduce Map Map Map MapMap Build windows Build windows Build windows Build windows Build windows Shuffle Convolute Convolute Convolute Convolute Convolute CONVOLUTION
- 60. CONVOLUTE-ADD Map (convolute with 0-padding) Reduce (add) Add values in overlapping regions 0 0 0 0 0 0 A A+B B B+C C
- 61. SEGMENTATION • You don’t need 100Hz data for everything • Approximate signal with linear segments • Key points: 0-crossings of 1st, 2nd, 3rd derivative • Maths: derivative of smoothed signal = convolution with derivative of kernel
- 62. 1st, 2nd,3rd degree derivatives signal convolution segmentation
- 64. VISUALIZATION
- 65. Twitter data
- 67. Geospacial data
- 68. OPEN SOURCETOOLS
- 69. modelling, testing, prototyping lubridate, zoo: dates, time series reshape2: reshape data ggplot2: visualize data RCurl, RJSONIO: find more data HMisc: miscellaneous DMwR, mlr: machine learning Forecast: time series forecasting garch: time series modelling quantmod: statistical financial trading xts: extensible time series igraph: study networks maptools: read and view maps R
- 70. scientific computing numpy: linear algebra scipy: optimization, signal/image processing, … scikits: toolkits for scipy scikit-learn: machine learning toolkit statsmodels: advanced statistic modelling matplotlib: plotting NLTK: natural language processing PyBrain: more machine learning PyMC: Bayesian inference Pattern:Web mining NetworkX: Study networks Pandas: easy-to-use data structures PYTHON