Shahid Lecture-2- MKAG1273

Statistical Hydrology
MAL1303/MKAG1273
Graphical Data Analysis
Dr. Shamsuddin Shahid
Associate Professor
Department of Hydraulics and Hydrology
Faculty of Civil Engineering
Room No.: M46-332;
Phone: 07-5531624; Mobile: 0182051586
Email: sshahid@utm.my
11/23/2015 Shamsuddin Shahid, FKA, UTM
You created this PDF from an application that is not licensed to print to novaPDF printer (http://www.novapdf.com)

• One measure of absolute skewness is difference between
mean and mode. A measure of such would not be true
meaningful because it depends of the units of
measurement.
• The simplest measure of skewness is the Pearson’s
coefficient of skewness:
Skewness
deviationStandard
Mode-Mean
skewnessoftcoefficiensPearson' 

• Skewness coefficient varies between -3.o to +3.0.
• There is no acceptable range of skewness to measure the
distribution of data.
• Some people says that rule of thumb is -1 to +1 being
acceptable (-2 to +2 is often used too) for normal
distribution.
Skewness

 Kurtosis measures how peaked the histogram is
 The kurtosis of a normal distribution is 0 (zero)
 Kurtosis characterizes the relative peakedness or flatness of a
distribution compared to the normal distribution
3
)(
4
4




ns
xx
kurtosis
n
i
i
Kurtosis

• Platykurtic– When the kurtosis < 0, the frequencies throughout the
curve are closer to be equal (i.e., the curve is more flat and wide).
Thus, negative kurtosis indicates a relatively flat distribution
• Leptokurtic– When the kurtosis > 0, there are high frequencies in
only a small part of the curve (i.e, the curve is more peaked). Thus,
positive kurtosis indicates a relatively peaked distribution
• Kurtosis is based on the size of a distribution's tails. Negative
kurtosis (platykurtic) – distributions with short tails. Positive
kurtosis (leptokurtic) – distributions with relatively long tails
leptokurticplatykurtic
Kurtosis

Coefficient of Kurtosis is the most important measure of kurtosis
which is based on the second and fourth moments :
Kurtosis
2
2
4
2


 
N
xxf
2
2
)( 

N
xxf
4
4
)( 

Where,
Second Momentum
Fourth Momentum
• If 2 -3 > 0, the distribution is leptokurtic.
• If , If 2 -3 < 0 the distribution is platykurtic.
• If , 2 -3 = 0 the distribution is mesokurtic (normal).
A kurtosis value of
+/-1 is considered
very good for most
uses, but +/-2 is
also usually
acceptable.

Chebyshev’s theorem
According to Chebyshev’s theorem,
At least of the measurements will fall within
[Mean – (k-1)*SD] to [Mean + (k-1)*SD], where K = 2
Empirical rule
Give a set of n measurements possessing a mound-shaped histogram,
then
the interval X  s contains approximately 68% of the measurements
the interval X  2s contains approximately 95% of the measurements
the interval X  3s contains approximately 99.7% of the measurements.
Chebyshev’s Rule

Empirical rule
Give a set of n measurements possessing a mound-shaped histogram, then
the interval X  s contains approximately 68% of the measurements
the interval X  2s contains approximately 95% of the measurements
the interval X  3s contains approximately 99.7% of the measurements.
Chebyshev’s Rule

Outlier
An outlier is an observation that lies an abnormal distance from other
values in a random sample from a population.
Outlier or an outlying observation, is one that appears to deviate
markedly from other members of the sample in which it occurs.
Outliers can have many anomalous causes:
• A physical apparatus for taking measurements may have suffered a
transient malfunction.
• There may have been an error in data transmission or transcription.
• Outliers arise due to changes in system behaviour, fraudulent
behaviour, human error, instrument error
• simply through natural deviations in populations.
• A sample may have been contaminated with elements from outside
the population being examined.

Identification of Outliers
There is no rigid mathematical definition of what constitutes an outlier.
Determining whether or not an observation is an outlier is ultimately a
subjective exercise.
Type 1 - Determine the outliers with no prior knowledge of the data. This is
essentially a learning approach. The approach processes the data as a
static distribution, pinpoints the most remote points, and flags them as
potential outliers.
Type 2 – Using model-based methods which assume that the data are from
a normal distribution, and identify observations which are deemed
"unlikely" based on mean and standard deviation.
• Chauvenet's criterion
• Grubbs' test
• Dixon's Q test

Chauvenet's criterion
• A value is measured experimentally in several trials as 9, 10, 10, 10,
11, and 50.
• The mean is 16.7 and the standard deviation 16.34.
• Value 50 differs from 16.7 by 33.3, slightly more than two standard
deviations.
• The probability of taking data more than two standard deviations from
the mean is roughly 0.05.
• Six measurements were taken, so the statistic value (data size
multiplied by the probability) is 0.05×6 = 0.3.
• Because 0.3 < 0.5, according to Chauvenet's criterion, the measured
value of 50 should be discarded (leaving a new mean of 10, with
standard deviation 0.7).

Grubbs' test detects one outlier at a time.
Gcalculated > Gtable then reject the questionable point.
Grubbs' test
Example: 9, 10, 10, 10, 11, and 50

Grubbs' test

To apply a Q test, arrange the data in order of increasing values
and calculate Q as defined:
Where gap is the absolute difference between the outlier in
question and the closest number to it.
If Qcalculated > Qtable then reject the questionable point.
Dixon's Q test, or simply the Q test
Example: 9, 10, 10, 10, 11, and 50

Dixon's Q test

Common Characteristics of Water Resources Data
1. A lower bound of zero. No negative values are possible.
2. Presence of 'outliers‘ regularly occur, specially outliers on the
high side are more common in water resources.
3. Non-normal distribution of data
4. Positive skewness is common.
5. Data reported only as below or above some threshold
(censored data). Examples include concentrations below one or
more detection limits, annual flood above a level, etc.
6. Seasonal patterns. Values tend to be higher or lower in certain
seasons of the year.
7. Positive autocorrelation. Consecutive observations tend to be
strongly correlated with each other. High values tend to follow
high values and low values tend to follow low values.
8. Dependence on other uncontrolled variables. Water discharge
from a well highly depends on hydraulic conductivity, sediment
grain size, or some other variable.

1. Data type
2. Mean, median and Mode
3. Data quality control
4. Outliers
5. Nature of Hydrological Data

General Characteristics of Water Resources Data
1. A lower bound of zero. No negative values are possible.
2. Presence of 'outliers‘ regularly occur, specially outliers on the
high side are more common in water resources.
3. Non-normal distribution of data
4. Positive skewness is common.
5. Data reported only as below or above some threshold
(censored data). Examples include concentrations below one or
more detection limits, annual flood above a level, etc.
6. Seasonal patterns. Values tend to be higher or lower in certain
seasons of the year.
7. Positive autocorrelation. Consecutive observations tend to be
strongly correlated with each other. High values tend to follow
high values and low values tend to follow low values.
8. Dependence on other uncontrolled variables. Water discharge
from a well highly depends on hydraulic conductivity, sediment
grain size, or some other variable.

1. Histogram
2. Scatter Plot
3. Box-Plot
4. Quantile Plot
5. Q-Q Plots
6. Enhancement of data presentation
7. Presentation of multivariate data.

Exploratory Data Analysis (EDA) is an approach/philosophy for
data analysis that employs a variety of techniques (mostly
graphical) to maximize insight into a data set, uncover
underlying structure, extract important variables, detect outliers
and anomalies, test underlying assumptions, etc.
The EDA approach is an approach, not a set of techniques, but
an attitude/philosophy about how a data analysis should be
carried out.
EDA is a philosophy as to how we dissect a data set; what we
look for; how we look; and how we interpret.
Exploratory Data Analysis

Data Summarization
A summary analysis is simply a numeric reduction of a data set. It is
quite passive. Quite commonly, its purpose is to simply arrive at a
few key statistics (for example, mean and standard deviation) which
may then either replace the data set or be added to the data set in
the form of a summary table.
Exploratory Data Analysis
In contrast, EDA has as its broadest goal the desire to gain insight
into the engineering/scientific process behind the data. EDA uses
the data to peer into the heart of the process that generated the
data. There is an archival role in the research for summary
statistics, but there is an enormously larger role for the EDA
approach.
Summarization and Exploratory data analysis

Exploratory data analysis
Exploratory data analysis mostly depends on graphical analysis.
The particular graphical techniques employed in EDA are often
quite simple, consisting of various techniques of:
Plotting the raw data (such as histograms, scatter plots, etc.)
Plotting simple statistics such as mean plots, standard deviation
plots, box plots, and main effects plots of the raw data.

Data analysis and interpretation cannot be completely automated,
particularly when making crucial modeling choices. The analyst
must use judgment and make decisions that require familiarity
with the data, the site, and the questions that need to be
answered.
The analysis of data typically starts by plotting the data and
calculating statistics that describe important characteristics of
the sample.
It does little help if we just look at tabulated data. However, the
human eye can recognize patterns from graphical displays of the
data.
We perform such an exploratory analysis to:
1. familiarize ourselves with the data and
2. detect patterns of regularity.
Why Graphical Data Analysis?

Why Graphical Data Analysis?

Summary statistics, medians and IQRs used in explanatory
data analysis are said to be resistant statistics. A resistant
statistic is relatively less affected by outliers than a
nonresistant statistic. The mean and standard deviation are
examples of nonresistant statistics.
Exploratory data analysis

Distribution of Data: Histogram
• A histogram is used to graphically summarize the distribution of a data set
• A histogram divides the range of values in a data set into intervals
• Over each interval is placed a bar whose height represents the frequency of
data values in the interval.

Negatively Skewed Positively Skewed
Histogram from 50 years of annual average river discharge data

PlatykurticLeptokurtic
For the normal distribution the kurtosis coefficient is 3. The normal distribution is said to be
mesokurtic. If a distribution has a relatively greater concentration of probability near the
mean than the normal distribution, the kurtosis coefficient will be greater than 3 and the
distribution is said to be leptokurtic. If a distribution has a relatively smaller concentration of
probability, the kurtosis coefficient will be less than 3 and the distribution is said to be
platykurtic.

Scatter Plot
A scatter plot is useful for studying the association between two
interval variables. It is a plot of the values of one variable against
the other.
Scatter plot can be used for:
• To suggest a relationship between the two variables, for instance
a linear or quadratic relation,
• It may help to identify patterns or clusters in the data.
• Inspect these plots may help to detect outliers.

At each depth two data are collected: Temperature and Nitrogen Concentration.
We obtained two scatter plot:
(i) Depth vs. Temperature;
(ii) Depth vs. Nitrogen Concentration.
In the first graph, it is observed that temperature is increasing with depth, as a general
tendency. This corresponds to a positive association.
In the second graph, Nitrogen concentration decreasing with depth. This corresponds to a
negative association.
Scatter Plot and Data Association

Scatter Plot and Data Association

Scatter Plot and Data Pattern

Scatter Plot and Outliers

A dot chart or dot plot is a statistical chart consisting of group of data
points plotted on a simple scale.
Dot Plot
• Dot plots are one of the simplest statistical plots, and are
suitable for small to moderate sized data sets.
• Dot plots are used for continuous, quantitative, univariate data.

Groundwater depth
(meter) at 12 locations
are given below:
9, 11, 18, 7, 12, 21, 15, 12,
23, 13, 12, 10
Use dot plot for EDA
Dot Plot
• They are useful for highlighting
clusters and gaps, as well as
outliers.
• Their other advantage is the
conservation of numerical
information.

Groundwater depth
(meter):
9, 11, 18, 7, 12, 21, 15, 13,
23, 14, 17, 10
Dot Plot • Data Distribution is often not clear.
• Can not be used for large data set

A stem-leaf plot is a technique for presenting
quantitative data in a graphical format.
Similar to a histogram, to assist in visualizing
the shape of a distribution.
A basic stem-leaf plot contains two columns
separated by a vertical line. The left column
contains the stems and the right column
contains the leaves.
Stem-leaf Plot

Groundwater Depth (ft)
at 16 locations:
39, 31, 18, 7, 22, 21, 35, 12,
23, 13, 23, 10, 41, 27, 24, 9
Stem-leaf Plot
Unlike histograms, stem-leaf plots
retain the original data to at least
two significant digits, and put the
data in order.

Stem-leaf Plot
Groundwater Depth (ft)
at 16 locations:
39, 31, 18, 7, 22, 21, 35, 12,
23, 13, 23, 10, 41, 27, 24, 9
Can break into more
than one group.

Data:
9, 11, 18, 7, 12, 21, 15, 12, 23, 13,
12, 10
Dot Plot and Stem-leaf Plot: Comparison

Data:
9, 11, 18, 7, 12, 21, 15, 12,
23, 13, 12, 10

Data:
9, 11, 18, 7, 12, 21, 15, 13,
23, 14, 17, 10

• Stem-leaf plots are useful for displaying the relative density and
shape of the data, giving the reader a quick overview of
distribution.
• They retain (most of) the raw numerical data, often with perfect
integrity.
• They are also useful for highlighting outliers and finding the
mode.
• However, stem and leaf plots are only useful for moderately
sized data sets (around 15-150 data points).
• With very small data sets a stem and leaf plot can be of little
use, as a reasonable number of data points are required to
establish definitive distribution properties.
Stem-leaf Plot

A boxplot is a graph of a data set that depicts the five-number
summary in a visual way.
The Five-Number Summary of a data set consists of the five
values { min value, Q1, Q2, Q3, max value }:
1. the smallest observation,
2. lower quantile (Q1),
3. median (Q2),
4. upper quantile (Q3), and
5. largest observation.
Box Plot
• It is also useful in helping you compare data sets.
• It is also sometimes referred to as a box-and-whisker-plot.

• Draw a horizontal measurement scale
• Place a rectangle above this axis: the left edge is the lower
fourth quartile(Q1), the right edge is the upper fourth quartile
(Q3)
• Place a vertical line inside the rectangle at the location of
median (Q2)
• Draw “whiskers” out from either end of the rectangle to the
smallest and largest observations that are not outliers
Constructing a Box Plot

Five point summary of river discharge values are given below:
Minimum discharge: 51.0 cumec
First quartile (Q1): 60.75 cumec
Median: 63.0 cumec
Third quartile (Q3): 65.0 cumec
Maximum Discharge: 70.0 cumec
Constructing a Box Plot

Simple Box Plot
In simple box-plot whisker is extended to maximum and minimum
data points.

In standard box-plot whisker is extended to 1.5 times the height of the
box (1.5 times the interquartile range). Observations between one and
two steps from the box in either direction, if present, are plotted
individually with an asterisk ("outside values"). Observations farther
than two steps beyond the box, if present, are distinguished by plotting
them with a small circle ("far-out values").
Standard Box Plot

Truncated Box Plot
In truncated box-plot the whiskers are drawn only to the 90th and 10th
percentiles of the data set. The largest 10 percent and smallest 10
percent of the data are not shown. It is used only when the extreme 20
percent of data are not of interest.

Interpretation of Box Plot
Normal distribution,
symmetrical data
Right skewed
Left-skewed

Information contain in a Box Plot
• Location and Spread
• More informative to use a comparative Box plot

Significance of Box Plot

Interpretation of Box
Plot

• Box plot can be used
for any size of data
set
• Only summary
values and spread
are visible
• Numerical
information of data
are lost.

Information Obtained from a Box Plot
• If the median is near the center of the box, the distribution is
approximately symmetric.
• If the median falls to the left of the center of the box, the distribution
is positively skewed.
• If the median falls to the right of the center, the distribution is
negatively skewed.
• If the whisker lines are about the same length, the distribution is
approximately symmetric.
• If the right whisker line is larger than the left whisker line, the
distribution is positively skewed.
• If the left whisker line is larger than the right whisker line, the
distribution is negatively skewed.

Quantile PlotQuantile Plot
To construct a quantile plot, cumulative frequency of sample data are
plotted against quantiles of the standardized theoretical distribution.

Quantile Plot

Symmetric Right Skewed
Left Skewed Uniform11/23/2015 Shamsuddin Shahid, FKA, UTM

Q-Q Plot
Quantile-Quantile Plot (Q-Q Plot) is a graphical method for
diagnosing differences between the probability distribution and the
sampling distribution or comparing two sample distribution.
This is a scatterplot with the quantiles of the variable on the
horizontal axis and the expected normal scores on the vertical axis.
Q-Q plot can be two types:
1. Normal Q-Q plot: The normal Q-Q plot graphically compares the
distribution of a given variable to the normal distribution.
2. Q-Q plot: The Q-Q plot graphically compares the distribution of two
variables.

• A normal distribution is often a reasonable model for the data.
• Without inspecting the data, however, it is risky to assume a
normal distribution.
• There are a number of graphs that can be used to check the
deviations of the data from the normal distribution. A histogram is
an example of a graph that can be used to check normality. Here,
the histogram should reveal a bell shaped curve.
• The most useful tool for assessing normality is a quantile-
quantile or QQ plot.
• Q-Q plot is also a important graphical method for identify the
outliers.
• Q-Q plot can also be used to identify the shape of the data
distribution, skewness, etc.
Why normal Q-Q Plot?

Normal QNormal Q--Q PlotsQ Plots
Construction
Order n observations from smallest to largest and given a rank (i)
according to its position.
Quantile values corresponding to each observation is calculated using
formula
Theoretical normal quantile values corresponding to each calculated
quantile values are obtained from normal distribution table.
Plot the pairs on a two-coordinate system
 x-axis: theoretical (distribution) quantiles
 y-axis: sample quantiles
Sample

QQ--Q PlotsQ Plots
Formulas used to calculate the quantiles:

Normal Q-Q Plot

QQ--Q PlotsQ Plots

Normal Q-Q Plot:
Weibull

Normal Q-Q Plot: Comparison of Formulas
Probability Plot using Blom, Cunnane and Gringorten Formulas

Normal Q-Q Plot:
Hazen

Q-Q Plot using:
Weibull Formula
Blom, Cunnane and
Gringorten Formulas
Hazen Formula

Interpretation of Normal Q-Q Plot

All but a few points fall on a line. -
outliers in the data
Left end of pattern is below the
line; right end of pattern is
above the line. - Long tails at
both ends of the data
distribution.
Interpretation of
Normal Q-Q Plot
For the normal distribution:
68% within 1 SD of the mean
95% within 2 SDs
99.7% within 3 SDs

Left end of pattern is above the
line; right end of pattern is below
the line. - Short tails at both ends
of the data distribution
Curved pattern with slope
increasing from left to right. -
Data distribution is skewed to
the right
Interpretation of
Normal Q-Q Plot

Interpretation of
Normal Q-Q Plot

Description of Point Pattern Possible Interpretation
all but a few points fall on a line outliers in the data
left end of pattern is below the line; right
end of pattern is above the line
long tails at both ends of
the data distribution
left end of pattern is above the line; right
end of pattern is below the line
short tails at both ends of
the data distribution
curved pattern with slope increasing from
left to right
data distribution is skewed
to the right
curved pattern with slope decreasing from
left to right
data distribution is skewed
to the left
staircase pattern (plateaus and gaps)
data have been rounded or
are discrete
Interpretation of Probability Plot

Q-Q Plot
Q-Q plot is similar to probability plot, except instead of comparing
one quantile with theoretical normal quantile, two quantiles are
compared.
It gives us an idea about dispersion of one set of observation with
other set.
It is possible to compare two sets of observation to make some
important interpretations.

Annual averaged rainfall for years (1960-
1980) at 15 stations are given in first column
(A). Annual averaged rainfall of same 15
stations for the years (1981-2000) are given in
second column (B). Use Q-Q plot to see the
changing pattern of rainfall over the time.
Q-Q Plot

Construction of Q-Q Plot

The Q-Q Plot

• If the two distributions being compared are identical, the Q-Q plot
follows the 45° line y = x.
• If the two distributions agree after linearly transforming the values
in one of the distributions, then the Q-Q plot follows some line, but
not necessarily the line y = x.
• If the general trend of the Q-Q plot is flatter than the line y = x, the
distribution plotted on the horizontal axis is more dispersed than
the distribution plotted on the vertical axis.
• Conversely, if the general trend of the Q-Q plot is steeper than the
line y = x, the distribution plotted on the vertical axis is more
dispersed than the distribution plotted on the horizontal axis.
• Q-Q plots are often arced, or "S" shaped, indicating that one of
the distributions is more skewed than the other, or that one of the
distributions has heavier tails than the other
The Q-Q Plot

• Multivariate data analysis techniques are appropriate when more
than one response is measured.
• Multivariate data is somewhat difficult to accurately visualize
because of the multidimensional nature of the problem.
• In a multivariate approach, each response variable adds another
dimension to the analysis problem.
Some examples of graphical techniques used for easily displaying
multivariate data in two-dimensions:
Profile Plot
Area Plot
Kite Graph
Star Plot
Glyph Plot
Etc.
Graphical Representation of Multivariate Data

A profile plot uses a series of vertical axes presented consecutively along the
base (x-axis) of the plot. Any number of response variables can be considered
with varying scales of measurement. The response variables are arranged
along the base of the Plot.
Profile Plot

Area Plot

Representation of multivariate data
Multiple variables can be displayed

Kite Graph
Four variables can be displayed

Star Plot
Multiple variables can be displayed

Glyph Plots
Color, size and type of marker, a whisker line with different length, angle,
and color are used to represent different variables

Representation of Directional Data

Shahid Lecture-2- MKAG1273

More Related Content

Viewers also liked (13)

Similar to Shahid Lecture-2- MKAG1273 (20)

Recently uploaded (20)

Shahid Lecture-2- MKAG1273