![signals; the results are only approximate but often close enough in practice).
Sometimes one needs to know whether two signals have the same shape, for example in
comparing the spectrum of an unknown to a stored reference spectrum. Most likely the
concentrations of the unknown and reference, and therefore the amplitudes of the spectra,
will be different. Therefore a direct overlay or subtraction of the two spectra will not be
useful. One possibility is to compute the point-by-point ratio of the two signals; if they
have the same shape, the ratio will be a constant. For example, examine Figure 2.
Figure 2. Do the two spectra on the left have the same shape? They certainly do not
look the same, but that may simply be due to that fact that one is much weaker that the
other. The ratio of the two spectra, shown in the right part (Window 2), is relatively
constant from 300 to 440 nm, with a value of 10 +/- 0.2. This means that the shape of
these two signals is very nearly identical over this wavelength range.
The left part (Window 1) shows two superimposed spectra, one of which is much weaker
than the other. But do they have the same shape? The ratio of the two spectra, shown in
the right part (Window 2), is relatively constant from 300 to 440 nm, with a value of 10
+/- 0.2. This means that the shape of these two signals is the same, within about +/-2 %,
over this wavelength range, and that top curve is about 10 times more intense than the
bottom one. Above 440 nm the ratio is not even approximately constant; this is caused by
noise, which is the topic of the next section.
Simple signal arithmetic operations such as these are easily done in a spreadsheet (e.g.
Excel or the free OpenOffice Calc), any general-purpose programming language, or in a
dedicated signal-processing program such as SPECTRUM, which is available for free
download.
SPECTRUM includes addition and multiplication of a signal with a constant; addition,
subtraction, multiplication, and division of two signals; normalization, and a large
number of other basic math functions (log, ln, antilog, square root, reciprocal, etc).
In Matlab, math operations on signals are especially powerful because the variables in
Matlab can be either scalar (single values), vector (like a row or a column in a
spreadsheet), representing one entire signal, spectrum or chromatogram, or matrix (like a
rectangular block of cells in a spreadsheet), representing a set of signals. For example, in
Matlab you could define two vectors a=[1 2 5 2 1] and b=[4 3 2 1 0]. Then
to subtract B from A you would just type a-b, which gives the result [-3 -1 3 1 1].
To multiply A times B point by point, you would type a.*b, which gives the result [4](https://image.slidesharecdn.com/introtosignalprocessing-140409234717-phpapp01/85/Intro-tosignalprocessing-4-320.jpg)
![6 10 2 0]. If you have an entire spectrum in the variable a, you can plot it just by
typing plot(a). And if you also had a vector w of x-axis values (such as wavelengths),
you can plot a vs w by typing plot(w,a). The subtraction of two spectra a and b, as in
Figure 1, can be performed simply by writing a-b. To plot the difference, you would
write plot(a-b). Likewise, to plot the ratio of two spectra, as in Figure 2, you would
write plot(a./b). Moreover, Matlab is a programming language that can automate
complex sequences of operations by saving them in scripts and functions.
Signals and noise
Experimental measurements are never perfect, even with sophisticated modern
instruments. Two main types or measurement errors are recognized: (a) systematic error,
in which every measurement is consistently less than or greater than the correct value by a
certain percentage or amount, and (b) random error, in which there are unpredictable
variations in the measured signal from moment to moment or from measurement to
measurement. This latter type of error is often called noise, by analogy to acoustic noise.
There are many sources of noise in physical measurements, such as building vibrations,
air currents, electric power fluctuations, stray radiation from nearby electrical apparatus,
static electricity, interference from radio and TV transmissions, turbulence in the flow of
gases or liquids, random thermal motion of molecules, background radiation from natural
radioactive elements, "cosmic rays" from outer space (seriously), and even the basic
quantum nature of matter and energy itself.
One of the fundamental problems in signal measurement is distinguishing the noise from
the signal. It is not always easy. The signal is the "important" part of the data that you
want to measure - it might be the average of the signal over a certain time period, or it
might be the height of a peak or the area under a peak that occurs in the data. For
example, in the absorption spectrum in the right-hand half of Figure 1 in the previous
section, the "important" parts of the data are probably the absorption peaks located at 520
and 550 nm. The height of either of those peaks might be considered the signal,
depending on the application. In this example, the height of the largest peak is about 0.08
absorbance units. The noise would be the standard deviation of that peak height from
spectrum to spectrum (if you had access to repeat measurememnts of the same spectrum).
But what if you had only one recording of that spectrum? In that case, you'd be forced to
estimate the noise in that single recording, based on the visible short-term fluctuations in
the signal (the little random wiggles superimposed on the smooth signal). In this case,
those fluctuations amount to about 0.005 units peak-to-peak or a standard deviation of
0.001. (For random fluctuations, the general rule of thumb is that the standard deviation is
approximately 1/5 of the peak-to-peak variation between the highest and the lowest
readings). As another example, the data on the right half of Figure 3, below, has a peak
in the center with a height of about 1.0. The peak-to-peak noise on the baseline is also
about 1.0, so the standard deviation of the noise is about 1/5th of that, or 0.2.
The quality of a signal is often expressed quantitatively as the signal-to-noise ratio
(SNR), which is the ratio of the true signal amplitude (e.g. the average amplitude or the
peak height) to the standard deviation of the noise. Thus the signal-to-noise ratio of the
spectrum in Figure 1 is about 0.08/0.001 = 80, and the signal in Figure 3 has a signal-to-](https://image.slidesharecdn.com/introtosignalprocessing-140409234717-phpapp01/85/Intro-tosignalprocessing-5-320.jpg)
![You can also create user-defined functions in Matlab to automate commonly-used
algorithms. Some examples that you can download and use are these user-defined
functions to calculate typical peak shapes commonly encountered in analytical chemistry,
gaussian and lorentzian, and typical types of random noise (whitenoise, pinknoise), which
can be useful in modeling and simulating analytical signals and testing measurement
techniques. (If you are viewing this document on-line, you can ctrl-click on these links to
inspect the code). Once you have created or downloaded those functions, you can use
them to plot a simulated noisy peak such as in Figure 3 by typing, for example,
x=[1:256];plot(x,gaussian(x,128,64)+whitenoise(x)).
Smoothing
In many experiments in physical science, the true signal amplitudes (y-axis values)
change rather smoothly as a function of the x-axis values, whereas many kinds of noise
are seen as rapid, random changes in amplitude from point to point within the signal. In
the latter situation it is common practice to attempt to reduce the noise by a process called
smoothing. In smoothing, the data points of a signal are modified so that individual points
that are higher than the immediately adjacent points (presumably because of noise) are
reduced, and points that are lower than the adjacent points are increased. This naturally
leads to a smoother signal. As long as the true underlying signal is actually smooth, then
the true signal will not be much distorted by smoothing, but the noise will be reduced.
Smoothing algorithms. The simplest smoothing algorithm is the rectangular or
unweighted sliding-average smooth; it simply replaces each point in the signal with the
average of m adjacent points, where m is a positive integer called the smooth width. For
example, for a 3-point smooth (m=3):
for j = 2 to n-1, where Sj the jth point in the smoothed signal, Yj the jth point in the
original signal, and n is the total number of points in the signal. Similar smooth
operations can be constructed for any desired smooth width, m. Usually m is an odd
number. If the noise in the data is "white noise" (that is, evenly distributed over all
frequencies) and its standard deviation is s, then the standard deviation of the noise
remaining in the signal after the first pass of an unweighted sliding-average smooth will
be approximately s over the square root of m (s/sqrt(m)), where m is the smooth width.
The triangular smooth is like the rectangular smooth, above, except that it implements a
weighted smoothing function. For a 5-point smooth (m=5):
for j = 3 to n-2, and similarly for other smooth widths. This is equivalent to two passes of
a 3-point rectangular smooth. This smooth is more effective at reducing high-frequency
noise in the signal than the simpler rectangular smooth. Note that again in this case, the
width of the smooth m is an odd integer and the smooth coefficients are symmetrically
balanced around the central point, which is important point because it preserves the x-](https://image.slidesharecdn.com/introtosignalprocessing-140409234717-phpapp01/85/Intro-tosignalprocessing-8-320.jpg)
![background=ones(size(ObservedSpectrum));
A=[background A1 A2 A3];
where A1, A2, A3... are the absorption spectra of the individual components. Performing a T-
weighted regression is also readily performed:
weight=T;
MeasuredAmp=([weight weight] .* A)(ObservedSpectrum .* weight);
where T is the transmission spectrum. Here, the matrix division backslash "" is used as a short-
cut to the classical least-squares matrix solution (c.f.
http://www.mathworks.com/help/techdoc/ref/mldivide.html).
The custom Matlab function "TFit3Demo.m" simulates the measurement of three absorbing
components by weighted linear regression, demonstrates the effect of the amount of noise in the
signal, the extent of overlap of the spectra, and the linearily of the analytical curves of each
component, and also compares the results to another method that applies curve fitting to the
transmission spectra rather than to the absorbance spectra.
The Inverse Least Squares (ILS) technique is demonstrated by this script and the graph below.
This is a real data set derived from the near infrared (NIR) reflectance spectroscopy of
agricultural wheat samples analyzed for protein content. In this example there are 50 calibration
samples measured at 6 wavelengths. The samples had already been analyzed by a reliable, but
laborious and time-consuming, reference method. The
purpose of this calibration is to establish whether near-
infrared reflectance spectroscopy, which can be measured
much more quickly on wheat paste preparations,
correlates to their protein content. These results indicate
that it does, at least for this set of 50 wheat samples, and
therefore is it likely that near-infrared spectroscopy
should do a pretty good job of estimating the protein
content of similar unknown samples. The key is that the
unknown samples must be similar to the calibrations
samples (except for the protein content). However, this is
a very common analytical situation in commerce, where large numbers of samples of a similar
predictable type must be analyzed.
Curve fitting C: Non-linear Iterative Curve Fitting
("spectral deconvolution" or "peak deconvolution")
The linear least squares curve fitting described in "Curve Fitting A" is simple and fast,
but it is limited to situations where the dependent variable can be modeled as a
polynomial with linear coefficients. We saw that in some cases a non-linear situation can
be converted into a linear one by a coordinate transformation, but this is possible only in
some special cases and, in any case, the resulting coordinate transformation of the noise
in the data can result in inaccuracies in the parameters measured in this way.](https://image.slidesharecdn.com/introtosignalprocessing-140409234717-phpapp01/85/Intro-tosignalprocessing-39-320.jpg)
![peaks. It turns out that the non-linear iterative curve fitting is much more difficult to do
and more prone to error, but it's necessary if you need to track shifts in peak position or
widths or to decompose a complex overlapping peak signal into fundamental components
knowing only their shape.
Note: you can right-click on any of the m-file links above and select Save Link As... to
download them to your computer for use within Matlab.
Interactive Peak Fitter
(http://terpconnect.umd.edu/~toh/spectrum/InteractivePeakFitter.htm)
This is a series of Matlab peak fitting programs for
time-series signals, which uses an unconstrained
non-linear optimization algorithm to decompose a
complex, overlapping-peak signal into its
component parts. The objective is to determine
whether your signal can be represented as the sum
of fundamental underlying peaks shapes. These
programs do not require the signal processing or
optimization toolboxes. They accept signals of any
length, including those with non-integer and non-uniform x-values, and can fits groups of
peaks with Gaussian, Lorentzian, Logistic, Pearson, and exponentially-broadened
Gaussian models (expandable to other shapes). There are three different versions, a
command line version, a keypress operated interactive version, and a version with mouse-
controlled sliders (which requires Matlab 6.5).
Accuracy and precision of peak parameter measurement
There are four major sources of error in measuring the peak parameters (peak positions,
heights, widths, and areas) by iterative curve fitting:
a. Model errors.
Peak shape. If you have the wrong model for your
peaks, the results can not be expected to be
accurate; for instance, if your actual peaks are
Lorentzian in shape, but you fit them with a
Gaussian model, or vice versa. For example, a
single isolated Gaussian peak at x=5, with a height
of 1.000 fits a Gaussian model virtually perfectly,
using the Matlab user-defined peakfit function, as
shown on the right. (The 5th input argument for
the peakfit function specifies the shape of peaks to
be used in the fit; "1" means Gaussian).
>> x=[0:.1:10];y=exp(-(x-5).^2);
>> [FitResults,MeanFitError]=peakfit([x' y'],5,10,1,1)
FitResults =
1 5 1 .6649 1.7724
MeanFitError =
0.001679](https://image.slidesharecdn.com/introtosignalprocessing-140409234717-phpapp01/85/Intro-tosignalprocessing-42-320.jpg)
![The "FitResults" are, from left to right, peak number, peak position, peak height, peak
width, and peak area. The MeanFitError, or just "fitting error", is the RMS difference
between the data and the best-fit model, as a percentage of the maximum signal in the
fitted region. (Note the good agreement of the area measurement (1.7724) with the
theoretical area under the curve of exp(-x2), which is the square root of p, or 1.7725...).
But this same peak, when fit with a Logistic model (peak shape number 3), gives a fitting
error of 1.4% and height and width errors of 3% and 6%, respectively:
>> [FitResults,MeanFitError]=peakfit([x' y'],5,10,1,3)
FitResults =
1 5.0002 0.96652 1.762 1.7419
MeanFitError =
1.4095
When fit with a Lorentzian model (peak shape 2), this peak gives a 6% fitting error and
height and width errors of 8% and 20%, respectively.
> [FitResults,MeanFitError]=peakfit([x' y'],5,10,1,2)
FitResults =
1 5 1.0876 1.3139 2.0579
MeanFitError =
5.7893
So clearly the larger the fitting errors, the larger
are the parameter errors, but the parameter
errors are of course not equal to the fitting error
(that would just be too easy). Also, clearly the
peak width and area are the parameters most
susceptible to errors. The peak positions, as you
can see here, are measured accurately even if
the model is way wrong, as long as the peak is
symmetrical and not highly overlapping with
other peaks.
Number of peaks. Another source of model error occurs if you have the wrong number
of peaks in your model, for example if the signal actually has two peaks but you try to fit
it with only one peak. In the example below, a bit of of Matlab code generates a
simulated signal with of two Gaussian peaks at x=4 and x=6 with peaks heights of 1.000
and 0.5000 respectively and widths of 1.665, plus random noise with a standard deviation
5% of the height of the largest peak (a signal-to-noise ratio of 20):
>> x=[0:.1:10];
>> y=exp(-(x-6).^2)+.5*exp(-(x-4).^2)+.05*randn(size(x));
In a real experiment you would not usually know the peak positions, heights, and widths;
you would be using curve fitting to measure those parameters. Let's assume that, on the
basis of previous experience or some preliminary trial fits, you have established that the
optimum peak shape model is Gaussian, but you don't know for sure how many peaks are
in this group. If you start out by fitting this signal with a single-peak Gaussian model, you
get:](https://image.slidesharecdn.com/introtosignalprocessing-140409234717-phpapp01/85/Intro-tosignalprocessing-43-320.jpg)
![>>
[FitResults,MeanFitError]=peakfit([x'
y'],5,10,1,1)
FitResults =
1 5.5291 0.86396
2.9789 2.7392
MeanFitError =
10.467
The residual plot shows a "wavy" structure that's
visible in the random scatter of points due to the
random noise in the signal. This means that the
fitting error is not limited by the random noise;
it is a clue that the model is not quite right.
But a fit with two peaks yields much better results (The 4th input argument for the peakfit
function specifies the number of peaks to be used in the fit).
>> [FitResults,MeanFitError]=peakfit([x' y'],5,10,2,1)
FitResults =
1 4.0165 0.50484 1.6982 0.91267
2 5.9932 1.0018 1.6652 1.7759
MeanFitError = 4.4635
Now the residuals have a random scatter
of points, as would be expected if the
signal is accurately fit except for the
random noise. Moreover, the fitting error
is much lower (less that half) of the error
with only one peak. In fact, the fitting
error is just about what we would expect
in this case based on the 5% random
noise in the signal (estimating the relative
standard deviation of the points in the
baseline visible at the edges of the
signal). Because this is a simulation in
which we know beforehand the true
values of the peak parameters (peaks at
x=4 and x=6 with peaks heights of 1.0 and 0.50 respectively and widths of 1.665), we can
actually calculate the parameter errors (the difference between the real peak positions,
heights, and widths and the measured values). We see that they are quite accurate (in this
case within about 1% relative on the peak height and 2% on the widths), which is actually
better than the 5% random noise in this signal because of the averaging effect of fitting to
multiple data points in the signal.
But if going from one peak to two peaks gave us a better fit, why not go to three peaks?
Changing the number of peaks to three gives these results:
>> [FitResults,MeanFitError]=peakfit([x' y'],5,10,3,1)
FitResults =
1 4.0748 0.51617 1.7874 0.98212
2 6.7799 0.089595 2.0455 0.19507](https://image.slidesharecdn.com/introtosignalprocessing-140409234717-phpapp01/85/Intro-tosignalprocessing-44-320.jpg)
![3 5.9711 0.94455 1.53 1.5384
MeanFitError =
4.3878
The fitting algorithm has now tried to fit
an additional low-amplitude peak
(numbered peak 2 in this case) located at
x=6.78. The fitting error is actually lower
that for the 2-peak fit, but only slightly
lower, and the residuals are no less
visually random that with a 2-peak fit.
So, knowing nothing else, a 3-peak fit
might be rejected on that basis alone. In
fact, there is a serious downside to fitting
more peaks than are present in the signal:
it increases the parameter measurement
errors of the peaks that are actually
present. Again, we can prove this because we know beforehand the true values of the
peak parameters: clearly the peak positions, heights, and widths of the two real peaks than
are actually in the signal (peaks 1 and 3) are significantly less accurate that of the 2-peak
fit.
Moreover, if we repeat that fit with the same signal but with a different sample of random
noise (simulating a repeat measurement of a stable experimental signal in the presence or
random noise), the additional third peak in the 3-peak fit will bounce around all over the
place (because the third peak is actually fitting the random noise, not an actual peak in the
signal).
>> x=[0:.1:10];
>> y=exp(-(x-6).^2)+.5*exp(-(x-4).^2)+.05*randn(size(x));
>> [FitResults,MeanFitError]=peakfit([x' y'],5,10,3,1)
FitResults =
1 4.115 0.44767 1.8768 0.89442
2 5.3118 0.093402 2.6986 0.26832
3 6.0681 0.91085 1.5116 1.4657
MeanFitError =
4.4089
With this new set of data, two of the peaks (numbers 1 and 3) have roughly the same
position, height, and width, but peak number 2 has changed substantially compared to the
previous run. Now we have an even more compelling reason to reject the 3-peak model:
the 3-peak solution is not stable. And because this is a simulation in which we know the
right answers, we can verify that the accuracy of the peak heights is substantially poorer
(about 10% error) than expected with this level of random noise in the signal (5%). If we
were to run a 2-peak fit on the same new data, we get much better measurements of the
peak heights.
>> [FitResults,MeanFitError]=peakfit([x' y'],5,10,2,1)
FitResults =
1 4.1601 0.49981 1.9108 1.0167
2 6.0585 0.97557 1.548 1.6076](https://image.slidesharecdn.com/introtosignalprocessing-140409234717-phpapp01/85/Intro-tosignalprocessing-45-320.jpg)
![MeanFitError =
4.4113
If this is repeated several times, it turns out that the peak parameters of the peaks at x=4
and x=6 are, on average, more accurately measured by the 2-peak fit. In practice, the best
way to evaluate a proposed fitting model is to fit several repeat measurements of the same
signal (if that is practical experimentally) and to compute the standard deviation of the
peak parameter values.
Peak width constraints. Finally, there is one more thing that we can do that might
improve the peak parameter measurement accuracy, and it concerns the peak widths. In
all the above simulations, the basic assumption that all the peak parameters were
unknown and independent of one another. In some types of measurements, however, the
peak widths of each group of adjacent peaks are all expected to be equal to each other, on
the basis of first principles or previous experiments. This is a common situation in
analytical chemistry, especially in atomic spectroscopy and in chromatography, where the
peak widths are determined largely by instrumental factors. In the current simulation, the
true peaks widths are in fact equal, but all the results above show that the measured peak
widths are close but not quite equal, due to random noise in the signal. But we can
introduce an equal-width constraint into the fit by using peak shape 6 (Equal width
Gaussians) or peak shape 7 (Equal width Lorentzians) in peakfit version 1.4 and later.
Using peak shape 6 on the same set of data as the previous example:
>> [FitResults,MeanFitError]=peakfit([x' y'],5,10,2,6)
FitResults =
1 4.0435 0.4942 1.6559 0.87113
2 6.0039 0.99218 1.6559 1.7489
MeanFitError =
4.5076
This "equal width" fit forces all the peaks within
one group to have exactly the same width. The
result is a slightly higher fitting error (in this
case 4.5 rather than 4.4), but - perhaps
surprisingly - the peak parameter measurements
are usually more accurate and more
reproducible (Specifically, the relative standard
deviations of a large number of repetitions of
measurements of peak parameters are on average
lower for the equal-width fit than for an
unconstrained-width fit to the same data,
assuming of course that the true underlying peak widths are really equal). This is an
exception to the general expectation that lower fitting errors result in lower peak
parameter errors. It is an illustration of the general rule that the more you know about the
nature of your signals, and the closer your chosen model adheres to that knowledge, the
better the results. In this case we knew that the peak shape was Gaussian (although we
could have verified that choice by trying other candidate peaks shapes). We determined
that the number of peaks was two by inspecting the residuals and fitting errors for 1, 2,
and 3 peak models. And finally we introduced the constraint of equal peak widths within](https://image.slidesharecdn.com/introtosignalprocessing-140409234717-phpapp01/85/Intro-tosignalprocessing-46-320.jpg)
![SlopeThreshold - Slope of the smoothed first-derivative that is taken to indicate a
peak. This discriminates on the basis of peak width. Larger values of this
parameter will neglect broad features of the signal. A reasonable initial value for
Gaussian peaks is 0.7*WidthPoints^-2, where WidthPoints is the number of data
points in the half-width of the peak.
AmpThreshold - Discriminates on the basis of peak height. Any peaks with
height less than this value are ignored.
SmoothWidth - Width of the smooth function that is applied to data before the
slope is measured. Larger values of SmoothWidth will neglect small, sharp
features. A reasonable value is typically about equal to 1/2 of the number of data
points in the half-width of the peaks.
FitWidth - The number of points around the "top part" of the (unsmoothed) peak
that are taken to estimate the peak heights, positions, and widths. A reasonable
value is typically about equal to 1/2 of the number of data points in the half-width
of the peaks. The minimum value is 3.
Smoothtype (added in Version 4, Sept. 2011) determines the smoothing
algorithm (see http://terpconnect.umd.edu/~toh/spectrum/Smoothing.html)
If smoothtype=1, rectangular (sliding-average or boxcar)
If smoothtype=2, triangular (2 passes of sliding-average)
If smoothtype=3, pseudo-Gaussian (3 passes of sliding-average)
Basically, higher values yield greater reduction in high-frequency noise, at the
expense of slower execution.
There is also a similar function for finding valleys (minima), called findvalleys.m, which
works the same way as findpeaks.m, except that it locates minima instead of maxima.
Right-click the link and select "Save link as...."
Examples:
>> x=[0:.01:50];y=cos(x);findpeaks(x,y,0,-1,5,5)
ans =
1 6.2832 1 2.3569
2 12.566 1 2.3569
3 18.85 1 2.3568
4 25.133 1 2.3569
5 31.416 1 2.3569
6 37.699 1 2.3568
7 43.982 1 2.3569
>> x=[0:.01:50];y=cos(x);findvalleys(x,y,0,-1,5,5)
ans =
1 3.1416 -1 2.3571
2 9.4248 -1 2.3571
3 15.708 -1 2.3571
4 21.991 -1 2.3571
5 28.274 -1 2.3571
6 34.558 -1 2.3571
7 40.841 -1 2.3571
Note: The accuracy of the measurements of peak position, height, and width by the
findpeaks function depends on the shape of the peaks, the extent of peak overlap, and
signal-to-noise ratio. The width measurements particularly are strongly influenced by](https://image.slidesharecdn.com/introtosignalprocessing-140409234717-phpapp01/85/Intro-tosignalprocessing-59-320.jpg)
![peak overlap, noise, and the choice of FitWidth. Isolated peaks of Gaussian shape are
measured most accurately. See "ipeakdemo" (below) for an accuracy trial. For the most
accurate measurements of highly overlapping peaks of various shapes, use the Interactive
Peak Fitter (ipf.m).
DemoFindPeak (DemoFindPeak.m)
A simple demonstration script using the findpeaks function on noisy synthetic data. Numbers the
peaks and prints out the P matrix in the Matlab command window:
Peak # Position Height Width
P =
1.0000 10.0235 0.1814 1.7506
2.0000 29.9964 0.6412 1.9931
3.0000 50.0205 0.4315 2.1433
4.0000 70.0120 1.3935 2.0202
5.0000 79.9567 0.2580 1.8267
6.0000 100.0086 1.0862 2.0306
......
The iPeak Function (ipeak.m), version 3.8
ZIP file available at http://terpconnect.umd.edu/~toh/spectrum/ipeak3.zip
iPeak is a keyboard-operated Interactive Peak Finder for time series data, based on the
"findpeaks.m" function. It displays the entire signal in the lower half of the Figure
window and an adjustable zoomed-in section in the upper window. (Press Enter to change
the plot color.) Simple keystrokes allow you to adjust the peak detection parameters
AmpThreshold (A/Z keys), SlopeThreshold (S/X), SmoothWidth (D/C), FitWidth (F/V),
and other controls. (See list of Keyboard Controls, below). This function can be used to
determine experimentally what values of those parameters give the most reliable peak
detection for a particular type of data, detecting the desired peaks and ignoring those that
are too small, too broad, or too narrow to be of interest. Detected peaks are numbered
from left to right. Returns the peak table in P (Peak #, Position, Height, Width, Area).
EXAMPLE 1: Two input arguments; data in separate x and y vectors
>> x=[0:.1:100];y=(x.*sin(x)).^2;ipeak(x,y);
EXAMPLE 2: One input argument; data in two columns of a matrix](https://image.slidesharecdn.com/introtosignalprocessing-140409234717-phpapp01/85/Intro-tosignalprocessing-60-320.jpg)
![>> x=[0:.01:5]';y=x.*sin(x.^2).^2;ipeak([x y])
EXAMPLE 3: One input argument; data in single vector
>> y=cos(.1:.1:100);ipeak(y)
Keystrokes allow you to pan and zoom the upper window, to inspect each peak in detail if
desired. You can set the initial values of pan and zoom in optional input arguments 7
('xcenter') and 8 ('xrange'). See example 6 below.
The Spacebar/Tab keys (version 3.2 and later) jump to the next/previous detected peak
and displays it in the upper window at the current zoom setting (use the up and down
cursor arrow keys to adjust the zoom range). The L key turns off and on peak parameter
labeling. The Y key (version 3.2) toggles between linear and log y-axis scale in the lower
window (a log axis is good for inspecting signals with high dynamic range; it effects only
the lower window display and has no effect on the peak detection or measurements). The
U key switches between peak and valley detection mode.
The T key toggles the autozero mode off and on (version 3.2 and later). When autozero
is OFF, peak heights are measured relative to zero. (If the peaks are superimposed on a
background, use the baseline subtract keys - B and G - first to subtract the background).
When autozero is ON, peak heights are automatically measured relative to the local
baseline on either side of the peak; use the zoom controls to isolate the peaks so that the
signal returns to the local baseline between the peaks as displayed in the upper window.
When autozero is ON, the peak heights, widths, and areas in the peak table (R or P keys)
will be automatically corrected for the baseline. (Autozero OFF will give better results
when the baseline is zero, or has been subtracted using the B key, even if the peaks are
partly overlapped. Autozero ON will work best ifthe peaks are well separated so that the
signal returns to the local baseline between the peaks. If the peaks are highly overlapped,
or if they are not Gaussian in shape, the best results will be obtained by using the curve
fitting function - the N or M keys).
Normal and Multiple Curve fitting: The N key applies variable-shape iterative curve
fittingto the detected peaks that are displayed in the upper window (referred to a
"Normal" curve fitting). If the peaks are superimposed on a background, turn on the
Autozero mode (T key). Then use the pan and zoom keys to select a peak or a group of
overlapping peaks in the upper window, with the signal returning all the way to the local
baseline at the ends of the upper window. Make sure that AmpThreshold,
SlopeThreshold,SmoothWidth are adjusted so that each peak is numbered once. Then
press the N key, type a number for the desired peak shape at the prompt in the Command
window and press Enter (1=Gaussian (default); 2=Lorentzian; 3=logistic, 4=Pearson;
5=exponentially broadened Gaussian, 6=equal-width Gaussians; 7=Equal-width
Lorentzians; 8=exponentially broadened equal-width Gaussian), then type in a number of
repeat trial fits and press Enter (the default is 1; start with that and then increase if
necessary). The program will perform the fit, display the results graphically in Figure 2,
and print out a table of results in the command window, e.g.:
Peak shape (1-8): 2](https://image.slidesharecdn.com/introtosignalprocessing-140409234717-phpapp01/85/Intro-tosignalprocessing-61-320.jpg)
![Number of trials: 1
Least-squares fit to Lorentzian peak model
Fitting Error 1.1581e-006%
Peak# Position Height Width Area
1 100 1 50 71.652
2 350 1 100 146.13
3 700 1 200 267.77
The use of the iterative least-squares function can result in more accurate peak parameter
measurements that the normal peak table (R or P keys), especially if the peaks are non-
Gaussian in shape or are highly overlapped.
There is also an experimental "Multiple" peak fit function (M key) that will attempt to
apply non-linear iterative curve fitting to to all the detected peaks in the signal
automatically. Before using this function, use the baseline correction function first (B
key) to remove the background signal. Then press M and proceed as for the normal curve
fit. With some signals, this function may not work as well as using the normal (N-key)
curve fitting function on each group of peaks separately. Note: The N and M key fitting
functions perform non-linear iterative curve fitting using the peakfit.m function. The
number of peaks and the starting values of peak positions and widths for the curve fit
function are automatically supplied by the the findpeaks function, so it is essential that the
peak detection variables in iPeak be adjust so that all the peaks in the selected ragion are
detected and numbered once. (For more flexible curve fitting, use ipf.m).
EXAMPLE 4: Additional scalar argument (after the data) to control peak sensitivity.
>> x=[0:.1:100];y=5+5.*cos(x)+randn(size(x));ipeak(x,y,10);
or >> ipeak([x;y],10);
or >> ipeak(humps(0:.01:2),3)
or >> x=[0:.1:10];y=exp(-(x-5).^2);ipeak([x' y'],1)
The additional argument is an estimate of the ratio of the typical peak width to the length
of the entire data record. Small values detect fewer peaks; larger values detect more
peaks. It effects only the starting values for the peak detection parameters. (This is just a
quick way to set reasonable initial values of the peak detection parameters, rather than
specifying each one individually as in the following example).
EXAMPLE 5: Six input arguments. As above, but input arguments 3 to 6 directly
specify the initial values of PeakD, AmpT, SlopeT, SmoothW, FitW. (PeakD is ignored
in this case, so just type a '0' as the second argument after the data matrix).
>> ipeak(datamatrix,0,.5,.0001,20,20);
EXAMPLE 6: Eight input arguments. As above, but input arguments 7 and 8 specifiy the
initial pan and zoom settings, 'xcenter' and 'xrange', respectively. In this example, the x-
axis data are wavelengths in nanometers (nm), and the upper window zooms in on a very
small 0.4 nm region centered on 249.7 nm. (These data, provided in the ZIP file, are from
a high-resolution atomic spectrum).
>> load ipeakdata.mat
>> ipeak(Sample1,0,110,0.06,3,4,249.7,0.4);](https://image.slidesharecdn.com/introtosignalprocessing-140409234717-phpapp01/85/Intro-tosignalprocessing-62-320.jpg)
![EXAMPLE 7: Nine input arguments. As example 6, but the 9th input argument turns
ON the autozero mode (equivalent to pressing the T key). If not specified, autozero is
initially OFF.
>> ipeak(Sample1,0,110,0.06,3,4,249.7,0.4,1);
Peak identification: There is an optional "peak identification" function if optional input
arguments 9 ('MaxError'), 10 ('Positions'), and 11 ('Names') are included. The "i" key
toggles this function ON and OFF. This function compares the found peak positions
(maximum x-values) to a database of known peaks, in the form of an array of known peak
maximum positions ('Positions') and matching cell array of names ('Names'). If the
position of a found peak in the signal is closer to one of the known peaks by less than the
specified maximun error ('MaxError'), then that peak is considered a match and its name
is displayed next to the peak in the upper window. When when the 'o' key is pressed, the
peak positions, names, errors, and amplitudes are printed out in a table in the command
window.
EXAMPLE 8: Eleven input arguments. As above, but also specifies 'MaxError',
'Positions', and 'Names' in optional input arguments 9, 10, and 11, for peak identification
function. Pressing the 'I' key toggles off and on the peak identification labels in the upper
window. Pressing "o" prints the peak positions, names, errors, and amplitudes in a table
in the command window. These data (provided in the ZIP file) are from a high-resolution
atomic spectrum (x-axis in nanometers).
>> load ipeakdata.mat
>> ipeak(Sample1,0,120,0.06,3,6,296,5,0.1,Positions,Names);
Note: The ZIP file contains the latest version of the iPeak function as well as some
sample data to demonstrate peak identification (Example 7). Obviously for your own
applications, it's up to you to provide your own array of known peak maximum positions
('Positions') and matching cell array of names ('Names') for your particular types of
signals.
Keyboard Controls:
Pan signal left and right...Coarse pan: < and >
Fine pan: left and right cursor arrow keys
Nudge left and right: [ and ]
Zoom in and out.............Coarse zoom: / and '
Fine zoom: up and down cursor arrow keys
Resets pan and zoom.........ESC
Change plot color...........Enter
Adjust AmpThreshold.........A,Z (A increases, Z decreases)
Adjust SlopeThreshold.......S,X (S increases, X decreases)
Adjust SmoothWidth..........D,C (D increases, C decreases)
Adjust FitWidth.............F,V (F increases, Z decreases)
Baseline correction.........B, then click baseline at multiple points
Restore original signal.....G to cancel previous background subtraction
Invert signal...............- Invert (negate) the signal (flip + and -)
Set minimum to zero.........0 (Zero) Sets minumun signal to zero
Toggle log y mode OFF/ON....Y Plot log Y axis on lower graph
Toggle autozero OFF/ON......T Auto background subtraction on upper graph
Toggle valley mode OFF/ON...U Switch to valley mode
Print peak table............P Prints Peak #, Position, Height, Width
Step through peaks..........Space/Tab Jumps to next/previous peak
Normal peak fit.............N Fit peaks in upper window with peakfit.m
Multiple peak fit...........M Fit all peaks in signal with peakfit.m
Print keyboard commands.....K Prints this list
Print findpeaks arguments...Q Prints findpeaks function with arguments.
Print ipeak arguments.......W Prints ipeak function with all arguments.](https://image.slidesharecdn.com/introtosignalprocessing-140409234717-phpapp01/85/Intro-tosignalprocessing-63-320.jpg)
![areas measured by the Peak Measure (P) command are based on ther assumption that the
baseline under the peaks is zero. The Backspace key starts background correction mode.
In the command window, type in the number of background points to click and press the
Enter key. The cursor changes to cross hairs; click along the presumed background,
starting to the left of the x axis and placing the last click to the right of the x axis. When
the last point is clicked, the linearly interpolated baseline between those points is
subtracted from the signal. To restore the original background (i.e. to correct an error or
to try again), press the '' key (just below the backspace key).
Saving the results
To save the processed signal as a .mat file, press the 'o' key, the type in the desired file
name, then press Enter or click Save.
Peak measurement
The P key toggles off and on the peak measure mode, which attempts to measure the one
peak that is centered in the upper window under the green cursor; a red "cap" on the peak
indicates that portion of the signal that is taken for the measurement. (Adjust the pan and
zoom controls so that the red cap overlays roughly the top half of the peak). The peak
position, height, width are measured by least-squares curve fitting of a Gaussian to the
isolated peak; the peak area is measured by the trapezoidal method over the selected
segment displayed in the upper window. (If the peak is noisy and/or non-Gaussian, the
results will only be approximate, but the position, height, and area values are pretty good
for any peak shape. If the peaks are superimposed on a non-zero background, subtract the
background before measuring peaks). Press the R key to print out the peak parameters in
the command window.
Other keystroke controls
The L key toggles off and on the Overlay mode, which overlays the current processed
signal with the original signal as a dotted line, for the purposes of comparison. The tab
key restores the original signal and cursor settings. The T key sets the selected region to
zero (to trim away artifacts and spikes). The "-" (minus sign) key is used to negate the
signal (flip + for -). Press H to toggle display of semilog y plot in the lower window,
useful for signals with very wide dynamic range (zero and negative points are ignored in
the log plot). Press K to see all the keyboard commands.
EXAMPLE 1: Single input argument; data in a two columns of a matrix [x;y] or in a
single y vector
>> isignal(y);
>> isignal([x;y]);
EXAMPLE 2: Two input arguments. Data in separate x
and y vectors.
>> isignal(x,y);
EXAMPLE 3: Three or four input arguments. The last
two arguments specify the initial values of pan (xcenter)
and zoom (xrange) in the last two input arguments. Using
data in the ZIP file:](https://image.slidesharecdn.com/introtosignalprocessing-140409234717-phpapp01/85/Intro-tosignalprocessing-68-320.jpg)
![>> load data.mat
>> isignal(DataMatrix,180,40); or
>> isignal(x,y,180,40);
EXAMPLE 4: As above, but additionally specifies
initial values of SmoothMode, SmoothWidth, ends, and
DerivativeMode in the last four input arguments.
>> isignal(DataMatrix,180,40,2,9,0,1);
EXAMPLE 5: As above, but additionally specifies
initial values of the peak sharpening parameters
Sharpen, Sharp1, and Sharp2 in the last three input
arguments. Press the E key to toggle sharpening on and
off for comparison.
>> isignal(DataMatrix,180,40,2,9,0,0,1,51,6000);
EXAMPLE 6: Using the built-in "humps" function:
>> x=[0:.005:2];y=humps(x);Data=[x;y];
4th derivative of the peak at x=0.9:
>> isignal(Data,0.9,0.5,1,3,1,4);
Peak sharpening applied to the peak at x=0.3:
>> isignal(Data,0.3,0.5,1,3,1,0,1,220,5400);
Press 'E' key to toggle sharpening ON/OFF to compare)
iSignal keyboard controls:
Pan signal left and right...Coarse pan: < and >
Fine pan: left and right cursor arrows
Nudge one point left or right: [ and ]
Zoom in and out.............Coarse zoom: / and "
Fine zoom: up and down cursor arrows
Reset pan and zoom..........ESC (resets to initial default values)
Adjust smooth width.........A,Z (A=>more smoothing, Z=>less smoothing)
Adjust smooth type..........S (None, Rectanglular, Triangle, Gaussian,
Savitzky-Golay)
Toggle smooth ends..........X (0=ends zeroed 1=ends smoothed
Adjust derivative order.....D (0th to 4th order)
Toggle peak sharpening......E (0=OFF 1=ON)
Sharpening for Gaussian.....Y Set sharpen settings for Gaussians
Sharpening for Lorentzian...U Set sharpen settings for Lorentzians
Adjust sharp1...............F,V F=>sharper, V=>less sharpening
Adjust sharp2 ............G,B G=>sharper, B=>less sharpening
Toggle peak labeling........P labels center peak in upper window
Toggle overlay mode.........L Overlays original signal in upper window
Toggle log y mode...........H Display semilog y plot in lower window
Toggle Autozero OFF/ON......T Subtracts background from upper window segment
Restores original signal....Tab key resets to original signal and modes
Baseline subtraction........Backspace, then click baseline at 8 points
Restore background.......... to cancel previous background subtraction
Invert signal...............- Invert (negate) the signal (flip + and -)
Remove offset...............0 (zero) set minimun signal) to zero
Trim region to zero.........; sets selected region to zero.
Print keyboard commands.....K prints this list
Print signal report.........Q prints signal info and current settings
Print isignal arguments.....W prints isignal (with current arguments)
Save output to disk.........O as .mat file containing processed signal](https://image.slidesharecdn.com/introtosignalprocessing-140409234717-phpapp01/85/Intro-tosignalprocessing-69-320.jpg)
![5. Interactive Peak Fitter
A Matlab peak fitting program for time-series signals, which uses an unconstrained non-
linear optimization algorithm to decompose a complex, overlapping-peak signal into its
component parts. The objective is to determine whether your signal can be represented as
the sum of fundamental underlying peaks shapes. Accepts signals of any length, including
those with non-integer and non-uniform x-values. Fits groups of up to 6 peaks with 8
distinct peak shapes (expandable to other shapes).
a. Command line function: peakfit.m
Peakfit.m is a user-defined command window peak fitting function, usable from a remote
terminal. It is written as a self-contained Matlab function in a single m-file. (To view of
download, click Peakfit.m). It takes data in the form of a 2 x n matrix that has the X
values in row 1 and the Y values in row 2, or as a single independent variable vector. A
ZIP file containing peakfit.m, DemoPeakFit.m, ipf.m, Demoipf.m, and some sample data
for testing is available at from http://terpconnect.umd.edu/~toh/spectrum/ipf57.zip
Peakfit can be called with several optional additional arguments
peakfit(signal);
Performs an iterative least-squares fit of a
single Gaussian peak to the entire data matrix
"signal", which has x values in row 1 and Y
values in row 2 (e.g. [x y]) or which may be a
single signal vector (in which case the data
points are plotted against their index numbers
on the x axis).
peakfit(signal,center,window);
Fits a single Gaussian peak to a portion of the
matrix "signal". The portion is centered on the
x-value "center" and has width "window" (in x
units).
In this and in all following examples, set "center" and "window" both to 0 to fit the entire
signal.
peakfit(signal,center,window,NumPeaks);
"NumPeaks" = number of peaks in the model (maximum 6; default is 1 if not specified).
peakfit(signal,center,window,NumPeaks,peakshape);
Specifies the peak shape of the model: "peakshape" = 1-8. (1=Gaussian, 2=Lorentzian,
3=logistic, 4=Pearson, 5=exponentionally broadened Gaussian; 6=equal-width Gaussians,
7=equal-width Lorentzians, 8=exponentionally broadened equal-width Gaussians),
9=exponential pulse, 10=sigmoid).
peakfit(signal,center,window,NumPeaks,peakshape,extra)
Specifies the value of 'extra', used in the Pearson and the exponentionally broadened](https://image.slidesharecdn.com/introtosignalprocessing-140409234717-phpapp01/85/Intro-tosignalprocessing-70-320.jpg)
![Gaussian shapes to fine-tune the peak shape.
peakfit(signal,center,window,NumPeaks,peakshape,extra,NumTrials);
Performs "NumTrials" trial fits and selects the best one (with lowest fitting error).
NumTrials can be any positive integer (default is 1).
peakfit(signal,center,window,NumPeaks,peakshape,extra,NumTrials,s
tart)
Specifies the first guesses vector "firstguess" for the peak positions and widths, e.g.
start=[position1 width1 position2 width2 ...]
peakfit(signal,center,window,NumPeaks,peakshape,extra,NumTrials,s
tart, autozero)
Specifies the AUTOZERO setting: 0 is OFF and 1 is ON. When ON, it removes linear
baseline from data segment. Default is ON (AUROZERO=1).
[FitResults,MeanFitError]=peakfit(signal,center,window...)
Returns the FitResults vector in the order peak number, peak position, peak height, peak
width, and peak area), and the MeanFitError (the percent RMS difference between the
data and the model in the selected segment of that data) of the best fit.
Optional output parameters:
1. FitResults: a table of model peak parameters, one row for each peak, listing Peak
number, Peak position, Height, Width, and Peak area.
2. LowestError: The rms fitting error of the best trial fit.
3. BestStart: the starting guesses that gave the best fit.
4. xi: vector containing 100 interploated x-values for the model peaks.
5. yi: matrix containing the y values of model peaks at each xi. Type plot(xi,yi(1,:)) to
plot peak 1 or plot(xi,yi) to plot all the peaks
Example 1:
Signal is a matrix' Fits exp(-x)^2 with a single Gaussian peak model.
> x=[0:.1:10];y=exp(-(x-5).^2);peakfit([x' y'])
ans =
1 5 1 1.665 1.7725
Peak number Peak position Height Width Peak area
Example 1:
Signal is a single vector: Create a small data set and fit to a Gaussian model
>> peakfit([0 0 0 1 2 4 6 7 6 4 2 1 0 0 0])
ans =
1 7.9741 6.8279 4.4068 32.023
Example 3: Like Example 1, except that random noise is added to the y data. Compare
output to Example 1.
> x=[0:.1:10];y=exp(-(x-5).^2)+.1*randn(1,length(x));peakfit([x' y'])
Example 4: Fits a noisy two-peak signal with a double Gaussian model (NumPeaks=2).
> x=[0:.1:10];y=exp(-(x-5).^2)+.5*exp(-(x-3).^2)+.1*randn(1,length(x));](https://image.slidesharecdn.com/introtosignalprocessing-140409234717-phpapp01/85/Intro-tosignalprocessing-71-320.jpg)
![> peakfit([x' y'],5,19,2,1,0,1)
ans =
1 3.0001 0.49489 1.642 0.86504
2 4.9927 1.0016 1.6597 1.7696
Example 5:
> x=[0:.005:1];y=humps(x);peakfit([x' y'],.3,.7,1,4,3);
Fits a portion of the humps function, 0.7 units wide and centered on x=0.3, with a single
(NumPeaks=1) Pearson function (peakshape=4) with extra=3 (controls shape of Pearson
function).
Example 6: Creates a data matrix 'smatrix', fits a portion to a two-peak Gaussian model,
takes the best of 10 trials. Returns FitResults and MeanFitError.
> x=[0:.005:1];y=(humps(x)+humps(x-.13)).^3;smatrix=[x' y'];
> [FitResults,MeanFitError]=peakfit(smatrix,.4,.7,2,1,0,10)
FitResults =
1 0.4128 3.1114e+008 0.10448 3.4605e+007
2 0.3161 2.8671e+008 0.098862 3.0174e+007
MeanFitError =
0.68048
Example 7: As above, but specifies the first-guess position and width of the two peaks,
in the order [position1 width1 position2 width2]
> peakfit([x' y'],.4,.7,2,1,0,10,[.3 .1 .5 .1]);
Example 8: As above, returns the vector x1 containing 100 interploated x-values for the
model peaks and the matrix y1 containing the y values of each model peak at each xi.
Type plot(xi,yi(1,:)) to plot peak 1 or plot(xi,yi) to plot all peaks.
>
[FitResults,LowestError,BestStart,xi,yi]=peakfit(smatrix,.4,.7,2,1,0,10)
Example 9: Like Example 7, but turns AUTOZERO off in the last argument. Default is
AUROZERO=1, removes linear baseline from data segment.
>> peakfit([x' y'],.4,.7,2,1,0,10,[.3 .1 .5 .1],0);
Example 10: Fits a group of three peaks near x=2400 in DataMatrix3 with three
equal-width exponentially-broadened Gaussians.
>> [FitResults,MeanFitError]=peakfit(DataMatrix3,2400,440,3,8,31,1)
FitResults =
1 2300.5 0.82546 60.535 53.188
2 2400.4 0.48312 60.535 31.131
3 2500.6 0.84799 60.535 54.635
MeanFitError =
0.19975
DemoPeakFit.m is a demonstration script for peakfit.m. It generates an overlapping
Gaussian peak signal, adds normally-distributed noise, fits it with the peakfit.m function
(in line 78), repeats this several times (NumRepeats in line 20), then compares the peak
parameters (position, height, width, and area) of the measurements to their actual values](https://image.slidesharecdn.com/introtosignalprocessing-140409234717-phpapp01/85/Intro-tosignalprocessing-72-320.jpg)
![In this example, pan and zoom controls are used to
isolate a segment of a chromatogram that contains three
weak peaks near 5.8 minutes The lower plot shows the
whole chromatogram and the upper plot shows the
segment. Only the peaks in that segment are subjected
to the fit.
Pan and zoom are adjusted so that the signal returns
to the local baseline at the ends of the segment;
pressing T turns on the autozero mode, causing the
program to compute and subtract a linear baseline
between those points. Pressing 3, E, F performs a 3-
peak exponentially-broadened Gaussian fit.
Operating instructions for ipf.m (Version 7.1)
1. At the command line, type ipf(x, y), (x = independent
variable, y = dependent variable) or ipf(datamatrix)
where "datamatrix" is a matrix that has x values in row
1 and y values in row 2. Or if you have only one signal
vector y, type ipf(y).
2. Use the four cursor arrow keys on the keyboard to pan
and zoom the signal to isolate the peak or group of
peaks that you want to fit in the upper window. (Use the
< and > and ? and " keys for coarse pan and zoom and
the square bracket keys [ and ] to nudge one point left
and right). The curve fitting operation applies only to the segment of the signal shown
in the top plot. The bottom plot shows the entire signal. Try not to get any undesired
peaks in the upper window or the program may try to fit them.
3. Press the 1, 2, 3, 4, 5, or 6 keys to choose the number of model peaks, that is, the
minimum number of peaks that you think might suffice to fit this segment of the
signal.
4. Select the desired model peak shape by pressing the following keys: Gaussian (G),
Equal-width Gaussians (H), Exponentially-broadened equal-width Gaussians (J);
Lorentzian (L), Equal-width Lorentzians (;), Logistic (O), Pearson (P), exponentially-
broadened Gaussian (E), and exponentially-broadened equal width Gaussians (J key) ;
exponential pulse (U) and sigmoid (S) models. If the peak widths of each group of
peaks is expected to be the same or nearly so, select the "equal-width" shapes. Equal-
width fits (available only for the Gaussian, Lorentzian, and equal-width Gaussians
shapes in version 5.6 and later) are faster, easier, and much more stable than regular
variable-width fits, especially if the number of model peaks is greater than 3 (because
there are fewer variable parameters for the program to adjust - only one width, rather
than a separate width for each peak).](https://image.slidesharecdn.com/introtosignalprocessing-140409234717-phpapp01/85/Intro-tosignalprocessing-74-320.jpg)
![However, in the TFit method, there is only one iterated variable (absorbance) per
measured component, and reasonable first guesses are easily available from the
conventional single-wavelength absorbance calculation or multiwavelength regression
methods. As a result, an iterative least-squares method works very well in this case.
Comparing the expression for absorbance given above for the TFit method to that for the
weighted regression method:
absorbance=([weight weight].*[Background ReferenceSpectra])(-
log10(yobsd).*weight)
you can see that, in addition to the ReferenceSpectra and observed transmission spectrum
(yobsd), the TFit method also requires a measurement of the Instrument function (spectral
bandpass) and the stray light (which the linear regression methods assume to be zero), but
these are characteristics of the spectrometer and need be done only once for a given
spectrometer. Finally, although the TFit method does make the computer work harder,
the computation time on a typical laboratory personal computer is only a fraction of a
second (roughly 25 µsec per spectral data point per component analyzed), using Matlab as
the computational environment.
TFitDemo.m: Interactive explorer for the Tfit method, version 2.1
Interactive explorer for the Tfit method, applied to the measurement of a single
component with a Lorentzian or Gaussian absorption peak, with keystroke controls for
adjusting the parameters while observing the effects graphically and numerically. The
adjustable parameters are: the true absorbance, "True A" (adjusted with the A/Z keys), the
spectral width of the absorption peak, "AbsWidth", (adjusted with the S/X keys), the
spectral width of the slit function, "SlitWidth" (adjusted with the D/C keys), the percent
stray light, "Straylight" (adjusted with the F/V keys), and the noise level, "Noise"
(adjusted with the G/B keys). (The wavelength scale, and the values of both AbsWidth
and SlitWidth, are in arbitrary units). The simulation includes the effect of photon noise,
unabsorbed stray light, and random background intensity shifts (light source flicker), and
it compares observed absorbances computed by the single-wavelength, "Single",
weighted multilinear regression,"WReg" (sometimes called Classical Least Squares in the
chemometrics literature), and the TFit methods. To run this file, right-click TFitDemo.m
click "Save link as...", save it in a folder in the Matlab path, then type "TFitDemo" at the
Matlab command prompt. With the figure window topmost on the screen, press K to get
a list of the keypress functions.](https://image.slidesharecdn.com/introtosignalprocessing-140409234717-phpapp01/85/Intro-tosignalprocessing-81-320.jpg)
Intro tosignalprocessing
- 1. An Introduction to Signal Processing with applications in Chemical Analysis An illustrated essay with software available for free download Last updated February, 2012 PDF format: http://terpconnect.umd.edu/~toh/spectrum/IntroToSignalProcessing.pdf Web format: http://terpconnect.umd.edu/~toh/spectrum/TOC.html Tom O'Haver Professor Emeritus Department of Chemistry and Biochemistry University of Maryland at College Park E-mail: toh@umd.edu Table of Contents Introduction 2 Signal arithmetic 3 Signals and noise 5 Smoothing 8 Differentiation 14 Resolution enhancement 22 Harmonic analysis 25 Convolution 27 Deconvolution 28 Fourier filter 29 Integration and peak area measurement 30 Linear least squares curve fitting 31 Multicomponent spectroscopy 36 Non-linear iterative curve fitting 39 SPECTRUM for Macintosh OS8 54 Matlab for PC/Mac/Linux 57 Matlab: Peak finding and measurement 58 Matlab: Interactive smooth, derivative, sharpen, etc. 65 Matlab: Interactive peak fitter (ipf.m) 70 The TFit Method for absorption spectroscopy 77 References 85
- 2. Introduction The interfacing of analytical measurement instrumentation to small computers for the purpose of on-line data acquisition has now become standard practice in the modern laboratory. Using widely-available, low-cost microcomputers and off-the-shelf add-in components, it is now easier than ever to acquire data quickly in digital form. In what ways is on-line digital data acquisition superior to the old methods such as the chart recorder? Some of the advantages are obvious, such as archival storage and retrieval of data and post-run re-plotting with adjustable scale expansion. Even more important, however, there is the possibility of performing post-run data analysis and signal processing. There are a large number of computer-based numerical methods that can be used to reduce noise, improve the resolution of overlapping peaks, compensate for instrumental artifacts, test hypotheses, optimize measurement strategies, diagnose measurement difficulties, and decompose complex signals into their component parts. These techniques can often make difficult measurements easier by extracting more information from the available data. Many of these techniques are based on laborious mathematical procedures that were not practical before the advent of computerized instrumentation. As personal computers become faster and more widely available, many of these techniques will eventually become routine. It is important for students to appreciate the capabilities and the limitations of these modern signal processing techniques. In the chemistry curriculum, signal processing may be covered as part of a course on instrumental analysis (1, 2), electronics for chemists (3), laboratory interfacing (4), or basic chemometrics (5). The purpose of this paper is to give a general introduction to some of the most widely used signal processing techniques and to give illustrations of their applications in analytical chemistry. This essay covers only elementary topics and is limited to only basic mathematics. For more advanced topics and for a more rigorous treatment of the underlying mathematics, refer to the extensive literature on chemometrics (references, page 85). This tutorial makes use of a freeware Macintosh signal-processing program called SPECTRUM that was used to produce many of the illustrations. Additional examples were developed in Matlab, a high-performance commercial numerical computing environment and programming language that is widely used by scientists, researchers, and engineers. Paragraphs in gray at the end of each section in this essay describe the related capabilities of each of these programs. The Appendix (page 54-84) describes the details of these programs. Go to http://terpconnect.umd.edu/~toh/spectrum/TOC.html for the latest version of this essay online and its related programs.
- 3. Signal arithmetic The most basic signal processing functions are those that involve simple signal arithmetic: point-by-point addition, subtraction, multiplication, or division of two signals or of one signal and a constant. Despite their mathematical simplicity, these functions can be very useful. For example, in the left part of Figure 1 (Window 1) the top curve is the absorption spectrum of an extract of a sample of oil shale, a kind of rock that is is a source of petroleum. Figure 1. A simple point-by--point subtraction of two signals allows the background (bottom curve on the left) to be subtracted from a complex sample (top curve on the left), resulting in a clearer picture of what is really in the sample (right). This spectrum exhibits two absorption bands, at about 515 nm and 550 nm, that are due to a class of molecular fossils of chlorophyll called porphyrins. (Porphyrins are used as geomarkers in oil exploration). These bands are superimposed on a background absorption caused by the extracting solvents and by non-porphyrin compounds extracted from the shale. The bottom curve is the spectrum of an extract of a non-porphyrin-bearing shale, showing only the background absorption. To obtain the spectrum of the shale extract without the background, the background (bottom curve) is simply subtracted from the sample spectrum (top curve). The difference is shown in the right in Window 2 (note the change in Y-axis scale). In this case the removal of the background is not perfect, because the background spectrum is measured on a separate shale sample. However, it works well enough that the two bands are now seen more clearly and it is easier to measure precisely their absorbances and wavelengths. In this example and the one below, the assumption is being made that the two signals in Window 1 have the same x-axis values, that is, that both spectra are digitized at the same set of wavelengths. Strictly speaking this operation would not be valid if two spectra were digitized over different wavelength ranges or with different intervals between adjacent points. The x-axis values much match up point for point. In practice, this is very often the case with data sets acquired within one experiment on one instrument, but the experimenter must take care if the instruments settings are changed or if data from two experiments or two instrument are combined. (Note: It is possible to use the mathematical technique of interpolation to change the number of points or the x-axis intervals of
- 4. signals; the results are only approximate but often close enough in practice). Sometimes one needs to know whether two signals have the same shape, for example in comparing the spectrum of an unknown to a stored reference spectrum. Most likely the concentrations of the unknown and reference, and therefore the amplitudes of the spectra, will be different. Therefore a direct overlay or subtraction of the two spectra will not be useful. One possibility is to compute the point-by-point ratio of the two signals; if they have the same shape, the ratio will be a constant. For example, examine Figure 2. Figure 2. Do the two spectra on the left have the same shape? They certainly do not look the same, but that may simply be due to that fact that one is much weaker that the other. The ratio of the two spectra, shown in the right part (Window 2), is relatively constant from 300 to 440 nm, with a value of 10 +/- 0.2. This means that the shape of these two signals is very nearly identical over this wavelength range. The left part (Window 1) shows two superimposed spectra, one of which is much weaker than the other. But do they have the same shape? The ratio of the two spectra, shown in the right part (Window 2), is relatively constant from 300 to 440 nm, with a value of 10 +/- 0.2. This means that the shape of these two signals is the same, within about +/-2 %, over this wavelength range, and that top curve is about 10 times more intense than the bottom one. Above 440 nm the ratio is not even approximately constant; this is caused by noise, which is the topic of the next section. Simple signal arithmetic operations such as these are easily done in a spreadsheet (e.g. Excel or the free OpenOffice Calc), any general-purpose programming language, or in a dedicated signal-processing program such as SPECTRUM, which is available for free download. SPECTRUM includes addition and multiplication of a signal with a constant; addition, subtraction, multiplication, and division of two signals; normalization, and a large number of other basic math functions (log, ln, antilog, square root, reciprocal, etc). In Matlab, math operations on signals are especially powerful because the variables in Matlab can be either scalar (single values), vector (like a row or a column in a spreadsheet), representing one entire signal, spectrum or chromatogram, or matrix (like a rectangular block of cells in a spreadsheet), representing a set of signals. For example, in Matlab you could define two vectors a=[1 2 5 2 1] and b=[4 3 2 1 0]. Then to subtract B from A you would just type a-b, which gives the result [-3 -1 3 1 1]. To multiply A times B point by point, you would type a.*b, which gives the result [4
- 5. 6 10 2 0]. If you have an entire spectrum in the variable a, you can plot it just by typing plot(a). And if you also had a vector w of x-axis values (such as wavelengths), you can plot a vs w by typing plot(w,a). The subtraction of two spectra a and b, as in Figure 1, can be performed simply by writing a-b. To plot the difference, you would write plot(a-b). Likewise, to plot the ratio of two spectra, as in Figure 2, you would write plot(a./b). Moreover, Matlab is a programming language that can automate complex sequences of operations by saving them in scripts and functions. Signals and noise Experimental measurements are never perfect, even with sophisticated modern instruments. Two main types or measurement errors are recognized: (a) systematic error, in which every measurement is consistently less than or greater than the correct value by a certain percentage or amount, and (b) random error, in which there are unpredictable variations in the measured signal from moment to moment or from measurement to measurement. This latter type of error is often called noise, by analogy to acoustic noise. There are many sources of noise in physical measurements, such as building vibrations, air currents, electric power fluctuations, stray radiation from nearby electrical apparatus, static electricity, interference from radio and TV transmissions, turbulence in the flow of gases or liquids, random thermal motion of molecules, background radiation from natural radioactive elements, "cosmic rays" from outer space (seriously), and even the basic quantum nature of matter and energy itself. One of the fundamental problems in signal measurement is distinguishing the noise from the signal. It is not always easy. The signal is the "important" part of the data that you want to measure - it might be the average of the signal over a certain time period, or it might be the height of a peak or the area under a peak that occurs in the data. For example, in the absorption spectrum in the right-hand half of Figure 1 in the previous section, the "important" parts of the data are probably the absorption peaks located at 520 and 550 nm. The height of either of those peaks might be considered the signal, depending on the application. In this example, the height of the largest peak is about 0.08 absorbance units. The noise would be the standard deviation of that peak height from spectrum to spectrum (if you had access to repeat measurememnts of the same spectrum). But what if you had only one recording of that spectrum? In that case, you'd be forced to estimate the noise in that single recording, based on the visible short-term fluctuations in the signal (the little random wiggles superimposed on the smooth signal). In this case, those fluctuations amount to about 0.005 units peak-to-peak or a standard deviation of 0.001. (For random fluctuations, the general rule of thumb is that the standard deviation is approximately 1/5 of the peak-to-peak variation between the highest and the lowest readings). As another example, the data on the right half of Figure 3, below, has a peak in the center with a height of about 1.0. The peak-to-peak noise on the baseline is also about 1.0, so the standard deviation of the noise is about 1/5th of that, or 0.2. The quality of a signal is often expressed quantitatively as the signal-to-noise ratio (SNR), which is the ratio of the true signal amplitude (e.g. the average amplitude or the peak height) to the standard deviation of the noise. Thus the signal-to-noise ratio of the spectrum in Figure 1 is about 0.08/0.001 = 80, and the signal in Figure 3 has a signal-to-
- 6. noise ratio of 1.0/0.2 = 5. So we would say that the signal-to-noise ratio of the signal in Figure 1 is better than that in Figure 3. Signal-to-noise ratio is inversely proportional to the relative standard deviation of the signal amplitude. Measuring the signal-to-noise ratio is much easier if the noise can be measured separately, in the absence of signal. Depending on the type of experiment, it may be possible to acquire readings of the noise alone, for example on a segment of the baseline before or after the occurrence of the signal. However, if the magnitude of the noise depends on the level of the signal, then the experimenter must try to produce a constant signal level to allows measurement of the noise on the signal. In a few cases, where it is possible to model the shape of the signal exactly by means of a mathematical function, the noise may be estimated by subtracting the model signal from the experimental signal. If possible, it's always better to determine the standard deviation of repeated measurements of the thing that you want to measure, rather than trying to estimate the noise from a single recording of the data. Sometimes the signal and the noise can be partly distinguished on the basis of frequency components: for example, the signal may contain mostly low-frequency components and the noise may be located a higher frequencies. This is the basis of filtering and smoothing. In both Figure 1 and Figure 3, the peaks contain mostly low-frequency components, whereas the noise is distributed over a much wider frequency range. One key thing that really distinguishes signal from noise is that random noise is not the same from one measurement of the signal to the next, whereas the genuine signal is at least partially reproducible. So if the signal can be measured more than once, use can be made of this fact by measuring the signal over and over again, as fast as is practical, and adding up all the measurements point-by-point, then dividing by the number of signal averaged. This is called ensemble averaging, and it is one of the most powerful methods for improving signals, when it can be applied. For this to work properly, the noise must be random and the signal must occur at the same time in each repeat. An example is shown in Figure 3. Another example (EnsembleAverage1.wmv) demonstrates the ensemble averaging of 1000 repeats of a signal, which improves the signal-to-noise by about 30 times. Figure 3. Window 1 (left) is a single measurement of a very noisy signal. There is actually a broad peak near the center of this signal, but it is not possible to measure its position, width, and height accurately because the signal-to-noise ratio is very poor (less
- 7. than 1). Window 2 (right) is the average of 9 repeated measurements of this signal, clearly showing the peak emerging from the noise. The expected improvement in signal- to-noise ratio is 3 (the square root of 9). Often it is possible to average hundreds of measurement, resulting is much more substantial improvement. Noise is also distinguished by its frequency spectrum. White noise has equal power at all frequencies. It derives its name from white light, which has equal brightness at all wavelengths in the vsible region. The noise in the example signal in Figure 3 is white. In the acoustical domain, white noise sounds like a hiss. In measurement science, is it common for noise to have a low-frequency-weighted character, that is, to have more power at low frequencies that high frequencies. This is often called "pink noise". In the acoustical domain, pink noise sounds more like a roar. A commonly-encountered sub- species of that type of noise is "1/f noise", where the noise power in inversly proportional to frequency. The application of smoothing and low-pass filtering to reduce noise is more effective for white noise than for pink noise. When pink noise is present, it is sometimes beneficial to apply modulation techniques, such as optical chopping or wavelength modulation, to convert a direct-current (DC) signal into an alternating current (AC) signal, thereby increasing the frequency of the signal to a frequency region where the noise is lower. In such cases it is common to use a lock-in amplifier, or the digital equivalent thereof, to measure the amplitude of the signal. In spectroscopy, three fundamental types of noise are recognized, based on their origin and on how they vary with light intensity: photon noise, detector noise, andflicker (fluctuation) noise. Photon noise (often the limiting noise in instruments that use photomultiplier detectors) is white and is proportional to the square root of light intensity, and therefore the SNR is proportional to the square root of light intensity and directly proportional to the monochromator slit width. Detector noise (often the limiting noise in instruments that use solid-state photodiode detectors) is independent of the light intensity and therefore the detector SNR is directly proportional to the light intensity and to the square of the monochromator slit width. Flicker noise, caused by light source instability, vibration, sample cell positioning errors, sample turbulence, light scattering by suspended particles, dust, bubbles, etc., is usually pink rather than white and is directly proportional to the light intensity, so the flicker signal-to-noise ratio is not decreased by increasing the slit width. Flicker noise can usually be reduced or eliminated by using specialized instrument designs such as double-beam, dual wavelength, derivative, and wavelength modulation. SPECTRUM includes several functions for measuring signals and noise, plus a signal- generator that can be used to generate artificial signals with Gaussian and Lorentzian bands, sine waves, and normally-distributed random noise. Matlab has built-in functions that can be used for measuring and plotting signals and noise, such as mean, max, min, std, plot, hist, rand, and randn. Just type "help" and the function name at the Matlab command >> prompt, e.g. "help mean". Most of these Matlab functions apply to vectors and matrices as well as scalar variables. So, for example if you have a set of signals in the rows of a matrix S, where each column represents the value of each signal at the same value of the independent variable (e.g. time), you can compute the ensemble average of those signals just by typing "mean(S)", which computes the mean of each column of S.
- 8. You can also create user-defined functions in Matlab to automate commonly-used algorithms. Some examples that you can download and use are these user-defined functions to calculate typical peak shapes commonly encountered in analytical chemistry, gaussian and lorentzian, and typical types of random noise (whitenoise, pinknoise), which can be useful in modeling and simulating analytical signals and testing measurement techniques. (If you are viewing this document on-line, you can ctrl-click on these links to inspect the code). Once you have created or downloaded those functions, you can use them to plot a simulated noisy peak such as in Figure 3 by typing, for example, x=[1:256];plot(x,gaussian(x,128,64)+whitenoise(x)). Smoothing In many experiments in physical science, the true signal amplitudes (y-axis values) change rather smoothly as a function of the x-axis values, whereas many kinds of noise are seen as rapid, random changes in amplitude from point to point within the signal. In the latter situation it is common practice to attempt to reduce the noise by a process called smoothing. In smoothing, the data points of a signal are modified so that individual points that are higher than the immediately adjacent points (presumably because of noise) are reduced, and points that are lower than the adjacent points are increased. This naturally leads to a smoother signal. As long as the true underlying signal is actually smooth, then the true signal will not be much distorted by smoothing, but the noise will be reduced. Smoothing algorithms. The simplest smoothing algorithm is the rectangular or unweighted sliding-average smooth; it simply replaces each point in the signal with the average of m adjacent points, where m is a positive integer called the smooth width. For example, for a 3-point smooth (m=3): for j = 2 to n-1, where Sj the jth point in the smoothed signal, Yj the jth point in the original signal, and n is the total number of points in the signal. Similar smooth operations can be constructed for any desired smooth width, m. Usually m is an odd number. If the noise in the data is "white noise" (that is, evenly distributed over all frequencies) and its standard deviation is s, then the standard deviation of the noise remaining in the signal after the first pass of an unweighted sliding-average smooth will be approximately s over the square root of m (s/sqrt(m)), where m is the smooth width. The triangular smooth is like the rectangular smooth, above, except that it implements a weighted smoothing function. For a 5-point smooth (m=5): for j = 3 to n-2, and similarly for other smooth widths. This is equivalent to two passes of a 3-point rectangular smooth. This smooth is more effective at reducing high-frequency noise in the signal than the simpler rectangular smooth. Note that again in this case, the width of the smooth m is an odd integer and the smooth coefficients are symmetrically balanced around the central point, which is important point because it preserves the x-
- 9. axis position of peaks and other features in the signal. (This is especially critical for analytical and spectroscopic applications because the peak positions are sometimes important measurement objectives). Note that we are assuming here that the x-axis intervals of the signal is uniform, that is, that the difference between the x-axis values of adjacent points is the same throughout the signal. This is also assumed in many of the other signal-processing techniques described in this essay, and it is a very common (but not necessary) characteristic of signals that are acquired by automated and computerized equipment. The Savitzky-Golay smooth is based on the least-squares fitting of polynomials to segments of the data. The algorithm is discussed in http://www.wire.tu- bs.de/OLDWEB/mameyer/cmr/savgol.pdf. Compared to the sliding-average smooths, the Savitzky-Golay smooth is less effective at reducing noise, but more effective at retaining the shape of the original signal. It is capable of differentiation as well as smoothing. The algorithm is more complex and the computional times are greater than the smooth types discussed above, but with modern computers the difference is not significant and code in various languages is widely available online. Noise reduction. Smoothing usually reduces the noise in a signal. If the noise is "white" (that is, evenly distributed over all frequencies) and its standard deviation is s, then the standard deviation of the noise remaining in the signal after one pass of a triangular smooth will be approximately s*0.8/sqrt(m), where m is the smooth width. Smoothing operations can be applied more than once: that is, a previously-smoothed signal can be smoothed again. In some cases this can be useful if there is a great deal of high-frequency noise in the signal. However, the noise reduction for white noise is less less in each successive smooth. For example, three passes of a rectangular smooth reduces white noise by a factor of approximately s*0.7/sqrt(m), only a slight improvement over two passes (triangular smooth). Edge effects and the lost points problem. Note in the equations above that the 3-point rectangular smooth is defined only for j = 2 to n-1. There is not enough data in the signal to define a complete 3-point smooth for the first point in the signal (j = 1) or for the last point (j = n) , because there are no data points before the first point or after the last point. Similarly, a 5-point smooth is defined only for j = 3 to n-2, and therefore a smooth can not be calculated for the first two points or for the last two points. In general, for an m- width smooth, there will be (m-1)/2 points at the beginning of the signal and (m-1)/2 points at the end of the signal for which a complete m-width smooth can not be calcu- lated. What to do? There are two approaches. One is to accept the loss of points and trim off those points or replace them with zeros in the smooth signal. (That's the approach taken in the figures in this paper). The other approach is to use progressively smaller smooths at the ends of the signal, for example to use 2, 3, 5, 7... point smooths for signal points 1, 2, 3,and 4..., and for points n, n-1, n-2, n-3..., respectively. The later approach may be preferable if the edges of the signal contain critical information, but it increases execution time. Examples of smoothing. A simple example of smoothing is shown in Figure 4. The left half of this signal is a noisy peak. The right half is the same peak after undergoing a triangular smoothing algorithm. The noise is greatly reduced while the peak itself is hardly changed. Smoothing increases the signal-to-noise ratio. The larger the smooth
- 10. width, the greater the noise reduction, but also the greater the possibility that the signal will be distorted by the smoothing operation. The optimum choice of smooth width Figure 4. The left half of this signal is a noisy peak. The right half is the same peak after undergoing a smoothing algorithm. The noise is greatly reduced while the peak itself is hardly changed, making it easier to measure the peak position, height, and width. depends upon the width and shape of the signal and the digitization interval. For peak- type signals, the critical factor is the smoothing ratio, the ratio between the smooth width m and the number of points in the half-width of the peak. In general, increasing the smoothing ratio improves the signal-to-noise ratio but causes a reduction in amplitude and in increase in the bandwidth of the peak. The figures above show examples of the effect of three different smooth widths on noisy Gaussian-shaped peaks. In the figure on the left, the peak has a (true) height of 2.0 and there are 80 points in the half-width of the peak. The red line is the original unsmoothed peak. The three superimposed green lines are the results of smoothing this peak with a triangular smooth of width (from top to bottom) 7, 25, and 51 points. Because the peak width is 80 points, the smooth ratios of these three smooths are 7/80 = 0.09, 25/80 = 0.31, and 51/80 = 0.64, respectively. As the smooth width increases, the noise is progressively reduced but the peak height also is reduced slightly. For the largest smooth, the peak width is slightly increased. In the figure on the right, the original peak (in red) has a true height of 1.0 and a half-width of 33 points. (It is also less noisy than the example on the left.) The three superimposed green lines are the results of the same three triangular smooths of width (from top to bottom) 7, 25, and 51 points. But because the peak width in this case is only 33 points, the smooth ratios of these three smooths are larger: 0.21, 0.76, and 1.55, respectively. You can see that the peak distortion effect (reduction of peak
- 11. height and increase in peak width) is greater for the narrower peak because the smooth ratios are higher. Smooth ratios of greater than 1.0 are seldom used because of excessive peak distortion. Note that even in the worst case, the peak positions are not effected (assuming that the original peaks were symmetrical and not overlapped by other peaks). It's important to point out that smoothing results such as illustrated in the figure above may be deceptively impressive because they employ a single sample of a noisy signal that is smoothed to different degrees. This causes the viewer to underestimate the contribution of low-frequency noise, which is hard to estimate visually because there are so few low-frequency cycles in the signal record. This error can be remedied by taking a large number of independent samples of noisy signal. This is illustrated in the DemoSmoothSider module for Matlab, which includes a "Resample" control that swaps the noise in the signal with different random noise samples, to demonstrate the low- frequency noise that remains in the signal after smoothing. This gives a much more realistic impression of the performance of smoothing. The figure on the right is another example signal that illustrates some of these principles. You can download the data file "udx" in TXT format or in Matlab MAT format. The signal consists of two Gaussian peaks, one located at x=50 and the second at x=150. Both peaks have a peak height of 1.0 and a peak half-width of 10, and a normally-distributed random white noise with a standard deviation of 0.1 has been added to the entire signal. The x-axis sampling interval, however, is different for the two peaks; it's 0.1 for the first peaks and 1.0 for the second peak. This means that the first peak is characterized by ten times more points that the second peak. It may look like the first peak is noisier than the second, but that's just an illusion; the signal- to-noise ratio for both peaks is 10. The second peak looks less noisy only because there are fewer noise samples there and we tend to underestimate the deviation of small samples. The result of this is that when the signal is smoothed, the second peak is much more likely to be distorted by the smooth (it becomes shorter and wider) than the first peak. The first peak can tolerate a much wider smooth width, resulting in a greater degree of noise reduction. Optimization of smoothing. Which is the best smooth ratio? It depends on the purpose of the peak measurement. If the objective of the measurement is to measure the true peak height and width, then smooth ratios below 0.2 should be used. (In the example on the left, the original peak (red line) has a peak height greater than the true value 2.0 because of the noise, whereas the smoothed peak with a smooth ratio of 0.09 has a peak height that is much closer to the correct value). But if the objective of the measurement is to measure the peak position (x-axis value of the peak), much larger smooth ratios can be employed if desired, because smoothing has no effect at all on the peak position of a symmetrical peak (unless the increase in peak width is so much that it causes adjacent peaks to overlap). In quantitative analysis applications, the peak height reduction caused by smoothing is
- 12. not so important, because in most cases calibration is based on the signals of standard solutions. If the same signal processing operations are applied to the samples and to the standards, the peak height reduction of the standard signals will be exactly the same as that of the sample signals and the effect will cancel out exactly. In such cases smooth widths from 0.5 to 1.0 can be used if necessary to further improve the signal-to-noise ratio. (The noise is reduced by approximately the square root of the smooth width). In practical analytical chemistry, absolute peak height measurements are seldom required; calibration against standard solutions is the rule. (Remember: the objective of a quantitative analytical spectrophotometric procedure is not to measure absorbance but rather to measure the concentration of the analyte.) It is very important, however, to apply exactly the same signal processing steps to the standard signals as to the sample signals, otherwise a large systematic error may result. When should you smooth a signal? There are two reasons to smooth a signal: (1) for cosmetic reasons, to prepare a nicer-looking graphic of a signal for visual inspection or publication, and (2) if the signal will be subsequently processed by an algorithm that would be adversely effected by the presence of too much high-frequency noise in the signal, for example if the location of peaks, valleys, or inflection points in the signal is to be automatically determined by detecting zero-crossings in derivatives of the signal. But one common situation where you should not smooth signals is prior to least-squares curve fitting, because all smoothing algorithms are at least slightly "lossy", entailing at least some change in signal shape and amplitude. If these requirements conflict, care must be used in the design of algorithms. For example, in a popular technique for peak finding and measurement, peaks are located by detecting downward zero-crossings in the smoothed first derivative, but the position, height, and width of each peak is determined by least-squares curve-fitting of a segment of original unsmoothed data in the vicinity of the zero-crossing. Thus, even if heavy smoothing is necessary to provide reliable discrimination against noise peaks, the peak parameters extracted by curve fitting are not distorted. Condensing oversampled signals. Sometimes signals are recorded more densely (that is, with smaller x-axis intervals) than really necessary to capture all the features of the signal. This results in larger-than-necessary data sizes, which slows down signal processing procedures and may tax storage capacity. To correct this, oversampled signals can be reduced in size either by eliminating data points (say, dropping every other point or every third point) or by replacing groups of adjacent points by their averages. The later approach has the advantage of using rather than discarding extraneous data points, and it acts like smoothing to provide some measure of noise reduction. (If the noise in the original signal is white, it is reduced in the condensed signal by the square root of n, with no change in frequency distribution of the noise). The Matlab custom user-defined function condense.m performs this function; condense(y,n) returns a condensed version of y in which each group of n points is replaced by its average, reducing the length of y by the factor n. (For x,y data sets, use this function on both independent variable x and dependent variable y so that the features of y will appear at the same x values). See the Matlab File Exchange. Video Demonstration. This 18-second, 3 MByte video (Smooth3.wmv) demonstrates the effect of triangular smoothing on a single Gaussian peak with a peak height of 1.0 and
- 13. peak width of 200. The initial white noise amplitude is 0.3, giving an initial signal-to- noise ratio of about 3.3. An attempt to measure the peak amplitude and peak width of the noisy signal, shown at the bottom of the video, are initially seriously inaccurate because of the noise. As the smooth width is increased, however, the signal-to-noise ratio improves and the accuracy of the measurements of peak amplitude and peak width are improved. However, above a smooth width of about 40 (smooth ratio 0.2), the smoothing causes the peak to be shorter than 1.0 and wider than 200, even though the signal-to-noise ratio continues to improve as the smooth width is increased. (This demonstration was created in Matlab 6.5 using "Interactive Smoothing for Matlab" module). SPECTRUM includes rectangular and triangular smoothing functions for any number of points. Smoothing in Matlab. The user-defined function fastsmooth implements all the types of smooths discussed above. (If you are viewing this document on-line, you can ctrl-click on this link to inspect the code). Fastsmooth is a Matlab function of the form s=fastsmooth(a,w,type,edge). The argument "a" is the input signal vector; "w" is the smooth width; "type" determines the smooth type: type=1 gives a rectangular (sliding-average or boxcar); type=2 gives a triangular (equivalent to 2 passes of a sliding average); type=3 gives a pseudo-Gaussian (equivalent to 3 passes of a sliding average). The argument "edge" controls how the "edges" of the signal (the first w/2 points and the last w/2 points) are handled. If edge=0, the edges are zero. (In this mode the elapsed time is independent of the smooth width. This gives the fastest execution time). If edge=1, the edges are smoothed with progressively smaller smooths the closer to the end. (In this mode the execution time increases with increasing smooth widths). The smoothed signal is returned as the vector "s". (You can leave off the last two input arguments: fastsmooth(Y,w,type) smooths with edge=0 and fastsmooth(Y,w) smooths with type=1 and edge=0). Compared to convolution-based smooth algorithms, fastsmooth typically gives much faster execution times, especially for large smooth widths; it can smooth a 1,000,000 point signal with a 1,000 point sliding average in less than 0.1 second. iSignal is a Matlab function that performs smoothing for time- series signals using the fastsmooth algorithm, among other funcitons, with keystrokes that allow you to adjust the smoothing parameters continuously while observing the effect on your signal dynamically. See http://terpconnect.umd.edu/~toh/spectrum/iSignal.html . Differentiation The symbolic differentiation of functions is a topic that is introduced in all elementary Calculus courses. The numerical differentiation of digitized signals is an application of this concept that has many uses in analytical signal processing. The first derivative of a signal is the rate of change of y with x, that is, dy/dx, which is interpreted as the slope of the tangent to the signal at each point. Assuming that the x-interval between adjacent points is constant, the simplest algorithm for computing a first derivative is:
- 14. (for 1< j <n-1). where X'j and Y'j are the X and Y values of the jth point of the derivative, n = number of points in the signal, and DX is the difference between the X values of adjacent data points. A commonly used variation of this algorithm computes the average slope between three adjacent points: (for 2 < j <n-1). The second derivative is the derivative of the derivative: it is a measure of the curvature of the signal, that is, the rate of change of the slope of the signal. It can be calculated by applying the first derivative calculation twice in succession. The simplest algorithm for direct computation of the second derivative in one step is (for 2 < j <n-1). Similarly, higher derivative orders can be computed using the appropriate sequence of coefficients: for example +1, -2, +2, -1 for the third derivative and +1, -4, +6, -4, +1 for the 4th derivative, although these derivatives can also be computed simply by taking successive lower order derivatives. The Savitzky-Golay smooth can also be used as a differentiation algorithm with the appropriate choice of input arguments; it combines differentiation and smoothing into one algorithm, which is sensible because smoothing is always required with differentiation. Basic Properties of Derivative Signals The figure on the left shows the results of the successive differentiation of a computer- generated signal. The signal in each of the four windows is the first derivative of the one before it; that is, Window 2 is the first derivative of Window 1, Window 3 is the first derivative of Window 2, Window 3 is the second derivative of Window 1, and so on. You can predict the shape of each signal by recalling that the derivative is simply the slope of the original signal: where a signal slopes up, its derivative is positive; where a signal slopes down, its derivative is negative; and where a signal has zero slope, its derivative is zero. The sigmoidal signal shown in Window 1 has an inflection point (point where where the slope is maximum) at the center of the x axis range. This corresponds to the
- 15. maximum in its first derivative (Window 2) and to the zero-crossing (point where the signal crosses the x-axis going either from positive to negative or vice versa) in the second derivative in Window 3. This behavior can be useful for precisely locating the inflection point in a sigmoid signal, by computing the location of the zero-crossing in its second derivative. Similarly, the location of maximum in a peak-type signal can be computed precisely by computing the location of the zero-crossing in its first derivative. Another important property of the differentiation of peak-type signals is the effect of the peak width on the amplitude of derivatives. The figure on the left shows the results of the successive differentiation of two computer-generated Gaussian bands. The two bands have the same amplitude (peak height) but one of them is exactly twice the width of the other. As you can see, the wider peak has the smaller derivative amplitude, and the effect becomes more noticeable at higher derivative orders. In general, it is found that that the amplitude of the nth derivative of a peak is inversely proportional to the nth power of its width. Thus differentiation in effect discriminates against wider peaks and the higher the order of differentiation the greater the discrimination. This behavior can be useful in quantitative analytical applications for detecting peaks that are superimposed on and obscured by stronger but broader background peaks. Applications of Differentiation A simple example of the application of differentiation of experimental signals is shown in Figure 5. This signal is typical of the type of signal recorded in amperometric titrations and some kinds of thermal analysis and kinetic experiments: a series of straight line segments of different slope. The objective is to determine how many segments there are, where the breaks between then fall, and the slopes of each segment. This is difficult to do from the raw data, because the slope differences are small and the resolution of the computer screen display is limiting. The task is much simpler if the first derivative (slope) of the signal is calculated (Figure 5, right). Each segment is now clearly seen as a separate step whose height (y-axis value) is the slope. The y-axis now takes on the units of dy/dx. Note that in this example the steps in the derivative signal are not completely flat, indicating that the line segments in the original signal were not perfectly straight. This is most likely due to random noise in the original signal. Although this noise was not particularly evident in the original signal, it is more noticeable in the derivative
- 16. . Figure 5. The signal on the left seems to be a more-or-less straight line, but its numerically calculated derivative (dx/dy), plotted on the right, shows that the line actually has several approximately straight-line segments with distinctly different slopes and with well-defined breaks between each segment. It is commonly observed that differentiation degrades signal-to-noise ratio, unless the differentiation algorithm includes smoothing that is carefully optimized for each application. Numerical algorithms for differentiation are as numerous as for smoothing and must be carefully chosen to control signal-to-noise degradation. A classic use of second differentiation in chemical analysis is in the location of endpoints in potentiometric titration. In most titrations, the titration curve has a sigmoidal shape and the endpoint is indicated by the inflection point, the point where the slope is maximum and the curvature is zero. The first derivative of the titration curve will therefore exhibit a maximum at the inflection point, and the second derivative will exhibit a zero-crossing at that point. Maxima and zero crossings are usually much easier to locate precisely than inflection points. Figure 6 The signal on the left is the pH titration curve of a very weak acid with a strong base, with volume in mL on the X-axis and pH on the Y-axis. The endpoint is the point of greatest slope; this is also an inflection point, where the curvature of the signal is zero. With a weak acid such as this, it is difficult to locate this point precisely from the original titration curve. The endpoint is much more easily located in the second derivative, shown on the right, as the the zero crossing. Figure 6 shows a pH titration curve of a very weak acid with a strong base, with volume in mL on the X-axis and pH on the Y-axis. The volumetric equivalence point (the "theoretical" endpoint) is 20 mL. The endpoint is the point of greatest slope; this is also
- 17. an inflection point, where the curvature of the signal is zero. With a weak acid such as this, it is difficult to locate this point precisely from the original titration curve. The second derivative of the curve is shown in Window 2 on the right. The zero crossing of the second derivative corresponds to the endpoint and is much more precisely measurable. Note that in the second derivative plot, both the x-axis and the y-axis scales have been expanded to show the zero crossing point more clearly. The dotted lines show that the zero crossing falls at about 19.4 mL, close to the theoretical value of 20 mL. Another common use of differentiation is in the detection of peaks in a signal. It's clear from the basic properties described in the previous section that the first derivative of a peak has a downward-going zero-crossing at the peak maximum, which can be used to locate the x-value of the peak. But the presence of random noise in real experimental signal will cause many false zero-crossings simply due to the noise. To avoid this problem, one popular technique smooths the first derivative of the signal first, before looking for downward-going zero-crossings, and then takes only those zero crossings whose slope exceeds a certain predetermined minimum (called the "slope threshold") at a point where the original signal amplitude exceeds a certain minimum (called the "amplitude threshold"). By carefully adjusting the smooth width, slope threshold, and amplitude threshold, it is possible to detect only the desired peaks and ignore peaks that are too small, too wide, or too narrow. Moreover, because smoothing can distort peak signals, reducing peak heights, and increasing peak widths, this technique determines the position, height, and width of each peak by least-squares curve-fitting of a segment of original unsmoothed signal in the vicinity of the zero-crossing. Thus, even if heavy smoothing is necessary to provide reliable discrimination against noise peaks, the peak parameters extracted by curve fitting are not distorted. Derivative Spectroscopy In spectroscopy, the differentiation of spectra is a widely used technique, particularly in infra-red, u.v.-visible absorption, fluorescence, and reflectance spectrophotometry, referred to as derivative spectroscopy. Derivative methods have been used in analytical spectroscopy for three main purposes: (a) spectral discrimination, as a qualitative fingerprinting technique to accentuate small structural differences between nearly identical spectra; (b) spectral resolution enhancement, as a technique for increasing the apparent resolution of overlapping spectral bands in order to more easily determine the number of bands and their wavelengths; (c) quantitative analysis, as a technique for the correction for irrelevant background absorption and as a way to facilitate multicomponent analysis. (Because differentiation is a linear technique, the amplitude of a derivative is proportional to the amplitude of the original signal, which allows quantitative analysis applications employing any of the standard calibration techniques). Most commercial spectrophotometers now have built-in derivative capability. Some instruments are designed to measure the spectral derivatives optically, by means of dual wavelength or wavelength modulation designs. Because of the fact that the amplitude of the nth derivative of a peak-shaped signal is inversely proportional to the nth power of the width of the peak, differentiation may be employed as a general way to discriminate against broad spectral features in favor of narrow components. This is the basis for the application of differentiation as a method of
- 18. correction for background signals in quantitative spectrophotometric analysis. Very often in the practical applications of spectrophotometry to the analysis of complex samples, the spectral bands of the analyte (i.e. the compound to be measured) are superimposed on a broad, gradually curved background. Background of this type can be reduced by differentiation. This is illustrated by the figure on the left, which shows a simulated UV spectrum (absorbance vs wavelength in nm), with the green curve representing the spectrum of the pure analyte and the red line representing the spectrum of a mixture containing the analyte plus other compounds that give rise to the large sloping background absorption. The first derivatives of these two signals are shown in the center; you can see that the difference between the pure analyte spectrum (green) and the mixture spectrum (red) is reduced. This effect is considerably enhanced in the second derivative, shown on the right. In this case the spectra of the pure analyte and of the mixture are almost identical. In order for this technique to work, it is necessary that the background absorption be broader (that is, have lower curvature) than the analyte spectral peak, but this turns out to be a rather common situation. Because of their greater discrimination against broad background, second (and sometimes even higher-order) derivatives are often used for such purposes. It is sometimes (mistakenly) said that differentiation "increases the sensitivity" of analysis. You can see how it would be tempting to say something like that by inspecting the three figures above; it does seems that the signal amplitude of the derivatives is greater (at least graphically) than that of the original analyte signal. However, it is not valid to compare the amplitudes of signals and their derivatives because they have different units. The units of the original spectrum are absorbance; the units of the first derivative are absorbance per nm, and the the units of the second derivative are absorbance per nm2. You can't compare absorbance to absorbance per nm any more than you can compare miles to miles per hour. (It's meaningless, for instance, to say that 30 miles per hour is greater than 20 miles.) You can, however, compare the signal-to- background ratio and the signal-to-noise ratio. For example, in the above example, it would be valid to say that the signal-to-background ratio is increased in the derivatives. Trace Analysis One of the widest uses of the derivative signal processing technique in practical analytical work is in the measurement of small amounts of substances in the presence of large amounts of potentially interfering materials. In such applications it is common that the
- 19. analytical signals are weak, noisy, and superimposed on large background signals. Measurement precision is often degraded by sample-to-sample baseline shifts due to non- specific broadband interfering absorption, non-reproducible cuvette positioning, dirt or fingerprints on the cuvette walls, imperfect cuvette transmission matching, and solution turbidity. Baseline shifts from these sources are usually either wavelength-independent (light blockage caused by bubbles or large suspended particles) or exhibit a weak wavelength dependence (small-particle turbidity). Therefore it can be expected that differentiation will in general help to discriminate relevant absorption from these sources of baseline shift. An obvious benefit of the suppression of broad background by differentiation is that variations in the background amplitude from sample to sample are also reduced. This can result in improved precision or measurement in many instances, especially when the analyte signal is small compared to the background and if there is a lot of uncontrolled variability in the background. An example of the improved ability to detect trace component in the presence of strong background interference is shown in Figure 7. Figure 7. The spectrum on the left shows a weak shoulder near the center due to a small concentration of the substance that is to be measured (e.g. the active ingredient in a pharmaceutical preparation). It is difficult to measure the intensity of this peak because it is obscured by the strong background caused by other substances in the sample. The fourth derivative of this spectrum is shown on the right. The background has been almost completely suppressed and the analyte peak now stands out clearly, facilitating measurement. The spectrum on the left shows a weak shoulder near the center due to the analyte. The signal-to-noise ratio is very good in this spectrum, but in spite of that the broad, sloping background obscures the peak and makes quantitative measurement very difficult. The fourth derivative of this spectrum is shown on the right. The background has been almost completely suppressed and the analyte peak now stands out clearly, facilitating measurement. An even more dramatic case is shown in Figure 8. This is essentially the same spectrum as in Figure 7, except that the concentration of the analyte is lower. The question is: is there a detectable amount of analyte in this spectrum? This is quite impossible to say from the normal spectrum, but inspection of the fourth derivative (right) shows that the answer is yes. Some noise is clearly evident here, but nevertheless the signal-to-noise ratio is sufficiently good for a reasonable quantitative measurement.
- 20. Figure 8. Similar to Figure 7, but in the case the peak is so weak that it can not even be seen in the spectrum on the left. The fourth derivative (right) shows that a peak is still there, but much reduced in amplitude (note the smaller y-axis scale). This use of signal differentiation has become widely used in quantitative spectroscopy, particularly for quality control in the pharmaceutical industry. In that application the analyte would typically be the active ingredient in a pharmaceutical preparation and the background interferences might arise from the presence of fillers, emulsifiers, flavoring or coloring agents, buffers, stabilizers, or other excipients. Of course, in trace analysis applications, care must be taken to optimize signal-to-noise ratio of the instrument as much as possible. The Importance of Smoothing Derivatives For the successful application of differentiation in quantitative analytical applications, it is essential to use differentiation in combination with sufficient smoothing, in order to optimize the signal-to-noise ratio. This is illustrated in the figure on the left. Window 1 shows a Gaussian band with a small amount of added noise. Windows 2, 3, and 4, show the first derivative of that signal with increasing smooth widths. As you can see, without sufficient smoothing, the signal-to-noise ratio of the derivative can be substantially poorer than the original signal. However, with adequate amounts of smoothing, the signal-to-noise ratio of the smoothed derivative can be better than that of the unsmoothed original. This effect is even more striking in the second derivative, as shown below. In this case, the signal-to-noise ratio of the unsmoothed second derivative (Window 2) is so poor you can not even see the signal visually. It makes no difference whether the smooth operation is applied before or after the
- 21. differentiation. What is important, however, is the nature of the smooth, its smooth ratio (ratio of the smooth width to the width of the original peak), and the number of times the signal is smoothed. The optimum values of smooth ratio for derivative signals is approximately 0.5 to 1.0. For a first derivative, two applications of a simple rectangular smooth or one application of a triangular smooth is adequate. For a second derivative, three applications of a simple rectangular smooth or two applications of a triangular smooth is adequate. The general rule is: for the nth derivative, use at least n+1 applications of rectangular smooth (or half that number of triangular smooths). Such heavy amounts of smoothing result in substantial attenuation of the derivative amplitude; in the figure on the right above, the amplitude of the most heavily smoothed derivative (in Window 4, above) is much less than its less-smoothed version (Window 3). However, this won't be a problem, as long as the standard (analytical) curve is prepared using the exact same derivative, smoothing, and measurement procedure as is applied to the unknown samples. (Because differentiation and smoothing are both linear techniques, the amplitude of a smoothed derivative is proportional to the amplitude of the original signal, which allows quantitative analysis applications employing any of the standard calibration techniques). SPECTRUM for Macintosh OS 8 includes first and second derivative functions, which can be applied successively to compute derivatives of any order. Differentiation functions such as described above can easily be created in Matlab. Some simple examples that you can download include: deriv, a first derivative using the 2-point central- difference method, deriv2, a simple second derivative using the 3-point central-difference method, a third derivative deriv3 using a 4-point formula, and deriv4, a 4th derivative using a 5- point formula. Each of these is a simple Matlab function of the form d=deriv(a); the input argument is a signal vector "a", and the differentiated signal is returned as the vector "d". If you are viewing this document on-line, you can ctrl-click on these links to inspect the code. IiSignal is a Matlab function that performs differentiation and smoothing for time-series signals, up to the 4th derivative, with keystrokes that allow you to adjust the smoothing parameters continuously while observing the effect on your signal dynamically. View the code here or download the ZIP file that adds sample data for testing. Interactive Derivative for Matlab 6.5 is a collection of Matlab functions and scripts for
- 22. interactive differentiation of time-series signals, with sliders that allow you to adjust the derivative order, smooth width, and scale expansion continuously while observing the effect on your signal dynamically. Requires Matlab 6.5. Click here to download the ZIP file "InteractiveDerivative.zip" that also includes supporting functions, self-contained demos to show how it works. Run InteractiveDerivativeTest to see how it works. Also includes DerivativeDemo.m, which demonstrates the application of differentiation to the detection of peaks superimposed on a strong, variable background. Generates a signal peak, adds random noise and a variable background, then differentiates and smooths it, and measures the signal range and signal-to-noise ratio (SNR). Interactive sliders allow you to control all the parameters. Derivative-based Peak Finding and Measurement in Matlab Appendix 3, page 58, describes a fast, customizable Matlab routine for locating and measuring the peaks in noisy time-series data sets. It detects peaks by looking for downward zero-crossings in the smoothed first derivative. Determines the position, height, and width of each peak by least-squares curve-fitting. (This is useful primarily for signals that have several data points in each peak, not for spikes that have only one or two points). It can find and measure 1000 peaks in a 1,000,000 point signal in 8 seconds. The routine is available in three different versions: (1) the basic findpeaks function (); (2) an interactive keypress-operated function (ipeak.m) for adjusting the peak detection criteria in real-time to optimize for any particular peak type; and (3) a script using mouse- controlled sliders for interactive control. You can also download it from the Matlab File Exchange. See Appendix 3, page 58, for details. Resolution enhancement (Peak Sharpening) Figure 9 shows a spectrum that consists of several poorly-resolved (that is, partly overlapping) bands. The extensive overlap of the bands makes the accurate measurement of their intensities and positions impossible, even though the signal-to-noise ratio is very good. Things would be easier if the bands were more completely resolved, that is, if the bands were narrower. Figure 9. A resolution enhancement algorithm has been applied to the signal on the left to artificially improve the apparent resolution of the peaks. In the resulting signal, right, the component bands are narrowed so that the intensities and positions can be measured.
- 23. Here use can be made of resolution enhancement algorithms to artificially improve the apparent resolution of the peaks. One of the simplest such algorithms is based on the weighted sum of the original signal and the negative of its second derivative. where Rj is the resolution-enhanced signal, Y is the original signal, Y'' is the second derivative of Y, and k is a user-selected weighting factor. It is left to the user to select the weighting factor k which gives the best trade-off between resolution enhancement, signal- to-noise degradation, and baseline undershoot. The optimum choice depends upon the width, shape, and digitization interval of the signal. (As a starting point, a reasonable value for k is w2/25 for peaks of Gaussian shape, or w2/6 for peaks of Lorentzian shape, where w is the number of data points in the half-width of the component peaks). The result of the application of this algorithm is shown on the right in Figure 9. The component bands have been artificially narrowed so that the intensities and positions can be measured. However, the signal-to-noise ratio is degraded. Here's how it works. The figure shows, in Window 1, a computer-generated peak (with a Lorentzian shape) in red, superimposed on the negative of its second derivative in green). The second derivative is amplified (by multiplying it by an adjustable constant) so that the negative sides of the inverted second derivative (from approximately X = 0 to 100 and from X = 150 to 250) are a mirror image of the sides of the original peak over those regions. In this way, when the original peak is added to the inverted second derivative, the two signals will approximately cancel out in the two side regions but will reinforce each other in the central region (from X = 100 to 150). The result, shown in Window 2, is a substantial (about 50%) reduction in the width, and an increase in height, of the peak. This works best with Lorentzian shaped peaks; with Gaussian-shaped peaks, the resolution enhancement is less dramatic (only about 20%). An interesting property of this procedure is that it does not change the total peak area (that is, the area under the peak) because the total area under the curve of the derivative of a peak-shaped signal is zero (the area under the negatives lobes cancels the area under the positive lobes). Because differentiation and smoothing are both linear techniques, the amplitude of a sharpened signal is therefore proportional to the amplitude of the original signal, which allows quantitative analysis applications employing any of the standard calibration techniques). As long as you apply the same signal-processing techniques to the standards as well as to the samples, everything works. Note: Another technique that can increase the resolution of overlapping peaks is deconvolution, which is applicable when the broadening function responsible for the overlap of the peaks is known. Deconvolution of the broadening function from the
- 24. broadened peaks is in principle capable of extracting the underlying peaks shapes, whereas this resolution enhancement technique can not be expected to do that. SPECTRUM includes this simple resolution-enhancement algorithm, with adjustable weighting factor and derivative smoothing width. Resolution enhancement in Matlab. The user-defined Matlab function enhance.m (ctrl-click on this link to inspect the code) uses a slightly more advanced algorithm that extends the above approach by adding in a small amount of the 4th derivative of the signal: R = Y - k2Y'' + k4Y'''' This function has the form: Enhancedsignal=enhance(signal,k2,k4,SmoothWidth), where signal is the original signal vector, the arguments k2 and k4 are 2nd and 4th derivative weighting factors, and SmoothWidth is the width of the built-in smooth. The resolution- enhanced signal is returned in the vector Enhancedsignal. iSignal is a Matlab function that performs resolution enhancement for time-series signals, using the above enhance function, with keystrokes that allow you to adjust the 2nd and 4th derivative weighting factors and the smoothing continuously while observing the effect on your signal dynamically. The E key turns the resolution enhancement function on and off. View the code here or download the ZIP file with sample data for testing. iSignal calculates the sharpening and smoothing settings for Gaussian and for Lorentzian peak shapes using the Y and U keys, respectively. Just isolate a single typical peak in the upper window using the pan and zoom keys, then press Y for Gaussian or U for Lorentzian peaks. (The optimum settings depends on the width of the peak, so if your signal has peaks of widely different widths, one setting will not be optimum for all the peaks). You can fine-tune the sharpening with theF/V and G/B keys and the smoothing with the A/Z keys. See http://terpconnect.umd.edu/~toh/spectrum/iSignal.html Video Demonstration. This 15-second, 1.7 MByte video (ResEnhance3.wmv) demonstrates the Matlab interactive resolution enhancement function InteractiveResEnhance.m. The signal consists of four overlapping, poorly-resolved Lorentzian bands. First, the 2nd derivative factor (Factor 1) is adjusted, then the 4th derivative factor (Factor 2) is adjusted, then the smooth width (Smooth) is adjusted, and finally the Factor 2 is tweaked again. Harmonic analysis and the Fourier Transform Some signals exhibit periodic components that repeat at fixed intervals throughout the signal, like a sine wave. It is often useful to describe the amplitude and frequency of such periodic components exactly. Actually, it is possible to analyze any arbitrary set of data into periodic components, whether or not the data appear periodic. Harmonic analysis is conventionally based on the Fourier transform, which is a way of expressing a signal as a sum of sine and cosine waves. It can be shown that any arbitrary discretely sampled signal can be described completely by the sum of a finite number of sine and cosine components whose frequencies are 0,1,2,3 ... n/2 times the fundamental frequency f=1/nDx, where Dx is the interval between adjacent x-axis values and n is the total number of points. The Fourier transform is simply the coefficients of these sine and cosine components.
- 25. The concept of the Fourier transform is involved in two very important instrumental methods in chemistry. In Fourier transform infrared spectroscopy (FTIR), the Fourier transform of the spectrum is measured directly by the instrument, as the interferogram formed by plotting the detector signal vs mirror displacement in a scanning Michaelson interferometer. In Fourier transform nuclear magnetic resonance spectroscopy (FTNMR), excitation of the sample by an intense, short pulse of radio frequency energy produces a free induction decay signal that is the Fourier transform of the resonance spectrum. In both cases the spectrum is recovered by inverse Fourier transformation of the measured signal. The power spectrum is a simple way of showing the total amplitude at each of these frequencies; it is calculated as the square root of the sum of the squares of the coefficients of the sine and cosine components. Figure 10. The signal on the left (x = time; y = voltage), which was expected to contain a single peak, is clearly very noisy. The power spectrum of this signal (x-axis = frequency in Hz) shows a strong component at 60 Hz, suggesting that much of the noise is caused by stray pick-up from the 60 Hz power line. The smaller peak at 120 Hz (the second harmonic of 60 Hz) probably comes from the same source. A signal with n points gives a power spectrum with only (n/2)+1 points. The x-axis is the harmonic number. The first point (x=0) is the zero-frequency (constant) component. The second point (x=1) corresponds to the fundamental frequency, the next point to twice the fundamental frequency, etc. An example of a practical application of the use of the power spectrum as a diagnostic tool is shown in Figure 10. The two figures above show a classic example of harmonic analysis of the annual variation in the number of observed sunspots, which have been recorded since the year 1700! In this case the time axis is in years. A plot of the power spectrum (on the left)
- 26. shows a strong peak at 0.09 cycles/year; if the data are replotted (on the right) with time (the reciprocal of frequency) on the x-axis, the plot shows the well-known 11-year cycle. (You can download these yearly sunspot data from NOAA at ftp://ftp.ngdc.noaa.gov/STP/SOLAR_DATA/SUNSPOT_NUMBERS/INTERNATIONA L/yearly/YEARLY.PLT). In all the examples shown here, the signal is a time-series signal with time as the independent variable. More generally, it is also possible to compute the Fourier transform and power spectrum of any signal, such as an optical spectrum, where the independent variable might be wavelength or wavenumber, or an electrochemical signal, where the independent variable might be volts. In such cases the units of the x-axis of the power spectrum are simply the reciprocal of the units of the x-axis of the original signal (e.g. nm-1 for a signal whose x-axis is in nm). SPECTRUM includes a power spectrum function, as well as forward and reverse Fourier transformation. Matlab has built-in functions for computing the Fourier transform (FFT and IFFT) and the power spectrum (powerspectrum(f,T)). Interactive Power Spectrum Demo for Matlab iPower is a keyboard-controlled interactive power spectrum demonstrator, useful for teaching and learning about the power spectra of different types of signals and the effect of signal duration and sampling rate. Single keystrokes allow you to select the type of signal (12 different signals included), the total duration of the signal, the sampling rate, and the global variables f1 and f2 which are used in different ways in the different signals. When the Enter key is pressed, the signal (y) is sent to the Windows WAVE audio device. Press K to see a list of all the keyboard commands. Tested in Matlab version 7.8 (R2009a). See http://terpconnect.umd.edu/~toh/spectrum/HarmonicAnalysis.html Click here to view or download. You can also download it from the Matlab File Exchange. T. C. O'Haver (toh@umd.edu), version 2, October 2011 Convolution Convolution is an operation performed on two signals which involves multiplying one signal by a delayed version of another signal, integrating or averaging the product, and repeating the process for different delays. Convolution is a useful process because it accurately describes some effects that occur widely in scientific measurements, such as the influence of a low-pass filter on an electrical signal or of a spectrometer on the shape of a spectrum. In practice the calculation is usually performed by multiplication of the two signals in the
- 27. Fourier domain. First, the Fourier transform of each signal is obtained. Then the two Fourier transforms are multiplied by the rules for complex multiplication and the result is then inverse Fourier transformed. Although this seems to be a round-about method, it turns out to be faster then the shift-and-multiply algorithm when the number of points in the signal is large. Convolution can be used as a very powerful and general algorithm for smoothing and differentiation. The example of Figure 11 shows how it can be used to predict the broadening effect of a spectrometer on an atomic line spectrum. Figure 11. Convolution is used here to determine how the atomic line spectrum in Window 1 (top left) will appear when scanned with a spectrometer whose slit function (spectral resolution) is described by the Gaussian function in Window 2 (top right). The Gaussian function has already been rotated so that its maximum falls at x=0. The resulting convoluted spectrum (bottom center) shows that the two lines near x=110 and 120 will not be resolved but the line at x=40 will be partly resolved. SPECTRUM includes convolution and auto-correlation (self-convolution) functions. Matlab has a built-in function for convolution: conv. This function can be used to create very general type of filters and smoothing functions, such as sliding-average and triangular smooths. Deconvolution Deconvolution is the converse of convolution in the sense that division is the converse of multiplication. In fact, the deconvolution of one signal from another is usually performed by dividing the two signals in the Fourier domain. The practical significance of deconvolution is that it can be used as an artificial (i.e. computational) way to reverse the result of a convolution occurring in the physical domain, for example, to reverse the signal distortion effect of an electrical filter or of the finite resolution of a spectrometer. Two examples of the application of deconvolution are shown in Figures 12 and 13.
- 28. Figure 12. Deconvolution is used here to remove the distorting influence of an exponential tailing response function from a recorded signal (Window 1, left) that is the result of an unavoidable RC low-pass filter action in the electronics. The response function (Window 2, center) is usually either calculated on the basis of some theoretical model or is measured experimentally as the output signal produced by applying an impulse (delta) function to the input of the system. The response function, with its maximum at x=0, is deconvoluted from the original signal . The result (right) shows a closer approximation to the real shape of the peaks; however, the signal-to-noise ratio is unavoidably degraded. Figure 13. A different application of the deconvolution function is to reveal the nature of an unknown data transformation function that has been applied to a data set by the measurement instrument itself. In this example, Window 1 (top left) is a uv-visible absorption spectrum recorded from a commercial photodiode array spectrometer (X-axis: nanometers; Y-axis: milliabsorbance). Window 2 (top right) is the first derivative of this spectrum produced by an (unknown) algorithm in the software supplied with the spectrometer. The signal in the bottom left is the result of deconvoluting the derivative spectrum (top right) from the original spectrum (top left). This therefore must be the convolution function used by the differentiation algorithm in the spectrometer's software. Rotating and expanding it on the x-axis makes the function easier to see (bottom right). Expressed in terms of the smallest whole numbers, the convolution series is seen to be +2, +1, 0, -1, -2. This simple example of "reverse engineering" would make it easier to compare results from other instruments or to duplicate these result on other equipment. When applying deconvolution to experimental data, to remove the effect of a known broadening or low-pass filter operator caused by the experimental system, a very serious signal-to-noise degradation commonly occurs. Any noise added to the signal by the system after the broadening or low-pass filter operator will be greatly amplified when the Fourier transform of the signal is divided by the Fourier transform of the broadening operator, because the high frequency components of the broadening operator (the denominator in the division of the Fourier transforms) are typically very small, resulting in a great amplification of high frequency noise in the resulting deconvoluted signal. This can be controlled but not completely eliminated by smoothing and by constraining the deconvolution to a frequency region where the signal has a sufficiently high signal-to-
- 29. noise ratio. Note: The term "deconvolution" is sometimes also used for the process of resolving or decomposing a set of overlapping peaks into their separate components by the technique of iterative least-squares curve fitting of a putative peak model to the data set. The process is actually conceptually distinct from deconvolution, because in deconvolution the underlying peak shape is unknown but the broadening function is assumed to be known; whereas in iterative least-squares curve fitting the underlying peak shape is assumed to be known but the broadening function is unknown. SPECTRUM includes a deconvolution function. Matlab has built-in function for deconvolution: deconv. Fourier filter The Fourier filter is a type of filtering or smoothing function that is based on the frequency components of a signal. It works by taking the Fourier transform of the signal, then cutting off all frequencies above a user-specified limit, then inverse transforming the result. The assumption is made here that the frequency components of the signal fall predominantly at low frequencies and those of the noise fall predominantly at high frequencies. The user tries to find a cut-off frequency that will allow most of the noise to be eliminated while not distorting the signal significantly. An example of the application of the Fourier filter is given in Figure 14. Figure 14. The signal at the far left seems to be only random noise, but its power spectrum (Window 2, second from left) shows that high-frequency components dominate the signal. The power spectrum is expanded in the X and Y directions to show more clearly the low-frequency region (Window 1, right). Working on the hypothesis that the components above the 20th harmonic are noise, the Fourier filter function can be used to delete the higher harmonics and to reconstruct the signal from the first 20 harmonics. The result (far right) shows the signal contains two bands at about x=200 and x=300 that are totally obscured by noise in the original signal. SPECTRUM includes a simple Fourier low-pass filter function with adjustable harmonic cut-off. The custom Matlab function FouFilter.m is a more flexible Fourier filter that can serve as a lowpass, highpass, bandpass, or bandreject (notch) filter with variable cut-off rate. Has the form ry=FouFilter(y,samplingtime,centerfrequency,frequencywidth,shape,mode) where y is the time-series signal vector, 'samplingtime' is the total duration of sampled signal in sec, millisec, or microsec; 'centerfrequency' and 'frequencywidth' are the center frequency and
- 30. Integration and peak area measurement The symbolic integration of functions and the calculation of definite integrals are topics that are introduced in elementary Calculus courses. The numerical integration of digitized signals finds application in analytical signal processing mainly as a method for measuring the areas under the curves of peak-type signals. Peak area measurements are very important in chromatography. Quantitation in chromatography is customarily carried out on the basis of peak area rather than peak height measurement. The reason for this is that peak area is less sensitive to the influence of peak broadening (dispersion) mechanisms. These broadening effects, which arise from many sources, cause chromatographic peaks to become shorter, broader, and more unsymmetrical, but have little effect on the total area under the peak. The peak area remains proportional to the total quantity of substance passing into the detector. Therefore peak area measurements are often found to be more reliable than peak height measurement. The simple numeric integration of a digital, e.g. by Simpson's rule, will convert a series of peaks into a series of steps, the height of each of which is proportional to the area under that peak. This is a commonly used method in proton NMR spectroscopy, where the area under each peak or multiplet is proportional to the number of equivalent hydrogen atoms responsible for that peak. But this works well only if the peaks are well separated from each other (e.g. well resolved). In chromatographic analysis one often has the problem of measuring the the area under the curve of the peaks when they are not well resolved or are superimposed on a background. For example, Figure 15 shows a series of four computer-synthesized Gaussian peaks that all have the same height, width, and area, but the separation between the peaks on the right is insufficient to achieve complete resolution. Figure 15. Peak area measurement for overlapping peaks, using the perpendicular drop method.
- 31. The classical way to handle this problem is to draw two vertical lines from the left and right bounds of the peak down to the x-axis and then to measure the total area bounded by the signal curve, the x-axis (y=0 line), and the two vertical lines. This is often called the the perpendicular drop method, and it is an easy task for a computer, although very tedious to do by hand. The idea is illustrated for the second peak from the left in Figure 15. The left and right bounds of the peak are usually taken as the valleys (minima) between the peaks. Using this method it is possible to estimate the area of the second peak in this example to an accuracy of about 0.3% and the second and third peaks to an accuracy of better than 4%, despite the poor resolution. Peak area measurement using Matlab. iSignal is a downloadable user-defined Matlab function that performs various signal processing functions described in this tutorial, including measurement of peak area using Simpson's Rule and the perpendicular drop method. Click to view or right-click > Save link as... here, or you can download the ZIP file with sample data for testing. For highly overlapped peaks of Gaussian or Lorentzian shapes, the most accurate measurements can be made with iterative least-squares peak fitting, for example using the Interactive Peak Fitter (http://terpconnect.umd.edu/~toh/spectrum/InteractivePeakFitter.htm).
- 32. Curve fitting A: Linear Least Squares The objective of curve fitting is to find a mathematical equation that describes a set of data and that is minimally influenced by random noise. The most common approach is the "linear least squares" method, also called "polynomial least squares", a well-known mathematical procedure for finding the coefficients of polynomial equations that are a "best fit" to a set of X,Y data. A polynomial equation expresses the dependent variable Y as a polynomial in the independent variable X, for example as a straight line (Y = a + bX, where a is the intercept and b is the slope), or a quadratic (Y = a + bX + cX2), or a cubic (Y = a + bX + cX2 + dX3), or higher-order polynomial. In all these cases, Y is a linear function of the parameters a,b,c, and d. This is why this is called "linear" least-squares fit, not because the plot of X vs Y is linear. Only for the first-order polynomial is the plot of X vs Y linear. (Sometimes this type of curve fitting is called "curvilinear"). The graph shows a simple straight-line example (Y = a + bX), a small set of data where X = concentration and Y = instrument reading. The blue dots are the data points. The least-squares algorithm computes the values of a (intercept) and b (slope) of the straight line that is a least-squares best fit to the data points. For this set of data, the slope is 9.7926 and the intercept is 0.199. The least squares procedure also calculates R2, the coefficient of determination, also called the correlation coefficient, which is an indicator of the "goodness of fit". R2 is exactly 1.0000 when the fit is perfect, less than that when the fit is imperfect. The closer to 1.0000 the better. The points do not all fall on the straight line because of random noise in the data. (In this example, the "true" value of the slope is 10 and of the intercept is zero, so the presence of the noise caused this particular measurement of slope to be off by about 2%. Had there been a larger number of points in this data set, the calculated values of slope and intercept would probably have been better). A simple way to evaluate a curve fit is to look at a plot of the "residuals", the differences between the y values in the original data and the y values computed by the fit equation. If the residuals are completely random and show no systematic variation, then the fit is as good as possible for that level of noise; but if the residual plot shows a regular curve, for example a U-shaped or an S-shaped curve, it may mean that another fitting function would be more appropriate. Least-squares best fits can be calculated by some hand-held calculators, spreadsheets, and dedicated computer programs (see below for details). Although it is possible to estimate the best-fit straight line by visual estimation and a straightedge, the least-square method is more objective and easier to automate. (If you were to give this set of data to five different people and ask them to estimate the best-fit line visually, you'd get five slightly
- 33. different answers, but it you gave the data set to five different computer programs, you'd get the exact same answer every time). Transforming non-linear relationships In some cases a fundamentally non-linear relationship can be transformed into a form that is amenable to polynomial curve fitting by means of a coordinate transformation (e.g. taking the log or the reciprocal of the data), and then the least-squares method can be applied to the resulting linear equation. For example, the points in Figure 16 are from the simulation of an exponential decay (X = time, Y = signal intensity) that has the mathematical form Y = a exp(bX), where a is the Y-value at X = 0 and b is the decay constant. In this particular example, a = 1 and b = -1. This is a fundamentally non-linear problem because Y is a non-linear function of the parameter b. However, by taking the natural log of both sides of the equation, we obtain ln(Y) = ln(a) + bX. In this equation, ln(Y) is a linear function of ln(a) and b, so it can be fit by the least squares method, from which values of a and b are easily calculated. An exponential least-squares fit (solid line) applied to a noisy data set (points) in order to estimate the decay constant. The best fit equation, shown by the green solid line in the figure, is Y = 0.9897 exp(- 0.98896 X), that is a = 0.9897 and b = -0.98896, very close to the expected values. Thus, even in the presence of substantial random noise, it is possible to get good estimates of the parameters of the underlying equation. The most important requirement is that the model be good, that is, that the equation selected for the model accurately describes the underlying behavior of the system (except for noise). Often that is the most difficult aspect, because the underlying models are not always known with certainty. Other examples of non-linear relationships that can be linearized by coordinate transformation include the logarithmic (Y = a ln(bX)) and power (Y=aXb) relationships. Methods of this type used to be very common back in the days before computers, when fitting anything but a straight line was difficult. It is still used today to extend the range of functional relationships that can be handled by common linear least-squares routines. (Only a few non-linear relationships can be handled this way, however. To fit any arbitrary custom function, you may have to resort to the more difficult Non-linear Iterative Curve Fitting method). Another example of the use of transformation to convert a non-linear relationship into a form that is amenable to polynomial curve fitting is the use of the natural log (ln) transformation to convert a Gaussian peak, which has the fundamental functional form exp(-x2), into a parabola of the form -x2, which can be fit with a second order polynomial (quadratic) function (y = a + bx + cx2). All three parameters of the Gaussian (height, maximum position, and width) can be calculated from the three quadratic coefficients a, b, and c; the peak height is given by exp(a-c*(b/(2*c))^2), the peak position by -b/(2*c), and the peak half-width by 2.35703/(sqrt(2)*sqrt(-c)). An example is of this type of Gaussian curve fitting is shown in the figure on the left. The signal is a Gaussian peak with a true peak height of 100 units, a true peak position of 100 units, and a true half-width of 100 units, but it is sparsely sampled only every 31 units on the x-axis. The resulting data set, shown by the red points in the upper left, has only 6
- 34. data points on the peak itself. If we were to take the maximum of those 6 points (the 3rd point from the left, with x=87, y=95) as the peak maximum, that would not be very close to the true values of peak position and height (100). If we were to take the distance between the 2nd the 5th data points as the peak width, we'd get 3*31=93, again not very close to the true value of 100. However, taking the natural log of the data (upper right) produces a parabola that can be fit with a quadratic least-squares fit (shown by the blue line in the lower left). From the three coefficients of the quadratic fit we can calculate much more accurate values of the Gaussian peak parameters, shown at the bottom of the figure (height=100.57; position=98.96; width=99.2). The plot in the lower right shows the resulting Gaussian fit (in blue) displayed with the original data (red points). The accuracy of those peak parameters (about 1% in this example) is limited only by the noise in the data. This figure was created in Matlab, using this script. Note: in order for this method to work properly, the data set must not contain any zeros or negatives points, and the original Gaussian peak signal must have a zero baseline, that is, must tend to zero far from the peak center. In practice this means that any non-zero baseline must be subtracted from the data set before applying this method. (A more general approach to fitting Gaussian peaks, which works for data sets with zeros and negative numbers and also for data with multiple overlapping peaks, is the non-linear iterative curve fitting method). Note 1: you can right-click on any of the m-file links above and select Save Link As... to download them to your computer for use within Matlab. Note 2: In the curve fitting techniques described here and in the next two sections, there is no requirement that the x-axis interval between data points be uniform, as is the assumption in many of the other signal processing techniques previously covered. Curve fitting algorithms typically accept a set of arbitrarily-spaced x-axis values and a corresponding set of y-axis values. Math details The least-squares best fit for an x,y data set can be computed using only basic arithmetic. Here are the relevant equations for computing the slope and intercept of the first-order best-fit equation, y = intercept + slope*x, as well as the coefficient of determination, "R2", which is an indicator of the "goodness of fit". (R2 is 1.0000 if the fit is perfect and less than that if the fit is imperfect): n = number of x,y data points sumx = Σx sumy = Σy sumxy = Σx*y sumx2 = Σx*x
- 35. meanx = sumx / n meany = sumy / n slope = (n*sumxy - sumx*sumy) / (n*sumx2 - sumx*sumx intercept = meany-(slope*meanx) ssy = Σ(y-meany)^2 ssr = Σ(y-intercept-slope*x)^2 R2 = 1-(ssr/ssy) Popular spreadsheets have facilities for computing polynomial least-squares curve fits. For example, the LINEST function in both Excel and OpenOffice Calc can be used to compute polynomial and other curvilinear least-squares fits. For some examples of applications to analytical chemistry, see Calibration Curve Fitting Methods in Absorption Spectroscopy and Error propagation in Analytical Calibration. You can also download the spreadsheets pictured above, in OpenOffice Calc format, that automate the computation of those equations and also plot the data and the best-fit line, requiring only that you type in (or paste in) the x-y data. There is one spreadsheet for linear fits (http://terpconnect.umd.edu/~toh/spectrum/LeastSquares.odt) and also a version for quadratic (parabolic) fits (http://terpconnect.umd.edu/~toh/spectrum/QuadraticLeastSquares.ods). For the application to analytical calibration, there are specific versions of these spreadsheets that also calculate the concentrations of the unknowns (http://terpconnect.umd.edu/~toh/models/CalibrationCurve.html). SPECTRUM includes least-squares curve fitting for polynomials of order 1 through 5, plus exponential, logarithmic, and power relationships. Matlab has simple built-in functions for least-squares curve fitting: polyfit and polyval. For example, if you have a set of x,y data points in the vectors "x" and "y", then the coefficients for the least-squares fit are given by coef=polyfit(x,y,n), where "n" is the order of the polynomial fit: n = 1 for a straight-line fit, 2 for a quadratic (parabola) fit, etc. For a straight-line fit (n=1), coef(1) is the slope ("b") and coef(2) is the intercept ("a"). For a quadratic fit (n=2), coef(1) is the x2 term ("c"), coef(2) is the x term ("b") and coef(3) is the constant term ("a"). The fit equation can be evaluated using the function polyval, for example fity=polyval(coef,x). This works for any order of polynomial fit ("n"). You can plot the data and the fitted equation together using the plot function: plot(x,y,'ob',x,polyval(coef,x),'-r'), which plots the data as blue circles and the fitted equation as a red line. You can plot the residuals by writing plot(x,y- polyval(coef,x)). (When the number of data points is small, you can get a smoother plot of the fitted equation, evaluated at more finely-divided values of x, by defining xx=linspace(min(x),max(x)); and then using xx rather than x to evaluate and plot the fit: plot(x,y,'ob',xx,polyval(coef,xx),'-r')).
- 36. The graph on the left (click to see a larger size) was generated by a simple user-defined function plotfit(x,y,polyorder), that uses all the techniques mentioned in the previous paragraph. It accepts data in the form of a single vector, or a pair of vectors "x" and "y", or a 2xn or nx2 matrix with x in first row or column and y in the second, and plots the data points as red dots, then if the optional input argument "polyorder" is provided, it fits a polynomial of order "polyorder" to the data and plots the fit as a green line and displays the fit coefficients and the goodness-of-fit measure R2 in the upper left corner of the graph. Curve fitting B: Multicomponent Spectroscopy The spectroscopic analysis of mixtures, when the spectra of the components overlap considerably, require special calibration methods based on a type of linear least squares called multiple linear regression. This method is widely used in multi-wavelength techniques such as diode-array, Fourier transform, and automated scanning spectrometers. Definitions: A = analytical signal c1, c2 = species 1, species 2, etc. e = analytical sensitivity l1, l2 = wavelength 1, wavelength 2, etc. c = molar concentration s1,s2 = sample 1, sample 2, etc. s = number of samples n = number of distinct chemical species in the mixture. w = number of wavelengths at which signal is measured Assumptions: The analytical signal, A (such as absorbance in absorption spectroscopy, fluorescence intensity in fluorescence spectroscopy, and reflectance in reflectance spectroscopy.) is directly proportional to concentration, c. The proportionality constant, which is the slope of a plot of A vs c, is e. A = ec The total signal is the sum of the signals for each component in a mixture: Atotal = Ac1 + Ac2 + ... for all n species. Classical Least Squares (CLS) calibration. This method is applicable to the quantitative analysis of a mixture of species when the spectra of the individual species are known. Measurement of the spectra of known concentrations of the separate species allows their analytical sensitivity e at each wavelength to be determined. Then it follows that: Al1 =ec1,l1 cc1 + ec2,l1 cc2 + ec3,l1 cc3 + ... for all n species. Al2 =ec1,l2 cc1 + ec2,l2 cc2 + ec3,l2 cc3 + ... and so on for all wavelengths. This set of linear equations is written compactly in matrix form: A = eC where A is the w-length vector of measured signals (i.e. the signal spectrum) of the mixture, e is the n x w rectangular matrix of the known e-values for each of the n species at each of the w wavelengths, and C is the n-length vector of concentrations of all the species. If you have a sample solution containing unknown amounts of those species, then you measure its spectrum A and seek to calculate the concentration vector C. In order to
- 37. solve the above matrix equation for C, the number of wavelengths w must be equal to or greater than the number of species n. If w = n, then we have a system of n equations in n unknowns which can be solved by pre-multiplying both sides of the equation by e-1 , the matrix inverse of e, and using the property that any matrix times its inverse is unity: C = e-1 A Because real experimental spectra are subject to random noise (e.g. photon noise and detector noise), the solution will be more precise if signals at a larger number of wavelengths are used, i.e. if w > n. But then the equation can not be solved by matrix inversion, because the e matrix is a w x n matrix and a matrix inverse exists only for square matrices. A solution can be obtained in this case by pre-multiplying both sides of the equation by the expression (eT e)-1 eT : (eT e) -1 eT A = (eT e) -1 eT eC = (eT e) -1 (eT e)C But the quantity (eT e)-1 (eT e) is a matrix times its inverse and is therefore unity. Thus: C = (eT e) -1 eT A In this expression, eT e is a square matrix of order n, the number of species. In most practical applications, n is typically only 2 to 5, this is not a very big matrix to invert, no matter how many wavelengths are used. In general, the more wavelengths are used the more effectively the random noise will be averaged out (although it won’t help to use wavelengths in spectral regions where none of the components produce analytical signals). The determination of the optimum wavelength region must usually be determined empirically. Two extensions of the CLS method are commonly made. First, in order to account for baseline shift caused by drift, background, and light scattering, a column of 1s is added to the e matrix. This has the effect of introducing into the solution an additional component with a flat spectrum; this is referred to as “background correction”. Second, in order to account for the fact that the precision of measurement may vary with wavelength, it is common to perform a weighted least squares solution that de- emphasizes wavelength regions where precision is poor: C = (eT V -1 e) -1 eT V -1 A where V is an w x w diagonal matrix of variances at each wavelength. In absorption spectroscopy, where the precision of measurement is poor in spectral regions where the absorbance is very high (and the light level and signal-to-noise ratio therefore low), it is common to use the transmittance T or its square T2 as weighting factors. The classical least-sqsuares method is method is in principle applicable to any number of overlapping components. Its accuracy is limited by how accurately the spectra of the individual components are known, the amount of noise in the signal, the extent of overlap of the spectra, and the linearily of the analytical curves of each component (the extent to which the signal amplitudes are proportional to concentration). The method is widely applied in absorption spectrophotometry, where the well-known deviations from analytical curve linearity set a limit to the performance to this method (and which can be avoided, in a related method, by applying curve fitting to the transmission spectra). Inverse Least Squares (ILS) calibration. ILS is a method that can be used to measure the concentration of an analyte in samples in which the spectrum of the analyte in the sample is not known beforehand. Whereas the classical least squares method models the signal at each wavelength as the sum of the concentrations of the analyte times the analytical sensitivity, the inverse least squares methods use the reverse approach and models the analyte concentration c in each sample as the sum of the signals A at each wavelength times calibration coefficients m that express how the concentration of that
- 38. species is related to the signal at each wavelength: cs1 = ml1 As1,l1 + ml2 As1,l2 + ml2 As1,l2 + ... for all w wavelengths. cs2 = ml1 As2,l1 + ml2 As2,l2 + ml2 As2,l2 + ... and so on for all s samples. In matrix form C = AM where C is the s-length vector of concentrations of the analyte in the s samples, A is the w x s matrix of measured signals at the w wavelengths in the s samples, and M is the w- length vector of calibration coefficients. Now, suppose that you have a set of standard samples that are typical of the type of sample that you wish to be able to measure and which contain a range of analyte concentrations that span the range of concentrations expected to be found in other samples of that type. This will serve as the calibration set. You measure the spectrum of each of the samples in this calibration set and put these data into a w x s matrix of measured signals A. You then measure the analyte concentrations in each of the samples by some reliable and independent analytical method and put those data into a s-length vector of concentrations C. Together these data allow you to calculate the calibration vector M by solving the above equation. If the number of samples in the calibration set is greater than the number of wavelengths, the least-squares solution is: M = (A T A) -1 A T C (Note that A T A is a square matrix of size w, the number of wavelengths, which must be less than s). This calibration vector can be used to compute the analyte concentrations of other samples, which are similar to but not in the calibration set, from the measured spectra of the samples: C = AM Clearly this will work well only if the analytical samples are similar to the calibration set. Most modern spreadsheets have basic matrix manipulation capabilities and can be used for multicomponent calibration, for example Excel, OpenOffice Calc. Here is an example of a multicomponent calibration calibration performed in a spreadsheet environment. But Matlab is really the natural computer approach to multicomponent analysis because it handles all types of matrix math so easily. In Matlab, the notation is a little different: transpose of a matrix A is A', the inverse of A is inv(A), and matrix multiplication is designated by *. Thus the solution to the classical least squares method above is written C = inv(E'*E)*E'*A, where E is the rectangular matrix of sensitivities at each wavelength for each component The script RegressionDemo.m demonstrates the classical least squares procedure for a simulated absorption spectrum of a 5-component mixture, illustrated on the left. In this example the dots represent the observed spectrum of the mixture (with noise) and the five colored bands represent the five components in the mixture, whose spectra are known but whose concentrations in the mixture are unknown. The black line represents the "best fit" to the observed spectrum calculated by the program. In this example the concentrations of the five components are measured to an accuracy of about 1% relative (limited by the noise in the observed spectrum). The extension to "background correction" is easily accomplished in Matlab by adding a column of 1s to the Amatrix containing the absorption spectrum of each of the components.:
- 39. background=ones(size(ObservedSpectrum)); A=[background A1 A2 A3]; where A1, A2, A3... are the absorption spectra of the individual components. Performing a T- weighted regression is also readily performed: weight=T; MeasuredAmp=([weight weight] .* A)(ObservedSpectrum .* weight); where T is the transmission spectrum. Here, the matrix division backslash "" is used as a short- cut to the classical least-squares matrix solution (c.f. http://www.mathworks.com/help/techdoc/ref/mldivide.html). The custom Matlab function "TFit3Demo.m" simulates the measurement of three absorbing components by weighted linear regression, demonstrates the effect of the amount of noise in the signal, the extent of overlap of the spectra, and the linearily of the analytical curves of each component, and also compares the results to another method that applies curve fitting to the transmission spectra rather than to the absorbance spectra. The Inverse Least Squares (ILS) technique is demonstrated by this script and the graph below. This is a real data set derived from the near infrared (NIR) reflectance spectroscopy of agricultural wheat samples analyzed for protein content. In this example there are 50 calibration samples measured at 6 wavelengths. The samples had already been analyzed by a reliable, but laborious and time-consuming, reference method. The purpose of this calibration is to establish whether near- infrared reflectance spectroscopy, which can be measured much more quickly on wheat paste preparations, correlates to their protein content. These results indicate that it does, at least for this set of 50 wheat samples, and therefore is it likely that near-infrared spectroscopy should do a pretty good job of estimating the protein content of similar unknown samples. The key is that the unknown samples must be similar to the calibrations samples (except for the protein content). However, this is a very common analytical situation in commerce, where large numbers of samples of a similar predictable type must be analyzed. Curve fitting C: Non-linear Iterative Curve Fitting ("spectral deconvolution" or "peak deconvolution") The linear least squares curve fitting described in "Curve Fitting A" is simple and fast, but it is limited to situations where the dependent variable can be modeled as a polynomial with linear coefficients. We saw that in some cases a non-linear situation can be converted into a linear one by a coordinate transformation, but this is possible only in some special cases and, in any case, the resulting coordinate transformation of the noise in the data can result in inaccuracies in the parameters measured in this way.
- 40. The most general way of fitting any model to a set of data is the iterative method, a kind of "trial and error" procedure in which the parameters of the model are adjusted in a systematic fashion until the equation fits the data as close as required. This sounds like a brute-force approach, and it's true that, in the days before computers, this method was only grudgingly applied. But its great generality, coupled with advances in computer speed and algorithm efficiency, means that iterative methods are more widely used now than ever before. Iterative methods proceed in the following general way: (1) the operator selects a model for the data; (2) first guesses of all the non-linear parameters are made; (3) a computer program computes the model and compares it to the data set, calculating a fitting error; (4) if the fitting error is greater that the required fitting accuracy, the program systematically changes one or more of the parameters and loops back around to step 3. This continues until the fitting error is less than the specified error. One popular technique for doing this is called the Nelder-Mead Modified Simplex. This is essentially a way of organizing and optimizing the changes in parameters (step 4, above) to shorten the time required to fit the function to the required degree of accuracy. With contemporary personal computers, the entire process typically takes only a fraction of a second. The main difficulty of the interactive methods is that they sometime fail to converge at an optimum solution in difficult cases. The standard approach to handle this is to restart the algorithm with another set of first guesses. Iterative curve fitting also takes longer than linear regression - with typical modern personal computers, an iterative fit might take fractions of a second where a regression would take fractions of a millisecond. Still, this is fast enough for many purposes. Note: the term "spectral deconvolution" or "band deconvolution" is often used to refer to this technique. Spreadsheets and stand-alone programs. There are a number of downloadable non- linear iterative curve fitting adds-ons and macros for Excel and OpenOffice, as well as some stand-alone freeware and commercial programs that perform this function. Matlab has a convenient and efficient function called FMINSEARCH that uses the Nelder-Mead method. It works in conjunction with a user-defined "fitting function" that computes the model, compares it to the data, and returns the fitting error. For example, writing options = optimset('TolX',0.1); parameter=FMINSEARCH('fitfunction',start,options,x,y) performs an iterative fit of the data in the vectors x,y to a model described in a previously-created function called fitfunction, using the first guesses in the vector start and stopping at the tolerance defined by the optimset function. The parameters of the fit are returned in the vector "parameters", in the same order that they appear in "start". A simple example is fitting the blackbody equation to the spectrum of an incandescent body for the purpose of estimating its color temperature. In this case there is only one nonlinear parameter, temperature. The script BlackbodyDataFit.m demonstrates the technique, placing the experimentally measured spectrum in the vectors "wavelength" and
- 41. "radiance" and then calling FMINSEARCH with the fitting function fitblackbody.m. Another application is demonstrated by Matlab's built-in demo fitdemo.m and its fitting function fitfun.m, which models the sum of two exponential decays. (To see this, just type "fitdemo" in the Matlab command window). Fitting peaks. Many instrumental methods of measurement produce signals in the form of peaks of various shapes; a common requirement is to measure the positions, heights, widths, and/or areas of those peaks, even when they are noisy or overlapped with one another. This cannot be done by linear least-squares methods, because such signals can not be modeled as polynomials with linear coefficients (the positions and widths of the peaks are not linear functions), so iterative curve fitting techniques are used instead, often using Gaussian, Lorentzian, or some other fundamental simple peak shapes as a model. The custom demonstration script Demofitgauss.m demonstrates fitting a Gaussian function to a set of data, using the fitting function fitgauss2.m. In this case there are two non-linear parameters, the peak position and the peak width (the peak height is a linear parameter and is determined by regression in line 9 of the fitting function fitgauss2.m and is returned in the global variable "c"). This is easily extended to fitting two overlapping Gaussians in Demofitgauss2.m (shown on the left) using the same fitting function (which easily adapts to any number of peaks, depending on the length of the first-guess "start" vector). All these functions call the user- defined peak shape function gaussian.m. Similar procedures can be defined for other peak shapes simply by calling the corresponding peak shape function, such as lorentzian.m. (Note: in order for scripts like Demofitgauss.m or Demofitgauss2.m to work on your version of Matlab, all the functions that they call must be loaded into Matlab beforehand, in this case fitgauss2.m and gaussian.m). You can create your own fitting functions for any purpose; they are not limited to single algebraic expressions, but can be arbitrary complex multi-step algorithms. For example, in the Tfit method in absorption spectroscopy (http://terpconnect.umd.edu/~toh/spectrum/TFit.html ), a model of the instrumentally- broadened transmission spectrum is fit to the observed transmission data, in order to extend the dynamic range and calibration linearity beyond the normal limits, using a fitting function that performs Fourier convolution of the transmission spectrum with the slit function of the spectrometer. It's instructive to compare this method with classical least-squares curve fitting, discussed in the previous section, which can also fit peaks in a signal. The difference is that in the classical least squares method, the positions, widths, and shapes of all the individual components are all known beforehand; the only unknowns are the amplitudes (e.g. peak heights) of the components in the mixture. In non-linear iterative curve fitting, on the other hand, the positions, widths, and heights of the peaks are all unknown beforehand; the only thing that is known is the fundamental underlying shape of the
- 42. peaks. It turns out that the non-linear iterative curve fitting is much more difficult to do and more prone to error, but it's necessary if you need to track shifts in peak position or widths or to decompose a complex overlapping peak signal into fundamental components knowing only their shape. Note: you can right-click on any of the m-file links above and select Save Link As... to download them to your computer for use within Matlab. Interactive Peak Fitter (http://terpconnect.umd.edu/~toh/spectrum/InteractivePeakFitter.htm) This is a series of Matlab peak fitting programs for time-series signals, which uses an unconstrained non-linear optimization algorithm to decompose a complex, overlapping-peak signal into its component parts. The objective is to determine whether your signal can be represented as the sum of fundamental underlying peaks shapes. These programs do not require the signal processing or optimization toolboxes. They accept signals of any length, including those with non-integer and non-uniform x-values, and can fits groups of peaks with Gaussian, Lorentzian, Logistic, Pearson, and exponentially-broadened Gaussian models (expandable to other shapes). There are three different versions, a command line version, a keypress operated interactive version, and a version with mouse- controlled sliders (which requires Matlab 6.5). Accuracy and precision of peak parameter measurement There are four major sources of error in measuring the peak parameters (peak positions, heights, widths, and areas) by iterative curve fitting: a. Model errors. Peak shape. If you have the wrong model for your peaks, the results can not be expected to be accurate; for instance, if your actual peaks are Lorentzian in shape, but you fit them with a Gaussian model, or vice versa. For example, a single isolated Gaussian peak at x=5, with a height of 1.000 fits a Gaussian model virtually perfectly, using the Matlab user-defined peakfit function, as shown on the right. (The 5th input argument for the peakfit function specifies the shape of peaks to be used in the fit; "1" means Gaussian). >> x=[0:.1:10];y=exp(-(x-5).^2); >> [FitResults,MeanFitError]=peakfit([x' y'],5,10,1,1) FitResults = 1 5 1 .6649 1.7724 MeanFitError = 0.001679
- 43. The "FitResults" are, from left to right, peak number, peak position, peak height, peak width, and peak area. The MeanFitError, or just "fitting error", is the RMS difference between the data and the best-fit model, as a percentage of the maximum signal in the fitted region. (Note the good agreement of the area measurement (1.7724) with the theoretical area under the curve of exp(-x2), which is the square root of p, or 1.7725...). But this same peak, when fit with a Logistic model (peak shape number 3), gives a fitting error of 1.4% and height and width errors of 3% and 6%, respectively: >> [FitResults,MeanFitError]=peakfit([x' y'],5,10,1,3) FitResults = 1 5.0002 0.96652 1.762 1.7419 MeanFitError = 1.4095 When fit with a Lorentzian model (peak shape 2), this peak gives a 6% fitting error and height and width errors of 8% and 20%, respectively. > [FitResults,MeanFitError]=peakfit([x' y'],5,10,1,2) FitResults = 1 5 1.0876 1.3139 2.0579 MeanFitError = 5.7893 So clearly the larger the fitting errors, the larger are the parameter errors, but the parameter errors are of course not equal to the fitting error (that would just be too easy). Also, clearly the peak width and area are the parameters most susceptible to errors. The peak positions, as you can see here, are measured accurately even if the model is way wrong, as long as the peak is symmetrical and not highly overlapping with other peaks. Number of peaks. Another source of model error occurs if you have the wrong number of peaks in your model, for example if the signal actually has two peaks but you try to fit it with only one peak. In the example below, a bit of of Matlab code generates a simulated signal with of two Gaussian peaks at x=4 and x=6 with peaks heights of 1.000 and 0.5000 respectively and widths of 1.665, plus random noise with a standard deviation 5% of the height of the largest peak (a signal-to-noise ratio of 20): >> x=[0:.1:10]; >> y=exp(-(x-6).^2)+.5*exp(-(x-4).^2)+.05*randn(size(x)); In a real experiment you would not usually know the peak positions, heights, and widths; you would be using curve fitting to measure those parameters. Let's assume that, on the basis of previous experience or some preliminary trial fits, you have established that the optimum peak shape model is Gaussian, but you don't know for sure how many peaks are in this group. If you start out by fitting this signal with a single-peak Gaussian model, you get:
- 44. >> [FitResults,MeanFitError]=peakfit([x' y'],5,10,1,1) FitResults = 1 5.5291 0.86396 2.9789 2.7392 MeanFitError = 10.467 The residual plot shows a "wavy" structure that's visible in the random scatter of points due to the random noise in the signal. This means that the fitting error is not limited by the random noise; it is a clue that the model is not quite right. But a fit with two peaks yields much better results (The 4th input argument for the peakfit function specifies the number of peaks to be used in the fit). >> [FitResults,MeanFitError]=peakfit([x' y'],5,10,2,1) FitResults = 1 4.0165 0.50484 1.6982 0.91267 2 5.9932 1.0018 1.6652 1.7759 MeanFitError = 4.4635 Now the residuals have a random scatter of points, as would be expected if the signal is accurately fit except for the random noise. Moreover, the fitting error is much lower (less that half) of the error with only one peak. In fact, the fitting error is just about what we would expect in this case based on the 5% random noise in the signal (estimating the relative standard deviation of the points in the baseline visible at the edges of the signal). Because this is a simulation in which we know beforehand the true values of the peak parameters (peaks at x=4 and x=6 with peaks heights of 1.0 and 0.50 respectively and widths of 1.665), we can actually calculate the parameter errors (the difference between the real peak positions, heights, and widths and the measured values). We see that they are quite accurate (in this case within about 1% relative on the peak height and 2% on the widths), which is actually better than the 5% random noise in this signal because of the averaging effect of fitting to multiple data points in the signal. But if going from one peak to two peaks gave us a better fit, why not go to three peaks? Changing the number of peaks to three gives these results: >> [FitResults,MeanFitError]=peakfit([x' y'],5,10,3,1) FitResults = 1 4.0748 0.51617 1.7874 0.98212 2 6.7799 0.089595 2.0455 0.19507
- 45. 3 5.9711 0.94455 1.53 1.5384 MeanFitError = 4.3878 The fitting algorithm has now tried to fit an additional low-amplitude peak (numbered peak 2 in this case) located at x=6.78. The fitting error is actually lower that for the 2-peak fit, but only slightly lower, and the residuals are no less visually random that with a 2-peak fit. So, knowing nothing else, a 3-peak fit might be rejected on that basis alone. In fact, there is a serious downside to fitting more peaks than are present in the signal: it increases the parameter measurement errors of the peaks that are actually present. Again, we can prove this because we know beforehand the true values of the peak parameters: clearly the peak positions, heights, and widths of the two real peaks than are actually in the signal (peaks 1 and 3) are significantly less accurate that of the 2-peak fit. Moreover, if we repeat that fit with the same signal but with a different sample of random noise (simulating a repeat measurement of a stable experimental signal in the presence or random noise), the additional third peak in the 3-peak fit will bounce around all over the place (because the third peak is actually fitting the random noise, not an actual peak in the signal). >> x=[0:.1:10]; >> y=exp(-(x-6).^2)+.5*exp(-(x-4).^2)+.05*randn(size(x)); >> [FitResults,MeanFitError]=peakfit([x' y'],5,10,3,1) FitResults = 1 4.115 0.44767 1.8768 0.89442 2 5.3118 0.093402 2.6986 0.26832 3 6.0681 0.91085 1.5116 1.4657 MeanFitError = 4.4089 With this new set of data, two of the peaks (numbers 1 and 3) have roughly the same position, height, and width, but peak number 2 has changed substantially compared to the previous run. Now we have an even more compelling reason to reject the 3-peak model: the 3-peak solution is not stable. And because this is a simulation in which we know the right answers, we can verify that the accuracy of the peak heights is substantially poorer (about 10% error) than expected with this level of random noise in the signal (5%). If we were to run a 2-peak fit on the same new data, we get much better measurements of the peak heights. >> [FitResults,MeanFitError]=peakfit([x' y'],5,10,2,1) FitResults = 1 4.1601 0.49981 1.9108 1.0167 2 6.0585 0.97557 1.548 1.6076
- 46. MeanFitError = 4.4113 If this is repeated several times, it turns out that the peak parameters of the peaks at x=4 and x=6 are, on average, more accurately measured by the 2-peak fit. In practice, the best way to evaluate a proposed fitting model is to fit several repeat measurements of the same signal (if that is practical experimentally) and to compute the standard deviation of the peak parameter values. Peak width constraints. Finally, there is one more thing that we can do that might improve the peak parameter measurement accuracy, and it concerns the peak widths. In all the above simulations, the basic assumption that all the peak parameters were unknown and independent of one another. In some types of measurements, however, the peak widths of each group of adjacent peaks are all expected to be equal to each other, on the basis of first principles or previous experiments. This is a common situation in analytical chemistry, especially in atomic spectroscopy and in chromatography, where the peak widths are determined largely by instrumental factors. In the current simulation, the true peaks widths are in fact equal, but all the results above show that the measured peak widths are close but not quite equal, due to random noise in the signal. But we can introduce an equal-width constraint into the fit by using peak shape 6 (Equal width Gaussians) or peak shape 7 (Equal width Lorentzians) in peakfit version 1.4 and later. Using peak shape 6 on the same set of data as the previous example: >> [FitResults,MeanFitError]=peakfit([x' y'],5,10,2,6) FitResults = 1 4.0435 0.4942 1.6559 0.87113 2 6.0039 0.99218 1.6559 1.7489 MeanFitError = 4.5076 This "equal width" fit forces all the peaks within one group to have exactly the same width. The result is a slightly higher fitting error (in this case 4.5 rather than 4.4), but - perhaps surprisingly - the peak parameter measurements are usually more accurate and more reproducible (Specifically, the relative standard deviations of a large number of repetitions of measurements of peak parameters are on average lower for the equal-width fit than for an unconstrained-width fit to the same data, assuming of course that the true underlying peak widths are really equal). This is an exception to the general expectation that lower fitting errors result in lower peak parameter errors. It is an illustration of the general rule that the more you know about the nature of your signals, and the closer your chosen model adheres to that knowledge, the better the results. In this case we knew that the peak shape was Gaussian (although we could have verified that choice by trying other candidate peaks shapes). We determined that the number of peaks was two by inspecting the residuals and fitting errors for 1, 2, and 3 peak models. And finally we introduced the constraint of equal peak widths within
- 47. each group of peaks (based on prior knowledge of the experiment rather than on inspection of residuals and fitting errors). Not every experiment yields peaks of equal width, but when it does, it's better to make use of that constraint. b. Background correction. The peaks that are measured in many scientific instruments are sometimes superimposed on a non-specific background. Ordinarily the experiment protocol is designed to minimize the background or to compensate for the background, for example by subtracting a "blank" signal from the signal of an actual specimen. But even so there is often a residual background that can not be eliminated completely experimentally. The origin and shape of that background depends on the specific measurement method, but often this background is a broad, tilted, or curved shape, and the peaks of interest are comparatively narrow features superimposed on that background. The presence of the background has little effect on the peak positions, but it is impossible to measure the peak heights, width, and areas accurately unless something is done to account for the background. There are various sophisticated methods described in the literature for estimating and subtracting the background in such cases. The simplest assumption is that the background is locally linear, that is, can be approximated as a straight line in the local region of group of peaks being fit together. This is easily accomplished if each group of peaks is sufficiently isolated from other groups that the signal returns to the local background between the groups of peaks, and this is the basis of the "autozero" mode in the ipf.m and iPeak.m functions. When the autozero mode of these functions is turned on (T key), a straight-line baseline connecting the two ends of the signal segment in the upper panel will be automatically subtracted as the pan and zoom controls are used to isolate the group of overlapping peaks to be fit. Example of an experimental chromatographic signal. From left to right, (1) Raw data with peaks superimposed on baseline. One group of peaks is selected using the the pan and zoom controls, adjusted so that the signal returns to the local background at the edges of the segment displayed in the upper window; (2) The linear baseline is subtracted when the "autozero" mode is turned on in ipf.m; (3) Fit with a three-peak Gaussian model, activated by pressing 3, G, F (3 peaks, Gaussian, Fit). Alternatively, it may be better to subtract the background from the entire signal first, before futher operations are performed. As before, the simplest assumption is that the background is piecewise linear, that is, can be approximated as a series of small straight line segments. This is the basis of the multi-point background subtraction mode in ipf.m, iPeak.m, and in iSignal. The user enters the number of points that is thought to be
- 48. sufficient to define the baseline, then clicks where the baseline is thought to be along the entire length of the signal in the lower whole-signal display (e.g. on the valleys between the peaks). After the last point is clicked, the program interpolates between the clicked points and subtracts the piecewise linear background from the original signal. Left: Raw data with peaks superimposed on baseline. Right: Background subtracted from the entire signal using the multi-point background subtraction function in iPeak.m (ipf.m and iSignal have the same function). In some cases the background may be caused by a broad peak whose maximum falls outside of the range of data acquired. This is not an uncommon situation with peaks shapes that have gradually sloping sides, such as the Lorentzian. There is no explicit provision for this in the current version of my programs; however, if the background peak has the same shape as the measured peaks, it is possible to fit the off-screen peak in ipf.m simply by increasing the number of peaks by one. It may help to the cursor-defined start position (C key) to define the start position for the background peak beyond the x-axis range of the plot, but still inside the Figure window. (Don't use the equal-width shapes for this, because it's likely that the background peak is broader than the measured peaks). c. Random noise in the signal. Any experimental signal has a certain amount of random noise, which means that the individual data points scatter randomly above and below their mean values. The assumption is ordinarily made that the scatter is equally above and below the true signal, so that the long-term average approaches the true mean value; the noise "averages to zero" as it is often said. The practical problem is that any given recording of the signal contains only one finite sample of the noise. If another recording of the signal is made, it will contain another independent sample of the noise. These noise samples are not infinitely long and therefore do not represent the true long-term nature of the noise. This presents two problems: (1) an individual sample of the noise will not "average to zero" and thus the parameters of the best-fit model will not necessarily equal the true values, and (2) the magnitude of the noise during one sample might not be typical; the noise might have been randomly greater or smaller than average during that time. This means that the mathematical "propagation of error" methods, which seek to estimate the likely error in the model parameters based on the noise in the signal, will be subject to error (underestimating the error if the noise happens to be lower than average and overestimating the errors if the noise happens to be larger than average).
- 49. A better way to estimate the parameter errors is to record multiple samples of the signal, fit each of those separately, compute the models parameters from each fit, and calculate the standard error of each parameter. This is exactly what the script DemoPeakfit.m (which requires the peakfit.m function) does for simulated noisy peak signals such as those illustrated below. It's easy to demonstrate that, as expected, the average fitting error precision and the relative standard deviation of the parameters increases directly with the random noise level in the signal. But the precision and the accuracy of the measured parameters also depend on which parameter it is (peak positions are always measured more accurately than their heights, widths, and areas) and on the peak height and extent of peak overlap (the two left-most peaks in this example are not only weaker but also more overlapped that the right-most peak, and therefore exhibit poorer parameter measurements). In this example, the fitting error is 1.6% and the percent relative standard deviation of the parameters ranges from 0.05% for the peak position of the largest peak to 12% for the peak area of the smallest peak. The errors in the values of peak parameters measured by curve fitting depend not only on the characteristics of the peaks in question and the signal-to-noise ratio, but also upon other peaks that are overlapping it. From left to right: (1) a single peak at x=100 with a peak height of 1.0 and width of 30 is fit with a Gaussian model, yielding a relative fit error of 4.9% and relative standard deviation of peak position, height, and width of 0.2%, 0.95%, and 1.5% , respectively. (2) The same peak, with the same noise level but with another peak overlapping it, reduces the relative fit error to 2.4% (because the addition if the second peak increases overall signal amplitude), but increases the relative standard deviation of peak position, height, and width to 0.84%, 5%, and 4% - a seemingly better fit, but with poorer precision for the first peak. (3) The addition of a third peak further reduces the fit error to 1.6% , but the relative standard deviation of peak position, height, and width of the first peak are still 0.8%, 5.8%, and 3.64%, about the same as with two peaks, because the third peak does not overlap the first one significantly. One way to reduce the effect of noise is to take more data. If the experiment makes it possible to reduce the x-axis interval between points, or to take multiple readings at each x-axis values, then the resulting increase in the number of data points in each peak should help reduce the effect of noise. As a demonstration, using the script DemoPeakfit.m to create a simulated overlapping peak signal like that shown above right, it's possible to change the interval between x values and thus the total number of data points in the signal. With a noise level of 1% and 75 points in the signal, the fitting error is 0.35 and the average parameter error is 0.8%. With 300 points in the signal and the same noise level, the fitting error is essentially the same, but the average parameter error drops to 0.4%, suggesting that the accuracy of the measured parameters varies inversely with the square root of the number of data points in the peaks.
- 50. The figure on the right illustrates the importance of sampling interval and data density. You can download the data file "udx" in TXT format or in Matlab MAT format. The signal consists of two Gaussian peaks, one located at x=50 and the second at x=150. Both peaks have a peak height of 1.0 and a peak half-width of 10, and normally- distributed random white noise with a standard deviation of 0.1 has been added to the entire signal. The x-axis sampling interval, however, is different for the two peaks; it's 0.1 for the first peaks and 1.0 for the second peak. This means that the first peak is characterized by ten times more points that the second peak. When you fit these peaks separately to a Gaussian model (e.g., using peakfit.m or ipf.m), you will find that all the parameters of the first peak are measured more accurately than the second, even though the fitting error is not much different: First peak: Second peak: Percent Fitting Error = 7.6434% Percent Fitting Error = 8.8827% Peak# Position Height Width Peak# Position Height Width 1 49.95 1.005 10.11 1 149.64 1.0313 9.94 d. Iterative fitting errors. Unlike multiple linear regression curve fitting, iterative methods may not converge on the exact same model parameters each time the fit is repeated with slightly different starting values (first guesses). The Interactive Peak Fitter makes it easy to test this, because it uses slightly different starting values each time the signal is fit (by pressing the F key in ipf.m, for example). Even better, by pressing the X key, the ipf.m function silently computes 10 fits with different starting values and takes the one with the lowest fitting error. A basic assumption of any curve fitting operation is that the fitting error (the RMS difference between the model and the data) is minimized, the parameter errors (the difference between the actual parameters and the parameters of the best-fit model) will also be minimized. This is generally a good assumption, as demonstrated by the graph to the right, which shows typical percent parameters errors as a function of fitting error for the left-most peak in one sample of the simulated signal generated by DemoPeakfit.m (shown in the previous section). The variability of the fitting error here is caused by random small variations in the first guesses, rather than by random noise in the signal. In many practical cases there is enough random noise in the signals that the iterative fitting errors within one sample of the signal are small compared to the random noise errors between samples.
- 51. Remember that the variability in measured peak parameters from fit to fit of a single sample of the signal is not a good estimate of the precision or accuracy of those parameters, for the simple reason that those results represent only one sample of the signal, noise, and background. The sample-to-sample variations are likely to be much greater than the within-sample variations due to the iterative curve fitting. (In this case, a "sample" is a single recording of signal). So, to sum up, we can make the following observations about the accuracy of model parameters: (1) the parameter errors depend on the peak shape and number of peaks; (2) the parameter errors are directly proportional to the noise in the data and to the fitting error; (3) the errors are typically least for peak position and worse for peak width and area; (4) the errors depend on the data density (number of independent data points in the width of each peak) and on the extent of peak overlap (the parameters of isolated peaks are easier to measure than highly overlapped peaks). Fitting signals that are subject to exponential broadening. DataMatrix2 (right) is a computer- generated test signal consisting of 16 symmetrical Gaussian peaks with random white noise added. The peaks occur in groups of 1, 2, or 3 overlapping peaks, but the peak maxima are located at exactly integer values of x from 300 to 3900 (on the 100's) and the peak widths are always exactly 60 units. The peak heights vary from 0.06 to 1.85. The standard deviation of the noise is 0.01. You can use this signal to test curve-fitting programs and to determine the accuracy of their measurements of peak parameters. Right-click and select "Save" to download this signal, put it in the Matlab path, then type "load DataMatrix2" to load it into the Matlab workspace. DataMatrix3 (left) is an exponentially broadened version of DataMatrix2, with a "time constant" of 33 points on the x-axis. The result of the exponential broadening is that all the peaks in this signal are asymmetrical, their peak maxima are shifted to longer x values, and their peak heights are smaller and their peak widths are larger than the corresponding peaks in DataMatrix2. Also, the random noise is damped in this signal compared to the original and is no longer "white", as a consequence of the broadening. This type of effect is common in physical measurements and often arises from some physical or electrical effect in the measurement system that is apart from the fundamental peak characteristics. In such cases it is usually desirable to compensate for the effect of the broadening, either by deconvolution or by curve fitting, in an attempt to measure what the peak parameters would have been before the broadening (and also to measure the broadening itself). This can be done for
- 52. Gaussian peaks that are exponentially broadened by using the "ExpGaussian" peak shape in peakfit.m and ifp.m. Right-click and select "Save" to download this signal, put it in the Matlab path, then type "load DataMatrix3" to load it into the Matlab workspace. The example illustrated on the right focuses on the single isolated peak whose "true" peak position, height, width, and area in the original unbroadened signal, are 2800, 0.52, 60, and 33.2 respectively. (The relative standard deviation of the noise is 0.01/0.52=2%.) In the broadened signal, the peak is visibly asymmetrical, the peak maximum is shifted to larger x values, and it has a shorter height and larger width, as demonstrated by the attempt to fit a normal (symmetrical) Gaussian to the broadened peak. (The peak area, on the other hand, is not much effected by the broadening). >> load DataMatrix3 >> ipf(DataMatrix3); Peak Shape = Gaussian Autozero ON Number of peaks = 1 Fitted range = 2640 - 2979.5 (339.5) (2809.75) Percent Error = 1.2084 Peak# Position Height Width Area 1 2814.832 0.45100549 68.441262 32.859436 The large "wavy" residual plot is a tip-off that the model is not quite right. Moreover, the fitting error (1.2%) is larger than expected for a peak with a half-width of 60 points and a 2% noise RSD (approximately 2%/sqrt(60)=0.25%). Fitting to an exponentially-broadened Gaussian (pictured on the right) gives a much lower fitting error ("Percent error") and a more nearly random residual plot. But the interesting thing is that it also recovers the original peak position, height, and width to an accuracy of a fraction of 1%. In performing this fit, the time constant ("extra") was experimentally determined from the broadened signal by adjusting it with the A and Z keys to give the lowest fitting error; that also gives a reasonably good measurement of the broadening factor (32.6, vs the actual value of 33).
- 53. Peak Shape = Exponentially-broadened Gaussian Autozero ON Number of peaks = 1 Extra = 32.6327 Fitted range = 2640 - 2979.5 (339.5) (2809.75) Percent Error = 0.21696 Peak# Position Height Width Area 1 2800.1302 0.51829906 60.086295 33.152429 Comparing the two methods, the exponentially-broadened Gaussian fit recovers all the underlying peak parameters quite accurately: Position Height Width Area Actual peak parameters 2800 0.52 60 33.2155 Gaussian fit to broadened signal 2814.832 0.45100549 68.441262 32.859436 ExpGaussian fit to broadened signal 2800.1302 0.51829906 60.086295 33.152429 Other peaks in the same signal, under the broadening influence of the same time constant, can be fit with similar settings, for example the set of three overlapping peaks near x=2400. As before, the peak positions are recovered almost exactly and even the width measurements are reasonably accurate (1% or better). (The smaller fitting error evident here is just a reflection of the larger peak heights in this particular group of peaks - the noise is the same everywhere in this signal). Peak Shape = Exponentially-broadened Gaussian Autozero OFF Number of peaks = 3 Extra = 31.9071 Fitted range = 2206 - 2646.5 (440.5) (2426.25) Percent Error = 0.11659 Peak# Position Height Width Area 1 2300.2349 0.83255884 60.283214 53.422354 2 2400.1618 0.4882451 60.122977 31.24918 3 2500.3123 0.85404245 60.633532 55.124839 The residual plots in both of these examples still have some "wavy" character, rather than being completely random and "white". The exponential broadening smooths out any white noise in the original signal that is introduced before the exponential effect, acting as a low-pass filter in the time domain and resulting in a low-frequency dominated "pink" noise, which is what remains in the residuals after the broadened peaks have been fit as well as possible. On the other hand, white noise that is introduced after the exponential effect would continue to appear white and random on the residuals. In real experimental data, both types of noise may be present in varying amounts.
- 54. One final caveat: peak asymmetry similar to exponential broadening can in principle be the result a pair of closely-spaced peaks of different peak heights. In fact, a single exponential broadened Gaussian peak can be fit with two symmetrical Gaussians to a fitting error at least as low as a single exponential broadened Gaussian fit. This makes it hard to distinguish between these two models on the basis of fitting error alone. However, this can be decided by inspecting the other peaks in the signal: in most experiments, the same exponential broadening applies to every peak in the signal, and the broadening is either constant or changes gradually over the length of the signal. On the other hand, it is very unlikely that every peak in the signal will be accompanied by a smaller side peak that varies in this way. So, if a only one or a few peaks exhibit asymmetry, and the others are symmetrical, it's most likely that the asymmetry is due to closely-spaced peaks of different peak heights. If all peaks have the same or similar asymmetry, it's more likely to be a broadening factor that applies to the entire signal. Appendix: Software details 1. SPECTRUM for Macintosh OS 7 or 8 Many of the figures in this essay are screen images from S.P.E.C.T.R.U.M. (Signal Processing for Experimental Chemistry Teaching and Research/ University of Maryland), a Macintosh program we have developed for teaching signal processing to chemistry students. SPECTRUM is designed for post-run (rather than real-time) processing of "spectral" or time- series data (y values at equally-spaced x intervals), such as spectra, chromatograms, electrochemical signals, etc. The program enhances the information content of instrument signals, for example by reducing noise, improving resolution, compensating for instrumental artifacts, testing hypotheses, and decomposing a complex signal into its component parts. SPECTRUM was the winner of two EDUCOM/NCRIPTAL national software awards in 1990, for Best Chemistry software and for Best Design.
- 55. Features • Reads one- or two- column (y-only or x-y) text data tables with either tab or space separators • Displays fast, labeled plots in standard resizable windows with full x- and y-axis scale expansion and a mouse- controlled measurement cursor • Addition, subtraction, multiplication, and division of two signals • Two kinds of smoothing. • Three kinds of differentiation • Integration • Resolution enhancement • Interpolation • Fused peak area measurement by perpendicular drop or tangent skim methods, with mouse-controlled setting of start and stop points • Fourier transformation • Power spectra • Fourier filtering • Convolution and deconvolution • Cross- and auto-correlation • Built-in signal simulator with Gaussian and Lorentzian bands, sine wave and normally-distributed random noise • A number of other useful functions, including: inspect and edit individual data points, normalize, histogram, interpolate, zero fill, group points by 2s, bridge segment, superimpose, extract subset of points, concatenate, reverse X-axis, rotate, set X axis values, reciprocal, log, ln, antilog, antiln, standard deviation, absolute value, square root SPECTRUM can be used both as a research tool and as an instructional aid in teaching signal processing techniques. The program and its associated tutorial was originally developed for students of analytical chemistry, but the program could be used in any field in which instrumental measurements are used: e.g. chemistry, biochemistry, physics, engineering, medical research, clinical psychology, biology, environmental and earth sciences, agricultural sciences, or materials testing. Machine Requirements: SPECTRUM runs only on older Macintosh models running OS 7 or 8, minimum 1 MByte RAM, any standard printer. Color screen desirable. SPECTRUM has been tested on most Macintosh models and on all versions of the operating system through OS 8.1. No PC version or more recent Mac version is available or planned. The full version of SPECTRUM 1.1 is now available as freeware, and can be downloaded from http://terpconnect.umd.edu/~toh/spectrum/. There are two versions: SPECTRUM 1.1e: Signals are stored internally as extended-precision real variables and there is a limit of 1024 points per signal. This version performs all its calculations in extended precision and thus has the best dynamic range and the smallest numeric round-off errors. The download address of this version in HQX format is http://terpconnect.umd.edu/~toh/spectrum/SPECTRUM11e.hqx.
- 56. SPECTRUM 1.1b: Signals are stored internally as single-precision real variables and there is a limit of 4000 points per signal. This version is less precise in its calculations (has more numerical round-off error) than the other version, but allows signals with data more points. The download address of this version in HQX format is http://terpconnect.umd.edu/~toh/spectrum/SPECTRUM11b.hqx. The two versions are otherwise identical. There is also a documentation package (located at http://terpconnect.umd.edu/~toh/spectrum/SPECTRUMdemo.hqx) consisting of: a. Reference manual. Macwrite format (Can be opened from within MacWrite, Microsoft Word, ClarisWorks, WriteNow, and most other full-featured Macintosh word processors). Explains each menu selection and describes the algorithms and mathematical formulae for each operation. The SPECTRUM Reference Manual is also available separately in PDF format at http://terpconnect.umd.edu/~toh/spectrum/SPECTRUMReferenceManual.pdf. b. Signal processing tutorial. Macwrite format (Can be opened from within MacWrite, Microsoft Word, ClarisWorks, WriteNow, and most other full-featured Macintosh word processors). Self-guided tutorial on the applications of signal processing in analytical chemistry. This tutorial is also available on the Web at (http://terpconnect.umd.edu/~toh/Chem498C/SignalProcessing.html) c. Tutorial signals: A library of prerecorded data files for use with the signal processing tutorial. These are plain decimal ascii (tab-delimited) data files. These files are binhex encoded: use Stuffit Expander to decode and decompress as usual. If you are downloading on a Macintosh, all this should happen completely automatically. If you are downloading on a Windows PC, shift-click on the download links above to begin the download. If you are using the ARDI Executor Mac simulator, download the "HQX" files to your C drive, launch Executor, then open the downloaded HQX files with Stuffit Expander, which is pre-loaded into the Executor Macintosh environment. Stuffit Expander will automatically decode and decompress the downloaded files. Note: Because it was developed for academic teaching application where the most modern and powerful models of computers may not be available, SPECTRUM was designed to be "lean and mean" - that is, it has a simple Macintosh-type user interface and very small memory and disk space requirements. It will work quite well on Macintosh models as old as the Macintosh II, and will even run on older monochrome models (with some cramping of screen space). It does not require a math co-processor. (c) 1989 T. C. O'Haver. This program is free and may be freely distributed. It may be included on CD-ROM collections or other archives.
- 57. 2. Matlab Matlab is a high-performance commercial numerical computing environment and programming language that is widely used in research and education. See http://en.wikipedia.org/wiki/MATLAB for a general description. There are several good on-line tutorials and collections of sample code. For example: a. Video Tutorials for New MATLAB Users (http://www.youtube.com/results? search_query=matlab+tutorial&aq=f ). b. An Introductory Guide to MATLAB. (http://www.cs.ubc.ca/spider/cavers/MatlabGuide/guide.html) c. Matlab Summary and Tutorial. (http://www.math.ufl.edu/help/matlab-tutorial/) d. A Practical Introduction to Matlab (http://www.math.mtu.edu/~msgocken/intro/intro.html) e. Matlab Chemometrics Index http://www.mathworks.com/matlabcentral/linkexchange/?term=chemometrics f. Introduction to Matlab: http://homepages.math.uic.edu/~jan/mcs320s07/ g. Practical Statistical Signal Processing using MATLAB. http://www.aticourses.com/sampler/Practical%20Signal%20Processing%20using %20MATLAB.pdf h. Multivariate Curve Resolution. http://www.mcrals.info/
- 58. 3. Peak Finding and Measurement in Matlab A common requirement in signal processing is to detect peaks in a signal and to measure their positions, heights, and widths. A common way to do this is to make use of the fact that the first derivative of a peak has a downward-going zero-crossing at the peak maximum. But the presence of random noise in real experimental signal will cause many false zero-crossing simply due to the noise. To avoid this problem, the technique described here first smooths the first derivative of the signal, before looking for downward-going zero-crossings, and then it takes only those zero crossings whose slope exceeds a certain predetermined minimum (called the "slope threshold") at a point where the original signal exceeds a certain minimum (called the "amplitude threshold"). By carefully adjusting the smooth width, slope threshold, and amplitude threshold, it's possible to detect only the desired peaks and ignore peaks that are too small, too wide, or too narrow. Moreover, because smoothing can distort peak signals, reducing peak heights and increasing peak widths, this technique estimates the position, height, and width of each peak by least-squares curve-fitting of a segment of the original unsmoothed signal in the vicinity of the zero-crossing. Thus, even if heavy smoothing of the first derivative is necessary to provide reliable discrimination against noise peaks, the peak parameters extracted by curve fitting are not distorted. (This is useful primarily for signals that have several data points in each peak, not for spikes that have only one or two points). This technique is capable of measuring peak positions and heights quite accurately, but the measurements of peak widths and areas is accurate only if the peaks are approximately Gaussian in shape. (For the most accurate measurement of highly overlapped peaks, iPeak version 3 is capable of utilizing non-linear iterative curve fitting). The routine is available in three different versions: (1) the basic findpeaks function (findpeaks.m); (2) an interactive keypress-operated function (ipeak.m) for adjusting the peak detection criteria in real-time to optimize for any particular peak type; and (3) a script using mouse-controlled sliders for interactive control. These are all detailed below. Click here to download the ZIP file "Peakfinder.zip" that includes supporting functions, several self-contained demos to show how it works, and two interactive versions. You can also download it from the Matlab File Exchange. The findpeaks function (findpeaks.m) P=findpeaks(x,y,SlopeThreshold,AmpThreshold,SmoothWidth,FitWidth, smoothtype) This is the basic function to locate and measure the positive peaks in a noisy data sets. It detects peaks by looking for downward zero-crossings in the smoothed first derivative that exceed SlopeThreshold and peak amplitudes that exceed AmpThreshold, and determines the position, height, and approximate width of each peak by least-squares curve-fitting the top part of the peak. Returns a list (in matrix P) containing the peak number and the estimated position, height, and width of each peak. The data are passed in the vectors x and y (x = independent variable, y = dependent variable). The other four parameters are user-adjustable:
- 59. SlopeThreshold - Slope of the smoothed first-derivative that is taken to indicate a peak. This discriminates on the basis of peak width. Larger values of this parameter will neglect broad features of the signal. A reasonable initial value for Gaussian peaks is 0.7*WidthPoints^-2, where WidthPoints is the number of data points in the half-width of the peak. AmpThreshold - Discriminates on the basis of peak height. Any peaks with height less than this value are ignored. SmoothWidth - Width of the smooth function that is applied to data before the slope is measured. Larger values of SmoothWidth will neglect small, sharp features. A reasonable value is typically about equal to 1/2 of the number of data points in the half-width of the peaks. FitWidth - The number of points around the "top part" of the (unsmoothed) peak that are taken to estimate the peak heights, positions, and widths. A reasonable value is typically about equal to 1/2 of the number of data points in the half-width of the peaks. The minimum value is 3. Smoothtype (added in Version 4, Sept. 2011) determines the smoothing algorithm (see http://terpconnect.umd.edu/~toh/spectrum/Smoothing.html) If smoothtype=1, rectangular (sliding-average or boxcar) If smoothtype=2, triangular (2 passes of sliding-average) If smoothtype=3, pseudo-Gaussian (3 passes of sliding-average) Basically, higher values yield greater reduction in high-frequency noise, at the expense of slower execution. There is also a similar function for finding valleys (minima), called findvalleys.m, which works the same way as findpeaks.m, except that it locates minima instead of maxima. Right-click the link and select "Save link as...." Examples: >> x=[0:.01:50];y=cos(x);findpeaks(x,y,0,-1,5,5) ans = 1 6.2832 1 2.3569 2 12.566 1 2.3569 3 18.85 1 2.3568 4 25.133 1 2.3569 5 31.416 1 2.3569 6 37.699 1 2.3568 7 43.982 1 2.3569 >> x=[0:.01:50];y=cos(x);findvalleys(x,y,0,-1,5,5) ans = 1 3.1416 -1 2.3571 2 9.4248 -1 2.3571 3 15.708 -1 2.3571 4 21.991 -1 2.3571 5 28.274 -1 2.3571 6 34.558 -1 2.3571 7 40.841 -1 2.3571 Note: The accuracy of the measurements of peak position, height, and width by the findpeaks function depends on the shape of the peaks, the extent of peak overlap, and signal-to-noise ratio. The width measurements particularly are strongly influenced by
- 60. peak overlap, noise, and the choice of FitWidth. Isolated peaks of Gaussian shape are measured most accurately. See "ipeakdemo" (below) for an accuracy trial. For the most accurate measurements of highly overlapping peaks of various shapes, use the Interactive Peak Fitter (ipf.m). DemoFindPeak (DemoFindPeak.m) A simple demonstration script using the findpeaks function on noisy synthetic data. Numbers the peaks and prints out the P matrix in the Matlab command window: Peak # Position Height Width P = 1.0000 10.0235 0.1814 1.7506 2.0000 29.9964 0.6412 1.9931 3.0000 50.0205 0.4315 2.1433 4.0000 70.0120 1.3935 2.0202 5.0000 79.9567 0.2580 1.8267 6.0000 100.0086 1.0862 2.0306 ...... The iPeak Function (ipeak.m), version 3.8 ZIP file available at http://terpconnect.umd.edu/~toh/spectrum/ipeak3.zip iPeak is a keyboard-operated Interactive Peak Finder for time series data, based on the "findpeaks.m" function. It displays the entire signal in the lower half of the Figure window and an adjustable zoomed-in section in the upper window. (Press Enter to change the plot color.) Simple keystrokes allow you to adjust the peak detection parameters AmpThreshold (A/Z keys), SlopeThreshold (S/X), SmoothWidth (D/C), FitWidth (F/V), and other controls. (See list of Keyboard Controls, below). This function can be used to determine experimentally what values of those parameters give the most reliable peak detection for a particular type of data, detecting the desired peaks and ignoring those that are too small, too broad, or too narrow to be of interest. Detected peaks are numbered from left to right. Returns the peak table in P (Peak #, Position, Height, Width, Area). EXAMPLE 1: Two input arguments; data in separate x and y vectors >> x=[0:.1:100];y=(x.*sin(x)).^2;ipeak(x,y); EXAMPLE 2: One input argument; data in two columns of a matrix
- 61. >> x=[0:.01:5]';y=x.*sin(x.^2).^2;ipeak([x y]) EXAMPLE 3: One input argument; data in single vector >> y=cos(.1:.1:100);ipeak(y) Keystrokes allow you to pan and zoom the upper window, to inspect each peak in detail if desired. You can set the initial values of pan and zoom in optional input arguments 7 ('xcenter') and 8 ('xrange'). See example 6 below. The Spacebar/Tab keys (version 3.2 and later) jump to the next/previous detected peak and displays it in the upper window at the current zoom setting (use the up and down cursor arrow keys to adjust the zoom range). The L key turns off and on peak parameter labeling. The Y key (version 3.2) toggles between linear and log y-axis scale in the lower window (a log axis is good for inspecting signals with high dynamic range; it effects only the lower window display and has no effect on the peak detection or measurements). The U key switches between peak and valley detection mode. The T key toggles the autozero mode off and on (version 3.2 and later). When autozero is OFF, peak heights are measured relative to zero. (If the peaks are superimposed on a background, use the baseline subtract keys - B and G - first to subtract the background). When autozero is ON, peak heights are automatically measured relative to the local baseline on either side of the peak; use the zoom controls to isolate the peaks so that the signal returns to the local baseline between the peaks as displayed in the upper window. When autozero is ON, the peak heights, widths, and areas in the peak table (R or P keys) will be automatically corrected for the baseline. (Autozero OFF will give better results when the baseline is zero, or has been subtracted using the B key, even if the peaks are partly overlapped. Autozero ON will work best ifthe peaks are well separated so that the signal returns to the local baseline between the peaks. If the peaks are highly overlapped, or if they are not Gaussian in shape, the best results will be obtained by using the curve fitting function - the N or M keys). Normal and Multiple Curve fitting: The N key applies variable-shape iterative curve fittingto the detected peaks that are displayed in the upper window (referred to a "Normal" curve fitting). If the peaks are superimposed on a background, turn on the Autozero mode (T key). Then use the pan and zoom keys to select a peak or a group of overlapping peaks in the upper window, with the signal returning all the way to the local baseline at the ends of the upper window. Make sure that AmpThreshold, SlopeThreshold,SmoothWidth are adjusted so that each peak is numbered once. Then press the N key, type a number for the desired peak shape at the prompt in the Command window and press Enter (1=Gaussian (default); 2=Lorentzian; 3=logistic, 4=Pearson; 5=exponentially broadened Gaussian, 6=equal-width Gaussians; 7=Equal-width Lorentzians; 8=exponentially broadened equal-width Gaussian), then type in a number of repeat trial fits and press Enter (the default is 1; start with that and then increase if necessary). The program will perform the fit, display the results graphically in Figure 2, and print out a table of results in the command window, e.g.: Peak shape (1-8): 2
- 62. Number of trials: 1 Least-squares fit to Lorentzian peak model Fitting Error 1.1581e-006% Peak# Position Height Width Area 1 100 1 50 71.652 2 350 1 100 146.13 3 700 1 200 267.77 The use of the iterative least-squares function can result in more accurate peak parameter measurements that the normal peak table (R or P keys), especially if the peaks are non- Gaussian in shape or are highly overlapped. There is also an experimental "Multiple" peak fit function (M key) that will attempt to apply non-linear iterative curve fitting to to all the detected peaks in the signal automatically. Before using this function, use the baseline correction function first (B key) to remove the background signal. Then press M and proceed as for the normal curve fit. With some signals, this function may not work as well as using the normal (N-key) curve fitting function on each group of peaks separately. Note: The N and M key fitting functions perform non-linear iterative curve fitting using the peakfit.m function. The number of peaks and the starting values of peak positions and widths for the curve fit function are automatically supplied by the the findpeaks function, so it is essential that the peak detection variables in iPeak be adjust so that all the peaks in the selected ragion are detected and numbered once. (For more flexible curve fitting, use ipf.m). EXAMPLE 4: Additional scalar argument (after the data) to control peak sensitivity. >> x=[0:.1:100];y=5+5.*cos(x)+randn(size(x));ipeak(x,y,10); or >> ipeak([x;y],10); or >> ipeak(humps(0:.01:2),3) or >> x=[0:.1:10];y=exp(-(x-5).^2);ipeak([x' y'],1) The additional argument is an estimate of the ratio of the typical peak width to the length of the entire data record. Small values detect fewer peaks; larger values detect more peaks. It effects only the starting values for the peak detection parameters. (This is just a quick way to set reasonable initial values of the peak detection parameters, rather than specifying each one individually as in the following example). EXAMPLE 5: Six input arguments. As above, but input arguments 3 to 6 directly specify the initial values of PeakD, AmpT, SlopeT, SmoothW, FitW. (PeakD is ignored in this case, so just type a '0' as the second argument after the data matrix). >> ipeak(datamatrix,0,.5,.0001,20,20); EXAMPLE 6: Eight input arguments. As above, but input arguments 7 and 8 specifiy the initial pan and zoom settings, 'xcenter' and 'xrange', respectively. In this example, the x- axis data are wavelengths in nanometers (nm), and the upper window zooms in on a very small 0.4 nm region centered on 249.7 nm. (These data, provided in the ZIP file, are from a high-resolution atomic spectrum). >> load ipeakdata.mat >> ipeak(Sample1,0,110,0.06,3,4,249.7,0.4);
- 63. EXAMPLE 7: Nine input arguments. As example 6, but the 9th input argument turns ON the autozero mode (equivalent to pressing the T key). If not specified, autozero is initially OFF. >> ipeak(Sample1,0,110,0.06,3,4,249.7,0.4,1); Peak identification: There is an optional "peak identification" function if optional input arguments 9 ('MaxError'), 10 ('Positions'), and 11 ('Names') are included. The "i" key toggles this function ON and OFF. This function compares the found peak positions (maximum x-values) to a database of known peaks, in the form of an array of known peak maximum positions ('Positions') and matching cell array of names ('Names'). If the position of a found peak in the signal is closer to one of the known peaks by less than the specified maximun error ('MaxError'), then that peak is considered a match and its name is displayed next to the peak in the upper window. When when the 'o' key is pressed, the peak positions, names, errors, and amplitudes are printed out in a table in the command window. EXAMPLE 8: Eleven input arguments. As above, but also specifies 'MaxError', 'Positions', and 'Names' in optional input arguments 9, 10, and 11, for peak identification function. Pressing the 'I' key toggles off and on the peak identification labels in the upper window. Pressing "o" prints the peak positions, names, errors, and amplitudes in a table in the command window. These data (provided in the ZIP file) are from a high-resolution atomic spectrum (x-axis in nanometers). >> load ipeakdata.mat >> ipeak(Sample1,0,120,0.06,3,6,296,5,0.1,Positions,Names); Note: The ZIP file contains the latest version of the iPeak function as well as some sample data to demonstrate peak identification (Example 7). Obviously for your own applications, it's up to you to provide your own array of known peak maximum positions ('Positions') and matching cell array of names ('Names') for your particular types of signals. Keyboard Controls: Pan signal left and right...Coarse pan: < and > Fine pan: left and right cursor arrow keys Nudge left and right: [ and ] Zoom in and out.............Coarse zoom: / and ' Fine zoom: up and down cursor arrow keys Resets pan and zoom.........ESC Change plot color...........Enter Adjust AmpThreshold.........A,Z (A increases, Z decreases) Adjust SlopeThreshold.......S,X (S increases, X decreases) Adjust SmoothWidth..........D,C (D increases, C decreases) Adjust FitWidth.............F,V (F increases, Z decreases) Baseline correction.........B, then click baseline at multiple points Restore original signal.....G to cancel previous background subtraction Invert signal...............- Invert (negate) the signal (flip + and -) Set minimum to zero.........0 (Zero) Sets minumun signal to zero Toggle log y mode OFF/ON....Y Plot log Y axis on lower graph Toggle autozero OFF/ON......T Auto background subtraction on upper graph Toggle valley mode OFF/ON...U Switch to valley mode Print peak table............P Prints Peak #, Position, Height, Width Step through peaks..........Space/Tab Jumps to next/previous peak Normal peak fit.............N Fit peaks in upper window with peakfit.m Multiple peak fit...........M Fit all peaks in signal with peakfit.m Print keyboard commands.....K Prints this list Print findpeaks arguments...Q Prints findpeaks function with arguments. Print ipeak arguments.......W Prints ipeak function with all arguments.
- 64. Print report................R Prints Peak table and parameters Peak labels ON/OFF......... L Displays peak parameters in upper window. Peak ID ON/OFF..............I Identifies closest peaks in 'Names' database. Print peak IDs..............O Prints table of peaks IDs iPeak Demo scripts The ZIP file at http://terpconnect.umd.edu/~toh/spectrum/ipeak3.zip contains several demo scripts (ipeakdemo.m, ipeakdemo1.m, etc) that illustrate various aspect of the iPeak function and how it can be used effectively. Run them by typing their name at the Matlab command windows prompt or by opening them in the Matlab editor and clicking the green "Run" button on the editor toolbar. ipeakdemo.m ipeakdemo: effect of the peak detection parameters Four Gaussian peaks with the same heights but different widths (10, 30, 50 and 70 units) This demonstrates the effect of SlopeThreshold and SmoothWidth on peak detection. Increasing SlopeThreshold (S key) will discriminate against the broader peaks. Increasing SmoothWidth (D key) will discriminate against the narrower peaks and noise. FitWidth (F/V keys) controls the number of points around the "top part" of the (unsmoothed) peak that are taken to estimate the peak heights, positions, and widths. A reasonable value is about equal to 1/2 of the number of data points in the half-width of the peaks. In this case, where the peak widths are quite different, set it to about 1/2 of the number of data points in the narrowest peak. ipeakdemo1.m ipeakdemo1: the autozero mode Demonstration of autozero background correction, for separated, narrow peaks on a large baseline. Each time you run this demo, you will get a different set of peaks and noise. A table of the actual peak positions, heights, widths, and areas is printed out in the command window. Jump to the next/previous peaks using the Spacebar/Tab keys. Hint: Turn on the Autozero mode (T key), adjust the zoom setting so that the peaks are shown one at a time in the upper window, then press the P key to display the peak table. ipeakdemo2.m ipeakdemo2: peak overlap and the curve fitting functions. Demonstration of error caused by overlapping peaks on a large offset baseline. Each time you run this demo, you will get a different set of peaks and noise. A table of the actual peak positions, heights, widths, and areas is printed out in the command window. Jump to the next/previous peaks using the Spacebar/Tab keys. Hint: Use the B key and click on the baseline points, then press the P key to display the peak table. Or turn on the Autozero mode (T key) and use the Normal curve fit (N key) or Multiple curve fit (M key). 240 260 280 300 320 340 0 1 2 3 4 iPeak 3.5. Autozero ON. Press K for keyboard commands Peak 1 Position: 300.4918 Height: 3.9459 Width: 27.9814 500 1000 1500 2000 2500 3000 3500 4000 5 10 15 Adjustment Keys: AmpT: A/Z SlopeT: S/X SmoothWidth: D/C FitWidth: F/V Linearmode AmpT: 0.89678 SlopeT: 0.00077778 SmoothW: 15 FitW: 15 1 2 3 4 5 6 7 8 9 10 11 12 13 1415 16 17 18 400 420 440 460 480 500 520 540 560 580 6 8 10 iPeak 3.5. Autozero OFF. Press K for keyboard commands 1 2 3 Peak 1 Position: 500.0927 Height: 10.7729 Width: 81.856 500 1000 1500 2000 2500 3000 3500 4000 2 4 6 8 10 Adjustment Keys: AmpT: A/Z SlopeT: S/X SmoothWidth: D/C FitWidth: F/V Linearmode AmpT: 0.53983 SlopeT: 0.00019444 SmoothW: 30 FitW: 30 1 2 3 4 5 6 7 8 910 11 12 13 14 15 16 17 40 60 80 100 120 140 160 0.2 0.4 0.6 0.8 1 iPeak 3.5. Autozero OFF. Press K for keyboard commands 1 2 3 4 Peak 1 Position: 100 Height: 1 Width: 10.0004 50 100 150 200 250 300 350 400 450 500 0.2 0.4 0.6 0.8 1 Adjustment Keys: AmpT: A/Z SlopeT: S/X SmoothWidth: D/C FitWidth: F/V Linearmode AmpT: 0.050175 SlopeT: 0.0004375 SmoothW: 20 FitW: 20 1 2 3 4
- 65. ipeakdemo3.m ipeakdemo3: Non-Gaussian peak shapes Demonstration of overlapping Lorentzian peaks, without an added background. Overlap of peaks causes significant errors in peak height, width, and area. Jump to the next/previous peaks using the Spacebar/Tab keys. Each time you run this demo, you will get a different set of noise. A table of the actual peak positions, heights, widths, and areas is printed out in the command window. Hint: turn OFF the Autozero mode (T key) and use the Normal curve fit (N key) or Multiple curve fit (M key) with peak shape 2 (Lorentzian). ipeakdemo4.m ipeakdemo4: dealing with very noisy signals Detection and measurement of four peaks in a very noisy signal. The signal-to-noise ratio of first peak is 2. Each time you run this demo, you will get a different set of noise. A table of the actual peak positions, heights, widths, and areas is printed out in the command window. Jump to the next/previous peaks using the Spacebar/Tab keys. The peak at x=100 is usually detected, but the accuracy of peak parameter measurement is poor because of the low signal-to-noise ratio. Hint: With very noisy signals it is usually best to increase SmoothWidth and FitWidth to help reduce the effect of the noise. Note: The ZIP file available at http://terpconnect.umd.edu/~toh/spectrum/ipeak3.zip contains findpeaks, DemoFindPeaks, ipeak.m, all the ipeakdemos described above, and the ipeakdata.mat file with the high-resolution atomic spectrum used in example 8. 4. iSignal: Interactive Smoothing, Differentiation, and Peak Sharpening iSignal is a Matlab function, written as a single self-contained m-file, for performing smoothing, differentiation, and peak sharpening (resolution enhancement) and other useful functions on time-series data. Using simple keystrokes, you can adjust the signal processing parameters continuously while observing the effect on your signal dynamically. The ZIP file for version 1.9 is available for download at http://terpconnect.umd.edu/~toh/spectrum/iSignal19.zip. The syntax is: Y=isignal(Data); or Y=isignal(Data,xcenter,xrange,SmoothMode,SmoothWidth,ends,DerivativeMod e,Sharpen,Sharp1,Sharp2); "Data" may be a 2-column matrix with the independent variable (x-values) in the first column and dependent variable (y values) in the second column, or separate x and y vectors, or a single y-vector (in which case the data points are plotted against their index numbers on the x axis). Only the first argument is required; all the others are optional. Returns the processed independent axis (Y) vector as the output argument. Plots the data in the figure window, the lower half of the window showing the entire signal, and the 40 60 80 100 120 140 160 0 0.5 1 iPeak 3.5. Autozero OFF. Press K for keyboard commands 1 Peak 1 Position: 100.1629 Height: 0.93683 Width: 13.1152 100 200 300 400 500 600 700 800 900 1000 1100 1200 0 5 10 Adjustment Keys: AmpT: A/Z SlopeT: S/X SmoothWidth: D/C FitWidth: F/V Linearmode AmpT: 0.54237 SlopeT: 0.00066272 SmoothW: 16 FitW: 16 1 2 3 4 5 6 7 8 9 10 40 60 80 100 120 140 160 -1 0 1 2 iPeak 3.5. Autozero OFF. Press K for keyboard commands 1 2 3 Peak 1 Position: 99.6398 Height: 1.0711 Width: 21.5707 50 100 150 200 250 300 350 400 450 500 0 2 4 Adjustment Keys: AmpT: A/Z SlopeT: S/X SmoothWidth: D/C FitWidth: F/V Linearmode AmpT: 0.21827 SlopeT: 0.00077778 SmoothW: 30 FitW: 30 1 2 3 4
- 66. upper half showing a selected portion controlled by the pan and zoom keystrokes, with the initial pan and zoom settings optionally controlled by input arguments 'xcenter' and 'xrange', respectively. Other keystrokes allow you to control the smooth type, width, and ends treatment, the derivative order (0th through 4th), and peak sharpening. (The initial values of all these parameters can be passed to the function via the optional input arguments SmoothMode, SmoothWidth, ends, DerivativeMode, Sharpen, Sharp1, and Sharp2. See the examples below). Press K to see all the keyboard commands. Smoothing Reference: http://terpconnect.umd.edu/~toh/spectrum/Smoothing.html The S key (or input argument "SmoothMode") cycles through four smoothing modes: If SmoothMode=0, the signal is not smoothed. If SmoothMode=1, rectangular (sliding-average or boxcar) If SmoothMode=2, triangular (2 passes of sliding-average) If SmoothMode=3, pseudo-Gaussian (3 passes of sliding-average) If SmoothMode=4, Savitzky-Golay smooth. The A and Z keys (or optional input argument SmoothWidth) control the SmoothWidth, w. The X key toggles "ends" between 0 and 1. This determines how the "ends" of the signal (the first w/2 points and the last w/2 points) are handled when smoothing. If ends=0, the ends are zero. If ends=1, the ends are smoothed with progressively smaller smooths the closer to the end. Generally, ends=1 is best, except in some cases using the derivative mode when ends=0 result in better vertical centering of the signal. Note: when smoothing peaks, you can easily measure the effect of smoothing on peak height and width by turning on peak measure mode (press P) and then press S to cycle through the smooth modes. Differentiation Reference: http://terpconnect.umd.edu/~toh/spectrum/Differentiation.html The D key (or optional input argument "DerivativeMode") cycles through the derivative orders 0, 1, 2, 3, 4, and then back to 0. The default is 0. Careful optimization of smoothing of derivatives is critical for acceptable signal-to-noise ratio. Peak sharpening (resolution enhancement) Reference: http://terpconnect.umd.edu/~toh/spectrum/ResolutionEnhancement.html The E key (or optional input argument "Sharpen") turns off and on peak sharpening (resolution enhancement). The sharpening strength is controlled by the F and V keys (or
- 67. optional input argument "Sharp1") and B and G keys (or optional argument "Sharp2"). The optimum values depend on the peak shape and width. For peaks of Gaussian shape, a reasonable value for Sharp1 is PeakWidth2/25 and for Sharp2 is PeakWidth4/800 (or PeakWidth2/6 and PeakWidth4/700 for Lorentzian peaks), where PeakWidth is the full- width at half maximum of the peaks expressed in number of data points. However, you don't need to do the math yourself; iSignal 1.6 can calculate sharpening and smoothing settings for Gaussian and for Lorentzian peak shapes using the Y and U keys, respectively. Just isolate a single typical peak in the upper window using the pan and zoom keys, then press Y for Gaussian or U for Lorentzian peaks. (The optimum settings depends on the width of the peak, so if your signal has peaks of widely different widths, one setting will not be optimum for all the peaks). You can fine-tune the sharpening with the F/V and G/B keys and the smoothing with the A/Z keys. You can expect a decrease in peak width (and corresponding increase in peak height) of about 20% - 50%, depending on the shape of the peak (the peak area is largely unchanged). Excessive sharpening leads to baseline artifacts and increased noise. iSignal allows you to experimentally determine the values of these parameters that give the best trade-off between sharpening, noise, and baseline artifacts, for your purposes. You can easily measure the effect of sharpening on peak width by turning on peak measure mode (press P) and then press E to toggle the sharpen mode off and on. Signal measurement The cursor keys control the position of the green cursor and the dotted red cursors that mark the selected range displayed in the upper graph window. The label under the top graph window shows the value of the signal (y) at the green cursor, the peak-to-peak (min and max) signal range, the area under the signal, and the standard deviation within the selected range (the dotted cursors). Pressing the Q key prints out a table of the signal information in the command window. Signal Noise Signal-to-noise ratio (SNR) measurement of a peak with very high SNR. Left: The peak height of the largest signal peak is measured by placing the green center cursor on the largest peak; peak-to-peak signal=66,000. Right: The noise is measured on a flat portion of the baseline: standard deviation of noise=0.01, therefore the SNR=66,000/.01 = 6,600,000 Background subtraction Background (or baseline) subtraction is important because the peak heights, widths, and
- 68. areas measured by the Peak Measure (P) command are based on ther assumption that the baseline under the peaks is zero. The Backspace key starts background correction mode. In the command window, type in the number of background points to click and press the Enter key. The cursor changes to cross hairs; click along the presumed background, starting to the left of the x axis and placing the last click to the right of the x axis. When the last point is clicked, the linearly interpolated baseline between those points is subtracted from the signal. To restore the original background (i.e. to correct an error or to try again), press the '' key (just below the backspace key). Saving the results To save the processed signal as a .mat file, press the 'o' key, the type in the desired file name, then press Enter or click Save. Peak measurement The P key toggles off and on the peak measure mode, which attempts to measure the one peak that is centered in the upper window under the green cursor; a red "cap" on the peak indicates that portion of the signal that is taken for the measurement. (Adjust the pan and zoom controls so that the red cap overlays roughly the top half of the peak). The peak position, height, width are measured by least-squares curve fitting of a Gaussian to the isolated peak; the peak area is measured by the trapezoidal method over the selected segment displayed in the upper window. (If the peak is noisy and/or non-Gaussian, the results will only be approximate, but the position, height, and area values are pretty good for any peak shape. If the peaks are superimposed on a non-zero background, subtract the background before measuring peaks). Press the R key to print out the peak parameters in the command window. Other keystroke controls The L key toggles off and on the Overlay mode, which overlays the current processed signal with the original signal as a dotted line, for the purposes of comparison. The tab key restores the original signal and cursor settings. The T key sets the selected region to zero (to trim away artifacts and spikes). The "-" (minus sign) key is used to negate the signal (flip + for -). Press H to toggle display of semilog y plot in the lower window, useful for signals with very wide dynamic range (zero and negative points are ignored in the log plot). Press K to see all the keyboard commands. EXAMPLE 1: Single input argument; data in a two columns of a matrix [x;y] or in a single y vector >> isignal(y); >> isignal([x;y]); EXAMPLE 2: Two input arguments. Data in separate x and y vectors. >> isignal(x,y); EXAMPLE 3: Three or four input arguments. The last two arguments specify the initial values of pan (xcenter) and zoom (xrange) in the last two input arguments. Using data in the ZIP file:
- 69. >> load data.mat >> isignal(DataMatrix,180,40); or >> isignal(x,y,180,40); EXAMPLE 4: As above, but additionally specifies initial values of SmoothMode, SmoothWidth, ends, and DerivativeMode in the last four input arguments. >> isignal(DataMatrix,180,40,2,9,0,1); EXAMPLE 5: As above, but additionally specifies initial values of the peak sharpening parameters Sharpen, Sharp1, and Sharp2 in the last three input arguments. Press the E key to toggle sharpening on and off for comparison. >> isignal(DataMatrix,180,40,2,9,0,0,1,51,6000); EXAMPLE 6: Using the built-in "humps" function: >> x=[0:.005:2];y=humps(x);Data=[x;y]; 4th derivative of the peak at x=0.9: >> isignal(Data,0.9,0.5,1,3,1,4); Peak sharpening applied to the peak at x=0.3: >> isignal(Data,0.3,0.5,1,3,1,0,1,220,5400); Press 'E' key to toggle sharpening ON/OFF to compare) iSignal keyboard controls: Pan signal left and right...Coarse pan: < and > Fine pan: left and right cursor arrows Nudge one point left or right: [ and ] Zoom in and out.............Coarse zoom: / and " Fine zoom: up and down cursor arrows Reset pan and zoom..........ESC (resets to initial default values) Adjust smooth width.........A,Z (A=>more smoothing, Z=>less smoothing) Adjust smooth type..........S (None, Rectanglular, Triangle, Gaussian, Savitzky-Golay) Toggle smooth ends..........X (0=ends zeroed 1=ends smoothed Adjust derivative order.....D (0th to 4th order) Toggle peak sharpening......E (0=OFF 1=ON) Sharpening for Gaussian.....Y Set sharpen settings for Gaussians Sharpening for Lorentzian...U Set sharpen settings for Lorentzians Adjust sharp1...............F,V F=>sharper, V=>less sharpening Adjust sharp2 ............G,B G=>sharper, B=>less sharpening Toggle peak labeling........P labels center peak in upper window Toggle overlay mode.........L Overlays original signal in upper window Toggle log y mode...........H Display semilog y plot in lower window Toggle Autozero OFF/ON......T Subtracts background from upper window segment Restores original signal....Tab key resets to original signal and modes Baseline subtraction........Backspace, then click baseline at 8 points Restore background.......... to cancel previous background subtraction Invert signal...............- Invert (negate) the signal (flip + and -) Remove offset...............0 (zero) set minimun signal) to zero Trim region to zero.........; sets selected region to zero. Print keyboard commands.....K prints this list Print signal report.........Q prints signal info and current settings Print isignal arguments.....W prints isignal (with current arguments) Save output to disk.........O as .mat file containing processed signal
- 70. 5. Interactive Peak Fitter A Matlab peak fitting program for time-series signals, which uses an unconstrained non- linear optimization algorithm to decompose a complex, overlapping-peak signal into its component parts. The objective is to determine whether your signal can be represented as the sum of fundamental underlying peaks shapes. Accepts signals of any length, including those with non-integer and non-uniform x-values. Fits groups of up to 6 peaks with 8 distinct peak shapes (expandable to other shapes). a. Command line function: peakfit.m Peakfit.m is a user-defined command window peak fitting function, usable from a remote terminal. It is written as a self-contained Matlab function in a single m-file. (To view of download, click Peakfit.m). It takes data in the form of a 2 x n matrix that has the X values in row 1 and the Y values in row 2, or as a single independent variable vector. A ZIP file containing peakfit.m, DemoPeakFit.m, ipf.m, Demoipf.m, and some sample data for testing is available at from http://terpconnect.umd.edu/~toh/spectrum/ipf57.zip Peakfit can be called with several optional additional arguments peakfit(signal); Performs an iterative least-squares fit of a single Gaussian peak to the entire data matrix "signal", which has x values in row 1 and Y values in row 2 (e.g. [x y]) or which may be a single signal vector (in which case the data points are plotted against their index numbers on the x axis). peakfit(signal,center,window); Fits a single Gaussian peak to a portion of the matrix "signal". The portion is centered on the x-value "center" and has width "window" (in x units). In this and in all following examples, set "center" and "window" both to 0 to fit the entire signal. peakfit(signal,center,window,NumPeaks); "NumPeaks" = number of peaks in the model (maximum 6; default is 1 if not specified). peakfit(signal,center,window,NumPeaks,peakshape); Specifies the peak shape of the model: "peakshape" = 1-8. (1=Gaussian, 2=Lorentzian, 3=logistic, 4=Pearson, 5=exponentionally broadened Gaussian; 6=equal-width Gaussians, 7=equal-width Lorentzians, 8=exponentionally broadened equal-width Gaussians), 9=exponential pulse, 10=sigmoid). peakfit(signal,center,window,NumPeaks,peakshape,extra) Specifies the value of 'extra', used in the Pearson and the exponentionally broadened
- 71. Gaussian shapes to fine-tune the peak shape. peakfit(signal,center,window,NumPeaks,peakshape,extra,NumTrials); Performs "NumTrials" trial fits and selects the best one (with lowest fitting error). NumTrials can be any positive integer (default is 1). peakfit(signal,center,window,NumPeaks,peakshape,extra,NumTrials,s tart) Specifies the first guesses vector "firstguess" for the peak positions and widths, e.g. start=[position1 width1 position2 width2 ...] peakfit(signal,center,window,NumPeaks,peakshape,extra,NumTrials,s tart, autozero) Specifies the AUTOZERO setting: 0 is OFF and 1 is ON. When ON, it removes linear baseline from data segment. Default is ON (AUROZERO=1). [FitResults,MeanFitError]=peakfit(signal,center,window...) Returns the FitResults vector in the order peak number, peak position, peak height, peak width, and peak area), and the MeanFitError (the percent RMS difference between the data and the model in the selected segment of that data) of the best fit. Optional output parameters: 1. FitResults: a table of model peak parameters, one row for each peak, listing Peak number, Peak position, Height, Width, and Peak area. 2. LowestError: The rms fitting error of the best trial fit. 3. BestStart: the starting guesses that gave the best fit. 4. xi: vector containing 100 interploated x-values for the model peaks. 5. yi: matrix containing the y values of model peaks at each xi. Type plot(xi,yi(1,:)) to plot peak 1 or plot(xi,yi) to plot all the peaks Example 1: Signal is a matrix' Fits exp(-x)^2 with a single Gaussian peak model. > x=[0:.1:10];y=exp(-(x-5).^2);peakfit([x' y']) ans = 1 5 1 1.665 1.7725 Peak number Peak position Height Width Peak area Example 1: Signal is a single vector: Create a small data set and fit to a Gaussian model >> peakfit([0 0 0 1 2 4 6 7 6 4 2 1 0 0 0]) ans = 1 7.9741 6.8279 4.4068 32.023 Example 3: Like Example 1, except that random noise is added to the y data. Compare output to Example 1. > x=[0:.1:10];y=exp(-(x-5).^2)+.1*randn(1,length(x));peakfit([x' y']) Example 4: Fits a noisy two-peak signal with a double Gaussian model (NumPeaks=2). > x=[0:.1:10];y=exp(-(x-5).^2)+.5*exp(-(x-3).^2)+.1*randn(1,length(x));
- 72. > peakfit([x' y'],5,19,2,1,0,1) ans = 1 3.0001 0.49489 1.642 0.86504 2 4.9927 1.0016 1.6597 1.7696 Example 5: > x=[0:.005:1];y=humps(x);peakfit([x' y'],.3,.7,1,4,3); Fits a portion of the humps function, 0.7 units wide and centered on x=0.3, with a single (NumPeaks=1) Pearson function (peakshape=4) with extra=3 (controls shape of Pearson function). Example 6: Creates a data matrix 'smatrix', fits a portion to a two-peak Gaussian model, takes the best of 10 trials. Returns FitResults and MeanFitError. > x=[0:.005:1];y=(humps(x)+humps(x-.13)).^3;smatrix=[x' y']; > [FitResults,MeanFitError]=peakfit(smatrix,.4,.7,2,1,0,10) FitResults = 1 0.4128 3.1114e+008 0.10448 3.4605e+007 2 0.3161 2.8671e+008 0.098862 3.0174e+007 MeanFitError = 0.68048 Example 7: As above, but specifies the first-guess position and width of the two peaks, in the order [position1 width1 position2 width2] > peakfit([x' y'],.4,.7,2,1,0,10,[.3 .1 .5 .1]); Example 8: As above, returns the vector x1 containing 100 interploated x-values for the model peaks and the matrix y1 containing the y values of each model peak at each xi. Type plot(xi,yi(1,:)) to plot peak 1 or plot(xi,yi) to plot all peaks. > [FitResults,LowestError,BestStart,xi,yi]=peakfit(smatrix,.4,.7,2,1,0,10) Example 9: Like Example 7, but turns AUTOZERO off in the last argument. Default is AUROZERO=1, removes linear baseline from data segment. >> peakfit([x' y'],.4,.7,2,1,0,10,[.3 .1 .5 .1],0); Example 10: Fits a group of three peaks near x=2400 in DataMatrix3 with three equal-width exponentially-broadened Gaussians. >> [FitResults,MeanFitError]=peakfit(DataMatrix3,2400,440,3,8,31,1) FitResults = 1 2300.5 0.82546 60.535 53.188 2 2400.4 0.48312 60.535 31.131 3 2500.6 0.84799 60.535 54.635 MeanFitError = 0.19975 DemoPeakFit.m is a demonstration script for peakfit.m. It generates an overlapping Gaussian peak signal, adds normally-distributed noise, fits it with the peakfit.m function (in line 78), repeats this several times (NumRepeats in line 20), then compares the peak parameters (position, height, width, and area) of the measurements to their actual values
- 73. and computes accuracy and precision (percent relative standard deviation). You can change any of the initial values in lines 13-30. Here is a typical result for a three-peak In the above results, you can see that the accuracy and precision of the peak position measurements are typically the best, followed by peak height and width and (the worst) peak area. b. Interactive keypress-operated version: ipf.m ipf.m is a version of the Interactive Peak Fitter that uses keyboard commands and the mouse cursor. It is written as a self-contained Matlab function, in a single m-file. The syntax is ipf(x,y), where x and y are the independent and dependent variables of your data set, or ipf(datamatrix) where "datamatrix" is a matrix that has x values in row 1 and y values in row 2. Demoipf.m is a demonstration script for ipf.m, with a built-in simulated signal generator. To download these m-files, right-click on these links, select Save Link As..., and click Save. You can also download a ZIP file containing ipf.m, Demoipf.m, and peakfit.m.
- 74. In this example, pan and zoom controls are used to isolate a segment of a chromatogram that contains three weak peaks near 5.8 minutes The lower plot shows the whole chromatogram and the upper plot shows the segment. Only the peaks in that segment are subjected to the fit. Pan and zoom are adjusted so that the signal returns to the local baseline at the ends of the segment; pressing T turns on the autozero mode, causing the program to compute and subtract a linear baseline between those points. Pressing 3, E, F performs a 3- peak exponentially-broadened Gaussian fit. Operating instructions for ipf.m (Version 7.1) 1. At the command line, type ipf(x, y), (x = independent variable, y = dependent variable) or ipf(datamatrix) where "datamatrix" is a matrix that has x values in row 1 and y values in row 2. Or if you have only one signal vector y, type ipf(y). 2. Use the four cursor arrow keys on the keyboard to pan and zoom the signal to isolate the peak or group of peaks that you want to fit in the upper window. (Use the < and > and ? and " keys for coarse pan and zoom and the square bracket keys [ and ] to nudge one point left and right). The curve fitting operation applies only to the segment of the signal shown in the top plot. The bottom plot shows the entire signal. Try not to get any undesired peaks in the upper window or the program may try to fit them. 3. Press the 1, 2, 3, 4, 5, or 6 keys to choose the number of model peaks, that is, the minimum number of peaks that you think might suffice to fit this segment of the signal. 4. Select the desired model peak shape by pressing the following keys: Gaussian (G), Equal-width Gaussians (H), Exponentially-broadened equal-width Gaussians (J); Lorentzian (L), Equal-width Lorentzians (;), Logistic (O), Pearson (P), exponentially- broadened Gaussian (E), and exponentially-broadened equal width Gaussians (J key) ; exponential pulse (U) and sigmoid (S) models. If the peak widths of each group of peaks is expected to be the same or nearly so, select the "equal-width" shapes. Equal- width fits (available only for the Gaussian, Lorentzian, and equal-width Gaussians shapes in version 5.6 and later) are faster, easier, and much more stable than regular variable-width fits, especially if the number of model peaks is greater than 3 (because there are fewer variable parameters for the program to adjust - only one width, rather than a separate width for each peak).
- 75. 5. A set of vertical dashed lines are shown on the plot, one for each model peak. Try to fine-tune the Pan and Zoom keys so that the signal goes to the baseline at both ends of the upper plot and so that the peaks (or bumps) in the signal roughly line up with the vertical dashed lines, as show in the figures on the left. This does not have to be exact. 6. Press F to initiate the curve-fitting calculation. Each time press F, another fit of the selected model to the data is performed with slightly different starting positions, so that you can judge the stability of the fit. Keep your eye on the residuals plot and on the "Error %" display. Do this several times, trying for the lowest error and the most unstructured random residuals plot. At any time, you can refine the signal region to be fit (step 2), the baseline position (step 9), change the number or peaks (step 3), or peak shape (step 4). 7. The model parameters of the last fit are displayed in the upper window. For example, for a 3-peak fit: Peak# Position Height Width Area 1 5.33329 14.8274 0.262253 4.13361 2 5.80253 26.825 0.326065 9.31117 3 6.27707 22.1461 0.249248 5.87425 The column are, left to right: the peak number, peak position, peak height, peak width, and the peak area. Press R to print this table out in the command window. Peaks are numbered from left to right. (The area of each component peak within the upper window is computed using the trapezoidal method and displayed after the width). Pressing Q prints out a report of settings and results in the command window, like so: Peak Shape = Gaussian Number of peaks = 3 Fitted range = 5 - 6.64 Percent Error = 7.4514 Peak# Position Height Width Area 1 5.33329 14.8274 0.262253 4.13361 2 5.80253 26.825 0.326065 9.31117 3 6.27707 22.1461 0.249248 5.87425 8. To select the Autozero mode, press the T key repeatedly; it cycles thorough Autozero OFF, Linear Autozero, and Quadratic Autozero. When autozero is linear, a straight- line baseline connecting the two ends of the signal segment in the upper panel will be automatically subtracted. When autozero is quadratic, a parabolic baseline connecting the two ends of the signal segment in the upper panel will be automatically subtracted. Use the quadratic autozero if the baseline is curved, as in these examples: Autozero OFF Linear autozero Quadratic autozero
- 76. 9. If you prefer to set the baseline manually, press the B key, then click on the baseline to the LEFT the peak(s), then click on the baseline to the RIGHT the peak(s). The new baseline will be subtracted and the fit re-calculated. (The new baseline remains in effect until you use the pan or zoom controls). Alternatively, you may use the multi- point background correction for the entire signal: press the Backspace key, type in the desired number of background points and press the Enter key, then click on the baseline starting at the left of the lowest x-value and ending to the right of the highest x-value. Press to restore the previous background to start over. 10. If you want to manually specify the first-guess peak positions, press C, then click on your estimates of the peak positions in the upper graph, once for each peak. A fit is automatically performed after the last click. Peaks are numbered in the order clicked. (The custom start positions remain in effect until you change the number of peaks or use the pan or zoom controls). 11.The A and Z keys is used only if you are using the Pearson or the exponentially- broadened Gaussian (ExpGaussian) shapes. It controls the extra shape factor of the Pearson (a value of 1.0 gives a Lorentzian shape, a value of 2.0 gives a shape roughly half-way between a Lorentzian and a Gaussian, and a larger values give a nearly Gaussian shape) and it also controls the time constant of the exponentially broadened Gaussian. You can either adjust this to minimize the Error % or set it to a previously- determined value. 12. For difficult fits, it may help to press X to perform ten silent trial fits with slightly different first guesses and take the one with the lowest fitting error. You can change the number of trials,"NumTrials", in or near line 80 (the default is 10). Remember: equal-width fits, available for the Gaussian (H key), Lorentzian (; key), and exponentially-broadened equal-width Gaussians (J key) shapes in ipf version 5.6 and later, are faster, easier, and much more stable than regular variable-width fits, so use equal-width fits whenever the peak widths are expected to be equal or nearly so. 13. Press Y to display the entire signal full screen without cursors, with the last fit displayed ini green. In version 5.6 and later, the residual is displayed in red, on the same y-axis scale as the entire signal. 14. Press M to switch the Y-axis of the full-signal plot in the bottom window between log and linear scales (added in version 7). This does not effect the fit, only the full- signal plot. 15. Press the D key to print out a table of model data in the command window (x, y1, y2, ..., where x is the column of x values of the fitted region and the y's are the y- values of each component, one for each peak in the model. You can then Copy and Paste this table into a spreadsheet or data plotting program of your choice. 16. Press W to print out the "peakfit" function in the command window, with the current values of all input arguments (including the last best-fit values of the first guess vector). You can copy and paste that function into you own code. Demoipf.m Demoipf.m is a demonstration script for ipf.m, with a built-in simulated signal generator. The true values of the simulated peak positions, heights, and widths are displayed in the Matlab command window, for comparison to the FitResults obtained by peak fitting. The default simulated signal contains six independent groups of peaks that you can use for
- 77. practice: a triplet near x = 150, a singlet at 400, a doublet near 600, a triplet near 850, and two broad single peaks at 1200 and 1700. Run this demo and see how close to the actual true peak parameters you can get. The useful thing about a simulation like this is that you can get a feel for the accuracy of peak parameter measurements, that is, the difference between the true and measured values of peak parameters. To download this m-file, right-click on the links, select Save Link As..., and click Save. To run it it, place both ipf.m and Demoipf in the Matlab path, then type Demoipf at the Matlab command prompt. An example of the use of this script is shown on the right. Here we focus in on the 3 fused peaks located near x=850. The true peak parameters (before the addition of the random noise) are: Position Height Width Area 800 3 30 95.808 850 2 40 85.163 900 1 50 53.227 When these peaks are isolated in the upper window and fit with three Gaussians, the results are Position Height Width Area 800.04 3.0628 29.315 95.583 850.15 1.9881 41.014 86.804 901.3 0.9699 46.861 48.376 So you can see that the accuracy of the measurements are excellent for peak position, good for peak height, and least good for peak width and area. It's no surprise that the least accurate measurements are for the smallest peak with the poorest signal-to-noise ratio. A ZIP file containing peakfit.m, DemoPeakFit.m, ipf.m, Demoipf.m, and some sample data for testing is available at from http://terpconnect.umd.edu/~toh/spectrum/ipf57.zip 6. The TFit Method for quantitative absorption spectroscopy This is a Matlab implementation of a computational method for quantitative analysis by multiwavelength absorption spectroscopy, called the transmission-fitting or “TFit” method, based on fitting a model of the instrumentally-broadened transmission spectrum to the observed transmission data, rather than direct calculation of absorbance as log10(Izero/I). The method is described in References 25, 26, and 27 on page 86.
- 78. Advantages of the TFit method compared to conventional methods are: (a) much wider dynamic range; (b) greatly improved calibration linearity; (c) ability to operate under conditions that are optimized for signal-to-noise ratio rather than for Beer's Law ideality (e.g. small spectrometers with lower dispersion and larger slit widths). Just like the multilinear regression (classical least squares) methods conventionally used in absorption spectroscopy, the Tfit method (a) requires an accurate reference spectrum of each analyte, (b) utilizes multi-wavelength data such as would be acquired on diode- array, Fourier transform, or automated scanning spectrometers, and (c) applies both to single-component and multi-component mixture analysis. The disadvantages of the Tfit method are: (a) it is computationally more intensive than the multilinear regression methods (but, on a typical personal computer, calculations take only a fraction of a second, even for the analysis of a mixture of three components); (b) it requires knowledge of the instrument function, i.e., the slit function or the resolution function of an optical spectrometer (however, this is a fixed characteristic of the instrument and is easily measured beforehand by scanning the spectrum of a narrow atomic line source such as a hollow cathode lamp); and (c) it is an iterative method that can under unfavorable circumstances converge on a local optimum (but this is handled by proper selection of the starting values, based on preliminary approximations calculated by conventional methods). Click here to download a self-contained demo m-file that works in recent versions of Matlab. (You can also download a ZIP file "TFit.zip" that contains both the newver version and the older one for Matlab 6.5) and you can also download it from the Matlab File Exchange. Background In absorption spectroscopy, the intensity I of light passing through an absorbing sample is given (in Matlab notation) by the Beer-Lambert Law: I = Izero.*10^-(alpha*L*c) where “Izero” is the intensity of the light incident on the sample, “alpha” is the absorption coefficient of the absorber, “L” is the distance that the light travels through the material (the path length), and “c” is the concentration of absorber in the sample. The variables I, Izero, and alpha are all functions of wavelength; L and c are scalar. In conventional applications, measured values of I and Izero are used to compute the quantity called "absorbance", defined as A = log10(Izero./I) Absorbance is defined in this way so that, when you combine this definition with the Beer-Lambert law, you get: A = alpha*L*c So, absorbance is proportional to concentration, ideally, which simplifies analytical calibration. However, any real spectrometer has a finite spectral resolution, meaning that the light beam passing through the sample is not truly monochromatic, with the result that an intensity reading at one wavelength setting is actually an average over a small spectral interval. More exactly, what is actually measured is aconvolution of the true spectrum of
- 79. the absorber and the instrument function. If the absorption coefficient "alpha" varies over that interval, then the calculated absorbance will no longer be linearly proportional to concentration (this is called the “polychromicity” error). The effect is most noticeable at high absorbances. In practice, many instruments will become non-linear starting at an absorbance of 2 (~1% Transmission). As the absorbance increases, the effect of unabsorbed stray light and instrument noise also becomes more significant. The theoretical best signal-to-noise ratio and absorbance precision for a photon-noise limited optical absorption instrument can be shown to be close to an absorbance of 1.0. However, if one attempts to arrange sample dilutions and absorption cell path lengths to obtain a working range centered on an absorbance of 1.0 , for example over the range .1 – 10, or 0.01 – 100, the measurments will obviously fail at the high end. (Clearly, the direct measurement of an absorbance of 100 is unthinkable, as it implies the measurement of light attenuation of 100 powers of ten - no real measurement system has a dynamic range even close to that). In practice, it is difficult to achieve an dynamic range even as high as 5 or 6 absorbance, so that much of the theoretically optimum absorbance range is actually unusable. (c.f.http://en.wikipedia.org/wiki/Absorbance). So, in conventional practice, greater sample diltuions and shorter path lengths are required to force the absorbance range to lower values, even if this means poorer signal-to-noise ratio and measurement precision at the low end. It is true that the non-linearity caused by polychromicity can be reduced by operating the instrument at the highest resolution setting (reducing the instrumental slit width). However, this has a serious undesired side effect: in dispersive instruments, reducing the slit width to increase the spectral resolution degrades the signal-to-noise substantially. The optimum signal-to-noise ratio is often achieved when the resolution of the instrument approximately matches the width of the analyte absorption, but operating the instrument in that way would result in very substantial non-linearity over most of the absorbance range becasue of the “polychromicity” error. The TFit method sidesteps the above problems by calculating the absorbance in a completely different way: it starts with the reference spectra (an accurate absorption spectrum for each analyte, also required by the multilinear regression methods), normalizes them to unit height, multiplies each by an adjustable coefficient, adds them up, computes the antilog, and convolutes it with the previously-measured slit function. The result, representing the instrumentally broadened transmission spectrum, is compared to the observed transmission spectrum. The coefficients (one for each unknown component in the mixture) are adjusted by the program until the computed transmission model is a least-squares best fit to the observed transmission spectrum. The best-fit coefficients are then equal to the absorbances determined under ideal optical conditions. Provision is also made to compensate for unabsorbed stray light and changes in background intensity (background absorption). These calculations are performed by the function fitM, which is used as a fitting function for Matlab's iterative non-linear fitting function fminsearch. The TFit method gives measurements of absorbance that are much closer to the "true" peak absorbance (in the absence of stray light and polychromatic light errors) and it allows linear and wide dynamic range measurements to be made, even if the slit width of the instrument is increased to optimize the signal-to-noise ratio.
- 80. Note: It's important to understand that the use of the TFit method does not guarantee a perfectly linear analytical curve at all absorbances, despite the impression given by these simulations. The TFit method simply removes the non-linearity caused by unabsorbed stray light and the polychromatic light effect. Other sources of non-linearity remain - in particular, chemical effects such as photolysis, equilibrium shifts, temperature effects, pH effects, binding, dimerization, polymerization, molecular phototropism, fluorescence, etc. The TFit method is based on the Beer-Lambert Law, but it calculates the absorbance in a different way that does not require the assumption that stray light and polychromatic radiation effects are zero. Because it allows larger slit widths to be used, it is possible to obtain greater signal-to-noise ratios, while still achieving a much wider linear dynamic range than previous methods. The fitM.m function function err = fitM(lam,yobsd,Spectra,InstFun,StrayLight) fitM is a fitting function for the Tfit method, for use with Matlab's fminsearch function. The input arguments of fitM are: lam = vector of adjustable parameters (in this case, peak absorbances) that are varied to obtained the best fit. yobsd = observed transmission spectrum of the mixture sample over the spectral range (column vector) Spectra = reference spectra for each component, over the same spectral range, one column/component, normalized to 1.00. InstFun = Zero-centered instrument function or slit function (column vector) StrayLight = fractional stray light (scalar or column vector, if it varies with wavelength) Note: yobsd, Spectra, and InstFun must have the same number of rows (wavelengths). Spectra must have one column for each absorbing component. Typical use: options = optimset('TolX',0.1); absorbance=fminsearch(@fitM,start,options,yobsd,ReferenceSpectra,... InstFunction,straylight); where start is the first guess (or guesses) of the absorbance(s) of the analyte(s); it's convenient to use the conventional log10(Izero/I) estimate of absorbance for start. The other arguments (described above) are passed on to FitM. In this example, fminsearch returns the value of absorbance that would have been measured in the absence of stray light and polychromatic light errors (which is either a single value or a vector of absorbances, if it is a multi-component analysis). The absorbance can then be converted into concentration by any of the usual calibration procedures (Beer's Law, external standards, standard addition, etc.) Iterative least-squares methods are ordinarily considered to be more difficult and less reliable than multilinear regression methods, and this can be true if there are more than one nonlinear variable that must be iterated, especially if those variables are correlated.
- 81. However, in the TFit method, there is only one iterated variable (absorbance) per measured component, and reasonable first guesses are easily available from the conventional single-wavelength absorbance calculation or multiwavelength regression methods. As a result, an iterative least-squares method works very well in this case. Comparing the expression for absorbance given above for the TFit method to that for the weighted regression method: absorbance=([weight weight].*[Background ReferenceSpectra])(- log10(yobsd).*weight) you can see that, in addition to the ReferenceSpectra and observed transmission spectrum (yobsd), the TFit method also requires a measurement of the Instrument function (spectral bandpass) and the stray light (which the linear regression methods assume to be zero), but these are characteristics of the spectrometer and need be done only once for a given spectrometer. Finally, although the TFit method does make the computer work harder, the computation time on a typical laboratory personal computer is only a fraction of a second (roughly 25 µsec per spectral data point per component analyzed), using Matlab as the computational environment. TFitDemo.m: Interactive explorer for the Tfit method, version 2.1 Interactive explorer for the Tfit method, applied to the measurement of a single component with a Lorentzian or Gaussian absorption peak, with keystroke controls for adjusting the parameters while observing the effects graphically and numerically. The adjustable parameters are: the true absorbance, "True A" (adjusted with the A/Z keys), the spectral width of the absorption peak, "AbsWidth", (adjusted with the S/X keys), the spectral width of the slit function, "SlitWidth" (adjusted with the D/C keys), the percent stray light, "Straylight" (adjusted with the F/V keys), and the noise level, "Noise" (adjusted with the G/B keys). (The wavelength scale, and the values of both AbsWidth and SlitWidth, are in arbitrary units). The simulation includes the effect of photon noise, unabsorbed stray light, and random background intensity shifts (light source flicker), and it compares observed absorbances computed by the single-wavelength, "Single", weighted multilinear regression,"WReg" (sometimes called Classical Least Squares in the chemometrics literature), and the TFit methods. To run this file, right-click TFitDemo.m click "Save link as...", save it in a folder in the Matlab path, then type "TFitDemo" at the Matlab command prompt. With the figure window topmost on the screen, press K to get a list of the keypress functions.
- 82. In the example shown on the above, the true peak absorbance is exactly 1.000, the absorption widths and slit function widths are equal, the unabsorbed stray light is 0.5%., and the photon noise is 5%. The results below the graphs show that the TFit method gives a much more accurate measurement (1.0056) than the single-wavelength method (0.6246) or weighted multilinear regression method (0.8883). Note: The original version of this demo, which uses sliders, works only on Matlab 6.5, but I recommend that you use TFitDemo.m instead, as it is more up-to-date and includes the statistics and calibration curve. KEYBOARD COMMANDS Peak shape....Q.....Toggles between Gaussian and Lorentzian absorption peak shape True peak A...A/Z...True absorbance of analyte at peak center, without instrumental broadening, stray light, or noise. AbsWidth......S/X...Width of absorption peak SlitWidth.....D/C...Width of instrument function (spectral bandpass) Straylight....F/V...Fractional unabsorbed stray light. Noise.........G/B...Random noise level Re-measure....Spacebar Re-measure signal Switch mode...W.....Switch between transmission and absorbance display Statistics....Tab...Prints table of statistics of 50 repeats Cal. Curve....M.....Displays analytical calibration curve in Figure 2 Keys..........K.....Print this list of keyboard commands Why does the noise on the graph change if I change the instrument function (slit width or InstWidth)? In the most common type of absorption spectrometer, the spectrometer's spectral bandwidth (InstWidth) is changed by changing the slit width, which also effects the light intensity and thus the signal-to-noise ratio. Therefore, in all these programs, when you change InstWidth, the photon noise is automatically changed accordingly just as it would in a real spectrophotometer. TFitStats.m: Statistics of methods compared Simple script that computes the statistics of the TFit method compared to single- wavelength (SingleW), simple regression (SimpleR), and weighted regression (WeightR) methods. Simulates photon noise, unabsorbed stray light and random background intensity shifts. Estimates the precision and accuracy of the four methods by repeating the calculations 50 times with different random noise samples. Computes the mean, relative percent standard deviation, and relative percent deviation from true absorbance. Parameters are easily changed in lines 19 - 26. Results are displayed in the MATLAB command window. Note: This statistics function is included as a keypress command (Tab key) in TFitDemo.m. In the typical results shown below (for a simulation with AbsWidth=SlitWidth=19; Straylight=0.5%, and Noise=5%) results for true absorbances of 0.001. 1.00, and 100 are compared, demonstrating that the accuracy and the precision of these methods over a 10,000-fold concentration range.
- 83. True A SingleW SimpleR WeightR TFit Mean result* .0010 .0003 .00057 .00070 .00097 % RSD** 435% 275% 40% 38% % Accuracy*** -70% -40% -30% 2.30% Mean result* 1.0000 0.599 0.656 0.841 1.001 % RSD** 0.69% 0.33% 0.27% 0.32% % Accuracy*** -40% -34% -16% 0.07% Mean result* 100 2.0038 3.7013 57.1530 99.9967 % RSD** 22.00% 23.00% 78.00% 6.80% % Accuracy*** -98.00% -96.00% -43.00% 0.33% * Average value of the 50 measured absorbances ** Percent relative standard deviation of the 50 measured absorbances *** Percent difference between the average of the 50 measured absorbances and the true absorbance TFitCalDemo.m: Comparison of analytical curves Function that compares the analytical curves for single-wavelength, simple regression, weighted regression, and the TFit method over any specified absorbance range (specified by the vector “absorbancelist” in line 20). Simulates photon noise, unabsorbed stray light and random background intensity shifts. Plots a log-log scatter plot with each repeat measurement plotted as a separate point (so you can see the scatter of points at low absorbances). The parameters can be changed in lines 43 - 51. In the sample result shown above, analytical curves for the four methods are computed over a 10,000-fold range, up to a peak absorbance of 100, demonstrating that the TFit method (shown by the green circles) is much more nearly linear over the whole range than the single-wavelength, simple regression, or weighted regression methods. Note: This calibration curve function is included as a keypress command (M key) in TFitDemo.m.
- 84. TFit3Demo.m: Application to a three-component mixture Interactive demonstration of the T-Fit method applied to the multicomponent absorption spectroscopy of a mixture of three absorbers, with keystrokes that allow you to adjust the parameters continuously while observing the effect dynamically. The adjustable parameters are: the absorbances of the three components (A1, A2, and A3, adjusted by the A/Z, S/X, and D/C keys respectively), spectral overlap between the component spectra ("Sepn", adjusted by the F/V keys), width of the instrument function ("InstWidth" adjusted by the G/B keys), and the noise level ("Noise", adjusted by the H/N keys). Compares quantitative measurement by weighted regression and TFit methods. Simulates photon noise, unabsorbed stray light and random background intensity shifts. Note: After executing this m-file, slide the "Figure No. 1" and "Figure No.2" windows side-by-side so that they don't overlap. Figure 1 shows a log-log scatter plot of the true vs. measured absorbances, with the three absorbers plotted in different colors and symbols. Figure 2 shows the transmission spectra of the three absorbers plotted in the corresponding colors. As you adjust the variable parameters in Figure No. 1, both graphs change accordingly. In the sample calculation shown above, component 2 (shown in blue) is almost completely buried by the stronger absorption bands on either side, giving a much weaker absorbance (0.1) that the other two components (3 and 5, respectively). Even in this case the TFit method gives a result within 1 to 2% of the correct value (A2=0.1). In fact, over most combinations of the three concentrations, the TFit methods works better, although of course nothing works if the spectral differences between the components is too small. (In this program, as in all of the above, when you change InstWidth, the photon noise is automatically changed accordingly just as it would in a real spectrophotometer). KEYBOARD COMMANDS A1 A/Z Increase/decrease true absorbance of component 1 A2 S/X Increase/decrease true absorbance of component 2 A3 D/C Increase/decrease true absorbance of component 3 Sepn F/V Increase/decrease spectral separation of the components InstWidth G/B Increase/decrease width of spectral bandpass Noise H/N Increase/decrease random noise level when InstWidth = 1 Peak shape Q Toggles between Gaussian and Lorentzian absorption peak shape Table Tab Print table of results Keys K Print this list of keyboard commands
- 85. Another run of the same simulation, showing the sample table of results obtained by pressing the Tab key: True Weighted TFit absorbance Regression method Component 1 3 2.06 3.001 Component 2 0.1 0.4316 0.09829 Component 3 5 2.464 4.998 References 1. Douglas A. Skoog, Principles of Instrumental Analysis, Third Edition, Saunders, Philadelphia, 1984. Pages 73-76. 2. Gary D. Christian and James E. O'Reilly, Instrumental Analysis, Second Edition, Allyn and Bacon, Boston, 1986. Pages 846-851. 3. Howard V. Malmstadt, Christie G. Enke, and Gary Horlick, Electronic Measurements for Scientists, W. A. Benjamin, Menlo Park, 1974. Pages 816-870. 4. Stephen C. Gates and Jordan Becker, Laboratory Automation using the IBM PC, Prentice Hall, Englewood Cliffs, NJ, 1989. 5. Muhammad A. Sharaf, Deborah L Illman, and Bruce R. Kowalski, Chemometrics, John Wiley and Sons, New York, 1986. 6. Peter D. Wentzell and Christopher D. Brown, Signal Processing in Analytical Chemistry, in Encyclopedia of Analytical Chemistry, R.A. Meyers (Ed.), p. 9764–9800, John Wiley & Sons Ltd, Chichester, 2000 (http://myweb.dal.ca/pdwentze/papers/c2.pdf) 7. Constantinos E. Efstathiou, Educational Applets in Analytical Chemistry, Signal Processing, and Chemometrics. (http://www.chem.uoa.gr/Applets/Applet_Index2.htm) 8. A. Felinger, Data Analysis and Signal Processing in Chromatography, Elsevier Science (19 May 1998). 9. Matthias Otto, Chemometrics: Statistics and Computer Application in Analytical Chemistry, Wiley-VCH (March 19, 1999). Some parts viewable in Google Books. 10. Steven W. Smith, The Scientist and Engineer's Guide to Digital Signal Processing. (Downloadable chapter by chapter in PDF format from http://www.dspguide.com/pdfbook.htm). This is a much more general treatment of the topic. 11. Robert de Levie, How to use Excel in Analytical Chemistry and in General Scientific Data Analysis, Cambridge University Press; 1 edition (February 15, 2001), ISBN-10: 0521644844. PDF excerpt . 12. Scott Van Bramer, Statistics for Analytical Chemistry, http://science.widener.edu/svb/stats/stats.html.
- 86. 13. Numerical Analysis for Chemical Engineers, Taechul Lee (http://www.cheric.org/ippage/e/ipdata/2001/13/lecture.html) 14. Educational Matlab GUIs, Center for Signal and Image Processing (CSIP), Georgia Institute of Technology. (http://users.ece.gatech.edu/mcclella/matlabGUIs/) 15. Digital Signal Processing Demonstrations in Matlab, Jan Allebach, Charles Bouman, and Michael Zoltowski, Purdue University (http://www.ecn.purdue.edu/VISE/ee438/demos/Demos.html) 16. Chao Yang , Zengyou He and Weichuan Yu, Comparison of public peak detection algorithms for MALDI mass spectrometry data analysis, http://www.biomedcentral.com/1471-2105/10/4 17. Michalis Vlachos, A practical Time-Series Tutorial with MATLAB, http://alumni.cs.ucr.edu/~mvlachos/PKDD05/PKDD05_Handout.pdf 18. Larry Larson, Evaluation of Calibration Curve Linearity, http://www.deq.state.va.us/waterguidance/pdf/96007.pdf. 19. Tobin A. Driscoll, A crash course in Matlab, http://www.math.umn.edu/~lerman/math5467/matlab_adv.pdf 20. P. E. S. Wormer, Matlab for Chemists, http://www.math.ru.nl/dictaten/Matlab/matlab_diktaat.pdf 21. Martin van Exter, Noise and Signal Processing, http://www.physics.leidenuniv.nl/sections/cm/ip/Onderwijs/SVR/bestanden/noise- final.pdf 22. Scott Sinex, Developer's Guide to Excelets, http://academic.pgcc.edu/~ssinex/excelets/ 23. R. de Levie, Advanced Excel for scientific data analysis, Oxford University Press, New York (2004) 25. T. C. O'Haver and J. Kindervater, J. of Analytical Atomic Spectroscopy 1, 89 (1986) 26. T. C. O'Haver and Jing-Chyi Chang, Spectrochim. Acta 44B, 795-809 (1989) 27. T. C. O'Haver, Anal. Chem. 68, 164-169 (1991).