![arXiv:1506.03706v1[astro-ph.IM]11Jun2015 Research Article Applied Optics 1
Smear correction of highly-variable,
frame-transfer-CCD images with application to
polarimetry
FRANCISCO A. IGLESIAS1,*, ALEX FELLER1, AND K. NAGARAJU1,2
1Max Planck Institute for Solar System Research, Justus-von-Liebig-Weg 3, 37077 Göttingen, Germany
2Indian Instiute of Astrophysics, Koramangala Second Block, Bengaluru-560034, India
*Corresponding author: iglesias@mps.mpg.de
Compiled June 12, 2015
Image smear, produced by the shutter-less operation of frame transfer CCD detectors, can be detrimental
for many imaging applications. Existing algorithms used to numerically remove smear, do not contem-
plate cases where intensity levels change considerably between consecutive frame exposures. In this
report we reformulate the smearing model to include specific variations of the sensor illumination. The
corresponding desmearing expression and its noise properties are also presented and demonstrated in the
context of fast imaging polarimetry. © 2015 Optical Society of America
OCIS codes: (040.1520) CCD, charge-coupled device; (100.3020) Image reconstruction-restoration; (110.5405) Polarimetric
imaging
http://dx.doi.org/10.1364/ao.XX.XXXXXX
1. INTRODUCTION
CCD sensors with frame transfer architecture are currently of
wide spread use in many imaging applications. A frame trans-
fer CCD collects photo-charges in a light sensitive area during
exposure time to then rapidly clock them towards a shielded,
frame storage area where readout is performed [1]. This kind
of detector readily become an interesting candidate when frame
rate, duty cycle and optical fill factor need to be simultaneously
maximized.
If the CCD is continuously illuminated during frame trans-
fer, the resulting image will be smeared reducing its contrast
and spatial resolution [1]. This artifact can be a draw back in
many applications [2–5] becoming particularly relevant when
high frame rate and duty cycle are required. Given its gen-
eral applicability, low cost and null effect on duty cycle among
others, numeric algorithms for post-facto correction of image
smear are an attractive alternative to the classic mechanical or
optoelectronic shutter systems.
The first desmearing algorithms were introduced by Powell
et al. [6], for standard, charge flush and reverse clocking CCD
operation modes, and Ruyten [7] only for charge flush mode. In
both of the above-mentioned works authors assumed, among
others, that pixel saturation has not taken place, noise in the im-
ages is negligible and the image illuminating the CCD does not
change within exposure and frame transfer time. The first of
these assumptions was investigated by Knox [8], who showed
that smearing can be used in specific cases to recover the val-
ues of saturated pixels in the image. More recently, different
smearing correction techniques have been developed that are
optimal for specific imaging applications [9, 10] or used for real
time correction [11].
In all the above-mentioned works, the original assumption
in [6, 8] of constant illumination is maintained. In this report we
firstly develop the smearing model to treat cases where sensor
illumination cannot be considered constant between consecu-
tive frames, provided the photon flux is constant between tran-
sitions, and the transitions occur in synchronization with the
frame transfers. Further we assume the flux transition profile
is symmetric with respect to its inflection point (Section 2). Sec-
ondly we study the corresponding desmearing algorithm and
its conditioning for the cases of non-periodic and periodic vari-
able scenes (Section 3).
The results of this work may be of utility to any application
where the scene is highly changing in synchronization with the
detector readout. An example are the fast polarimetric measure-
ments we use to demonstrate our results (Section 4).
2. FRAME TRANSFER MODEL WITH VARIABLE ILLUMI-
NATION
We will assume frame transfer is performed in columns direc-
tion and neither pixel saturation nor blooming has occurred.
Under these assumptions the analyses in the present and fol-](https://image.slidesharecdn.com/smearcorrectionofhighly-variable-170403224843/85/Smear-correction-of-highly-variable-1-320.jpg)
![Research Article Applied Optics 3
deriving in a causal difference equation also with matrix coeffi-
cients A and B.
In case the value of tt it is not known and can not be directly
measured, some techniques have been developed to allow its
experimental estimation for specific scenarios [7, 12].
When operating the CCD in charge flush mode, the charge
wells of the light sensitive area are flushed after the frame trans-
fer is finished. Eq. 5 can be modified to model this scenario by
doing r1 = 0. In reverse clocking mode the wells of the light sen-
sitive area are transfered to a charge drain, located at the top of
the sensor, by means of a reverse clocking. The latter is done
after the usual frame transfer. To consider this case in Eq. 5, A
has to be replaced by AT and r1 and r2 adjusted appropriately
if the clock rates, or charge transfer inefficiencies, of the usual
frame transfer and reverse sweep are different.
A. Noise properties of the smeared images
To investigate the noise properties of the smearing process, we
consider the unsmeared columns in Eq. 5 as independent ran-
dom vectors. We also include an independent, additive noise
term, Nk to the right hand side of Eq. 5 to account for extra
noise sources added during or after the charge wells are read,
e.g. readout noise. For the above-described scenario one can
show that,
Var[ ˆYk
] = AVar[Yk
]AT
+ BVar[Yk+1
]BT
+ Var[Nk
]
Cov[ ˆYk
, ˆYk+1
] = BVar[Yk+1
]AT
,
(6)
where Var and Cov represent the variance-covariance and cross-
covariance operations respectively. Note that Eq. 6 evidences
the spatial and temporal correlation introduced by the smear-
ing process.
We further assume in Eq. 6, that Var[Yk+1] = γVar[Yk] with
γ ∈ R, e.g. due to photon noise variation between frames with
very different illumination levels. Then, a simple upper limit
for the noise degradation can be written as follows,
| Var[ ˆYk
] |≤ η | Var[Yk
] | + | Var[Nk
] |, (7)
where | . | denotes the euclidean matrix norm and,
η = (1 + α + 2Mδ1/π)2
+ γ(α + 2Mδ2/π)2
for M ≫ 1. (8)
In the latter we used the fact that A = (1 + α − δ1)I + δ1∆ and
B = (α − δ2)I + δ2∆T, where I is the MxM identity matrix and
∆ an MxM upper triangular matrix with ∆i,j = 1 for j ≥ i and
∆i,j = 0 for j < i.
Fig. 2 shows curves of
√
η versus Mδ for different values
of α and γ (with r1 = r2 = 1). When interpreting Figure 2,
and similar ones in the next section, recall that by definition
for a fixed frame rate and exposure time, the sum 2α + Mδ is
constant.
3. IMAGE RESTORATION
A. Non-periodic illumination
In this section we discuss the solution of Eq. 5 to
an arbitrary time series of smeared columns, ˆY =
( ˆY0)T, ( ˆY1)T, . . . , ( ˆYK−1)T
T
, with K ≥ 2 ∈ N. Given
the shape of matrices A and B, the restored columns,
¯Y = ( ¯Y0)T, ( ¯Y1)T, . . . , ( ¯YK−1)T
T
, can be efficiently ob-
tained by solving iteratively from pixel no. M − 1 to 0 and
frame no. K − 1 to 0 if the final condition, ¯YK, is known.
0 5 10 15 20
Mδ %
1.0
1.1
1.2
1.3
1.4
√η
α=0.0%
α=10.0%
γ=10.0
γ=1.0
γ=0.1
γ=10.0
γ=1.0
γ=0.1
Fig. 2. Upper boundary for the noise degradation due to the
smearing process, versus the parameters product Mδ. The
curves in each set, continuous for α = 0.0% and dotted for
α = 10.0%, correspond to γ = 0.1, 1.0 and 10.0 from bottom to
top respectively. See Eq. 7-8 for extra details.
M=512 and Mδ ∈{0.5, 7.0, 13.5, 20.0}%
0 1 2 3 4 5 6
K−k
10
−5
10
−4
10
−3
10
−2
10
−1
10
0
|H
K−k
|
Mδ=0.5%
Mδ=20.0%
α=0.0%
α=10.0%
Fig. 3. Upper boundary for the propagation of errors in the fi-
nal condition of Eq. 5, versus index of the restored frame. The
curves in each set, continuous for α = 0.0% and dotted for
α = 10.0%, correspond to Mδ = 0.5, 7.0, 13.5 and 20.0% from
left to right respectively
. Note the vertical axis is in logarithmic scale. See Eq. 9 for extra
details.
The difficulties to obtain ¯YK can be avoided after considering
the general solution of Eq. 5,
¯Yk
=
K−k
∑
j=1
HK−k−j
A−1 ˆYK−j
+ HK−k ¯YK
, (9)
where Hk = (−A−1B)k, has a highly damped homogeneous
term for practical values of its parameters. Note that we use
upper scripts of time independent matrices to denote exponen-
tiation. Fig. 3 presents | HK−k | as function of K − k. The curves
were computed for M = 512, although they are practically in-
sensitive to M for M >> 1, and different values of α and δ (with
r1 = r2 = 1). As can be appreciated, the error introduced by a
wrong estimation of the final condition, e.g. ¯YK = ˆYK−1, can be
reduced to negligible levels provided few frames at the end of
the series are dropped.
To study the overall conditioning of the restoration algo-
rithm, with respect to uncertainties in the measured columns,](https://image.slidesharecdn.com/smearcorrectionofhighly-variable-170403224843/85/Smear-correction-of-highly-variable-3-320.jpg)
![Research Article Applied Optics 4
M=512, K=10 and α ∈{0, 4, 8, 12, 16, 20}%
0 5 10 15 20
Mδ %
1.00
1.02
1.04
1.06
1.08
1.10
1.12
1.14
Cond(U)
α=20%
α=0%
Fig. 4. Conditioning of the desmearing algorithm with respect
to errors in the measured images, versus the parameters prod-
uct Mδ. The curves correspond to α = 0, 4, 8, 12, 16 and 20%
from bottom to top respectively. See Eq. 10 for extra details.
let us write it using block matrices as follows (see Eq. 9),
¯Y = U ˆY + H ¯YK
, (10)
where H = (HK)T, (HK−1)T, . . . , (H1)T
T
,
U =
A−1 H1A−1 H2A−1 . . . HK−1A−1
0 A−1 H1A−1 . . . HK−2A−1
...
...
...
...
...
0 0 0 . . . A−1
,
and 0 is the M × M null matrix.
According to Eq. 10, when neglecting the contributions of
H ¯YK (see Fig. 3), the propagation of relative errors in the mea-
sured ˆY is upper bounded by the condition number of U, i.e.
the ratio of the largest to the smallest singular values of U [13].
The latter is plotted in Fig. 4 versus Mδ (with r1 = r2 = 1) for
M = 512, K = 10 and different values of α. The curves are
practically insensitive to M and K for M >> 1 and K >> 1. As
expected, the restoration becomes monotonically worse condi-
tioned for increasing δ and α.
B. Periodic illumination
Of particular interest for this work is the solution of Eq. 5 to a
time-periodic series of smeared columns, i.e. the illumination
pattern repeats cyclically after a fixed amount of frames. As-
suming periodicity in the scene allows the usage of image accu-
mulation for noise reduction purposes. This is specially impor-
tant for high-frame-rate applications that are photon starved.
In contrast to Section 3A, in the periodic illumination sce-
nario, image restoration can be performed for a single period of
the input, ˆY = ( ˆY0)T, ( ˆY1)T, . . . , ( ˆY
˜K−1)T
T
, where ˜K ≥ 2 ∈
N is the period. The corresponding desmearing expression can
be written in block matrix as,
¯Y = ˜U−1 ˆY, (11)
where,
˜U =
A B 0 . . . 0
0 A B . . . 0
...
...
...
...
...
B 0 0 . . . A
,
is a M ˜K × M ˜K, block circulant matrix [14]. Note Eq. 11 is linear
in ˆY and thus it can be applied after averaging any number of
periods, reducing this way the computational load.
On one hand, numeric inversion of ˜U will not be in general
a problem because it has to be done only once for each setup,
i.e. values of δ1, δ2 and α. There are, however, Fourier-based
methods used to invert block circulant matrices that can consid-
erably reduce computation time and storage while increasing
numeric stability [14, 15].
On the other hand, the product in the right side of Eq. 11
may become computationally expensive because it has to be
performed once per image column. Two possible approaches
to cope with this issue are: find only an approximate solution
iteratively by first guessing ˆY0 similar to Section 3A; or use the
fact that ˜U−1 is also block circulant and band dominant [15] to
implement an efficient multiplication algorithm.
4. APPLICATION TO FAST IMAGING POLARIMETRY
We applied the desmearing expression given in Section 3B to
correct a set of images taken by the Fast Solar Polarimeter (FSP)
[16]. We will not describe the instrument details here, only the
relevant aspects related to the image smearing process.
FSP uses a 264x264 pixels, split-frame-transfer pnCCD [17]
that can record up to 800 frames per second (fps). The pnCCD
operates in standard mode and its readout is synchronized with
a four-states beam modulator [18] that will introduce an inten-
sity variation according to the polarization state of the incom-
ing light. The intensity switching profiles and synchronization
between pnCCD and modulator meet the assumptions detailed
in section 2 for Eq. 5.
Any complete polarimetric measurement requires taking at
least four consecutive exposures, one during each modulator
state, that can have very different light levels for strongly polar-
ized sources. The four recorded frames are linearly combined
in a later step to retrieve the polarization signal. The sensor is
constantly illuminated during measurements. If the incoming
polarization state is constant, a periodic series of four images is
obtained.
The upper row of Fig. 5 presents an example measurement
taken at 700 fps of a USAF target illuminated by a light source
of constant linear polarization. Only a fraction of the total sen-
sor area is shown. Each of the four exposures, denoted with
labels State 1 to State 4, corresponds to an average of several
measurement periods. Smearing is obvious in all images, note
in particular its variable spatial distribution among different
states. The latter can be easily explained using Eq. 5 and con-
sidering that the mean light levels in the bright areas, i.e. above
two sigmas from the image mean, of each state are very differ-
ent. Namely 1950, 2828, 2825 and 297 arbitrary counts approxi-
mately for state 1 to state 4 respectively.
The lower row of Fig. 5 displays the restored images ob-
tained after applying Eq. 11 with ˜K = 4, M = 264, α = 3.9%,
δ1 = 0.05% and δ2 = 0.03% to the images in the upper row.
Reduction of the smearing is evident with only faint residuals.](https://image.slidesharecdn.com/smearcorrectionofhighly-variable-170403224843/85/Smear-correction-of-highly-variable-4-320.jpg)
Smear correction of highly variable,
- 1. arXiv:1506.03706v1[astro-ph.IM]11Jun2015 Research Article Applied Optics 1 Smear correction of highly-variable, frame-transfer-CCD images with application to polarimetry FRANCISCO A. IGLESIAS1,*, ALEX FELLER1, AND K. NAGARAJU1,2 1Max Planck Institute for Solar System Research, Justus-von-Liebig-Weg 3, 37077 Göttingen, Germany 2Indian Instiute of Astrophysics, Koramangala Second Block, Bengaluru-560034, India *Corresponding author: iglesias@mps.mpg.de Compiled June 12, 2015 Image smear, produced by the shutter-less operation of frame transfer CCD detectors, can be detrimental for many imaging applications. Existing algorithms used to numerically remove smear, do not contem- plate cases where intensity levels change considerably between consecutive frame exposures. In this report we reformulate the smearing model to include specific variations of the sensor illumination. The corresponding desmearing expression and its noise properties are also presented and demonstrated in the context of fast imaging polarimetry. © 2015 Optical Society of America OCIS codes: (040.1520) CCD, charge-coupled device; (100.3020) Image reconstruction-restoration; (110.5405) Polarimetric imaging http://dx.doi.org/10.1364/ao.XX.XXXXXX 1. INTRODUCTION CCD sensors with frame transfer architecture are currently of wide spread use in many imaging applications. A frame trans- fer CCD collects photo-charges in a light sensitive area during exposure time to then rapidly clock them towards a shielded, frame storage area where readout is performed [1]. This kind of detector readily become an interesting candidate when frame rate, duty cycle and optical fill factor need to be simultaneously maximized. If the CCD is continuously illuminated during frame trans- fer, the resulting image will be smeared reducing its contrast and spatial resolution [1]. This artifact can be a draw back in many applications [2–5] becoming particularly relevant when high frame rate and duty cycle are required. Given its gen- eral applicability, low cost and null effect on duty cycle among others, numeric algorithms for post-facto correction of image smear are an attractive alternative to the classic mechanical or optoelectronic shutter systems. The first desmearing algorithms were introduced by Powell et al. [6], for standard, charge flush and reverse clocking CCD operation modes, and Ruyten [7] only for charge flush mode. In both of the above-mentioned works authors assumed, among others, that pixel saturation has not taken place, noise in the im- ages is negligible and the image illuminating the CCD does not change within exposure and frame transfer time. The first of these assumptions was investigated by Knox [8], who showed that smearing can be used in specific cases to recover the val- ues of saturated pixels in the image. More recently, different smearing correction techniques have been developed that are optimal for specific imaging applications [9, 10] or used for real time correction [11]. In all the above-mentioned works, the original assumption in [6, 8] of constant illumination is maintained. In this report we firstly develop the smearing model to treat cases where sensor illumination cannot be considered constant between consecu- tive frames, provided the photon flux is constant between tran- sitions, and the transitions occur in synchronization with the frame transfers. Further we assume the flux transition profile is symmetric with respect to its inflection point (Section 2). Sec- ondly we study the corresponding desmearing algorithm and its conditioning for the cases of non-periodic and periodic vari- able scenes (Section 3). The results of this work may be of utility to any application where the scene is highly changing in synchronization with the detector readout. An example are the fast polarimetric measure- ments we use to demonstrate our results (Section 4). 2. FRAME TRANSFER MODEL WITH VARIABLE ILLUMI- NATION We will assume frame transfer is performed in columns direc- tion and neither pixel saturation nor blooming has occurred. Under these assumptions the analyses in the present and fol-
- 2. Research Article Applied Optics 2 lowing sections are restricted to a single sensor column without losing generality. Fig. 1 illustrates the operation in standard mode of a CCD with M rows during the acquisition of a single pixel in row no. m of frame no. k. The upper sketch represents the position of charge well no. m within the column. The lower profile ex- emplifies the values of the photo charges flux in pixel no. m as function of time, Sm(t). The considered frame extends from time instants tk to tk+1 with tk+1 − tk being the reciprocal of the CCD frame rate. The sensor is assumed to be permanently illuminated. The black boxes delimit the intervals where frame transfers are performed. Fig. 1. Process required to acquire pixel no. m of frame no. k, in a frame transfer CCD with M rows operating in standard mode. Upper sketch illustrates the position within the column of charge well no. m during frame acquisition. The lower pro- file exemplifies the model we adopted for the photo charges flux at pixel no. m, Sm(t). Namely constant during frame trans- fer (black boxes with duration Mtt) and exposure time (te), and variable during switching time (ts). The transition profile is assumed symmetric with respect to the point (black circle) with coordinates {tk+1 − ts/2, (Sk m + Sk+1 m )/2}. See the text for additional details. Acquisition of frame no. k starts during the transfer of frame no. k − 1. An empty charge well is shifted from the top of the ar- ray to position no. m during a fraction of the total frame transfer time given by (M − m + 1)tt, where tt is the period of the charge transfer clock. After this, static accumulation at position no. m takes place during the exposure time, te. The flux is assumed to be constant and equal to Sk m during frame transfer and expo- sure time. A transition of the flux to its constant value in the next frame, Sk+1 m , is then contemplated. The flux transition oc- curs during the switching time ts, with the charge well still at position no. m. The flux transition profile is assumed to be symmetric with respect to the point with Cartesian coordinates {tk+1 − ts/2, (Sk m + Sk+1 m )/2} in Fig. 1 (black circle). In the final step, once the flux is stable again, the charge well is transfered to the storage area of the sensor. For the above-described process and neglecting noise contri- butions (see Sections 2A and 3A), pixel saturation, charge trans- fer inefficiencies and instabilities of the charge transfer clock, the value of the smeared signal acquired in pixel no. m of frame no. k can be expressed as, ˆYk m = r1tt M−1 ∑ j=m+1 gjSk j + gmteSk m + 1 2 gmts(Sk m + Sk+1 m )+ dm(te + ts) + bm + r2tt m−1 ∑ j=0 gjSk+1 j + tt M−1 ∑ j=0 j=m dj, (1) where the time constants gj, dj and bj denote the gain, dark cur- rent flux and bias of pixel no. j respectively. The two ad-hoc coefficients r1 and r2, have been introduced to facilitate experi- mental tuning of the model, e.g. to: contemplate possible differ- ences in the photo-charges-generation efficiencies of the static and transferring clocking states; compensate the effects of syn- chronization errors between the CCD readout and flux switch- ing; model other CCD operation modes; etc. The simple form of the third term in the right hand side of Eq. 1, which accounts for the signal accumulated during ts, is a direct consequence of the symmetry restriction imposed on the flux transition profile. The later has been defined by the applica- tion that motivated this work. In spite of that, notice it also cov- ers other cases of possible practical interest like very fast switch- ing (step-like profile), linear switching (ramp-like profile), con- stant illumination, etc. In the following we will assume the acquired images have been offset corrected using the dark frame, ˆDm = dm(te + ts) + bm + tt M−1 ∑ j=0 j=m dj, (2) which in practice can be obtained by averaging a long sequence of dark images, i.e. with Sm(t) = 0. In addition, let us define the unsmeared signal acquired dur- ing the static and constant illumination exposure as, Yk m = gmteSk m (3) and constants, α = ts 2te , δ1 = r1tt te and δ2 = r2tt te . (4) Note that Eq. 3 implies any gain table corrections should be applied after the desmearing process (see Section 3). Using Eq. 1-4 and redefining ˆYk m − ˆDm as ˆYk m, we can write a smeared column of the sensor in matrix form as, ˆYk = AYk + BYk+1 , (5) where, ˆYk = ˆYk 0 , ˆYk 1 , . . . , ˆYk M−1 T , Yk = Yk 0 , Yk 1 , . . . , Yk M−1 T , A = 1 + α δ1 . . . δ1 0 1 + α . . . δ1 ... ... ... ... 0 0 0 1 + α , B = α 0 . . . 0 δ2 α . . . 0 ... ... ... ... δ2 δ2 δ2 α , and the superscript T denotes the transpose. Note that, an analogue analysis can be carried out for the case where frame transfer takes place before the flux switching,
- 3. Research Article Applied Optics 3 deriving in a causal difference equation also with matrix coeffi- cients A and B. In case the value of tt it is not known and can not be directly measured, some techniques have been developed to allow its experimental estimation for specific scenarios [7, 12]. When operating the CCD in charge flush mode, the charge wells of the light sensitive area are flushed after the frame trans- fer is finished. Eq. 5 can be modified to model this scenario by doing r1 = 0. In reverse clocking mode the wells of the light sen- sitive area are transfered to a charge drain, located at the top of the sensor, by means of a reverse clocking. The latter is done after the usual frame transfer. To consider this case in Eq. 5, A has to be replaced by AT and r1 and r2 adjusted appropriately if the clock rates, or charge transfer inefficiencies, of the usual frame transfer and reverse sweep are different. A. Noise properties of the smeared images To investigate the noise properties of the smearing process, we consider the unsmeared columns in Eq. 5 as independent ran- dom vectors. We also include an independent, additive noise term, Nk to the right hand side of Eq. 5 to account for extra noise sources added during or after the charge wells are read, e.g. readout noise. For the above-described scenario one can show that, Var[ ˆYk ] = AVar[Yk ]AT + BVar[Yk+1 ]BT + Var[Nk ] Cov[ ˆYk , ˆYk+1 ] = BVar[Yk+1 ]AT , (6) where Var and Cov represent the variance-covariance and cross- covariance operations respectively. Note that Eq. 6 evidences the spatial and temporal correlation introduced by the smear- ing process. We further assume in Eq. 6, that Var[Yk+1] = γVar[Yk] with γ ∈ R, e.g. due to photon noise variation between frames with very different illumination levels. Then, a simple upper limit for the noise degradation can be written as follows, | Var[ ˆYk ] |≤ η | Var[Yk ] | + | Var[Nk ] |, (7) where | . | denotes the euclidean matrix norm and, η = (1 + α + 2Mδ1/π)2 + γ(α + 2Mδ2/π)2 for M ≫ 1. (8) In the latter we used the fact that A = (1 + α − δ1)I + δ1∆ and B = (α − δ2)I + δ2∆T, where I is the MxM identity matrix and ∆ an MxM upper triangular matrix with ∆i,j = 1 for j ≥ i and ∆i,j = 0 for j < i. Fig. 2 shows curves of √ η versus Mδ for different values of α and γ (with r1 = r2 = 1). When interpreting Figure 2, and similar ones in the next section, recall that by definition for a fixed frame rate and exposure time, the sum 2α + Mδ is constant. 3. IMAGE RESTORATION A. Non-periodic illumination In this section we discuss the solution of Eq. 5 to an arbitrary time series of smeared columns, ˆY = ( ˆY0)T, ( ˆY1)T, . . . , ( ˆYK−1)T T , with K ≥ 2 ∈ N. Given the shape of matrices A and B, the restored columns, ¯Y = ( ¯Y0)T, ( ¯Y1)T, . . . , ( ¯YK−1)T T , can be efficiently ob- tained by solving iteratively from pixel no. M − 1 to 0 and frame no. K − 1 to 0 if the final condition, ¯YK, is known. 0 5 10 15 20 Mδ % 1.0 1.1 1.2 1.3 1.4 √η α=0.0% α=10.0% γ=10.0 γ=1.0 γ=0.1 γ=10.0 γ=1.0 γ=0.1 Fig. 2. Upper boundary for the noise degradation due to the smearing process, versus the parameters product Mδ. The curves in each set, continuous for α = 0.0% and dotted for α = 10.0%, correspond to γ = 0.1, 1.0 and 10.0 from bottom to top respectively. See Eq. 7-8 for extra details. M=512 and Mδ ∈{0.5, 7.0, 13.5, 20.0}% 0 1 2 3 4 5 6 K−k 10 −5 10 −4 10 −3 10 −2 10 −1 10 0 |H K−k | Mδ=0.5% Mδ=20.0% α=0.0% α=10.0% Fig. 3. Upper boundary for the propagation of errors in the fi- nal condition of Eq. 5, versus index of the restored frame. The curves in each set, continuous for α = 0.0% and dotted for α = 10.0%, correspond to Mδ = 0.5, 7.0, 13.5 and 20.0% from left to right respectively . Note the vertical axis is in logarithmic scale. See Eq. 9 for extra details. The difficulties to obtain ¯YK can be avoided after considering the general solution of Eq. 5, ¯Yk = K−k ∑ j=1 HK−k−j A−1 ˆYK−j + HK−k ¯YK , (9) where Hk = (−A−1B)k, has a highly damped homogeneous term for practical values of its parameters. Note that we use upper scripts of time independent matrices to denote exponen- tiation. Fig. 3 presents | HK−k | as function of K − k. The curves were computed for M = 512, although they are practically in- sensitive to M for M >> 1, and different values of α and δ (with r1 = r2 = 1). As can be appreciated, the error introduced by a wrong estimation of the final condition, e.g. ¯YK = ˆYK−1, can be reduced to negligible levels provided few frames at the end of the series are dropped. To study the overall conditioning of the restoration algo- rithm, with respect to uncertainties in the measured columns,
- 4. Research Article Applied Optics 4 M=512, K=10 and α ∈{0, 4, 8, 12, 16, 20}% 0 5 10 15 20 Mδ % 1.00 1.02 1.04 1.06 1.08 1.10 1.12 1.14 Cond(U) α=20% α=0% Fig. 4. Conditioning of the desmearing algorithm with respect to errors in the measured images, versus the parameters prod- uct Mδ. The curves correspond to α = 0, 4, 8, 12, 16 and 20% from bottom to top respectively. See Eq. 10 for extra details. let us write it using block matrices as follows (see Eq. 9), ¯Y = U ˆY + H ¯YK , (10) where H = (HK)T, (HK−1)T, . . . , (H1)T T , U = A−1 H1A−1 H2A−1 . . . HK−1A−1 0 A−1 H1A−1 . . . HK−2A−1 ... ... ... ... ... 0 0 0 . . . A−1 , and 0 is the M × M null matrix. According to Eq. 10, when neglecting the contributions of H ¯YK (see Fig. 3), the propagation of relative errors in the mea- sured ˆY is upper bounded by the condition number of U, i.e. the ratio of the largest to the smallest singular values of U [13]. The latter is plotted in Fig. 4 versus Mδ (with r1 = r2 = 1) for M = 512, K = 10 and different values of α. The curves are practically insensitive to M and K for M >> 1 and K >> 1. As expected, the restoration becomes monotonically worse condi- tioned for increasing δ and α. B. Periodic illumination Of particular interest for this work is the solution of Eq. 5 to a time-periodic series of smeared columns, i.e. the illumination pattern repeats cyclically after a fixed amount of frames. As- suming periodicity in the scene allows the usage of image accu- mulation for noise reduction purposes. This is specially impor- tant for high-frame-rate applications that are photon starved. In contrast to Section 3A, in the periodic illumination sce- nario, image restoration can be performed for a single period of the input, ˆY = ( ˆY0)T, ( ˆY1)T, . . . , ( ˆY ˜K−1)T T , where ˜K ≥ 2 ∈ N is the period. The corresponding desmearing expression can be written in block matrix as, ¯Y = ˜U−1 ˆY, (11) where, ˜U = A B 0 . . . 0 0 A B . . . 0 ... ... ... ... ... B 0 0 . . . A , is a M ˜K × M ˜K, block circulant matrix [14]. Note Eq. 11 is linear in ˆY and thus it can be applied after averaging any number of periods, reducing this way the computational load. On one hand, numeric inversion of ˜U will not be in general a problem because it has to be done only once for each setup, i.e. values of δ1, δ2 and α. There are, however, Fourier-based methods used to invert block circulant matrices that can consid- erably reduce computation time and storage while increasing numeric stability [14, 15]. On the other hand, the product in the right side of Eq. 11 may become computationally expensive because it has to be performed once per image column. Two possible approaches to cope with this issue are: find only an approximate solution iteratively by first guessing ˆY0 similar to Section 3A; or use the fact that ˜U−1 is also block circulant and band dominant [15] to implement an efficient multiplication algorithm. 4. APPLICATION TO FAST IMAGING POLARIMETRY We applied the desmearing expression given in Section 3B to correct a set of images taken by the Fast Solar Polarimeter (FSP) [16]. We will not describe the instrument details here, only the relevant aspects related to the image smearing process. FSP uses a 264x264 pixels, split-frame-transfer pnCCD [17] that can record up to 800 frames per second (fps). The pnCCD operates in standard mode and its readout is synchronized with a four-states beam modulator [18] that will introduce an inten- sity variation according to the polarization state of the incom- ing light. The intensity switching profiles and synchronization between pnCCD and modulator meet the assumptions detailed in section 2 for Eq. 5. Any complete polarimetric measurement requires taking at least four consecutive exposures, one during each modulator state, that can have very different light levels for strongly polar- ized sources. The four recorded frames are linearly combined in a later step to retrieve the polarization signal. The sensor is constantly illuminated during measurements. If the incoming polarization state is constant, a periodic series of four images is obtained. The upper row of Fig. 5 presents an example measurement taken at 700 fps of a USAF target illuminated by a light source of constant linear polarization. Only a fraction of the total sen- sor area is shown. Each of the four exposures, denoted with labels State 1 to State 4, corresponds to an average of several measurement periods. Smearing is obvious in all images, note in particular its variable spatial distribution among different states. The latter can be easily explained using Eq. 5 and con- sidering that the mean light levels in the bright areas, i.e. above two sigmas from the image mean, of each state are very differ- ent. Namely 1950, 2828, 2825 and 297 arbitrary counts approxi- mately for state 1 to state 4 respectively. The lower row of Fig. 5 displays the restored images ob- tained after applying Eq. 11 with ˜K = 4, M = 264, α = 3.9%, δ1 = 0.05% and δ2 = 0.03% to the images in the upper row. Reduction of the smearing is evident with only faint residuals.
- 5. Research Article Applied Optics 5 State 1 State 2 State 3 State 4 Fig. 5. Smeared (upper row) and corrected (lower row) im- ages, displayed using logarithmic scales to enhance the smear- ing. The gray scales of the images in each column are the same. The mean values of the bright areas in the upper-row images are 1950, 2828, 2825 and 297 arbitrary counts for states 1 to 4 respectively. The dashed and continuous horizontal lines in states 3 and 4 images, delimit the cuts shown in Fig. 6 . (a) State 3 horizontal cut 20 40 60 80 100 Pixels 0 1 2 3 4 5 6 Intensity% Smear. Corr. Diff. (b) State 4 horizontal stripe average 20 30 40 50 60 Pixels 0 2 4 6 8 10 12 14 Intensity% Smear. Corr. Diff. Fig. 6. Horizontal cut of the state 3 images (a) and vertical av- erage of a 4-pixel-wide horizontal stripe of state 4 images (b). The position of the cut and the center of stripe are indicated by the dashed and continuous, white lines in Fig. 5. In both plots the values for the smeared (dashed) and corrected (continu- ous) images, as well as their difference (dotted) are shown. All profiles are expressed in percentage of the mean value in the bright areas of the corresponding smeared image. The quality of the restoration and the differences in the arti- fact levels between state 3 and state 4 images, are exemplified by the profiles shown in Fig. 6. Firstly, from the corrected pro- files, note there is not clear residual smearing above the noise levels. Given the USAF target pattern, and in complete absence of smearing and stray light, the profiles of the selected cuts are expected to be flat except for noise induced variations. Secondly, from the smeared profiles in Fig. 6, note that the maximum artifact levels, relative to the mean in the bright ar- eas of each image, are larger in State 4 (∼ 12%) than in State 3 (∼ 5%). To understand this, recall that each frame accu- mulates smearing signal during both the pre-exposure and the post-exposure frame transfers (see black boxes in Fig. 1). Fur- ther, note that the smearing signal acquired during the post- exposure transfer depends on the illumination level of the fol- lowing frame. In this way, smearing level in State 3 is low because the consecutive frame (state 4) has approximately 9.5 times fainter illumination. On the contrary, the artifact level in State 4 is high because the consecutive frame (State 1, recall the input is periodic) has approximately 6.6 times larger intensity. 5. CONCLUSIONS We developed a model for the smearing, introduced in frame transfer CCDs, that accounts for variations in the sensor illumi- nation provided the changes take place in synchronization with the detector readout and have a transition profile that is an odd function. The derived model depends only on three physical parameters, namely the period of the charge transfer clock, the image exposure time and the photon flux switching time. In addition, we showed that smearing introduces not only a spatial but also a temporal correlation in the images and de- rived a simple upper boundary for the noise degradation. We also studied the corresponding desmearing algorithm and its conditioning for the cases of non-periodic and periodic variable scenes. The latter was successfully applied to restore a set of polarimetric measurements taken by the Fast Solar Po- larimeter. The present work does not take in to account the effects of pixel saturation, pixel blooming, and charge transfer inefficien- cies. F. A. Iglesias would like to acknowledge the International Max Planck Research School for Solar System Science for sup- porting his participation in this project. REFERENCES 1. G. C. Hoslst, CCD Arrays Cameras and Displays (SPIE Optical Engi- neering Press, 1998), chap. 3.3.3. 2. A. A. Dorrington, M. J. Cree, and D. A. Carnegie, “The importance of CCD readout smear in heterodyne imaging phase detection applica- tions,” in “Proceedings of Image and Vision Computing New Zealand (IVCNZ),” (2005). 3. E. Choi, J. Choi, and M. G. Kang, “Super-resolution approach to over- come physical limitations of imaging sensors: An overview,” Int. J. Imaging Syst. Technol 14, 36 – 46 (2004). 4. M. Legrand, J. Nogueira, A. Vargas, R. Ventas, and M. del Carmen Rodriguez-Hídalgo, “CCD image sensor induced error in PIV applica- tions,” Meas. Sci. Technol. 25, 13 (2012). 5. H. Zhao, C. Li, Y. Zhang, and Z. Xu, “Improvement in the synchro- nization between the radio frequency signal and the image detector in an acousto-optic tunable filter imaging spectrometer,” Applied Optics (2014).
- 6. Research Article Applied Optics 6 6. K. Powell, D. Chana, D. Fish, and C. Thompson, “Restoration and fre- quency analysis of smeared CCD images,” Applied Optics 38, 1343– 1347 (1999). 7. W. Ruyten, “Smear correction for frame transfer charged-coupled- device cameras,” Optics Letters 24, 878–880 (1999). 8. K. Knox, “Recovering saturated pixels blurred by CCD image smear,” in “Proceedings of the Advanced Maui Optical and Space Surveillance Technologies Conference,” , S. Ryan, ed. (2007), p. E59. 9. J. Sun and J. Zhou, “A novel method for smearing intensity estimation and elimination,” in “Intelligent Science and Intelligent Data Engineer- ing,” , vol. 7202 of Lecture Notes in Computer Science, Y. Zhang, Z.-H. Zhou, C. Zhang, and Y. Li, eds. (Springer Berlin Heidelberg, 2012), pp. 466–472. 10. J. Gao, Z. Zhang, R. Yao, J. Sun, and Y. Zhang, “A robust smear removal method for inter-frame charge-coupled device star images,” in “2011 Seventh International Conference on Natural Computation (ICNC),” (2011). 11. Y. S. Han, E. Choi, and M. G. Kang, “Smear removal algorithm using the optical black region for CCD imaging sensors,” IEEE Transactions on Consumer Electronics (2009). 12. W. Feng and L. Chen, “Smear correction of frame transfer CCD cam- era based on the integrating sphere,” Optik 124, 1805– 1807 (2012). 13. W. H. Press, S. A. Teukolsky, W. T. Vetterling, and B. P. Flannery, Nu- merical Recipes 3rd Edition: The Art of Scientific Computing (Cam- bridge University Press, 2007), chap. 2.6, p. 69. 14. T. de Mazancourt and D. Gerlic, “The inverse of block-circulant ma- trix,” IEEE Transactions on Antennas and Propagation AP-31, 808– 810 (1983). 15. R. Vescovo, “Inversion of block-circulant matrices and circular array ap- proach,” IEEE Transactions on Antennas and Propagation 45, 1565– 1567 (1997). 16. A. Feller, F. A. Iglesias, K. Nagaraju, S. K. Solanki, and S. Ihle, “Fast Solar Polarimeter: Description and first results,” in “ASP Conference Series,” , vol. 498 (2014), vol. 498, p. 271. 17. R. Hartman, W. Buttler, H. Gorke, S. Herrmann, P. Holl, N. Meidinger, H. Soltau, and L. Strüder, “A high speed pnCCD detector system for optical applications,” Nuclear Instruments & Methods in Physics Re- search 568, 118–123 (2006). 18. C. U. Keller, J. W. Harvey, and the SOLIS Team, “The SOLIS Vector- Spectromagnetograph,” in “ASP Conference Series,” , vol. 307 (2003), vol. 307, p. 13.