![Empirical Best Prediction and Best Linear Unbiased Prediction 21
3.3 Empirical Best Prediction and Best Linear Unbiased
Prediction of the Population Total
For the homogeneous population model, there is no auxiliary information. That
is, we cannot identify a variable Z whose population values vary and are all
known and is such that the conditional distribution of Y given Z (and in par-
ticular E(Y |Z)) varies with Z. From (2.2) we know that the minimum mean
squared error predictor of ty is t∗
y = tys +E [tyr |yi, i ∈ s]. For model (3.1), this is
given by t∗
y = tys + (N − n)μ. Of course, we do not know μ and so must replace
this parameter by an estimate in order to define the EB predictor. Intuitively,
in an exchangeable population all sample values provide the same information
about μ, and so it seems sensible to use the sample mean ȳs of Y as our estimator
of μ. This leads to the predictor:
tE
y = tys + (N − n)μ̂ = tys + (N − n)ȳs =
N
n
tys . (3.2)
The predictor defined by (3.2) above is commonly called the expansion estimator.
Note that the EB predictor is not necessarily unique, as there may be several
possible estimators of unknown parameters such as μ. However, it provides a
simple way to construct predictors, and will be statistically efficient if a sensible
method of parameter estimation is used. An alternative, more complex approach,
called Best Linear Unbiased Prediction, can also be used. This method does yield
a unique best predictor, called the Best Linear Unbiased Predictor, or BLUP. In
many cases, including model (3.1) and estimator (3.2), the BLUP is also an EB
predictor.
To define the BLUP, we will first define linear predictors to be those that
can be written as a linear combination of the values of Y associated with sample
units. Linear predictors are used extensively in survey sampling, mainly because
of their simplicity of use. The BLUP t̂BLUP
y of ty under a specified model satisfies
three conditions:
• It is a linear predictor; that is it can be written in the form t̂BLUP
y =
s wiyi,
where the wi are weights that have to be determined. Note that there is no
restriction on these weights, except that they must not depend on any values
of Y . In particular, they can, and often do, depend on the population units
that make up the sample s, and the auxiliary variable Z in cases where the
model includes Z.
• It is unbiased for ty, that is its sample error has an expectation of zero,
E t̂BLUP
y − ty = 0.
• For any sample s its sample error has minimum variance among the sam-
ple errors of all unbiased linear predictors of ty, that is Var t̂BLUP
y − ty ≤
Var(t̂y − ty) where t̂y is any other unbiased linear predictor of ty.
It turns out that to derive the BLUP, we only need to assume that different
observations are uncorrelated, rather than the stronger assumption of indepen-
dence in (3.1c). We first note that for any linear predictor of ty we have the](https://image.slidesharecdn.com/2648523-250526162053-c6d86f76/85/An-Introduction-To-Modelbased-Survey-Sampling-With-Applications-1st-Edition-Ray-Chambers-40-320.jpg)
![26 Homogeneous Populations
Substituting this estimate in the expression for n in (3.8) above, and discarding
lower order terms leads to
n = (B2
/A2
)m.
That is, the ratio of the estimated relative standard errors from both surveys is
the inverse of the ratio of the square roots of the respective sample sizes for the
surveys.
3.7 Selecting a Simple Random Sample
How do we go about selecting a simple random sample? The simplest way, given
the population and sample sizes involved are small, is to use a table of random
numbers. For example, Fisher and Yates (1963) provide a list of two digit ran-
dom numbers, together with instructions on how to use them to select a simple
random sample.
When sample sizes and populations are large, it is usually most convenient to
use a computer to select the sample. Most computer packages include a pseudo-
random number generator, which can be used in this regard. A simple way of
selecting a SRS of size n, using a computer-based random number generator, is
to randomly order the population units on the sampling frame, then take the
first n of these randomly ordered units to be the sample. This random ordering
is easily accomplished by independently assigning a pseudo-uniform [0,1] random
variate to each of the N units on the sampling frame. We then re-order these N
units according to these random values.
The above so-called ‘shuffle algorithm’ has the disadvantage that it requires
a pass through the population (to allocate the random numbers), then another
pass to re-order the population. This can be expensive (in computer time) in very
large populations. Another procedure, therefore, is to generate random numbers
between 1 and N until n distinct numbers are generated. These then define the
labels of the selected sample units. Vitter (1984) contains a discussion of several
efficient algorithms for computer-based selection of a simple random sample by
one sequential pass through a computer list of the population labels.
Implicit in the preceding discussion about methods of selecting a simple ran-
dom sample of size n is that the same population unit cannot occur more than
once in any particular selected sample. In other words any sample we select
must contain n distinct population units and all samples of the same size must
be equally likely. This method of sampling is typically referred to as Simple
Random Sampling Without Replacement (SRSWOR).
3.8 A Generalisation of the Homogeneous Model
Model (3.1) implies that the values of Y are uncorrelated for distinct units. Sup-
pose that we generalise the model, by allowing a uniform correlation ρ between
every pair of values:](https://image.slidesharecdn.com/2648523-250526162053-c6d86f76/85/An-Introduction-To-Modelbased-Survey-Sampling-With-Applications-1st-Edition-Ray-Chambers-45-320.jpg)
![A Year of Sadness
and Gloom
is, to say the least, exceedingly improbable, that the son of an
immigrant Alsatian shoemaker should have obtained entrée upon the
supposed terms of intimacy in a household in which the oldest child
was some six years younger than himself, and which belonged to the
highest social, if not titled rank, until he by the force of his talents,
culture, and high character, had risen to its level. That, after so
rising, the obscurity of his birth was forgotten and the only daughter
became his wife, is alike honorable to both parties. It is unnecessary
to pursue the point farther; the reader, having his attention drawn to
it, will observe for himself the many less prominent, but strongly
corroborating circumstances of the narrative, which confirm the
chronology adopted in it. At all events it must stand until new and
decisive facts against it be found.[38]
“My journey cost me a great deal, and I have not
the smallest hope of earning anything here. Fate is
not propitious to me in Bonn.” In poverty, ill,
melancholy, despondent, motherless, ashamed of and depressed by
his father’s ever increasing moral infirmity, the boy, prematurely old
from the circumstances in which he had been placed since his
eleventh year, had yet to bear another “sling and arrow of
outrageous fortune.” The little sister, now a year and a half old—but
here is the notice from the “Intelligenzblatt”:—“Died, November 25,
Margareth, daughter of the Court Musician Johann van Beethoven,
aged one year.” And so faded the last hope that the passionate
tenderness of Beethoven’s nature might find scope in the purest of
all relations between the sexes—that of brother and sister.
Thus, in sadness and gloom, Beethoven’s seventeenth year ended.](https://image.slidesharecdn.com/2648523-250526162053-c6d86f76/85/An-Introduction-To-Modelbased-Survey-Sampling-With-Applications-1st-Edition-Ray-Chambers-59-320.jpg)
![by what we learn of her influence upon the youthful Beethoven.
Court Councillor von Breuning perished in a fire in the electoral
palace on January 15, 1777. The young widow (she had barely
attained her 28th year), continued to live in the house of her
brother, Abraham von Kerich, with her three children, to whom was
added a fourth in the summer of 1777. Immediately after the death
of the father, his brother, the canon Lorenz von Breuning, changed
his residence from Neuss to Bonn and remained in the same house
as guardian and tutor of the orphaned children. These were:
1. Christoph, born May 13, 1771, a student of jurisprudence at Bonn,
Göttingen and Jena, municipal councillor in Bonn, notary, president
of the city council, professor at the law school in Coblenz, member
of the Court of Review in Cologne, and, finally, Geheimer Ober-
Revisionsrath in Berlin. He died in 1841.
2. Eleonore Brigitte, born April 23, 1772. On March 28, 1802, she
was married to Franz Gerhard Wegeler of Beul-an-der-Ahr, and died
on June 13, 1841, at Coblenz.
3. Stephan, born August 17, 1774. He studied law at Bonn and
Göttingen, and shortly before the end of the electorship of Max
Franz was appointed to an office in the Teutonic Order at
Mergentheim. In the spring of 1801 he went to Vienna, where he
renewed his acquaintance with Beethoven. They had simultaneously
been pupils of Ries in violin playing. The Teutonic Order offering no
chance of advancement to a young man, he was given employment
with the War Council and became Court Councillor in 1818. He died
on June 4, 1827. His first wife was Julie von Vering, daughter of
Ritter von Vering, a military physician; she died in the eleventh
month of her wedded life. He then married Constanze Ruschowitz,
who became the mother of Dr. Gerhard von Breuning, born August
28, 1813, author of “Aus dem Schwarzspanierhause.”
4. Lorenz (called Lenz, the posthumous child), born in the summer
of 1777, studied medicine and was in Vienna in 1794-97
simultaneously with Wegeler and Beethoven. He died on April 10,
1798 in Bonn.[39]](https://image.slidesharecdn.com/2648523-250526162053-c6d86f76/85/An-Introduction-To-Modelbased-Survey-Sampling-With-Applications-1st-Edition-Ray-Chambers-61-320.jpg)
![Madame von Breuning, who died on December 9, 1838, after a
widowhood of 61 years, lived in Bonn until 1815, then in Kerpen,
Beul-an-der-Ahr, Cologne and finally with her son-in-law, Wegeler, in
Coblenz.
The acquaintance between Beethoven and Stephan von Breuning
may have had some influence in the selection of the young musician
as pianoforte teacher for Eleonore and Lorenz,[40] an event (in
consideration of circumstances already detailed and of the ages, real
and reputed, of pupils and master) which may be dated at the close
of the year 1787, and which was, perhaps, the greatest good that
fate, now become propitious, could have conferred upon him; for he
was now so situated in his domestic relations, and at such an age,
that introduction into so highly refined and cultivated a circle was of
the highest value to him both morally and intellectually. The recent
loss of his mother had left a void in his heart which so excellent a
woman as Madame von Breuning could alone in some measure fill.
He was at an age when the evil example of his father needed a
counterbalance; when the extraordinary honors so recently paid to
science and letters at the inauguration of the university would make
the strongest impression; when the sense of his deficiencies in
everything but his art would begin to be oppressive; when his
mental powers, so strong and healthy, would demand some change,
some recreation, from that constant strain in the one direction of
music to which almost from infancy they had been subjected; when
not only the reaction upon his mind of the fresh and new intellectual
life now pervading Bonn society, but his daily contact with so many
of his own age, friends and companions now enjoying advantages
for improvement denied to him, must have cost him many a pang;
when a lofty and noble ambition might be aroused to lead him ever
onward and upward; when, the victim of a despondent melancholy,
he might sink into the mere routine musician, with no lofty aims, no
higher object than to draw from his talents means to supply his
necessities and gratify his appetites.](https://image.slidesharecdn.com/2648523-250526162053-c6d86f76/85/An-Introduction-To-Modelbased-Survey-Sampling-With-Applications-1st-Edition-Ray-Chambers-62-320.jpg)
![of humanity, hurried from Dux, took charge of affairs and was to be
found wherever the danger was greatest.” It was between May 4
and June 17, 1787, that Waldstein parted from his widowed mother
and journeyed to the place of his novitiate. His name may easily
have become known to Wegeler before the latter’s departure from
Bonn for Vienna.[41] Here follows what the good doctor says of the
Count—to what degree correct or mistaken, the reader can
determine for himself:
The first, and in every respect the most important, of the
Mæcenases of Beethoven was Count Waldstein, Knight of
the Teutonic Order, and (what is of greater moment here)
the favorite and constant companion of the young Elector,
afterwards Commander of the Order at Virnsberg and
Chancellor of the Emperor of Austria. He was not only a
connoisseur but also a practitioner of music. He it was
who gave all manner of support to our Beethoven, whose
gifts he was the first to recognize worthily. Through him
the young genius developed the talent to improvise
variations on a given theme. From him he received much
pecuniary assistance bestowed in such a way as to spare
his sensibilities, it being generally looked upon as a small
gratuity from the Elector.
Beethoven’s appointment as organist, his being sent to
Vienna by the Elector, were the doings of the Count. When
Beethoven at a later date dedicated the great and
important Sonata in C major, Op. 53, to him, it was only a
proof of the gratitude which lived on in the mature man. It
is to Count Waldstein that Beethoven owed the
circumstance that the first sproutings of his genius were
not nipped; therefore we owe this Mæcenas Beethoven’s
later fame.
Frau Karth remembered distinctly the 17th of June upon which
Waldstein entered the order, the fact being impressed upon her mind
by a not very gentle reminder from the stock of a sentinel’s musket](https://image.slidesharecdn.com/2648523-250526162053-c6d86f76/85/An-Introduction-To-Modelbased-Survey-Sampling-With-Applications-1st-Edition-Ray-Chambers-65-320.jpg)
![Ludwig the Head of
the Family
that the palace chapel was no place for children on such an
occasion. She remembered Waldstein’s visits to Beethoven in the
years following in his room in the Wenzelgasse and was confident
that he made the young musician a present of a pianoforte.
To save his line from extinction the Count obtained a dispensation
from his vows and married (May 9, 1812) Maria Isabella, daughter of
Count Rzewski. A daughter, Ludmilla, was born to him; but no son.
He died on August 29, 1823, and the family of Waldsteins of Dux
disappears. While all that Wegeler says of this man’s kindness in
obtaining the place of organist for Beethoven and of his influence
upon his musical education is one grand mistake,[42] there is no
reason whatever to doubt that those qualities which made the youth
a favorite with the Breunings, added to his manifest genius, made
their way to the young count’s heart and gained for Beethoven a
zealous, influential and active friend. Still, in June, 1778, Waldstein
possessed no such influence as to render a petition for increase of
salary, offered by his protégé, successful. That document has
disappeared, but a paper remains, dated June 5, concerning the
petition, which is endorsed “Beruhet.” Whatever this word may here
mean it is certain that Ludwig’s salary as organist remained at the
old point of 100 thalers, which, with the 200 received by his father,
the three measures of grain and the small sum that he might earn
by teaching, was all that Johann van Beethoven and three sons, now
respectively in their eighteenth, fifteenth and twelfth years, had to
live upon; and therefore so much the more necessity for the exercise
of Waldstein’s generosity.
After the death of the mother, says Frau Karth, a
housekeeper was employed and the father and
sons remained together in the lodgings in the
Wenzelgasse. Carl was intended for the musical profession; Johann
was put apprentice to the court apothecary, Johann Peter Hittorf.
Two years, however, had hardly elapsed when the father’s infirmity
compelled the eldest son, not yet nineteen years of age, to take the
extraordinary step of placing himself at the head of the family. One](https://image.slidesharecdn.com/2648523-250526162053-c6d86f76/85/An-Introduction-To-Modelbased-Survey-Sampling-With-Applications-1st-Edition-Ray-Chambers-66-320.jpg)
![Joseph Haydn’s
Visit to Bonn
implements and listened with obvious pleasure. Sit ei terra
levis!
The greatest musical event of the year (1790) in
Bonn occurred just at its close—the visit of Joseph
Haydn, on his way to London with Johann Peter
Salomon, whose name so often occurs in the preliminary chapters of
this work. Of this visit, Dies has recorded Haydn’s own account:
In the capital, Bonn, he was surprised in more ways than
one. He reached the city on Saturday [Christmas,
December 25] and set apart the next day for rest. On
Sunday, Salomon accompanied Haydn to the court chapel
to listen to mass. Scarcely had the two entered the church
and found suitable seats when high mass began. The first
chords announced a product of Haydn’s muse. Our Haydn
looked upon it as an accidental occurrence which had
happened only to flatter him; nevertheless it was
decidedly agreeable to him to listen to his own
composition. Toward the close of the mass a person
approached and asked him to repair to the oratory, where
he was expected. Haydn obeyed and was not a little
surprised when he found that the Elector, Maximilian, had
had him summoned, took him at once by the hand and
presented him to the virtuosi with the words: “Here I
make you acquainted with the Haydn whom you all revere
so highly.” The Elector gave both parties time to become
acquainted with each other, and, to give Haydn a
convincing proof of his respect, invited him to dinner. This
unexpected invitation put Haydn into an embarrassing
position, for he and Salomon had ordered a modest little
dinner in their lodgings, and it was too late to make a
change. Haydn was therefore fain to take refuge in
excuses which the Elector accepted as genuine and
sufficient. Haydn took his leave and returned to his
lodgings, where he was made aware in a special manner
of the good will of the Elector, at whose secret command](https://image.slidesharecdn.com/2648523-250526162053-c6d86f76/85/An-Introduction-To-Modelbased-Survey-Sampling-With-Applications-1st-Edition-Ray-Chambers-75-320.jpg)
An Introduction To Modelbased Survey Sampling With Applications 1st Edition Ray Chambers
- 1. An Introduction To Modelbased Survey Sampling With Applications 1st Edition Ray Chambers download https://ebookbell.com/product/an-introduction-to-modelbased- survey-sampling-with-applications-1st-edition-ray- chambers-5297046 Explore and download more ebooks at ebookbell.com
- 2. Here are some recommended products that we believe you will be interested in. You can click the link to download. An Introduction To Modelbased Cognitive Neuroscience 1st Edition Birte U Forstmann https://ebookbell.com/product/an-introduction-to-modelbased-cognitive- neuroscience-1st-edition-birte-u-forstmann-5140000 An Introduction To Management Studies Based On The St Gallen Management Model Thomas Bieger https://ebookbell.com/product/an-introduction-to-management-studies- based-on-the-st-gallen-management-model-thomas-bieger-49138874 Kripkes Worlds An Introduction To Modal Logics Via Tableaux 1st Edition Olivier Gasquet https://ebookbell.com/product/kripkes-worlds-an-introduction-to-modal- logics-via-tableaux-1st-edition-olivier-gasquet-4625076 An Introduction To Optimal Control Problems In Life Sciences And Economics From Mathematical Models To Numerical Simulation With Matlab 1st Edition Sebastian Ania https://ebookbell.com/product/an-introduction-to-optimal-control- problems-in-life-sciences-and-economics-from-mathematical-models-to- numerical-simulation-with-matlab-1st-edition-sebastian-ania-2243448
- 3. An Introduction To Continuoustime Stochastic Processes Theory Models And Applications To Finance Biology And Medicine 2nd Edition Vincenzo Capasso https://ebookbell.com/product/an-introduction-to-continuoustime- stochastic-processes-theory-models-and-applications-to-finance- biology-and-medicine-2nd-edition-vincenzo-capasso-2624272 An Introduction To Continuoustime Stochastic Processes Theory Models And Applications To Finance Biology And Medicine 2nd Edition Vincenzo Capasso https://ebookbell.com/product/an-introduction-to-continuoustime- stochastic-processes-theory-models-and-applications-to-finance- biology-and-medicine-2nd-edition-vincenzo-capasso-2630650 An Introduction To Continuoustime Stochastic Processes Theory Models And Applications To Finance Biology And Medicine 3rd Edition Vincenzo Capasso https://ebookbell.com/product/an-introduction-to-continuoustime- stochastic-processes-theory-models-and-applications-to-finance- biology-and-medicine-3rd-edition-vincenzo-capasso-5140118 An Introduction To The Model Penal Code Second Edition Dubber https://ebookbell.com/product/an-introduction-to-the-model-penal-code- second-edition-dubber-6638226 An Introduction To Velocity Model Building Ian F Jones https://ebookbell.com/product/an-introduction-to-velocity-model- building-ian-f-jones-4578154
- 6. OXFORD STATISTICAL SCIENCE SERIES Series Editors A. C. ATKINSON R. J. CARROLL D. J. HAND D. M. TITTERINGTON J.-L. WANG
- 7. OXFORD STATISTICAL SCIENCE SERIES For a full list of titles please visit http:/ /www.oup.co.uk/academic/science/maths/series/osss/ 10. J.K. Lindsey: Models for Repeated Measurements 11. N.T. Longford: Random Coefficient Models 12. P.J. Brown: Measurement, Regression, and Calibration 13. Peter J. Diggle, Kung-Yee Liang, and Scott L. Zeger: Analysis of Longitudinal Data 14. J.I. Ansell and M.J. Phillips: Practical Methods for Reliability Data Analysis 15. J.K. Lindsey: Modelling Frequency and Count Data 16. J.L. Jensen: Saddlepoint Approximations 17. Steffen L. Lauritzen: Graphical Models 18. A.W. Bowman and A. Azzalini: Applied Smoothing Techniques for Data Analysis 19. J.K. Lindsey: Models for Repeated Measurements, Second Edition 20. Michael Evans and Tim Swartz: Approximating Integrals via Monte Carlo and Deterministic Methods 21. D.F. Andrews and J.E. Stafford: Symbolic Computation for Statistical Inference 22. T.A. Severini: Likelihood Methods in Statistics 23. W.J. Krzanowski: Principles of Multivariate Analysis: A User’s Perspective, Revised Edition 24. J. Durbin and S.J. Koopman: Time Series Analysis by State Space Methods 25. Peter J. Diggle, Patrick Heagerty, Kung-Yee Liang, and Scott L. Zeger: Analysis of Longitudinal Data, Second Edition 26. J.K. Lindsey: Nonlinear Models in Medical Statistics 27. Peter J. Green, Nils L. Hjort, and Sylvia Richardson: Highly Structured Stochastic Systems 28. Margaret Sullivan Pepe: The Statistical Evaluation of Medical Tests for Classification and Prediction 29. Christopher G. Small and Jinfang Wang: Numerical Methods for Nonlinear Estimating Equations 30. John C. Gower and Garmt B. Dijksterhuis: Procrustes Problems 31. Margaret Sullivan Pepe: The Statistical Evaluation of Medical Tests for Classification and Prediction, Paperback 32. Murray Aitkin, Brian Francis, and John Hinde: Statistical Modelling in GLIM4, Second Edition 33. Anthony C. Davison, Yadolah Dodge, N. Wermuth: Celebrating Statistics: Papers in Honour of Sir David Cox on his 80th Birthday 34. Anthony Atkinson, Alexander Donev, and Randall Tobias: Optimum Experimental Designs, with SAS 35. M. Aitkin, B. Francis, J. Hinde, and R. Darnell: Statistical Modelling in R 36. Ludwig Fahrmeir and Thomas Kneib: Bayesian Smoothing and Regression for Longitudinal, Spatial and Event History Data 37. Raymond L. Chambers and Robert G. Clark: An Introduction to Model-Based Survey Sampling with Applications
- 8. An Introduction to Model-Based Survey Sampling with Applications Raymond L. Chambers Centre for Statistical and Survey Methodology, University of Wollongong, Australia Robert G. Clark Centre for Statistical and Survey Methodology, University of Wollongong, Australia 1
- 9. 3 Great Clarendon Street, Oxford ox2 6dp Oxford University Press is a department of the University of Oxford. It furthers the University’s objective of excellence in research, scholarship, and education by publishing worldwide in Oxford New York Auckland Cape Town Dar es Salaam Hong Kong Karachi Kuala Lumpur Madrid Melbourne Mexico City Nairobi New Delhi Shanghai Taipei Toronto With offices in Argentina Austria Brazil Chile Czech Republic France Greece Guatemala Hungary Italy Japan Poland Portugal Singapore South Korea Switzerland Thailand Turkey Ukraine Vietnam Oxford is a registered trade mark of Oxford University Press in the UK and in certain other countries Published in the United States by Oxford University Press Inc., New York © Raymond L. Chambers and Robert G. Clark 2012 The moral rights of the author have been asserted Database right Oxford University Press (maker) First published 2012 All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, without the prior permission in writing of Oxford University Press, or as expressly permitted by law, or under terms agreed with the appropriate reprographics rights organization. Enquiries concerning reproduction outside the scope of the above should be sent to the Rights Department, Oxford University Press, at the address above You must not circulate this book in any other binding or cover and you must impose the same condition on any acquirer British Library Cataloguing in Publication Data Data available Library of Congress Cataloguing in Publication Data Data available Typeset by SPI Publisher Services, Pondicherry, India Printed and bound by CPI Group (UK) Ltd, Croydon, CR0 4YY ISBN 978–0–19–856662–5 1 3 5 7 9 10 8 6 4 2
- 10. Preface The theory and methods of survey sampling are often glossed over in statistics education, with undergraduate programmes in statistics mainly concerned with introducing students to designs and procedures for choosing statistical models, checking model fit to available data, and estimating and making inferences about model parameters. Students may learn about models of considerable complexity, for example generalised linear models can be used for modelling the relationship of a range of explanatory variables to a response variable that can be continuous, binary or categorical. Increasingly, students are introduced to mixed models, time series models and models for spatial data, all of which are suitable for complex, correlated data sets. Non-parametric and semi-parametric methods based on kernel smoothing and spline smoothing are also increasingly important topics. In contrast, survey sampling is often only covered relatively briefly, and in contrast to these other topics, models either do not appear or are simple and are de-emphasised. This is surprising because survey sampling is one of the most satisfying and useful fields of statistics: • The target of inference is satisfyingly solid and observable. In clas- sical modelling theory, the focus is on estimating model parameters that are intrinsically unobservable. In contrast, a primary aim in surveys is to estimate quantities defined on a finite population – quantities that can in principle be directly observed by carrying out a census of this population. For example, an aim in classical statistical modelling might be to estimate the expected value of income, assuming that the distribution of income can be characterised by a specified distributional family; in contrast, the aim in a survey could be to estimate the mean income for the population of working age citizens of a country at a certain point in time. This mean income actually exists, and so it is possible to check the performance of statistical procedures for estimating its value in specific populations, in a way that is not possible when estimating model parameters. The practicalities of running a survey are also considered by the statistician and the statistical researcher, perhaps more so than in other fields of statistics. • Survey sampling is a major field of application of statistics, and is one of the great success stories of mathematical statistics. Before the mid-twentieth century, national statistics were based by and large on complete censuses of populations. This was enormously expensive and meant that only a limited range of data could be collected. Since then, the use of samples
- 11. vi Preface has become widely accepted, due mainly to the leadership of mathematical statisticians in government statistical agencies, and to the rapid development of a body of theory and methods for probability sampling. Surveys remain a major area of application of statistics – probably the major area in terms of dollars spent. A high proportion of graduates from statistics programmes spend some of their career at organisations that conduct surveys. • The rich range of models and associated methods used in ‘main- stream’ statistics can also be used in survey sampling. The key inferential objectives in survey sampling are fundamentally about prediction, and it is not difficult to transfer theoretical insights from mainstream sta- tistics to survey sampling. Unfortunately, however, this remains a rather under-developed area because the use of models is often de-emphasised in undergraduate courses on survey sampling. One of the reasons why modelling does not play much part when students are first taught sampling theory is that this theory has essentially evolved within the so-called design-based paradigm, with little or no reliance on models for inference. Instead, inference is based on the repeated sampling properties of estimators, where the repeated sampling is from a fixed finite population con- sisting of arbitrary data values. This is an attractive and logically consistent approach to inference, but is limiting because methods are required to work for virtually any population, and so cannot really exploit the properties of the par- ticular population at hand. The model-assisted framework, which has existed in some form since the early 1970s, makes use of models for the population at hand, but in a limited way, so that the potential risks and benefits from modelling are likewise limited. This book is an introduction to the model-based approach to survey sampling, where estimators and inference are based on a model that is assumed to summarise the population of interest for a survey. One way of presenting the model-based approach is to start with a very general linear model, or even generalised linear model, allowing also for any correlation structure in the population. Most of the methods in general use would then be special cases of this general model. We have instead chosen to start with simple models and build up from there, discussing the models suitable to different practical situations. With this aim in mind, this book is divided into three parts, with Part 1 focusing on estimating population totals under a range of models. Chapters 1 and 2 introduce survey sampling, and the model-based approach, respectively. Chapter 3 considers the simplest possible model, the homogenous population model. Chapter 4 extends this model to stratified populations. The stratified model is also quite simple, but nevertheless is very widely used in practice and is a good approximation to many populations of interest. Chapter 5 discusses linear regression models for populations with a single auxiliary variable, and Chapter 6 considers two level hierarchical populations made up of units grouped into clusters, with sampling carried out in two stages. Chapter 7 then integrates these results via the general linear population model. The approach in
- 12. Preface vii Chapters 3 through 7 is to present a model and discuss its applicability, to derive efficient predictors of a population total, and then to explore sample design issues for these predictors. Robustness to incorrectly specified models is of crucial importance in model- based survey sampling, particularly since much of the sample surveys canon has been model-free. Part 2 of this book therefore considers the properties of estima- tors based on incorrectly specified models. In practice, all statistical models are incorrect to a greater or lesser extent. To quote from Box and Draper (1987, page 74), ‘all models are wrong; the practical question is how wrong do they have to be to not be useful’. Chapter 8 shows that robust sample designs exist, and that, under these designs, predictors of population totals will still be approximately unbiased (although perhaps less efficient), even if the assumed model is incorrect. Chapter 9 extends this exploration of robustness to the important problem of robustifying prediction variance estimators to model misspecification. Chapter 10 completes Part 2 of the book with an exploration of how survey sampling meth- ods can be made robust to outliers (extreme observations not consistent with the assumed model), and also how flexible modelling methods like non-parametric regression can be used in survey sampling. Parts 1 and 2 of this book are concerned with the estimation of popula- tion totals, and more generally with linear combinations of population values. This has historically been the primary objective of sample surveys, and still remains very important, but other quantities are becoming increasingly impor- tant. Part 3 therefore explores how model-based methods can be used in a variety of new problem areas of modern survey sampling. Chapter 11 discusses predic- tion of non-linear population quantities, including non-linear combinations of population totals, and population medians and quantiles. Prediction variance estimation for such complex statistics is the focus of Chapter 12, which discusses how subsampling methods can be used for this purpose. In practice, most sur- veys are designed to estimate a range of quantities, not just a single population total, and Chapter 13 considers issues in design and estimation for multipurpose surveys. Chapter 14 discusses prediction for domains, and Chapter 15 explores small area estimation methods, which are rapidly becoming important for many survey outputs. Finally, in Chapters 16 and 17 we consider efficient prediction of population distribution functions and the use of transformations in survey inference. The book is designed to be accessible to undergraduate and graduate level students with a good grounding in statistics, including a course in the theory of linear regression. Matrix notation is not introduced until Chapter 7, and is avoided where possible to support readers less familiar with this notation. The book should also be a useful introduction to applied survey statisticians with some familiarity with surveys and statistics and who are looking for an introduction to the use of models in survey design and estimation. Using models for survey sampling is a challenge, but a rewarding one. It can go wrong – if the model is not checked carefully against sample data, or
- 13. viii Preface if samples are chosen poorly, then estimates and inferences will be misleading. But if the model is chosen well and sampling is robust, then the rich body of knowledge that exists on modelling can be used to understand the population of interest, and to exploit this understanding through tailored sample designs and estimators. We hope that this book will help in this process. Ray Chambers and Robert Clark April 2011
- 14. Acknowledgements This book owes its existence to the many people who have influenced our careers in statistics, and particularly our work in survey sampling. In this context, Ken Foreman stands out as the person whose inspiration and support in Ray’s early years in the field set him on the path that eventually led to this book and to the model-based ideas that it promotes, while Ken Brewer and our many colleagues at the Australian Bureau of Statistics provided us with the theoretical and prac- tical challenges necessary to ensure that these ideas were always grounded in reality. Early in Ray’s career he was enormously privileged to study under Richard Royall, who opened his eyes to the power of model-based ideas in survey sam- pling, and Alan Ross, who convinced him that it was just as necessary to ensure that these ideas were translated into practical advice for survey practitioners. To a large extent, the first part of this book is our attempt to achieve this aim. The book itself has its origin in a set of lectures that Ray presented to Eustat in Bilbao in 2003. Subsequently David Holmes was invaluable in providing advice on how these lectures should be organised into a book and with preparation of the exercises. Robert also had the privilege to work with some great colleagues and mentors. Frank Yu of the Australian Bureau of Statistics encouraged Robert to undertake study and research into the use of models in survey sampling. David Steel’s supervision of Robert’s PhD developed his knowledge and interest in this area, as did a year in the stimulating environment of the University of Southampton, enriched by interaction with too many friends and colleagues to mention. Robert would also like to express his appreciation of his parents for their lifelong love and support, and for passing on their belief in education. Many research colleagues have contributed over the years to the different applications that are described in this book, and we have tried to make sure that their inputs have been acknowledged in the text. However, special thanks are due to Hukum Chandra who helped considerably with the material pre- sented in Chapter 15 on prediction for small areas and to Alan Dorfman whose long-standing collaboration on the use of transformations in sample survey inference eventually led to Chapter 17, and whose insightful and support- ive comments on the first draft of the book resulted in it being significantly improved. The book itself has been a long time in preparation, and we would like to thank the editorial team at Oxford University Press, and in particular Keith
- 15. x Acknowledgements Mansfield, Helen Eaton, Alison Jones and Elizabeth Hannon, for their patience and support in bringing it to a conclusion. Finally, we would express our sincere thanks to our wives, Pat and Linda, for freely giving us the time that we needed to develop the ideas set out in this book. Without their support this book would never have been written.
- 16. Contents PART I BASICS OF MODEL-BASED SURVEY INFERENCE 1. Introduction 3 1.1 Why Sample? 4 1.2 Target Populations and Sampling Frames 5 1.3 Notation 6 1.4 Population Models and Non-Informative Sampling 9 2. The Model-Based Approach 14 2.1 Optimal Prediction 16 3. Homogeneous Populations 18 3.1 Random Sampling Models 19 3.2 A Model for a Homogeneous Population 20 3.3 Empirical Best Prediction and Best Linear Unbiased Prediction of the Population Total 21 3.4 Variance Estimation and Confidence Intervals 23 3.5 Predicting the Value of a Linear Population Parameter 24 3.6 How Large a Sample? 24 3.7 Selecting a Simple Random Sample 26 3.8 A Generalisation of the Homogeneous Model 26 4. Stratified Populations 28 4.1 The Homogeneous Strata Population Model 29 4.2 Optimal Prediction Under Stratification 30 4.3 Stratified Sample Design 31 4.4 Proportional Allocation 31 4.5 Optimal Allocation 34 4.6 Allocation for Proportions 35 4.7 How Large a Sample? 36 4.8 Defining Stratum Boundaries 37 4.9 Model-Based Stratification 40 4.10 Equal Aggregate Size Stratification 42 4.11 Multivariate Stratification 43 4.12 How Many Strata? 45 5. Populations with Regression Structure 49 5.1 Optimal Prediction Under a Proportional Relationship 49
- 17. xii Contents 5.2 Optimal Prediction Under a Linear Relationship 52 5.3 Sample Design and Inference Under the Ratio Population Model 53 5.4 Sample Design and Inference Under the Linear Population Model 55 5.5 Combining Regression and Stratification 56 6. Clustered Populations 61 6.1 Sampling from a Clustered Population 62 6.2 Optimal Prediction for a Clustered Population 63 6.3 Optimal Design for Fixed Sample Size 66 6.4 Optimal Design for Fixed Cost 68 6.5 Optimal Design for Fixed Cost including Listing 70 7. The General Linear Population Model 72 7.1 A General Linear Model for a Population 72 7.2 The Correlated General Linear Model 74 7.3 Special Cases of the General Linear Population Model 76 7.4 Model Choice 79 7.5 Optimal Sample Design 80 7.6 Derivation of BLUP Weights 81 PART II ROBUST MODEL-BASED SURVEY METHODS 8. Robust Prediction Under Model Misspecification 85 8.1 Robustness and the Homogeneous Population Model 85 8.2 Robustness and the Ratio Population Model 88 8.3 Robustness and the Clustered Population Model 93 8.4 Non-parametric Prediction 95 9. Robust Estimation of the Prediction Variance 101 9.1 Robust Variance Estimation for the Ratio Estimator 101 9.2 Robust Variance Estimation for General Linear Estimators 103 9.3 The Ultimate Cluster Variance Estimator 105 10. Outlier Robust Prediction 108 10.1 Strategies for Outlier Robust Prediction 108 10.2 Robust Parametric Bias Correction 110 10.3 Robust Non-parametric Bias Correction 113 10.4 Outlier Robust Design 114 10.5 Outlier Robust Ratio Estimation: Some Empirical Evidence 115 10.6 Practical Problems with Outlier Robust Estimators 117 PART III APPLICATIONS OF MODEL-BASED SURVEY INFERENCE 11. Inference for Non-linear Population Parameters 121 11.1 Differentiable Functions of Population Means 121
- 18. Contents xiii 11.2 Solutions of Estimating Equations 123 11.3 Population Medians 125 12. Survey Inference via Sub-Sampling 129 12.1 Variance Estimation via Independent Sub-Samples 130 12.2 Variance Estimation via Dependent Sub-Samples 131 12.3 Variance and Interval Estimation via Bootstrapping 135 13. Estimation for Multipurpose Surveys 139 13.1 Calibrated Weighting via Linear Unbiased Weighting 140 13.2 Calibration of Non-parametric Weights 141 13.3 Problems Associated With Calibrated Weights 143 13.4 A Simulation Analysis of Calibrated and Ridged Weighting 145 13.5 The Interaction Between Sample Weighting and Sample Design 151 14. Inference for Domains 156 14.1 Unknown Domain Membership 156 14.2 Using Information about Domain Membership 158 14.3 The Weighted Domain Estimator 159 15. Prediction for Small Areas 161 15.1 Synthetic Methods 162 15.2 Methods Based on Random Area Effects 164 15.3 Estimation of the Prediction MSE of the EBLUP 169 15.4 Direct Prediction for Small Areas 173 15.5 Estimation of Conditional MSE for Small Area Predictors 177 15.6 Simulation-Based Comparison of EBLUP and MBD Prediction 180 15.7 Generalised Linear Mixed Models in Small Area Prediction 184 15.8 Prediction of Small Area Unemployment 185 15.9 Concluding Remarks 192 16. Model-Based Inference for Distributions and Quantiles 195 16.1 Distribution Inference for a Homogeneous Population 195 16.2 Extension to a Stratified Population 197 16.3 Distribution Function Estimation under a Linear Regression Model 198 16.4 Use of Non-parametric Regression Methods for Distribution Function Estimation 201 16.5 Imputation vs. Prediction for a Wages Distribution 204 16.6 Distribution Inference for Clustered Populations 209 17. Using Transformations in Sample Survey Inference 214 17.1 Back Transformation Prediction 214 17.2 Model Calibration Prediction 215
- 19. xiv Contents 17.3 Smearing Prediction 218 17.4 Outlier Robust Model Calibration and Smearing 219 17.5 Empirical Results I 221 17.6 Robustness to Model Misspecification 225 17.7 Empirical Results II 227 17.8 Efficient Sampling under Transformation and Balanced Weighting 229 Bibliography 233 Exercises 241 Index 261
- 20. PART I Basics of Model-Based Survey Inference Statistical models for study populations were used in survey design and infer- ence almost from the very first scientific applications of the sampling method in the late nineteenth century. Following publication of Neyman’s influential paper (Neyman, 1934), however, randomisation or design-based methods became the dominant paradigm in scientific and official surveys in the mid-twentieth cen- tury, and models were effectively relegated to the secondary role of ‘assisting’ in the identification of efficient estimators for unknown population quantities. See Lohr (1999) for a development of sampling theory based on this approach. This situation has changed considerably over the last 30 years, with a resurgence of interest in the explicit use of models in finite population inference. Valliant et al. (2000) provide a comprehensive overview of the use of models in sample survey inference. To a large extent, this interest in the use of models is due to two mutu- ally reinforcing trends in modern sample surveys. The first is the need to provide sample survey solutions for inferential problems that lie outside the domain of design-based theory, particularly situations where standard probability-based sampling methods are not possible. The second is the need for methods of sur- vey inference that can efficiently integrate the increasing volume and complexity of data sources provided by modern information technology. In particular, it has been the capacity of the model-based paradigm to allow inference under a wider and more realistic set of sampling scenarios, as well as its capacity to efficiently integrate multiple sources of information about the population of interest, that has driven this resurgence. This book aims to provide the reader with an introduction to the basic con- cepts of model-based sample survey inference as well as to illustrate how it is being used in practice. In particular, in Part 1 of this book we introduce the reader to model-based survey ideas via a focus on four basic model ‘types’ that are in wide use. These are models for homogeneous populations, stratified popu- lations, populations with regression structure and clustered populations, as well as combinations of these basic structures.
- 21. This page intentionally left blank
- 22. 1 Introduction The standard method of scientific investigation is via controlled experimentation. That is, if a theory suggests some basic principles leading to outcomes that can be measured or observed, then if at all possible one attempts to validate it by carrying out an experiment where all extraneous effects influencing these outcomes are explicitly controlled or accounted for in the measurement process. By suitably modifying the conditions of the experiment, one can verify the theory by checking to see whether the observed responses are consistent with outcomes predicted under these conditions. In many situations, however, such experimentation is impossible. All one can do is observe the behaviour of the objects of interest, and infer general principles from this behaviour. Astronomy and geology are two fields of study where this type of situation holds. Another reason for not carrying out experimentation, especially in the bio- logical sciences, is because ethical and legal structures prevent most kinds of deliberate experimentation, especially on human populations. In other cases, the fact of applying some sort of treatment alters the fundamental nature of the class of objects under study, and thereby precludes any useful analysis. This is often the case in the social and natural sciences. Finally, there is a large class of problems for which controlled experimenta- tion is not meaningful. These are problems relating to the description of study populations. For such problems what is usually needed is the calculation of some summary statistic defined in terms of the values of a characteristic of interest over the population. Problems of this type form the basis for most official statistical collections. Collection of data by survey may be suitable in many of the instances where controlled experimentation is not possible. A survey can be defined as the planned observation of objects that are not themselves deliberately treated or controlled by the observer. In essence, ‘nature’ is assumed to have applied the treatments, and all the analyst can do is observe the consequences. A survey can be of the complete population of interest, in which case it is often called a census, or can be of a subset of this population, in which case it is usually referred to as a sample survey. In this latter case the problem of inferring about some behaviour in the population given what has been observed on the sample must be considered. This is the domain of sample survey theory and the focus of this book.
- 23. 4 Introduction Note that sample surveys are typically multipurpose in nature. A sample survey may be used to summarise characteristics of the study population as well to collect data for developing and/or evaluating theories about the mech- anism underlying these characteristics. In this book we tackle the first of these objectives, that is the use of sample surveys for presenting summary population information, rather than their use as a data collection tool for research purposes. That is, we emphasise the enumerative aspects of sample surveys rather than their analytic aspects. By and large this reflects the way sample survey theory has evolved. In particular, the statistical theory for sample surveys that has been developed over the last 50 years has tended to concentrate largely on their enu- merative use, and only recently has the problem of their analytic use received much attention. See Skinner et al. (1989) and Chambers and Skinner (2003) for recent developments on analysis of sample survey data. 1.1 Why Sample? Why only look at a subset (i.e. a sample) when one could look at the complete set (i.e. the population)? Cochran (1977) lists four reasons: (a) Reduced cost – samples, properly constructed, are usually much cheaper than censuses. This is especially true when the underlying population is very large, and where a sample that is only a small fraction of the population may still be large in terms of sample size and hence lead to highly precise sample estimates. (b) Greater speed – data from a sample that is a relatively small fraction of a population can be collected, summarised and published much more quickly than comparable data from a census. Timeliness of output of survey results is usually of primary importance in official data collections. (c) Greater scope and flexibility – the smaller size of sample survey operations means that greater effort can be invested in data collection for each sampled unit. Samples can thus be used to collect difficult to measure data that would be impracticable to collect via a census. (d) Greater accuracy – the smaller scale of sample surveys means that greater effort can be put into ensuring personnel of higher quality can be employed and given more intensive training and supervision. More effort can also be put into quality control when the survey data are processed. The end result is that use of a sampling approach may actually produce higher quality data and more accurate results than would be possible under a census. Of course there is a reverse side to these arguments. If accurate information is wanted for very rare population characteristics or for many very small groups in the population then the sample size needed to do the job may be so large that a census would be the most appropriate data collection vehicle anyway. However, this type of situation is the exception rather than the rule, and the use of the survey sample as a cost efficient method for data collection has
- 24. Target Populations and Sampling Frames 5 proliferated within the last 50 years. Censuses are still taken, but relatively infrequently, and in many cases as a method for benchmarking the much more frequent sample surveys that are the prime source of data for analysis. 1.2 Target Populations and Sampling Frames The most basic concept of survey sampling theory is that of the underlying target population. Simply speaking, this is the aggregate of elements or units about which we wish to make an inference. Some examples of target populations for a particular country, in this case the United Kingdom (UK), are: • all UK farming businesses in a particular year; • all current adult residents of the UK; • all transactions carried out by a UK business in a financial year; • all long stay (greater than a week) patients admitted to UK public hospitals in a particular year; • all animals of a particular species to be found in the UK county of Hampshire; • all UK registered fishing boats operating in the English Channel. Target populations can be finite or infinite. In this book we shall be concerned with finite populations, and all the examples above are of this type. However, target populations are not necessarily populations that can be surveyed. Often the units in a target population are only fuzzily defined. The actual population that is surveyed is the survey population. Thus, for the examples above the corresponding survey populations might be: • all farm businesses that responded to the UK Agricultural Census in that year; • all adults living in private dwellings and certain selected types of special dwelling (e.g. hotels, nursing homes, prisons, army barracks) in the UK on the night of 30 June; • all records (e.g. computer records) of transactions by the business in that financial year; • all hospital records showing more than seven days difference between first admission data and last discharge data for those patients discharged from UK public hospitals in that year; • all ‘visible’ (i.e. excluding extremely young) members of a mobile species (e.g. a species of bird) that can be found in Hampshire; • all current commercial fishing license holders for the UK Channel Fishery. Ideally the survey and target populations should coincide. However, as the above examples show, this is hardly ever the case. To the extent that the two population definitions differ, the results from the survey will not truly reflect the full target population. It is vital, therefore, that at the planning stages of a survey all efforts are made to ensure that there is the strongest possible link between these two populations. In any event, the results of the survey should always explicitly define the survey population.
- 25. 6 Introduction The next important concept in survey sampling is that of the sampling frame. This is the list, or series of lists, which enumerate the survey population and form the basis of the sample selection process. Again, referring to the examples above, some suitable frames might be: • a computer list of unique identifiers associated with all businesses that responded to the UK Agricultural Census; • a multiple level private dwelling list, with a first level consisting of a partition of the UK into small geographic areas (Enumeration Districts or EDs), a second level consisting of a list of all private dwellings within selected EDs, and a third level consisting of a list of all eligible adults within a selected sample of dwellings taken from the second level list; there may also be associated lists of special dwellings; • a computer list of transactions; • photocopies of hospital records for long stay patients, with name and address deleted to preserve confidentiality; • impossible to construct a sampling frame; • a computer printout of boat names, business addresses and license numbers for all commercial fishing license holders in the Fishery. Note that for the animal population example above no sampling frame was available. Consequently, list or frame-based methods for survey sampling are inappropriate for this situation. The theory of survey sampling of natural pop- ulations, where frames are usually not available, is not covered in this book. Interested readers are referred to Thompson (1992). Sampling frames can be complete or incomplete. The material presented in this book will assume a complete frame, that is one such that every element of the survey population is listed once and only once on the frame. Complete frames can be made up of a number of non-overlapping and exhaustive sub- frames, each covering a particular sub-class of elements in the population. An important ingredient of a complete frame is a unique identifier or label that can be associated with each element of the survey population and that enables the survey analyst to draw the specified sample from the frame. That is, the sample drawn is made up of a subset of the labels on the frame. Incomplete frames, by definition, do not cover the whole survey population. Often, a survey analyst can have two or more incomplete and overlapping sub- frames, each covering an unknown proportion of the survey population, and such that the union of these sub-frames completely covers the population. Although procedures for handling incomplete frames can be based on the ideas described in this book, we do not explicitly consider this issue here. 1.3 Notation Given a complete frame containing unique identifiers for elements of a finite survey population, we will denote this population by U and, without any loss
- 26. Notation 7 of generality, associate each label in U with an integer in the set {1, 2, . . . , N}, where N is the total survey population size. Because of the one to one association between the uniquely defined labels on the framework and the set {1, 2, . . . , N}, we can treat U as being indexed by the elements of this set, that is by i = 1, 2, . . . , N. The sample s is then a subset of U and therefore a subset of these indices, that is a subset of the integers between 1 and N. The number of elements in s (the sample size) will be denoted by n, and the set of N – n indices for the non-sampled elements, the complement of s, will be denoted r. Thus s ∪ r = U and s ∩ r = ∅. Surveys of human populations are usually targeted at attributes of the indi- viduals making up the population. These can be demographic (e.g. age, gender, marital status, racial/ethnic group), socio-economic (e.g. income, employment status, education) or personal (e.g. political preference, health conditions and behaviours, time use). Surveys of economic populations tend to focus on the physical inputs and outputs for the economic entities making up the survey pop- ulation, as well as financial performance measures like profit, debt and so on. We refer to any attribute measured in a survey as a variable, and use upper case to denote variables defined on a population and lower case to denote values specific to particular population units. Thus yi denotes the value of a variable Y associated with the ith population element. Following standard convention, we do not distinguish between realisations and corresponding variables, and so write E(yi), Var(yi), E(yi|zi) to represent the expected value, variance and conditional expected value of yi. We also distinguish between two classes of vari- ables. Survey variables correspond to measurements made in the survey whose population values exist but are only known on the sample. Auxiliary variables correspond to variables whose population values are known, although in prac- tice it is often sufficient to know the sample values and the population totals of these variables. In this book, we will generally write survey variables as Y , and auxiliary variables as X or Z. For example, in a survey of business incomes, the survey variables may include income, number of employees, profit and other financial variables from the most recent financial year. Auxiliary variables may include the number of employees or business income from an earlier period, perhaps obtained from taxation records. In a survey of employee satisfaction at a company, the survey variables would be various dimensions of satisfaction and morale. Auxiliary variables might con- sist of indicator variables summarising the age, gender, department and rank of each employee, available from personnel records. In a survey of the general pop- ulation, auxiliary variables often consist of indicator variables summarising the age, gender and geographical region of each person in the population. Population totals for these indicator variables are then population counts by age, gender and region – these counts may be available from official population counts produced by a national statistics office. Sample surveys are typically not concerned with the individual yi values themselves, but with making inferences about suitable aggregates summarising
- 27. 8 Introduction the distribution of these values in the survey population. The population total of these values, ty = U yi (1.1) and the population mean, ȳU = N−1 ty (1.2) are typically of interest. Sometimes the finite population distribution function defined by these values, FNy (t) = N−1 U I(yi ≤ t) (1.3) is also required. Here t is a dummy variable and I(yi ≤ t) is the indicator function for the event yi ≤ t, that is it takes the value 1 when yi is greater than or equal to t and the value zero when yi is less than t. Associated with this distribution function are the finite population quantiles. These are values QNy (α) such that QNy (α) = inf t {FNy (t) α} (1.4) where α is an index taking values between 1/N and (N −1)/N. That is, QNy (α) is the smallest value of t for which at least 100α% of the population yi values are less than or equal to that value. Note that α = 0.5 defines the finite population median. The quantities ty, ȳU , FNy (t) and QNy (α) specified by (1.1)–(1.4) are called finite population parameters. In general, a finite population parameter is any well-defined function of the population values associated with one or more survey characteristics. Thus, for example, the ratio of the population totals (or averages) of two survey variables, Ryx = U yi U xi (1.5) defines a finite population parameter, as does the average of the individual ratios of these characteristics, r̄U = N−1 U yi/xi. (1.6) A number of the arguments in this book are based on asymptotic considerations. That is, they relate to properties of statistics when the sample size n is large. Of course, since samples are always taken from populations, which are, by defini- tion, finite, these arguments implicitly assume that the population size N is also large, in the sense that the difference N − n is large. Since rigorous asymptotic arguments tend to be littered with technical conditions that are usually impossi- ble to verify in any practical application, we avoid them in this book. Instead, we make free use of ‘big oh’ notation to indicate order of magnitude conditions that need to apply before results can be expected to hold. In particular, given two sequences of numbers {αn} and {βn}, we say that {αn} is O(βn) if the sequence {αn/βn} remains bounded as n increases without limit. Thus, a statement of
- 28. Population Models and Non-Informative Sampling 9 the form Var(θ̂) = O(n−1 ) for some statistic θ̂ is just a shorthand way of saying that, as the sample size increases, the variance of this statistic decreases at the same rate as the inverse of the sample size. An important aspect of asymptotic arguments is that they allow approx- imations. This is very useful when exact results about the distribution of a statistic are too complex to derive. In this book we will make frequent use of two approximations that are pervasive in statistics. The first is the central limit approximation, which essentially states that if sample sizes are large enough, the distributions of many statistics (and particularly linear statistics) are well approximated by normal distributions. The second is approximation of moments by leading terms in their Taylor series expansions (often also referred to as Tay- lor linearisation). A reader who is unsure about these concepts should refer to Serfling (1980) for a thorough examination of their application in statistics. 1.4 Population Models and Non-Informative Sampling A finite population parameter is a quantity whose value would be known exactly if a census of the survey population were carried out. However, there is another type of statistical object that can be associated with the values that make up a finite population that is less well defined. This is a statistical model for these val- ues, often referred to as a superpopulation model. In this book a statistical model for a population is defined broadly as a specification of the statistical properties of the population values of the survey variables of interest. In some cases this model may be tightly specified, in the sense that it explicitly identifies a stochas- tic process that generated these population values. More generally, such a model is usually rather weakly specified, in the sense that it only identifies some of the statistical properties (e.g. first and second order moments) of the distribution of the population values of the survey variables. In either case there will be parame- ters associated with the model specification (superpopulation parameters) whose values are unknown. For example, it may be reasonable to postulate that the values y1, y2, . . . , yN are in fact N independent and identically distributed realisations of a random variable with mean μ and variance σ2 . In this case μ and σ2 are hypothet- ical constructs that could never be observed exactly even if a census of the survey population was carried out. Another example of a population model is a regression model. Given two survey variables with population values yi and xi, i = 1, 2, . . . , N, it may be reasonable to assume that the conditional expec- tation of yi given xi is linear in xi. That is, one can write E(yi|xi) = β0 + β1xi where E(.) denotes expectation relative to an underlying stochastic process that led to the population values yi. The superpopulation parameters here are β0 and β1. Again, we note that these parameters are hypothetical. They only exist as a convenient way of characterising how the population values of Y tend to change as the corresponding population values of X change. The stochastic process that actually generated the population values of these variables is unspecified.
- 29. 10 Introduction Given a model for a population, standard (infinite population) statistical theory provides various methods for efficient estimation of the parameters of this model. However, nearly all these methods assume the sample data are a random sample of realisations from the stochastic process defined by the population model. In practice this means that there is no systematic relationship between the values generated by the model and the method used to decide which of them are actually observed. With data obtained from survey samples this assumption may not be valid. Very often, the sample design used in the survey will favour observation of particular types of sample values. Ignoring this design information and analysing the survey data as if it had been obtained by some form of random sampling can lead to biased inference. As noted earlier in this chapter, methods for making inferences about super- population parameters such as β0 and β1 are not the focus of this book. However, this does not mean that such parameters are unimportant for finite population inference (i.e. for inference about finite population parameters). In fact, this book is essentially about how our knowledge of these parameters (and the sta- tistical models they characterise) can be used to develop efficient methods for finite population inference. The key concept used to relate parametric models for populations to infer- ence about finite population parameters is that of non-informative sampling. Broadly speaking, a method of sampling is non-informative for inference about the parameters of a superpopulation model for a variable if the same superpop- ulation model also holds for the sample values of this variable. That is, we can make valid inferences about these parameters on the basis of fitting the super- population model to the sample data. More formally, let θ be a finite population parameter defined by the population values of a (possibly multivariate) variable Y , let X U consist of auxiliary information about the population, let Y U be the values of Y for the population, let s be the set of n units selected using some sampling method, and let Y s be the values of Y for the sampled units. We say that a method of sampling is non-informative for inference about θ given X U if the joint conditional distribution of Y s given X U is the same as the joint condi- tional distribution of Y U given X U restricted to just those units in s. In effect, the method of sampling only influences inference about the parameters of the joint conditional population distribution of Y by determining which population units make up the sample. The outcome of the sampling process (the set s of sample labels) contains no further information about these parameters. The Conditionality Principle (Cox and Hinkley, 1974, p. 38) states that one should always condition on ancillary variables in inference. An ancillary vari- able is one whose distribution depends on parameters that are distinct from those associated with the distribution of the variable of interest. As an ancillary statistic, s should therefore be treated as fixed in inference about the parame- ters of the joint distribution of the population values of Y given the auxiliary information.
- 30. Population Models and Non-Informative Sampling 11 Probability sampling methods form an important class of non-informative sampling methods. These are methods that use a probability mechanism to decide whether or not to include a particular population unit in sample, and where this mechanism only depends on the population values of an auxiliary variable Z (which can be vector valued). In this case, once we condition on the population values of Z, it is clear that the outcome of the selection process is independent of the values of any of the survey variables, and so the method of sampling is non-informative given Z. Note that simple random sampling, where every singleton, pair, triple and so on of population units has exactly the same chance of turning up in sample as any other singleton, pair, triple, and so on, is non-informative. The importance of non-informative sampling to the model-based approach to finite population inference cannot be overstressed. This is because it allows valid inference for parameters of the conditional distribution of non-sampled popula- tion values of Y on the basis of models for the same conditional distribution of sampled values of Y. Thus, for example, we may have a model for a population that says that the regression of a variable Y on an auxiliary variable Z is linear. In effect the population values of these two variables satisfy E(yi|zi) = β0 + β1zi. (1.7) If our method of sampling is non-informative given the population values of Z, we can then immediately say that (1.7) holds in both the sampled and non-sampled parts of the population. As a consequence, sample estimates of β0 and β1 can be validly used to estimate the regression of Y on Z in the non-sampled part of the population. As will become clear in the next chapter, this ability to use information about superpopulation parameters derived from the sample to make statements about the distribution of the non-sampled part of the population is critical for application of the model-based approach to survey inference. A word of caution, however. Sampling methods are usually assumed to be non-informative conditional on Z. This is achieved by a combination of appropri- ate sample design, and the inclusion of relevant variables in Z, which can explain any differences between the sampled and non-sampled units. Very few sampling methods would be completely non-informative if Z was empty. However, there should always be some level of information about the outcome of the sampling process that allows us to distinguish the sampled population units from those that have not been sampled. Provided this information is included in Z, then we can safely ignore the sampling process in inference and treat sample and non- sample Y -values as drawn from the same distribution. To illustrate, suppose that (i) we have access to the population values of an auxiliary variable Z; (ii) we expect (1.7) to hold, with non-zero β1; and (iii) we use a method of sampling such that the sample distribution of Z differs from its non-sample distribution. Then, under the model-based approach, we expect the marginal sample and non- sample distributions of Y to be different, and so we must condition on Z in our
- 31. 12 Introduction inference. If we did not condition on the auxiliary variable, then sampling would be informative, and our inferences would be invalid. A classic example of this situation is where the probability of a particular population unit being included in sample depends on its value of Z, in the sense that units with larger values of Z tend to be included in sample more often than units with small values of Z. In such a case, we expect the sample and non-sample distributions of Z, and hence of Y , to be quite different. However, the conditional distribution of Y given Z is the same for both sets of units. That is, this method of sampling is non-informative given the population values of Z. We can then base our inference about a population characteristic of Y on this conditional distribution. What about if the sampling method is informative? Here conditioning on Z is not sufficient to ensure that population and sample distributions of the variable Y are the same. In this case we have two options. Sometimes (typi- cally not very often) we have sufficient information on the method of sampling to allow us to specify (and fit) a model for the distribution of the non-sample values of Y . Inference can then proceed on the basis of this model. An exam- ple is Sverchkov and Pfeffermann (2004). The other option is essentially our only choice when we do not have sufficient information to implement option one. This is to adopt robust methods of inference that allow for differences between the sample and non-sample distributions of Y . We discuss such robust methods of finite population inference later in this book. It should be noted, however, that such methods only work if the distribution of the non-sample values of Y is not too different from that of the sample values. No method of robust inference can protect against a total disconnect between the sample and non-sample distributions of the survey variable. In this context, it is advisable, if we suspect that an informative sampling method has been used, to collect enough additional information about the non-sampled part of the population to ensure that the sampling method then becomes non-informative, at least approximately. Applying good survey practices can also be used to reduce the potential for sampling to be informative. Steps that can be taken include: • Selecting a sample using probability sampling or some other non-subjective method. Designs where an expert chooses a set of units believed to be rep- resentative should be avoided, as in this case the sampling procedure will probably depend on variables other than Z, so that the sample distribution of Y |Z could differ from the population distribution. If expert knowledge is avail- able, it should be used to select which variables Z are likely to be relevant, rather than to select the actual sample. A sampling procedure based on Z should then be used. Extreme designs, for example where only the units with the largest values of Z are selected, should also be avoided, since assuming non-informativeness would then be equivalent to extrapolating the model for Y |Z to an unobserved part of the domain of Z. Appropriate sample designs
- 32. Population Models and Non-Informative Sampling 13 reflecting this approach will be suggested for different situations throughout this book. • Achieving a high response rate. When we say ‘sample’, we really mean the responding sample, that is those units who were selected, contacted, and agreed to participate in the survey. Of the initially selected sample, some units will be uncontactable or will decline to participate. The characteristics of the units who respond could well be different from those who do not. If response depends only on Z, then the sample will not be informative, but it may also depend on Y and other variables, leading to an informative sample. Achieving a high response rate will reduce the informativeness of the sampling process. This can be achieved by: using a sufficient number of callbacks when selected households do not answer in telephone or face-to-face interviewer sur- veys; well-designed questionnaires or interviews which do not overly burden the respondent; professional conduct, appearance and manner of interviewers; believable and justified assurances that respondents’ data will be used only for statistical purposes and not for identifying individuals; a concise statement to the respondent of the value of the survey to the community; maintaining a public reputation for trustworthiness, professionalism and relevance; the use of pre-approach letters; and offering incentives for respondents to participate. Some national statistical offices also have the power to make surveys compul- sory, which in conjunction with the other methods mentioned can lead to high response rates. For more information on survey methods, the reader is referred to Salant and Dillman (1994) and Groves et al. (2004).
- 33. 2 The Model-Based Approach In this chapter we develop the essentials of the model-based approach to sample survey design and estimation. In doing so, we focus on the population total ty = U yi of a survey variable Y , and we denote an estimator of this quantity by t̂y. Before we start our investigation of efficient estimators of this population total, it is useful to remind ourselves about which quantities are held fixed and which are allowed to be random under the model-based approach: • Population values yi are assumed to be generated by a stochastic model (the so-called superpopulation model) and are random. For example, (1.7) is a partial specification of such a model, giving the expected value of yi. • All expectations and variances are conditional on the outcome of the sample selection process. That is, the selected sample s is treated as a constant. • The sample values of yi are also random variables. • The population total ty is a sum of random variables and is therefore a random variable itself. Estimation of ty is equivalent to prediction of the value of this random variable using the data available. • Predictors t̂y of ty are functions of the sampled values {yi, i ∈ s} as well as of the auxiliary information {zi, i = 1, · · · , N}. The sampled values of Y are random variables, and so t̂y is a random variable. • Parameters of the model, such as β0 and β1 in (1.7), are assumed to be ‘fixed but unknown’ constants. In enumerative inference, we generally need to esti- mate model parameters, but only as a means to the end of predicting ty and other finite population quantities. It is important to be clear on what is treated as fixed or random, because other approaches to survey sampling do this differently. For example, the design- based (or randomization) (Cochran, 1977) and model-assisted (Särndal et al., 1992) approaches treat the population values of Y as unknown constants and the sample selected as the only source of randomness. The Bayesian approach (Ghosh, 2009; Ghosh and Meeden, 1997) treats all quantities as random vari- ables, including model parameters. For a recent comparison of the model-based and design-based approaches, see Brewer and Gregoire (2009). A recent overview of the model-based approach is given in Valliant (2009); the two pioneering references are Brewer (1963) and Royall (1970). In the following chapters we use the model-based approach to show how good predictors t̂y of ty can be constructed under some widely applicable models.
- 34. The Model-Based Approach 15 In particular, we show how the first two moments of t̂y − ty can be obtained under these models, and we then use this knowledge to design efficient sampling strategies for t̂y. To start, we note that both ty and t̂y are realisations of random variables whose joint distribution is determined by two processes – the first one, assumed random, that led to the actual population values of Y , and the second the process (possibly random, possibly not) that was used to determine which population units were selected for the sample s, and which were not. The sample s will be defined throughout the book to contain those units which were selected for the survey and which fully responded. We assume a (superpopulation) model for the population-generating process. Typically, we do not model the sample selection (and non-response) process, assuming instead that this process is non- informative given the values of an auxiliary variable Z whose values are related to those of Y and are known for all units making up the population. As noted in the previous chapter, this means that the conditional distribution of Y given Z in the population is the same as that in the sample. This circumstance considerably simplifies our inference and will be taken as given unless specified otherwise. Ideally, we want t̂y to be close to ty, or, equivalently, we want the sample error t̂y − ty to be close to zero. Of course, we do not know the value of this sample error, but under the model-based approach the statistical properties of t̂y −ty follow from the probability structure of the assumed model. In particular, the expected value and variance of t̂y − ty are of interest, in the sense that one would like the estimator t̂y to lead to a small expected value and a small variance for t̂y − ty. But which expected value and which variance? It turns out that, provided the method of sampling is non-informative given the population values of Z, then it is the mean and variance of t̂y − ty given these population Z-values that are relevant. The first step in developing the model-based approach to prediction of ty is to realise that this total can be decomposed as ty = s yi + r yi = tys + tyr . (2.1) That is, ty is the sum of the sample total tys of the Y -values and the correspond- ing non-sample total tyr . After the sample has been selected we obviously know tys so the basic problem is to predict tyr . If we denote such a prediction by t̂yr , then the corresponding predictor of ty satisfies t̂y = tys + t̂yr . In this context there are two basic questions one can ask: • Given the assumed model, what is the ‘best’ predictor t̂yr of tyr ? • Given this model and this predictor, what is the best way to choose the sample s in order to ‘minimise’ the sample error t̂y − ty = t̂yr − tyr ? The answers to these questions will depend on our interpretation of ‘best’ and ‘minimise’ above. Here we use ‘best’ in the sense that (i) t̂y is a member of a class of ‘acceptable’ predictors of ty; and
- 35. 16 The Model-Based Approach (ii) t̂y generates the smallest value of E(t̂y − ty)2 within this class given the sample s, where the expectation in (ii) above is with respect to the assumed model. Furthermore, we seek to ‘minimise’ t̂y − ty by choosing s in order minimise E(t̂y −ty)2 over the set of all ‘feasible’ samples, that is those samples that practi- cality and resources constraints allow. The combination of an optimal predictor and the optimal sample s to choose given this estimator is then an optimal sampling strategy for ty under the assumed model. 2.1 Optimal Prediction As noted earlier, under the model-based approach the statistical properties of t̂y as a predictor of ty are defined by the distribution of the sample error t̂y − ty under the assumed model for the population. Thus, the prediction bias of t̂y is the mean of this distribution, E(t̂y − ty), while the prediction variance of t̂y is the variance of this distribution, Var(t̂y − ty). The prediction mean squared error of t̂y is E(t̂y − ty)2 = Var(t̂y − ty) + E(t̂y − ty) 2 . Recollect that both ty and t̂y are random variables here! The predictor t̂y is said to be unbiased under the assumed model if its corresponding prediction bias E(t̂y − ty) is zero, in which case its prediction mean squared error is just its prediction variance, Var(t̂y − ty). Our aim is to identify an optimal sampling strategy for ty under the assumed model. The first step in this process is identification of an optimal predictor of tyr for any given s. In order to do so, we use the following well-known statistical result. Result 2.1 The minimum mean squared error predictor of a random variable W given the value of another random variable V is E(W|V ). See exercise E.1 for proof of this result. We can immediately apply it to the problem of predicting tyr (and hence prediction of ty). We put W equal to ty and V equal to our ‘observed data’, that is the sample Y -values and the population values of the auxiliary variable Z. The minimum mean squared error predictor of ty is then t∗ y = E(ty|yi, i ∈ s; zi, i = 1, · · · , N) = tys + E(tyr |yi, i ∈ s; zi, i = 1, · · · , N). (2.2) Clearly the conditional expectation in this result will depend on unknown para- meters of the assumed model, so t∗ y is impossible to compute in practice. For example, if (1.7) holds, then (2.2) becomes t∗ y = tys + r (β0 + β1zi), which depends on β0 and β1. However, observe that these parameters will be those defining the conditional distribution of Y given Z, and our assumption of
- 36. Optimal Prediction 17 non-informative sampling given Z implies that we can estimate them efficiently using the sample values of Y and Z. Substituting these estimated parameter values for unknown true values and computing the conditional expectation on the right hand side of (2.2) then leads to a ‘plug-in’ approximation to t∗ y, which is sometimes referred to as an empirical best (EB) predictor t̂EB y of ty. In the following chapters we explore specifications for EB predictors of ty under a number of widely used models for survey populations. We also consider specification of corresponding optimal model-based sampling strategies.
- 37. 3 Homogeneous Populations The first model for a survey population that we consider is the most basic that one might expect to encounter. This corresponds to a finite population where there are no auxiliary variables, or when it is clear a priori that any auxiliary variables are unrelated to Y . In this case, the distribution of Y |Z is assumed not to depend on Z, so that the model for yi is the same for every unit i in the population. We refer to this type of population as homogeneous. This does not imply that the distribution of Y has to have low variance, or to follow a well-behaved distribution. The following examples illustrate when the homogenous model would be used: • A crate of oranges. There is no sampling frame, and no auxiliary information. Oranges might be selected by physical sampling. Because there is no auxiliary information, the homogenous model is the only possible one, even though the oranges may vary considerably with respect to weight, colour, presence of mould and so on. • Children in a classroom. The frame could be a class roll and might include date of birth, so age would be a potential auxiliary variable. Date of birth might be assumed to be unrelated to variables measured on the children, because all of the children would be of approximately the same age. In this case a homoge- nous model would be used, because there are no relevant auxiliary variables which would allow modelling of a different distribution for different children. There might be many characteristics which would be related to the Y values of the children, such as sex, racial/ethnic origin, home background and physical limitations (e.g. short sight). However, these variables are not available for the population, and so do not form part of Z. The homogenous model applies because the distribution of yi|zi is the same for every child in the classroom, due to the paucity of information available in Z. (If it was thought that age might be a relevant variable, then a model other than the homogenous model would be used. Some non-homogenous models will be discussed in Chapters 4 through 7.) • Items on an assembly line. Y might be the weight of the item. If there were no auxiliary variables, then the homogenous model would apply. If the assem- bly line is dedicated to the production of a single item, then Y might be expected to follow a fairly well-behaved distribution, as production and legal standards are usually such that the items produced have to be as alike as pos- sible. In a multi-item assembly line, the homogeneity model would still apply as {yi|zi} would follow the same distribution for all i because zi is empty.
- 38. Random Sampling Models 19 However, the distribution would probably be an inconvenient one, with high variance and multiple modes. The theory in this chapter would still be applica- ble, but the large sample confidence intervals of Section 3.4 would perhaps require a larger sample size to be reliable than would otherwise be the case. (If the type of item was available for all items in the population, then type could be used as an auxiliary variable. The stratified model, which will be described in Chapter 4, would probably be the most appropriate.) The common thread in these examples is that zi does not contain any information which would allow (yi|zi) to be different for different i. Whenever this is the case, the homogenous model applies. 3.1 Random Sampling Models This lack of information in the sample labels means that all samples of the same size are equally informative. There is no reason for the survey designer to prefer any one sample to any other. For this reason, a random sampling method that gives equal probability of selection to all possible samples of the same size seems an intuitively sensible way of sampling from a homogeneous population. In addition, there are a number of strong, but essentially pragmatic, arguments for adopting such an approach that will be discussed later in this book. Such a random sampling method will be referred to as simple random sampling, or just SRS, in what follows. An additional argument for utilising a random approach to sample selection in this situation is that it sometimes makes it straightforward to derive a probability model directly from the probability sampling method. For example, consider a typical urn problem. An urn is known to contain N balls, some of which are white and some are black. It is required to estimate the proportion of white balls in the urn. Beyond knowing that the urn contains N balls, and that these are either black or white, nothing else is known about the distribution of the balls in the urn. However, by the simple expedient of vigorously stirring the balls in the urn, and then selecting a sample of n distinct balls ‘at random’ to observe, one can immediately generate a known distribution for the random variable cor- responding to the number of white balls observed in the sample. This is the hypergeometric distribution with parameters N = the total number of balls in the urn (known), W = the total number of white balls in the urn (unknown), and n = the sample number of balls taken from the urn (known). Under this model the probability that w white balls turn up in the sample is: p(w) = W w N − W n − w N n . An unbiased predictor of W under this model is Ŵ = N(w/n). Derivation of the variance of Ŵ under this model is left to exercise E.2.
- 39. 20 Homogeneous Populations Note that the use of the random selection procedure above is sufficient to guarantee a hypergeometric distribution for the sample data. However, this does not mean that such a random selection procedure is necessary to guarantee such a distribution. We may know of other factors which imply that the urn is already ‘randomly mixed’. If so, selection of any sample, not necessarily a random one, still allows use of the hypergeometric model. For example, one could select the n balls at the ‘top’ of the urn. However, even in this case, using a random selection procedure seems a wise precaution, in the event that our knowledge of these other factors may be imperfect, for example white balls might be slightly lighter than black balls and so tend to congregate more at the top of the urn. Application of this seemingly trivial model is widespread in survey sample practice. For example, it forms the basis of sample inference in opinion polls. Here the urns correspond to selected polling booths, the balls correspond to votes cast at these booths, the colours correspond to the candidates (or political parties) endorsed by these votes, and randomisation is necessary because of possible trends in the sequence in which the votes for particular candidates are cast at the booth. 3.2 A Model for a Homogeneous Population A general model for a homogeneous population starts with the concept of exchangeability. The random variables whose realisations are the population values yi of Y are said to be exchangeable up to order K if the joint distribution of {yi; i ∈ A} is the same for any permutation A of k = 1, 2, . . . , K distinct labels from the population. It is easy to see that in an exchangeable population all moments of products of population Y -values up to order K are the same. In particular, if K is greater than or equal to two, then all units in the population have Y -values with the same mean and the same variance, and all pairs of distinct units in the population have Y -values with the same covariance. Such a population will be referred to as second order homogeneous (or just homogeneous) in what follows. We will assume for now that values from different units are independent so that all covariances are zero, although Section 3.8 will remove this restriction. The second order homogeneous population model represents the basic ‘build- ing block’ for more complex models which we will describe in later chapters that can be used to represent real world variability. The population Y -values under the homogeneous model satisfy E(yi) = μ (3.1a) Var(yi) = σ2 (3.1b) yi and yj independent when i = j. (3.1c)
- 40. Empirical Best Prediction and Best Linear Unbiased Prediction 21 3.3 Empirical Best Prediction and Best Linear Unbiased Prediction of the Population Total For the homogeneous population model, there is no auxiliary information. That is, we cannot identify a variable Z whose population values vary and are all known and is such that the conditional distribution of Y given Z (and in par- ticular E(Y |Z)) varies with Z. From (2.2) we know that the minimum mean squared error predictor of ty is t∗ y = tys +E [tyr |yi, i ∈ s]. For model (3.1), this is given by t∗ y = tys + (N − n)μ. Of course, we do not know μ and so must replace this parameter by an estimate in order to define the EB predictor. Intuitively, in an exchangeable population all sample values provide the same information about μ, and so it seems sensible to use the sample mean ȳs of Y as our estimator of μ. This leads to the predictor: tE y = tys + (N − n)μ̂ = tys + (N − n)ȳs = N n tys . (3.2) The predictor defined by (3.2) above is commonly called the expansion estimator. Note that the EB predictor is not necessarily unique, as there may be several possible estimators of unknown parameters such as μ. However, it provides a simple way to construct predictors, and will be statistically efficient if a sensible method of parameter estimation is used. An alternative, more complex approach, called Best Linear Unbiased Prediction, can also be used. This method does yield a unique best predictor, called the Best Linear Unbiased Predictor, or BLUP. In many cases, including model (3.1) and estimator (3.2), the BLUP is also an EB predictor. To define the BLUP, we will first define linear predictors to be those that can be written as a linear combination of the values of Y associated with sample units. Linear predictors are used extensively in survey sampling, mainly because of their simplicity of use. The BLUP t̂BLUP y of ty under a specified model satisfies three conditions: • It is a linear predictor; that is it can be written in the form t̂BLUP y = s wiyi, where the wi are weights that have to be determined. Note that there is no restriction on these weights, except that they must not depend on any values of Y . In particular, they can, and often do, depend on the population units that make up the sample s, and the auxiliary variable Z in cases where the model includes Z. • It is unbiased for ty, that is its sample error has an expectation of zero, E t̂BLUP y − ty = 0. • For any sample s its sample error has minimum variance among the sam- ple errors of all unbiased linear predictors of ty, that is Var t̂BLUP y − ty ≤ Var(t̂y − ty) where t̂y is any other unbiased linear predictor of ty. It turns out that to derive the BLUP, we only need to assume that different observations are uncorrelated, rather than the stronger assumption of indepen- dence in (3.1c). We first note that for any linear predictor of ty we have the
- 41. 22 Homogeneous Populations decomposition t̂y = s wiyi = s yi + s (wi − 1)yi = tys + s uiyi where ui = wi − 1. Consequently the sample error can be expressed as t̂y − ty = s uiyi − r yi. We can think of ui as essentially defining the ‘prediction weight’ of unit i, that is the weight attached to its Y -value when predicting the non-sample total of Y . Clearly, in order to define the BLUP, all we need to do is work out the weights wi, or equivalently the prediction weights ui, that define this predictor. By definition there are two restrictions on these weights – they should lead to an unbiased predictor, and they should lead to the smallest possible prediction variance for such an unbiased predictor. To start, we focus on unbiasedness. This condition is equivalent to saying that for any linear predictor, including the BLUP, we have Bias(t̂y) = E(t̂y − ty) = μ s ui − (N − n) = 0 which is true only if s ui − (N − n) = 0. (3.3) Next, we seek to minimise the prediction variance. From standard statistical manipulations, we obtain Var(t̂y − ty) = Var(t̂yr − tyr ) = Var(t̂yr ) − 2Cov(t̂yr , tyr ) + Var(tyr ) where Var(t̂yr ) = σ2 s u2 i (3.4a) Var(tyr ) = (N − n)σ2 (3.4b) Cov(t̂yr , tyr ) = 0. (3.4c) Note that the last result (3.4c) makes use of the fact that the sample and non- sample values of Y are uncorrelated under model (3.1). Since Var(tyr ) and Cov(t̂yr , tyr ) are not functions of the ui, it follows that Var(t̂y − ty) will be minimised with respect to these weights when Var(t̂yr ) is minimised. That is, optimal values of ui (and hence wi) are obtained by minimising Var(t̂yr ) defined by (3.4a), or equivalently s u2 i , subject to the unbiasedness constraint (3.3). In order to do so, we form the Lagrangian L for this minimisation problem: L = s u2 i − 2λ s ui − (N − n) .
- 42. Variance Estimation and Confidence Intervals 23 Differentiating L with respect to ui and equating to zero we obtain ui = λ Substituting this expression into the unbiasedness constraint (3.3) and solving for λ leads to λ = N − n n , which implies ui = N−n n and hence wi = N n . That is, the BLUP t̂BLUP y of ty under the homogeneous population model (3.1) is the expansion estimator (3.2). 3.4 Variance Estimation and Confidence Intervals Substituting the optimal prediction weights ui = N−n n in (3.4a) leads to Var t̂E ry = σ2 (N − n)2 n and hence Var t̂E y − ty = Var t̂E yr + Var(tyr ) − 2Cov t̂E yr , tyr = σ2 (N−n)2 n + (N − n) = σ2 (N − n) N n . That is, the prediction variance of the expansion estimator (3.2) under the homogeneous population model (3.1) is Var t̂E y − ty = N2 n 1 − n N σ2 . (3.5) In order to create confidence intervals for ty based on (3.2), we need to be able to estimate (3.5). An unbiased estimator of σ2 in (3.1) is the sample variance of Y , s2 y = 1 n − 1 s (yi − ȳs)2 . This implies that an unbiased estimator of the prediction variance (3.5) is V̂ t̂E y = N2 n 1 − n N s2 y. (3.6) See exercise E.3 for a proof of the unbiasedness of (3.6). For large sample sizes, standard central limit theory implies that the distribution of the z statistic z = t̂E y − ty V̂ t̂E y
- 43. 24 Homogeneous Populations is (approximately) normal with zero mean and unit standard deviation. Conse- quently, an approximate 100(1 − α)% confidence interval for ty is t̂E y ± qα/2 V̂ t̂E y where qα/2 is the (1 − α/2)-quantile of an N(0, 1) distribution. Since ty is a random variable, such an interval is often referred to as a prediction interval. 3.5 Predicting the Value of a Linear Population Parameter Suppose that we are interested in predicting the value of A = N i=1 aiyi = U aiyi where a1, . . . , aN is a set of N known constants. For example, A could be the mean ȳU of the population Y -values in which case ai = N−1 . Often, A corre- sponds to a mean of Y for some identifiable subgroup of size M of the population, in which case ai = M−1 when unit i is in the subgroup and is zero otherwise. The BLUP for A under the homogeneous population model is then  = s aiyi + ȳs r ai. (3.7) It can be shown that the prediction variance of (3.7) is Var( − A) = σ2 n−1 r ai 2 + r a2 i which has the unbiased estimator V̂ (Â) = n−1 r ai 2 + r a2 i (n − 1)−1 s (yi − ȳs)2 . We may sometimes be interested in predicting other population parameters such as ratios of population means or totals. For example, the economic indicator ‘average weekly earnings’ is often defined as the sum of the ‘wages paid’ variable for a population of businesses, divided by the population sum of ‘number of employees’. See Section 11.1 for extensions of the above results to estimation of ratios. 3.6 How Large a Sample? As noted earlier, simple random sampling (SRS) is an obvious way of selecting a sample from a homogeneous population. Certainly, this method of sampling is
- 44. How Large a Sample? 25 one of the simplest probability-based methods of sample selection. There are two basic design questions that need to be answered before a sample can be selected via this method. These are: • How big a sample should we take? • How do we go about selecting a sample via SRS? The answer to the first question depends on the constraints imposed on the sam- ple design process. For example, suppose that it is required to select a sample of sufficient size so as to ensure that the expansion estimator t̂E y has a rela- tive standard error (RSE) of A percent. The RSE of a predictor (also known as its coefficient of variation, or CV ) is the square root of its prediction vari- ance, expressed as a percentage of the value of the population quantity being predicted. So RSE t̂E y = Var t̂E y − ty /ty × 100. By substituting Var t̂E y − ty from (3.5) above, then setting this expression equal to A, and solving for n, we obtain n = N−1 + (A2 /104 )ȳ2 U /σ2 −1 (3.8) where ȳU is the mean of the population Y -values. Thus, the required sample size to meet the RSE objective depends on the value of the ratio C = σ2 /ȳ2 U (3.9) and hence on the population Y -values. Typically, this ratio can be estimated directly from data obtained in a pilot study preceding the main survey, or, if such an assumption seems reasonable, by setting it equal to the relative variance of another population variable, say Z, whose values are known for all population elements (i.e. Z is an auxiliary variable). In many cases, these values are the historical values of Y from a past census of the population. Alternatively, if the survey is a continuing one, then an estimate of the RSE of t̂E y based on data from the immediately preceding survey can be calculated and, assuming the relative variance of Y has remained unchanged, substituted into (3.8) to give the sample size required for the target RSE of the present survey. To illustrate this situation, consider the case where in a past survey of the same population (or one very much like it), with sample size m, say, an estimated RSE equal to B was obtained. Assuming that the relative variance of Y is the same in both populations, we can then estimate the ratio C in (3.9) by Ĉ = B2 /1002 m−1 − N−1 .
- 45. 26 Homogeneous Populations Substituting this estimate in the expression for n in (3.8) above, and discarding lower order terms leads to n = (B2 /A2 )m. That is, the ratio of the estimated relative standard errors from both surveys is the inverse of the ratio of the square roots of the respective sample sizes for the surveys. 3.7 Selecting a Simple Random Sample How do we go about selecting a simple random sample? The simplest way, given the population and sample sizes involved are small, is to use a table of random numbers. For example, Fisher and Yates (1963) provide a list of two digit ran- dom numbers, together with instructions on how to use them to select a simple random sample. When sample sizes and populations are large, it is usually most convenient to use a computer to select the sample. Most computer packages include a pseudo- random number generator, which can be used in this regard. A simple way of selecting a SRS of size n, using a computer-based random number generator, is to randomly order the population units on the sampling frame, then take the first n of these randomly ordered units to be the sample. This random ordering is easily accomplished by independently assigning a pseudo-uniform [0,1] random variate to each of the N units on the sampling frame. We then re-order these N units according to these random values. The above so-called ‘shuffle algorithm’ has the disadvantage that it requires a pass through the population (to allocate the random numbers), then another pass to re-order the population. This can be expensive (in computer time) in very large populations. Another procedure, therefore, is to generate random numbers between 1 and N until n distinct numbers are generated. These then define the labels of the selected sample units. Vitter (1984) contains a discussion of several efficient algorithms for computer-based selection of a simple random sample by one sequential pass through a computer list of the population labels. Implicit in the preceding discussion about methods of selecting a simple ran- dom sample of size n is that the same population unit cannot occur more than once in any particular selected sample. In other words any sample we select must contain n distinct population units and all samples of the same size must be equally likely. This method of sampling is typically referred to as Simple Random Sampling Without Replacement (SRSWOR). 3.8 A Generalisation of the Homogeneous Model Model (3.1) implies that the values of Y are uncorrelated for distinct units. Sup- pose that we generalise the model, by allowing a uniform correlation ρ between every pair of values:
- 46. A Generalisation of the Homogeneous Model 27 E(yi) = μ (3.10a) Var(yi) = σ2 (3.10b) Cov(yi, yj) = ρσ2 when i = j. (3.10c) It turns out that the expansion estimator (3.2) is still the BLUP for ty under this more general homogeneous model. Furthermore, (3.6) is still unbiased for the prediction variance of t̂E y . For proof, see exercise E.3. We have focused on the more restrictive model (3.1) rather than (3.10) in this chapter because a uniform correlation between all pairs of units in a population does not have a sensible interpretation for most populations in the real world. However, correlations between pairs of units will be relevant in Chapter 6 where we consider sampling from a population made up of units grouped into clusters, because we then use a superpopulation model where values for units from the same cluster satisfy (3.10).
- 47. 4 Stratified Populations The reality of sample survey practice is that target populations, and especially the large populations of interest in social and economic data collections, are almost never homogeneous. In many cases, these target populations can be mod- elled as being made up of a number of distinct and non-overlapping groups of units, each one of which could be considered to be internally homogeneous, but which may differ considerably from one another. These groups, each one of which is usually referred to as a stratum, and collectively as strata, are often large in size with the average value of Y varying significantly across the strata. As a result, every stratum is sampled, since information about the distribution of Y obtained from units in the sampled strata tells us very little about the distribution of Y in the non-sampled strata. In many cases, strata are ‘naturally defined’. For example, if units are busi- nesses, then strata might be industries, and if units are people, strata might be states or provinces. In other cases, there may be information on the population frame that allows a choice of how the population can be stratified. Typically, this information consists of the known values, listed on the frame, of one or more aux- iliary or benchmark variables defined on the population. It is known that there is systematic variation in the survey variables associated with the variation of the benchmark variables on the frame. By judicious choice of appropriate ranges of values for these benchmarks, the survey analyst can define strata within which one can assume that the survey variables have small systematic variation relative to their variation across the population as a whole. Stratification of target populations is extremely common in survey sam- pling. Typically, samples are then selected independently from each stratum, referred to as stratified sampling. Aside from the statistical reason of stratify- ing in order to control for systematic heterogeneity in the target population, there are many practical reasons why stratified sampling is adopted as a sample survey technique. Cochran (1977) lists three sensible reasons for the use of this technique: • Domains of interest. The subpopulations defining the strata can be of interest in themselves, and estimates of known precision may be required for them. For example, states or provinces are often considered important output categories in national household surveys. • Efficient survey management. In many situations the target population is spread across a wide geographic area and administrative convenience may
- 48. The Homogeneous Strata Population Model 29 dictate the use of stratification; for example, the agency conducting the sur- vey may have field offices, each one of which can supervise the survey for part of the population. • Different methods of sampling. Sampling problems may differ markedly in different parts of the population. With human populations, people living in institutions (e.g. hospitals, army bases, prisons) are often placed in a different stratum from people living in ordinary homes because a different approach to sampling respondents is appropriate for the two situations. In sampling businesses we may place the largest firms in a separate stratum because the level of detail in the data we require from these firms may be quite different from the data we require from smaller firms. Some examples of stratified populations are: • Children in a school system. Children can be stratified on the basis of the level of school (primary/secondary/college), the type of school (govern- ment/private/other), and class levels within each school. Depending on the information available to the survey analyst, stratification on the basis of gender could also be considered. • Households in a city. Households can be stratified on the basis of the wards making up the city; special strata can also be constructed for special dwelling types like caravans, hotels, armed forces bases and institutional dwelling arrangements (e.g. hospitals, prisons). • Businesses in a sector of a country’s economy. In this case stratification is usu- ally on the basis of the industries (or groups of industries) to which businesses making up the sector belong, the physical locations of the businesses them- selves (based on an appropriate geographic identifier) and their sizes (measured in some appropriate way). • Books in a library, or files in an archive. This type of example comes up in surveys for auditing purposes. Stratification would often involve physical location, for example books could be stratified by room and shelf, and files could be stratified by location, filing cabinet and drawer. 4.1 The Homogeneous Strata Population Model A model for a stratified population follows naturally from our definition of strata as made up of population elements that are homogeneous with respect to other elements of the same stratum and heterogeneous with respect to elements of other strata. We will use the following model for the distribution of population Y -values across strata h = 1, . . ., H: E(yi|i ∈ h) = μh (4.1a) Var(yi|i ∈ h) = σ2 h (4.1b) yi and yj are independent when i = j. (4.1c)
- 49. 30 Stratified Populations Here i ∈ h indicates that population unit i is in stratum h. We refer to this model as the homogeneous strata population model in what follows. The assump- tion (4.1c) means that distinct population units are independent as far as their Y -values are concerned. We will see that (4.1c) is necessary to derive an EB predictor, while a weaker assumption of zero covariance is sufficient to derive the BLUP. 4.2 Optimal Prediction Under Stratification As in Section 3.3, we develop an EB predictor for this case. Since each stratum constitutes a separate homogeneous population, following model (3.1), the sam- ple mean of Y within each of the strata is an EB predictor of the corresponding stratum population mean. The predictor t̂EB y of the overall population total ty is then the sum of the individual stratum level expansion estimators t̂E yh , since these are EB predictors of the stratum population totals tyh of Y , that is t̂EB y = t̂S y = h t̂E yh = h Nhȳsh . (4.2) Here h indexes the strata, Nh is the stratum population size, nh is the stratum sample size and ȳsh is the sample mean of Y in stratum h. The predictor t̂S y defined by (4.2) above is usually called the stratified expansion estimator. Using the same derivation as in Section 3.3, it is straightforward to show that the stratified expansion estimator is also the BLUP under model (4.1). It is sufficient to assume that covariances are zero between different units, rather than the stronger assumption of independence in (4.1c). Note that if some strata are not represented in the sample, then it is not possible to use the stratified expansion estimator, since ȳsh will not be defined for those strata. In this case, there is no unbiased estimator of ty. Ideally, we should select our sample in such a way that every stratum is represented, so that this problem does not arise. In order to compute the prediction variance of t̂S y , and hence develop an estimator for it, we observe that since distinct population units are mutually uncorrelated the prediction variance of t̂S y under the stratified population model (4.1) is the sum of the individual prediction variances of the stratum specific BLUPs t̂E yh , and each of these is given by (3.5) with the addition of a stratum subscript, that is Var t̂S y − ty = h Var t̂E yh − tyh = h N2 h/nh (1 − nh/Nh)σ2 h. (4.3) Unbiased estimation of (4.3) is straightforward. One just sums unbiased stratum level estimators of the prediction variances of the t̂E yh (see (3.6)) to get V̂ t̂S y = h V̂ t̂E yh = h N2 h/nh (1 − nh/Nh)s2 yh . (4.4)
- 50. Stratified Sample Design 31 where s2 yh = 1 nh−1 sh (yi − ȳsh )2 denotes the unbiased estimator of the variance σ2 h of Y -values in stratum h. Here sh denotes the sample units in stratum h. Provided the strata population and sample sizes are large enough, the Central Limit Theorem applies within each stratum, and so applies overall, allowing us to write: t̂S y − ty / V̂ t̂S y ∼ N(0, 1). Confidence intervals for ty follow directly: an approximate 100(1−α)% confidence interval for ty is t̂S y ± qα/2 V̂ t̂S y . 4.3 Stratified Sample Design The stratified expansion estimator (4.2) can be used whenever model (4.1) is a reasonable approximation to the population. We do not necessarily have to use the strata in the sample design and selection. However, we want to ensure that every stratum is represented in the sample. Also, given that the strata are the most important feature of the population, it makes sense to build this into the design. We will see that we can develop efficient stratified designs that lead to low variance for the stratified expansion estimator when the model is true. In some surveys, it is not feasible to base selection on strata. The most common reason for this is that strata are not known in advance of sampling for every unit in the population. The stratified expansion estimator (4.2) can still be used (as long as we have at least one unit in sample from each stratum). To calculate its value, all we need to know are the population stratum sizes, and the stratum memberships of sampled units—this information is sometimes available even if the stratum membership of every population unit is not. The estimator (4.2) is sometimes called a post-stratified estimator in this scenario, as the strata are only formed after the sample is selected. The remainder of this chapter is concerned with how to design the sample when strata are known in advance. Some questions which we will consider are: how many units to select from each stratum; how to decide on the total sample size; how to form strata by categorising a continuous variable; how many such strata should be used; and how to construct strata when there are multiple auxiliary variables. 4.4 Proportional Allocation An intuitive method of allocating the sample to the strata is via proportional allocation, where the stratum sample proportion fh = nh/n is equal to the stra- tum population proportion Fh = Nh/N. This implies a stratum h sample size nh = nFh. Under proportional allocation the stratified expansion estimator (4.2) reduces to the simple expansion estimator. Note, however, that the prediction
- 51. 32 Stratified Populations Table 4.1 Population counts of 64 cities (in 1000s) in 1920 and 1930. Note that cities are arranged in the same order in both years. Z = 1920 population count Y = 1930 population count h = 1 h = 2 h = 1 h = 2 797 314 172 121 900 364 209 113 773 298 172 120 822 317 183 115 748 296 163 119 781 328 163 123 734 258 162 118 805 302 253 154 588 256 161 118 670 288 232 140 577 243 159 116 1238 291 260 119 507 238 153 116 573 253 201 130 507 237 144 113 634 291 147 127 457 235 138 113 578 308 292 100 438 235 138 110 487 272 164 107 415 216 138 110 442 284 143 114 401 208 138 108 451 255 169 111 387 201 136 106 459 270 139 163 381 192 132 104 464 214 170 116 324 180 130 101 400 195 150 122 315 179 126 100 366 260 143 134 variance of the stratified estimator is still based on (4.3) and is not the pre- diction variance (3.5) of the simple expansion estimator under a homogeneous population model. For example, consider the following population, taken from Cochran (1977, page 94). This population consists of 64 cities in the USA, with the variable of interest, Y , being their 1930 population counts (in 1000s). The total of these counts (which would, in practice, not be known by the sampler) is ty = 19, 568. It is assumed that the sampler knows the corresponding 1920 population counts for these cities, which we denote by Z, and can use this information for stratifying the population. The total of these ‘auxiliary’ counts is 16,290. Values of the 1920 and 1930 counts for the 64 cities are shown in Table 4.1. Consider two different approaches to stratification of this population, both resulting in two strata: 1. Put the 16 cities with the largest values of Z (1920 population counts) into one stratum and the remaining 48 cities into another. Call this size strat- ification. The two size strata are shown in Table 4.1, with stratum h = 1 containing the cities with the largest 1920 population counts, and stratum h = 2 containing the remainder. 2. Randomly allocate 16 of the 64 cities to stratum 1 and the remaining 48 to stratum 2. The Y -values (1930 population counts) of the 16 randomly chosen
- 52. Proportional Allocation 33 cities making up stratum 1 are shown in italic boldface in Table 4.1. Call this random stratification. Let Uh denote summation over all the population units in stratum h. We assume the homogenous strata model, (4.1). The strata variance parameters σ2 h are approximately equal to σ̂2 h = S2 yh = (Nh − 1)−1 Uh (yi − ȳh)2 where ȳh denotes the average value of Y in stratum h. Under size stratification, S2 y1 = 53, 843, while S2 y2 = 5, 581. On the other hand, under random stratification, S2 y1 = 52, 144 with S2 y2 = 53, 262. Since the population variance S2 y is 52,448, it is clear that the strata defined via size stratification are internally less variable (or at least stratum 2 is) than the overall population. This is not the case for the strata defined by random stratification, where we see no real reduction in variability within either stratum compared to that of the population as a whole. How can we assess the impact of reduced within-strata variability brought about by size stratification? Consider taking a sample of n = 16 cities from this population for the purpose of estimating the total 1930 count of the 64 cities. Suppose we assume that these cities are homogeneous with respect to their 1930 counts, take a simple random sample from all 64 (i.e. ignore stratification), and use the simple expansion estimator (3.2) to generate our estimate. The the- ory developed in Section 3.4 can then be applied, with the population variance S2 y substituted for the corresponding Y -variance σ2 , to show that the predic- tion variance (3.5) of the simple expansion estimator based on such a sample is approximately: Var t̂E y − ty ≈ N2 n 1 − n N S2 y = 642 16 1 − 16 64 × 52, 448 = 10, 070, 010 under the homogenous population model (3.1). Now, suppose that instead we use stratified sampling with our two size strata, with proportional allocation. The population sizes of the two strata are N1 = 16 and N2 = 48, so we obtain the following sample sizes: n1 = nN1/N = 16 × 16/64 = 4 n2 = nN2/N = 16 × 48/64 = 12 So, the variance of the stratified expansion estimator would be approximately equal to Var t̂S y − ty ≈ h N2 h/nh (1 − nh/Nh)S2 h = 162 /4 (1 − 4/16) × 53, 843 + 482 /12 (1 − 12/16) × 5, 581 = 3, 388, 113
- 53. 34 Stratified Populations assuming the homogenous strata population model (4.1) holds, where strata are given by our two size strata. Alternatively, we might consider stratified sampling with our two random strata, with proportional allocation. The population sizes of the two strata are still N1 = 16 and N2 = 48, so we again obtain n1 = 4 and n2 = 12. The variance of our estimator can then be approximated by: Var t̂S y − ty ≈ h N2 h/nh (1 − nh/Nh)S2 h = 162 /4 (1 − 4/16) × 52, 144 + 482 /12 (1 − 12/16) × 53, 262 = 10, 172, 572 assuming the homogenous strata population model (4.1) holds, where strata are now given by our two random strata. We can compare the three variance approximations we have obtained to give an indication of the relative efficiencies of the three designs. This comparison suggests that for the Cities’ population and a sample size of n = 16, size stratifi- cation with two strata and proportional allocation is about three times as efficient as random stratification with two strata and the same allocation. In fact, the latter sample design is virtually equivalent (in terms of prediction variance) to not stratifying at all. (We should note that these three variances are not strictly comparable as they are based on different models. Ideally we should have decided on a best model, and compared the relative performance of the three designs and estimators based on this model, but this would have been much more complex.) It follows that there are considerable gains to be had in stratifying so that the resulting strata are more homogeneous than the original population. 4.5 Optimal Allocation Can we do better than size stratification and proportional allocation? If our aim is to minimise the prediction variance of the stratified expansion estimator (4.2) subject to an overall sample size of n, that is h nh = n, the answer is yes. To see how, consider again the formula for the prediction variance (4.3) of this estimator. We see that it can be decomposed into two terms, Var t̂S y − ty = h N2 hσ2 h/nh − h Nhσ2 h Only the first term depends on the nh, and minimising Var t̂S y − ty is there- fore equivalent to choosing nh in order to minimise h N2 hσ2 h/nh subject to the restriction h nh = n. It can be shown (see exercise E.6) that this minimum occurs when nh ∝ Nhσh, which implies nh = nNhσh/ g Ngσg. (4.5) This optimal method of allocation is often referred to as Neyman Allocation, after Neyman (1934), whose fundamental paper gave the method wide prominence.
- 54. Allocation for Proportions 35 Notice that for two strata of the same size, (4.5) allocates a greater sample size to the more variable stratum. If the σ2 h is the same (or approximately the same) in each stratum then this method of allocation is equivalent to proportional allocation. Applying the method to the Cities’ population, and again substituting stra- tum variances for Y -variances, we see that for size stratification N1Sy1 = 3717 and N2Sy2 = 3586, and hence an optimal allocation corresponds to n1 = 8.1 and n2 = 7.9 which would be rounded to n1 = n2 = 8. On the other hand, for random stratification N1Sy1 = 3654 and N2Sy2 = 11078, leading to an optimal allocation defined by n1 = 4 and n2 = 12, that is proportional allocation. This result is hardly surprising given that the stratum variances in the two random strata are approximately the same and equal to the overall population variance. Under optimal allocation, it can easily be calculated that the prediction vari- ance (4.3) of the stratified expansion estimator (with σ2 h replaced by S2 yh ) for the Cities’ population is 2,200,908 under size stratification and 10,172,572 under random stratification. Comparing these figures with those for proportional allo- cation, we note a further improvement in precision under size stratification but not under random stratification. Both these results are consistent with the theory developed above. The formula (4.5) is actually a special case of a more general optimal allo- cation formula, which minimises the prediction variance (4.3) subject to a fixed survey budget rather than a fixed sample size, where the cost is assumed to be a linear combination of the stratum sample sizes. See exercise E.8. In the case of (4.5) there is the implicit assumption that there is no cost differential when sampling in different strata. True optimal allocation assumes knowledge of the variances σ2 h. In practice, of course, these quantities will not be known. However, estimates of them can often be obtained either from a preliminary pilot study of the population, or, since it is just the relative sizes of these variances between the strata that are needed, we can assume that these are unchanged from past studies of the same population, or are the same as the relative sizes of the stratum variances of another variable whose values are known for all population elements. For example, in the case of the Cities’ population, we know the 1920 counts for all 64 cities, and can base an ‘optimal’ allocation on these counts. See exercise E.7 for an examination of the efficiency of such an approach. 4.6 Allocation for Proportions An important type of survey variable is one that takes the value one or zero depending on whether the corresponding population element has, or does not have, a particular characteristic. That is, Y can be modelled as a Bernoulli variable. The population mean of such a variable is the proportion of population elements with this characteristic, and its distribution satisfies
- 55. Random documents with unrelated content Scribd suggests to you:
- 56. that.” “Later,” adds Ries, “I learned that, the family being greatly in need, my father had been helpful to him on this occasion in every way.” A petition of Johann van Beethoven, offered before the death of his wife, describing his pitiable condition and asking aid from the Elector, has not been discovered; but the substance of it is found in a volume of “Geheime Staats-Protocolle” for 1787 in form following: Your Elec. Highness has taken possession of this petition. July 24, 1787 Court Musician makes obedient representation that he has got into a very unfortunate state because of the long-continued sickness of his wife and has already been compelled to sell a portion of his effects and pawn others and that he no longer knows what to do for his sick wife and many children. He prays for the benefaction of an advance of 100 rthlr. on his salary. No record is found in the Düsseldorf archives of any grant of aid to the distressed family; hence, so far as now appears, the only successful appeal for assistance was made to Franz Ries, then a young man of 32 years, who generously aided in “every way” his unfortunate colleague. Where then was the Breuning family? Where Graf Waldstein? To these questions the reply is that Beethoven was still unknown to them—a reply which involves the utter rejection of the chronology adopted by Dr. Wegeler, in his “Notizen,” of that part
- 57. Dr. Wegeler’s Chronology Corrected of the composer’s life. This mistake, if indeed it prove to be such, is one which has been adopted without hesitation by all who have written upon the subject. The reader here, for the first time, finds Wegeler’s account of Beethoven’s higher intellectual development and his introduction into a more refined social circle placed after, instead of before, the visit to Vienna; and his introduction to the Breunings and Waldstein dated at the time when the youth was developing into the man, and not at a point upon the confines of childhood and youth. This demands some explanation. The history of Beethoven’s Bonn life would be so sadly imperfect without the “Notizen” of Dr. Wegeler, which bear in every line such an impress of perfect candor and honesty, that they can be read only with feelings of gratefullest remembrance of their author and with fullest confidence in their authenticity. But no more in his case than in others can the reminiscences of an aged man be taken as conclusive evidence in regard to facts and occurrences of years long since past, when opposed to contemporary records, or involving confusion of dates. Some slight lapse of memory, misapprehension, or unlucky adoption of another’s mistake, may lead astray and be the abundant source of error. Still, it is only with great diffidence and extreme caution that one can undertake to correct an original authority so trustworthy as Dr. Wegeler. Such corrections must be made, however; for only by this can many a difficulty be removed. An error in the Doctor’s chronology might easily be occasioned by the long accepted false date of Beethoven’s birth, insensibly influencing his recollections; and certainly when Dr. Wegeler, Madame von Breuning and Franz Ries, all alike venerable in years as in character, sit together discussing in 1837-8 occurrences of 1785-8, with nothing to aid their memories or control their reminiscences but an old Court Calendar or two, they may well to some extent have confounded times and seasons in the vague and misty distance of so many years; the more easily because the error is one of but two or three years at most. Bearing upon the point in question is the fact
- 58. that Frau Karth—who distinctly remembers the death of Madame van Beethoven—has no recollections of the young Breunings and Waldstein until after that event. Some words of Dr. Wegeler in an unprinted letter to Beethoven (1825): “inasmuch as the house of my mother-in-law was more your domicile than your own, especially after you lost your noble mother,” seem to favor the usually accepted chronology: but if Beethoven was thus almost a member of the Breuning family as early as 1785 or 1786, how can the tone of the letter to Dr. Schaden be explained? Or how account for the fact, that, when he reached Bonn again and found his mother dying, and his father “in a very unfortunate state” and “compelled to sell a portion of his effects and pawn others and knew not what to do,” it was to Franz Ries he turned for aid? The good Doctor is certainly mistaken as to the time when Beethoven found Mæcenases in the Elector and Waldstein; why not equally so in relation to the Breuning family? If, now, his own account of his intimacy with the young musician— given in the preface to the “Notizen”—be examined, it will be found to strengthen what has just been said: “Born in Bonn in 1765, I became acquainted in 1782 with the twelve years old lad, who, however, was already known as an author, and lived in most intimate association with him uninterruptedly until September, 1787” (and still he could forget that friend’s absence in Vienna only a few months before), “when, to finish my medical studies, I visited the Vienna schools and institutions. After my return in October, 1789, we continued to live together in an equally cordial association until Beethoven’s later departure for Vienna towards the close of 1792, whither I also emigrated in October, 1794.” For more than two years, then, and just at this period, Dr. Wegeler was not in Bonn. Let still another circumstance be noted: Nothing has been discovered, either in the “Notizen” or elsewhere, which necessarily implies that Wegeler himself intimately knew the Breunings until after his return from Vienna in 1789; moreover, in those days, when the distinctions of rank were so strongly marked, it
- 59. A Year of Sadness and Gloom is, to say the least, exceedingly improbable, that the son of an immigrant Alsatian shoemaker should have obtained entrée upon the supposed terms of intimacy in a household in which the oldest child was some six years younger than himself, and which belonged to the highest social, if not titled rank, until he by the force of his talents, culture, and high character, had risen to its level. That, after so rising, the obscurity of his birth was forgotten and the only daughter became his wife, is alike honorable to both parties. It is unnecessary to pursue the point farther; the reader, having his attention drawn to it, will observe for himself the many less prominent, but strongly corroborating circumstances of the narrative, which confirm the chronology adopted in it. At all events it must stand until new and decisive facts against it be found.[38] “My journey cost me a great deal, and I have not the smallest hope of earning anything here. Fate is not propitious to me in Bonn.” In poverty, ill, melancholy, despondent, motherless, ashamed of and depressed by his father’s ever increasing moral infirmity, the boy, prematurely old from the circumstances in which he had been placed since his eleventh year, had yet to bear another “sling and arrow of outrageous fortune.” The little sister, now a year and a half old—but here is the notice from the “Intelligenzblatt”:—“Died, November 25, Margareth, daughter of the Court Musician Johann van Beethoven, aged one year.” And so faded the last hope that the passionate tenderness of Beethoven’s nature might find scope in the purest of all relations between the sexes—that of brother and sister. Thus, in sadness and gloom, Beethoven’s seventeenth year ended.
- 60. Beethoven’s Friends: The von Breunings Chapter VII The von Breuning Family—Beethoven Brought Under Refining Influences—Count Waldstein, His Mæcenas—The Young Musician is Forced to Become Head of the Family. In 1527, the year in which the administration of the office of Hochmeister of the Teutonic Order was united with that of the Deutschmeister, whose residence had already been fixed at Mergentheim in 1525, this city became the principal seat of the order. From 1732 to 1761 Clemens Augustus was Hoch- und Deutschmeister of the order; according to the French edition of the Court Calendar of 1761, Christoph von Breuning was Conseiller d’État et Référendaire, having succeeded his father-in-law von Mayerhofen in the office. Christoph von Breuning had five sons: Georg Joseph, Johann Lorenz, Johann Philipp, Emanuel Joseph and Christoph. Lorenz became chancellor of the Archdeanery of Bonn, and the Freiadliges Stift at Neuss; after the death of his brother Emanuel he lived in Bonn so that, as head of the family, he might care for the education of the latter’s children. He died there in 1796. Johann Philipp, born 1742 at Mergentheim, became canon and priest at Kerpen, a place on the old highway from Cologne to Aix-la-Chapelle, where he died June 12, 1831. Christoph was court councillor at Dillingen. Emanuel Joseph continued in the electoral service at Bonn; at the early age of 20 years he was already court councillor (Conseiller actuel). He married Hélène von Kerich, born January 3, 1750, daughter of Stephan von Kerich, physician to the elector. Her brother, Abraham von Kerich, canon and scholaster of the archdeanery of Bonn, died in Coblenz in 1821. A high opinion of the intellect and character of Madame von Breuning is enforced upon us
- 61. by what we learn of her influence upon the youthful Beethoven. Court Councillor von Breuning perished in a fire in the electoral palace on January 15, 1777. The young widow (she had barely attained her 28th year), continued to live in the house of her brother, Abraham von Kerich, with her three children, to whom was added a fourth in the summer of 1777. Immediately after the death of the father, his brother, the canon Lorenz von Breuning, changed his residence from Neuss to Bonn and remained in the same house as guardian and tutor of the orphaned children. These were: 1. Christoph, born May 13, 1771, a student of jurisprudence at Bonn, Göttingen and Jena, municipal councillor in Bonn, notary, president of the city council, professor at the law school in Coblenz, member of the Court of Review in Cologne, and, finally, Geheimer Ober- Revisionsrath in Berlin. He died in 1841. 2. Eleonore Brigitte, born April 23, 1772. On March 28, 1802, she was married to Franz Gerhard Wegeler of Beul-an-der-Ahr, and died on June 13, 1841, at Coblenz. 3. Stephan, born August 17, 1774. He studied law at Bonn and Göttingen, and shortly before the end of the electorship of Max Franz was appointed to an office in the Teutonic Order at Mergentheim. In the spring of 1801 he went to Vienna, where he renewed his acquaintance with Beethoven. They had simultaneously been pupils of Ries in violin playing. The Teutonic Order offering no chance of advancement to a young man, he was given employment with the War Council and became Court Councillor in 1818. He died on June 4, 1827. His first wife was Julie von Vering, daughter of Ritter von Vering, a military physician; she died in the eleventh month of her wedded life. He then married Constanze Ruschowitz, who became the mother of Dr. Gerhard von Breuning, born August 28, 1813, author of “Aus dem Schwarzspanierhause.” 4. Lorenz (called Lenz, the posthumous child), born in the summer of 1777, studied medicine and was in Vienna in 1794-97 simultaneously with Wegeler and Beethoven. He died on April 10, 1798 in Bonn.[39]
- 62. Madame von Breuning, who died on December 9, 1838, after a widowhood of 61 years, lived in Bonn until 1815, then in Kerpen, Beul-an-der-Ahr, Cologne and finally with her son-in-law, Wegeler, in Coblenz. The acquaintance between Beethoven and Stephan von Breuning may have had some influence in the selection of the young musician as pianoforte teacher for Eleonore and Lorenz,[40] an event (in consideration of circumstances already detailed and of the ages, real and reputed, of pupils and master) which may be dated at the close of the year 1787, and which was, perhaps, the greatest good that fate, now become propitious, could have conferred upon him; for he was now so situated in his domestic relations, and at such an age, that introduction into so highly refined and cultivated a circle was of the highest value to him both morally and intellectually. The recent loss of his mother had left a void in his heart which so excellent a woman as Madame von Breuning could alone in some measure fill. He was at an age when the evil example of his father needed a counterbalance; when the extraordinary honors so recently paid to science and letters at the inauguration of the university would make the strongest impression; when the sense of his deficiencies in everything but his art would begin to be oppressive; when his mental powers, so strong and healthy, would demand some change, some recreation, from that constant strain in the one direction of music to which almost from infancy they had been subjected; when not only the reaction upon his mind of the fresh and new intellectual life now pervading Bonn society, but his daily contact with so many of his own age, friends and companions now enjoying advantages for improvement denied to him, must have cost him many a pang; when a lofty and noble ambition might be aroused to lead him ever onward and upward; when, the victim of a despondent melancholy, he might sink into the mere routine musician, with no lofty aims, no higher object than to draw from his talents means to supply his necessities and gratify his appetites.
- 63. There must have been something very engaging in the character of the small, pockmarked youth, or he could not have so won his way into the affections of the Widow von Breuning and her children. In his “Notizen” Wegeler writes: In this house reigned an unconstrained tone of culture in spite of youthful wilfulness. Christoph von Breuning made early essays in poetry, as was the case (and not without success) with Stephan von Breuning much later. The friends of the family were distinguished by indulgence in social entertainments which combined the useful and the agreeable. When we add that the family possessed considerable wealth, especially before the war, it will be easy to understand that the first joyous emotions of Beethoven found vent here. Soon he was treated as one of the children of the family, spending in the house not only the greater part of his days, but also many nights. Here he felt that he was free, here he moved about without constraint, everything conspired to make him cheerful and develop his mind. Being five years older than Beethoven I was able to observe and form a judgment on these things. It must not be forgotten that besides Madame von Breuning and her children the scholastic Abraham von Kerich and the canon Lorenz von Breuning were members of the household. The latter especially seems to have been a fine specimen of the enlightened clergy of Bonn who, according to Risbeck, formed so striking a contrast to the priests and monks of Cologne; and it is easy to trace Beethoven’s life-long love for the ancient classics—Homer and Plutarch at the head—to the time when the young Breunings would be occupied with them in the original under the guidance of their accomplished tutor and guardian. The uncle, Philipp von Breuning, may also have been influential in the intellectual progress of the young musician, for to him at Kerpen “the family von Breuning and their friends went annually for a vacation of five or six weeks. There, too, Beethoven several times spent a few weeks right merrily, and was frequently
- 64. Count Waldstein’s Arrival in Bonn urged to play the organ,” as Wegeler tells us in the “Notizen.” There let him be left enjoying and profiting by his intimacy with that family, and returning their kindness in some measure by instructing Eleonore and Lenz in music, while a new friend and benefactor is introduced. Emanuel Philipp, Count Waldstein and Wartemberg von Dux, and his wife, a daughter of Emanuel Prince Lichtenstein, were parents of eleven children. The fourth son was Ferdinand Ernst Gabriel, born March 24, 1762. Uniting in his veins the blood of many of the houses of the Austrian Empire, there was no career, no line of preferment open to younger sons of titled families, which was not open to him, or to which he might not aspire. It was determined that he should seek activity in the Teutonic Order, of which Max Franz was Grand Master. According to the rules and regulations of the order, the young nobleman came to Bonn to pass his examinations and spend his year of novitiate. Could the time of his arrival there be determined with certainty, the date would have a most important bearing either to confirm or disprove the chronological argument of some of our earlier pages; but one may well despair of finding so unimportant an event as the journey of a young man of 25 from Vienna to the Rhine anywhere upon record. One thing bearing directly upon this point may be read in the “Wiener Zeitung” of July 2, 1788. A correspondent in Bonn says that on “the day before yesterday,” i.e., June 17, 1788, “our gracious sovereign, as Hoch- und Deutschmeister, gave the accolade with the customary ceremonies to the Count von Waldstein, who had been accepted in the Teutonic Order.” Allowing for the regular year of novitiate, the Count was certainly in Bonn before the 17th of June, 1787. The misfortune of two unlucky Bohemian peasants, strange as it may seem, gives us, after the lapse of a century, a satisfactory solution of the difficulty. Some one reports in the “Wiener Zeitung” of May, 19, 1787, that on the 4th of that month two peasant houses were destroyed by fire in the village of Likwitz belonging to Osegg, and adds: “Count Ferdinand von Waldstein, moved by a noble spirit
- 65. of humanity, hurried from Dux, took charge of affairs and was to be found wherever the danger was greatest.” It was between May 4 and June 17, 1787, that Waldstein parted from his widowed mother and journeyed to the place of his novitiate. His name may easily have become known to Wegeler before the latter’s departure from Bonn for Vienna.[41] Here follows what the good doctor says of the Count—to what degree correct or mistaken, the reader can determine for himself: The first, and in every respect the most important, of the Mæcenases of Beethoven was Count Waldstein, Knight of the Teutonic Order, and (what is of greater moment here) the favorite and constant companion of the young Elector, afterwards Commander of the Order at Virnsberg and Chancellor of the Emperor of Austria. He was not only a connoisseur but also a practitioner of music. He it was who gave all manner of support to our Beethoven, whose gifts he was the first to recognize worthily. Through him the young genius developed the talent to improvise variations on a given theme. From him he received much pecuniary assistance bestowed in such a way as to spare his sensibilities, it being generally looked upon as a small gratuity from the Elector. Beethoven’s appointment as organist, his being sent to Vienna by the Elector, were the doings of the Count. When Beethoven at a later date dedicated the great and important Sonata in C major, Op. 53, to him, it was only a proof of the gratitude which lived on in the mature man. It is to Count Waldstein that Beethoven owed the circumstance that the first sproutings of his genius were not nipped; therefore we owe this Mæcenas Beethoven’s later fame. Frau Karth remembered distinctly the 17th of June upon which Waldstein entered the order, the fact being impressed upon her mind by a not very gentle reminder from the stock of a sentinel’s musket
- 66. Ludwig the Head of the Family that the palace chapel was no place for children on such an occasion. She remembered Waldstein’s visits to Beethoven in the years following in his room in the Wenzelgasse and was confident that he made the young musician a present of a pianoforte. To save his line from extinction the Count obtained a dispensation from his vows and married (May 9, 1812) Maria Isabella, daughter of Count Rzewski. A daughter, Ludmilla, was born to him; but no son. He died on August 29, 1823, and the family of Waldsteins of Dux disappears. While all that Wegeler says of this man’s kindness in obtaining the place of organist for Beethoven and of his influence upon his musical education is one grand mistake,[42] there is no reason whatever to doubt that those qualities which made the youth a favorite with the Breunings, added to his manifest genius, made their way to the young count’s heart and gained for Beethoven a zealous, influential and active friend. Still, in June, 1778, Waldstein possessed no such influence as to render a petition for increase of salary, offered by his protégé, successful. That document has disappeared, but a paper remains, dated June 5, concerning the petition, which is endorsed “Beruhet.” Whatever this word may here mean it is certain that Ludwig’s salary as organist remained at the old point of 100 thalers, which, with the 200 received by his father, the three measures of grain and the small sum that he might earn by teaching, was all that Johann van Beethoven and three sons, now respectively in their eighteenth, fifteenth and twelfth years, had to live upon; and therefore so much the more necessity for the exercise of Waldstein’s generosity. After the death of the mother, says Frau Karth, a housekeeper was employed and the father and sons remained together in the lodgings in the Wenzelgasse. Carl was intended for the musical profession; Johann was put apprentice to the court apothecary, Johann Peter Hittorf. Two years, however, had hardly elapsed when the father’s infirmity compelled the eldest son, not yet nineteen years of age, to take the extraordinary step of placing himself at the head of the family. One
- 67. of Stephan von Breuning’s reminiscences shows how low Johann van Beethoven had sunk: viz., that of having seen Ludwig furiously interposing to rescue his intoxicated father from an officer of police. Here again the petition has disappeared, but its contents are sufficiently made known by the terms of the decree dated November 20, 1789: His Electoral Highness having graciously granted the prayer of the petitioner and dispensed henceforth wholly with the services of his father, who is to withdraw to a village in the electorate, it is graciously commanded that he be paid in accordance with his wish only 100 rthr. of the annual salary which he has had heretofore, beginning with the approaching new year, and that the other 100 thlr. be paid to the suppliant’s son besides the salary which he now draws and the three measures of grain for the support of his brothers. It is probable that there was no intention to enforce this decree in respect of the withdrawal of the father from Bonn, and that this clause was inserted in terrorem in case he misbehaved himself; for he continued, according to Frau Karth, to dwell with his children, and his first receipt, still preserved, for the reduced salary is dated at Bonn—a circumstance, however, which alone would prove little or nothing.
- 68. Opera under Elector Max Franz Chapter VIII The National Theatre of Max Franz—Beethoven’s Artistic Associates—Practical Experience in the Orchestra—The “Ritterballet”—The Operatic Repertory of Five Years. Early in the year 1788, the mind of the Elector, Max Franz, was occupied with the project for forming a company of Hofschauspieler; in short, with the founding of a National Theatre upon the plan adopted by his predecessor in Bonn and by his brother Joseph in Vienna. His finances were now in order, the administration of public affairs in able hands and working smoothly, and there was nothing to hinder him from placing both music and theatre upon a better and permanent footing; which he now proceeded to do. The Klos troupe, which had left Cologne in March, played for a space in Bonn, and on its dispersal in the summer several of its better actors were engaged and added to others who had already settled in Bonn. The only names which it is necessary to mention here are those of significance in the history of Beethoven. Joseph Reicha was director; Neefe, pianist and stage-manager for opera; in the orchestra were Franz Ries and Andreas Romberg (violin), Ludwig van Beethoven (viola), Bernard Romberg (violoncello), Nicolaus Simrock (horn) and Anton Reicha (flute). A comparison of the lists of the theatrical establishment with that of the court chapel as printed in the Court Calendars for 1778 and the following years, shows that the two institutions were kept distinct, though the names for the greater part appear in both. Some of the singers in the chapel played in the theatrical orchestra, while certain of the players in the chapel sang upon the stage. Other names appear in but one of the lists. As organist the name of Beethoven appears still in the Court Calendar, but as viola player he had a place in both the orchestras.
- 69. Thus, for a period of full four years, he had the opportunity of studying practically orchestral compositions in the best of all schools —the orchestra itself. This body of thirty-one members, under the energetic leadership of Reicha, many of them young and ambitious, some already known as virtuosos and still keeping their places in musical history as such, was a school for instrumental music such as Handel, Bach, Mozart and Haydn had not enjoyed in their youth; that its advantages were improved both by Beethoven and others of the younger men, all the world knows. One fact worthy of note in relation to this company is the youth of most of the new members engaged. Maximilian seems to have sought out young talent, and when it proved to be of true metal, gave it a permanent place in his service, adopted wise measures for its cultivation, and thus laid a foundation upon which, but for the outbreak of the French Revolution, and the consequent dispersion of his court, would in time have risen a musical establishment, one of the very first in Germany. This is equally true of the new members of his orchestra. Reicha himself was still rather a young man, born in 1757. He was a virtuoso on the violoncello and a composer of some note; but his usefulness was sadly impaired by his sufferings from gout. The cousins Andreas and Bernhard Romberg, Maximilian had found at Münster and brought to Bonn. They had in their boyhood, as virtuosos upon their instruments—Andreas violin, Bernhard ’cello— made a tour as far as Paris, and their concerts were crowned with success. Andreas was born near Münster in 1767, and Ledebur (“Tonkünstler Berlins”) adopts the same year as the date also of Bernhard’s birth. They were, therefore, three years older than Beethoven and now just past 21. Both were already industrious and well-known composers and must have been a valuable addition to the circle of young men in which Beethoven moved. The decree appointing them respectively Court Violinist and Court Violoncellist is dated November 19, 1790.
- 70. Anton Reicha, a fatherless nephew of the concertmaster, born at Prague, February 27, 1770, was brought by his uncle to Bonn. He had been already for some years in that uncle’s care and under his instruction had become a good player of the flute, violin and pianoforte. In Bonn, Reicha became acquainted with Beethoven, who was then organist at court. “We spent fourteen years together,” says Reicha, “united in a bond like that of Orestes and Pylades, and were continually side by side in our youth. After a separation of eight years we saw each other again in Vienna, and exchanged confidences concerning our experiences.” At the age of 17 composing orchestral and vocal music for the Electoral Chapel, a year later flautist in the theatre, at nineteen both flautist and violinist in the chapel and so intimate a friend of Beethoven, who was less than a year his junior—were Reicha’s laurels no spur to the ambition of the other? The names of several of the performers upon wind-instruments were new names in Bonn, and the thought suggests itself that the Elector brought with him from Vienna some members of the Harmoniemusik which had won high praise from Reichardt, and it will hereafter appear that such a band formed part of the musical establishment in Bonn—a fact of importance in its bearing upon the questions of the origin and date of various known works both of Beethoven and of Reicha, and of no less weight in deciding where and how these men obtained their marvellous knowledge of the powers and effects of this class of instruments. The arrangements were all made in 1788, but not early enough to admit of the opening of the theatre until after the Christmas holidays, namely, on the evening of January 3, 1789. The theatre had been altered and improved. An incendiary fire threatened its destruction the day before, but did not postpone the opening. The opening piece was “Der Baum der Diana” by Vincenzo Martin. It may be thought not very complimentary to the taste of Maximilian that the first season of his National Theatre was opened thus, instead of with one of Gluck’s or Mozart’s masterpieces. It suffices to say that he, in his capacity of Grand Master of the Teutonic Order, had spent
- 71. a good part of the autumn at Mergentheim and only reached Bonn on his return on the last day of January. Hence he was not responsible for that selection. The season which opened on January 3, 1789, closed on May 23. Within this period the following operas were performed, Beethoven taking part in the performances as a member of the orchestra: “Der Baum der Diana” (L’Arbore di Diana), Martin; “Romeo und Julie,” Georg Benda; “Ariadne” (duo-drama by Georg Benda); “Das Mädchen von Frascati” (La Frascatana), Paisiello; “Julie,” Desaides; “Die drei Pächter” (Les trois Fermiers), Desaides; “Die Entführung aus dem Serail,” Mozart; “Nina,” Dalayrac; “Trofonio’s Zauberhöhle” (La grotta di Trofonio), Salieri; “Der eifersüchtige Liebhaber” (L’Amant jaloux), Grétry; “Der Schmaus” (Il Convivo), Cimarosa; “Der Alchymist,” Schuster; “Das Blendwerk” (La fausse Magie), Grétry. The second season began October 13, 1789, and continued until February 23, 1790. On the 24th of February news reached Bonn of the death of Maximilian’s brother, the Emperor Joseph II, and the theatre was closed. The repertory for the season comprised “Don Giovanni,” Mozart (which was given three times); “Die Colonie” (L’Isola d’Amore), Sacchini; “Der Barbier von Sevilla” (Il Barbiere di Siviglia), Paisiello; “Romeo und Julie,” Georg Benda; “Die Hochzeit des Figaro” (Le Nozze di Figaro), Mozart (given four times); “Nina,” Dalayrac; “Die schöne Schusterin,” Umlauf; “Ariadne,” Georg Benda; “Die Pilgrimme von Mecca,” Gluck; “Der König von Venedig” (Il Re Teodoro), Paisiello; “Der Alchymist,” Schuster; “Das listige Bauernmädchen” (La finta Giardiniera), Paisiello; “Der Doktor und Apotheker,” Dittersdorf. A letter to the “Berliner Annalen des Theaters” mentions three operas which are not in the list of the theatrical calendar and indicates that the theatre was opened soon after receipt of the intelligence of the death of Joseph, and several pieces performed, among them Il Marchese Tulipano by Paisiello. The writer also mentions performances of Anfossi’s (or Sarti’s) Avaro inamorato, Pergolese’s Serva padrona and La Villanella di spirito,
- 72. composer unmentioned, by an Italian company headed by Madame Bianchi. The third season began October 23, 1790, and closed on March 8, 1791. Between the opening and November 27, performances of the following musical-dramatic works are recorded: “König Theodor in Venedig” (Il Re Teodoro), Paisiello; “Die Wilden” (Azemia), Dalayrac; “Der Alchymist,” Schuster; “Kein Dienst bleibt unbelohnt,” (?); “Der Barbier von Sevilla,” Paisiello; “Die schöne Schusterin,” Umlauf; “Lilla,” Martini; “Die Geitzigen in der Falle,” Schuster; “Nina,” Dalayrac; “Dr. Murner,” Schuster. On March 8, the season closed with a ballet by Horschelt, “Pyramus und Thisbe.” The reporter in the “Theaterkalender” says: On Quinquagesima Sunday (March 6) the local nobility performed in the Ridotto Room a characteristic ballet in old German costume. The author, His Excellency Count Waldstein, to whom the composition and music do honor, had shown in it consideration for the chief proclivities of our ancestors for war, the chase, love and drinking. On March 8, all the nobility attended the theatre in their old German dress and the parade made a great, splendid and respectable picture. It was also noticeable that the ladies would lose none of their charms, were they to return to the costumes of antiquity. Before proceeding with this history a correction must be made in this report: the music to the “Ritterballet,” which was the characteristic ballet referred to, was not composed by Count Waldstein but by Ludwig van Beethoven. We shall recur to it presently. Owing to a long-continued absence of the Elector, the principal singers and the greater part of the orchestra, the fourth season did not begin till the 28th of December, 1791. Between that date and February 20, 1792, the following musical works were performed: “Doktor und Apotheker,” Dittersdorf; “Robert und Caliste,” Guglielmi; “Félix,” Monsigny; “Die Dorfdeputirten,” Schubauer; “Im Trüben ist gut Fischen” (Fra due Litiganti, il Terzo gode), Sarti; “Das rothe
- 73. Operas at Bonn in 1792 Käppchen,” Dittersdorf; “Lilla,” Martini; “Der Barbier von Sevilla,” Paisiello; “Ende gut, Alles gut,” music by the Electoral Captain d’Antoin; “Die Entführung aus dem Serail,” Mozart; “Die beiden Savoyarden” (Les deux petits Savoyards), Dalayrac. The fifth season began in October, 1792. Of the nine operas given before the departure of Maximilian and the company to Münster in December, “Die Müllerin” by De la Borde, “König Axur in Ormus” by Salieri, and “Hieronymus Knicker” by Dittersdorf, were the only ones new to Bonn; and in only the first two of these could Beethoven have taken part, unless at rehearsals; for at the beginning of November he left Bonn—and, as it proved, forever. Probably Salieri’s masterpiece was his last opera within the familiar walls of the Court Theatre of the Elector of Cologne. Beethoven’s eighteenth birthday came around during the rehearsals for the first season, of this theatre; his twenty-second just after the beginning of the fifth. During four years (1788-1792) he was adding to his musical knowledge and experience in a direction wherein he has usually been represented as deficient—as active member of an operatic orchestra; and the catalogue of works performed shows that the best schools of the day, save that of Berlin, must have been thoroughly mastered by him in all their strength and weakness. Beethoven’s titanic power and grandeur would have marked his compositions under any circumstances; but it is very doubtful if, without the training of those years in the Electoral “Toxal, Kammer und Theater” as member of the orchestra, his works would have so abounded in melodies of such profound depths of expression, of such heavenly serenity and repose and of such divine beauty as they do, and which give him rank with the two greatest of melodists, Handel and Mozart.
- 74. Chapter IX Gleanings of Musical Fact and Anecdote—Haydn in Bonn— A Rhine Journey—Abbé Sterkel—Beethoven Extemporises —Social and Artistic Life in Bonn—Eleonore von Breuning —The Circle of Friends—Beethoven Leaves Bonn Forever— The Journey to Vienna. As a pendant to the preceding sketches of Bonn’s musical history a variety of notices belonging to the last three years of Beethoven’s life in his native place are here brought together in chronological order. Most of them relate to him personally, and some of them, through errors of date, have been looked upon hitherto as adding proofs of the precocity of his genius. Prof. Dr. Wurzer communicated to the “Kölnische Zeitung” of August 30, 1838, the following pleasant anecdote: In the summer of the year 1790 or 1791 I was one day on business in Godesberger Brunnen. After dinner Beethoven and another young man came up. I related to him that the church at Marienforst (a cloister in the woods behind Godesberg) had been repaired and renovated, and that this was also true of the organ, which was either wholly new or at least greatly improved. The company begged him to give them the pleasure of letting them hear him play on the instrument. His great good nature led him to grant our wish. The church was locked, but the prior was very obliging and had it unlocked for us. B. now began to play variations on themes given him by the party in a manner that moved us profoundly; but what was much more significant, poor laboring folk who were cleaning out the débris left by the work of repair, were so greatly affected by the music that they put down their
- 75. Joseph Haydn’s Visit to Bonn implements and listened with obvious pleasure. Sit ei terra levis! The greatest musical event of the year (1790) in Bonn occurred just at its close—the visit of Joseph Haydn, on his way to London with Johann Peter Salomon, whose name so often occurs in the preliminary chapters of this work. Of this visit, Dies has recorded Haydn’s own account: In the capital, Bonn, he was surprised in more ways than one. He reached the city on Saturday [Christmas, December 25] and set apart the next day for rest. On Sunday, Salomon accompanied Haydn to the court chapel to listen to mass. Scarcely had the two entered the church and found suitable seats when high mass began. The first chords announced a product of Haydn’s muse. Our Haydn looked upon it as an accidental occurrence which had happened only to flatter him; nevertheless it was decidedly agreeable to him to listen to his own composition. Toward the close of the mass a person approached and asked him to repair to the oratory, where he was expected. Haydn obeyed and was not a little surprised when he found that the Elector, Maximilian, had had him summoned, took him at once by the hand and presented him to the virtuosi with the words: “Here I make you acquainted with the Haydn whom you all revere so highly.” The Elector gave both parties time to become acquainted with each other, and, to give Haydn a convincing proof of his respect, invited him to dinner. This unexpected invitation put Haydn into an embarrassing position, for he and Salomon had ordered a modest little dinner in their lodgings, and it was too late to make a change. Haydn was therefore fain to take refuge in excuses which the Elector accepted as genuine and sufficient. Haydn took his leave and returned to his lodgings, where he was made aware in a special manner of the good will of the Elector, at whose secret command
- 76. Welcome to our website – the perfect destination for book lovers and knowledge seekers. We believe that every book holds a new world, offering opportunities for learning, discovery, and personal growth. That’s why we are dedicated to bringing you a diverse collection of books, ranging from classic literature and specialized publications to self-development guides and children's books. More than just a book-buying platform, we strive to be a bridge connecting you with timeless cultural and intellectual values. With an elegant, user-friendly interface and a smart search system, you can quickly find the books that best suit your interests. Additionally, our special promotions and home delivery services help you save time and fully enjoy the joy of reading. Join us on a journey of knowledge exploration, passion nurturing, and personal growth every day! ebookbell.com