![1 Introduction
1.1 Motivation
Dependable Distributed Systems Distributed systems have become crucial in
most application domains of computing systems. Their success stems from, for
example, being more cost-efficient, more powerful, and more scalable than stand-
alone systems. Distributed systems are characterised by consisting of active com-
ponents that are spatially distributed and share information via some means of
communication. The components are active in the sense that they perform com-
putations that are of interest for the user of the system. The range of the compo-
nents’ spatial distribution is wide, from micro-scale embedded systems-on-a-chip
to globally – and possibly further – distributed systems.
As distributed systems more and more pervade our daily life, we increasingly
depend on their correct service. Many distributed systems deliver critical services,
whose failures are not acceptable. Prominent examples include air traffic control
systems, power grid systems, and patient monitoring systems. Failures of such
systems may entail excessive costs or even cause loss of life. Hence, the depend-
ability of distributed systems is essential for their acceptance in the future.
Dependability is a fundamental property of computing systems besides, for ex-
ample, performance and costs. Avižienis et al. [2004] have spent significant effort
on a comprehensive dependability taxonomy over the past two decades. They
describe dependability as the ability of a system to deliver service that can jus-
tifiably be trusted. Dependability is a general concept that subsumes different
more specific attributes such as availability, reliability, and integrity. For exam-
ple, availability deals with the “readiness for correct service,” reliability with the
“continuity of correct service,” and integrity with the “absence of improper system
alterations.” These attributes are threatened by faults, errors, and failures. A fail-
ure is an event that occurs when the delivered service of a system deviates from
correct service. An error is a part of the system state that may lead to a failure. A
fault is the cause of an error.
Means of Fault Tolerance With raising needs for dependable distributed sys-
tems, the demand for means to attain dependability increases. Avižienis et al.
[2004] divide such means into four classes: fault forecasting, fault prevention, fault
T. Warns, Structural Failure Models for Fault-Tolerant Distributed Computing,
DOI 10.1007/978-3-8348-9707-7_1,
© Vieweg+Teubner Verlag | Springer Fachmedien Wiesbaden GmbH 2010](https://image.slidesharecdn.com/94575-250509155417-a47a89df/85/Structural-Failure-Models-for-Fault-Tolerant-Distributed-Computing-1st-Edition-Timo-Warns-Auth-19-320.jpg)
![2 1 Introduction
removal, and fault tolerance. Despite all efforts of prevention and removal, each
distributed system consists of components that are bound to fail eventually. Means
of fault tolerance allow to avoid that component failures – being system faults –
lead to system failures; a dependable system can be built from undependable com-
ponents. Ideally, a system should completely mask the occurrence of faults from
external observers of the system. However, other forms (e.g., remaining in a safe
state despite faults) may be acceptable while being more cost-efficient.
A prerequisite for any type of fault tolerance is some form of redundancy [Gärt-
ner, 1999]: A fault-tolerant design must incorporate some entities that are not
required per se to deliver the desired service. These entities are only added for the
sake of fault tolerance. A simple example is a checksum that is added to some
data for purposes of error detection and correction. Another example is to have an
algorithm executed by different processors and vote on the result to mask a failed
processor. The extent of such redundancy in a system determines the costs of the
system and the resilience of the system to faults.
The Need for Models While designing and evaluating a distributed system are
already complex tasks, fault tolerance makes the tasks even more complex. If
these tasks are not done properly, the fault tolerance mechanisms themselves can
become the source of failures. For example, Mackall [1988] has reported that the
fault detection logic in a flight-crucial control system caused failures of an aircraft:
Each of overall three communication channels declared the other two channels
as failed although no actual hardware failure occurred. Only a manual selection
of a backup system allowed to land the aircraft safely. Such examples illustrate
that designing and evaluating fault tolerance mechanisms are complex as well as
critical tasks and, therefore, require a rigorous treatment.
Models as means of abstraction are crucial for a rigorous treatment and are the
key to master the complexity of fault-tolerant distributed systems. For example,
Bolosky et al. [2007] have reported that formal modelling was essential for de-
signing the distributed directory service of a distributed file system. While this
task took several months and 19 design iterations, it would have required even
more time without formal modelling.
A model is an abstract representation of an object of interest. A good model
describes the relevant aspects of the object but abstracts away the irrelevant ones.
Schneider [1993a] explains that the challenge in finding a good model lies in find-
ing a model that is both, accurate and tractable. A model is accurate if evaluations
based on the model yield results that do not only hold for the model, but also for
the actual object. A model is tractable if evaluations are possible at all. A model](https://image.slidesharecdn.com/94575-250509155417-a47a89df/85/Structural-Failure-Models-for-Fault-Tolerant-Distributed-Computing-1st-Edition-Timo-Warns-Auth-20-320.jpg)
![1.1 Motivation 3
that is not accurate or not tractable is useless, because it yields invalid results or
does not allow to obtain any results.
Fault Assumptions No matter to which extent a fault-tolerant design incorpo-
rates redundancy, no system can tolerate arbitrary faults. If faults are too severe
or occur too frequently, then the redundancy required for fault tolerance will get
exhausted: The system fails. For example, if all components of a system fail with
an arbitrary behaviour, the system cannot behave as desired anymore. Therefore,
it is essential for the design and the evaluation of a fault-tolerant system to make a
fault assumption on the faults to tolerate.
Fault assumptions are represented by fault models that – for the sake of tractabil-
ity – rely on simplifying assumptions. For example, such assumptions include:
• At most components may fail.
• Component failures are identically distributed for all components.
• Component failures are stochastically independent.
• There is no propagation of failures among components.
Of course, such assumptions must be justified for the sake of accuracy. If a sys-
tem is designed or evaluated under invalid assumptions, the system may fail even
if its correctness has been formally verified. For example, Mackall [1988] has
reported that, for the flight-crucial control system mentioned above, dual simulta-
neous component failures had been ruled out as impossible. Hence, the design of
the system did not account for them. However, such faults did occur during a test-
ing phase. The assumption that at most one component fails was not valid leading
to a failure of the overall system.
While simplifications as listed above are frequently found in the literature, there
is empirical evidence that they are not valid in many real-world systems. For ex-
ample, Tang and Iyer [1992] evaluated two DEC VAX-cluster systems and found
correlated failures due to errors in shared resources. Dobson et al. [2004, 2005]
argue that large-scale blackouts of power grids are typically caused by propagating
failures. It is likely that propagating failures in power grids manifest as propagat-
ing failures in distributed systems that are connected to these grids.
Dependent and Propagating Faults Correlated and, therefore, dependent com-
ponent failures occur over the whole range of spatial distribution. For example,
failure correlation coefficients up to 0.92 have been found in the globally dis-
tributed PlanetLab system [Warns et al., 2008]. If the failures had been stochas-
tically independent, the coefficients would have been approximately equal to 0.](https://image.slidesharecdn.com/94575-250509155417-a47a89df/85/Structural-Failure-Models-for-Fault-Tolerant-Distributed-Computing-1st-Edition-Timo-Warns-Auth-21-320.jpg)
![4 1 Introduction
Likewise, Bakkaloglu et al. [2002] found correlated failures when measuring the
availability of globally distributed web servers. Amir and Wool [1996] have found
strongly correlated failures when evaluating quorum systems on 14 computers lo-
cated in two geographical sites with a 50 km distance between both sites. With
an increasing integration density of embedded systems, faults such as electric dis-
charges are more likely to affect several neighboured components at the same time
and cause dependent failures [Limbourg et al., 2007].
The impact of correlated failures is significant. Although the average failure cor-
relation coefficient in the study on PlanetLab is rather low with 0.06, a prediction
underestimates the probability that exactly one node fails (in the next 5 minutes)
by four orders of magnitude under the assumption that failures are independent.
Likewise, Yalagandula et al. [2004] have provided empirical evidence that corre-
lated failures significantly hurt the availability of a system. Tang and Iyer [1993]
have shown that this is the case even if the correlation coefficients are low.
Fault Models for Dependence and Propagation The relevance of dependent
and propagating component failures has raised the interest for suitable fault mod-
els. Fault models that address the extent of faults are called structural failure
models. Most of these models are probabilistic and, for example, rely on corre-
lation coefficients as input parameters. Some examples are the models of Tang
and Iyer [1992] and Bakkaloglu et al. [2002]. Other models cover dependences
by explicitly considering the causes for dependent faults. For example, Limbourg
et al. [2007] make the assumption that the spatial arrangement of components has
an impact on fault dependences. They construct a probabilistic fault model by
explicitly considering the spatial arrangement. Junqueira [2006] has presented a
model for dependent faults that associates a set of attributes to each process of a
distributed system. Intuitively, these attributes capture causes for process failures:
If an attribute “is activated”, all processes that have this attribute fail.
Despite such efforts, threshold models are predominant in the literature on fun-
damentals of fault-tolerant distributed computing. Describing the extent of faults
by a simple threshold eases the design and the evaluation of a system and allows
to refrain from a probabilistic system model. Considering probabilistic behaviour
introduces additional complexities.
As Keidar and Marzullo [2002] have criticised, threshold models only allow to
model identically distributed and independent faults. Due to the relevance of de-
pendence and propagation in practice, such phenomena deserve to be considered
when studying the fundamentals of fault-tolerant distributed computing. Hence,
there is a need for simple fault models that allow to describe dependent and prop-
agating faults.](https://image.slidesharecdn.com/94575-250509155417-a47a89df/85/Structural-Failure-Models-for-Fault-Tolerant-Distributed-Computing-1st-Edition-Timo-Warns-Auth-22-320.jpg)
![8 1 Introduction
1.4 Remarks on Notation
Structured Proofs We formulate our proofs as structured proofs to ease their
readability. As introduced by Lamport [1993], each structured proof is a sequence
of hierarchically numbered steps. Each step has a proof on its own that may include
additional steps on a lower level. The notation for the number of a step gives its
level and its rank within its level. For example, a step numbered 12 is the second
step on the first level. The hierarchical structure provides the general outline of the
proof on high levels; the details are on low levels. Readers who are not interested
in the details of a proof can skip the lower levels.
Bulleted Formulae List We use the bulleted-list notation of TLA [Lamport,
2002] for conjunctions and disjunctions to make complicated formulae more read-
able. A list of formulae that are bulleted with ∧ or ∨ equals the conjunction or
disjunction of the formulae. The ∧ or ∨ symbols in a bulleted-list must line up
exactly and indentation is used to eliminate (some) parentheses. For example, the
list
∨ = 42
∨∧ = 3
∧ = 3
∨ ≤ 200
denotes the formulae
= 42 ∨ ( = 3 ∧ = 3) ∨ ≤ 200.](https://image.slidesharecdn.com/94575-250509155417-a47a89df/85/Structural-Failure-Models-for-Fault-Tolerant-Distributed-Computing-1st-Edition-Timo-Warns-Auth-26-320.jpg)
![10 2 Modelling Fault-Tolerant Distributed Systems
Contribution In this chapter, we introduce our system model that forms the
foundation for the rest of the thesis. We give a brief overview of the different
concerns of modelling fault-tolerant distributed systems. Basic terms (e.g., state,
trace, and property) are formally defined for distributed systems and their compo-
nents. Note that our system model is not particularly new: Variants of this model
are common in the literature on fault-tolerant distributed computing. Therefore,
the results that are obtained under our model also hold in settings addressed by
previous models (or can be easily adapted). In particular, our system model is de-
rived from the models of Abadi and Lamport [1991, 1995], and Lamport [1994],
which also serve as the semantic model for TLA. Restricting their models, we
limit ourselves to interleaving properties and closed systems, which are sufficient
for our purposes.
Overview In Section 2.1, we describe how processes communicate by passing
messages over channels. Section 2.2 gives formal definitions for states, traces, and
properties that are used to describe the behaviour of distributed systems. TLA is
briefly summarised in Section 2.3. Fault models and fault tolerance are formally
defined in Section 2.4 and Section 2.5, respectively. Section 2.6 gives an overview
of different timing models, before we summarise this chapter in Section 2.7.
2.1 Interprocess Communication
Processes Distributed systems are generally described by process models that
represent a system by concurrent executions of sequential processes [Lamport and
Lynch, 1990]. Sequential processes model “active” entities of a system that per-
form computations. For example, a process may represent a computing system, a
processor, or an operating system thread depending on the distributed system to be
modelled. For our system model, we assume that a distributed system consists of
a finite set of processes, denoted by Π = {1,...,}, 1.
Interprocess Communication Different process models can be distinguished
by their style of interprocess communication. For example, processes may com-
municate via shared memory, remote procedure calls, or message passing. For
our system model, we assume communication via message passing. This style
of communication is fundamental for distributed system. For example, Lamport
and Lynch [1990] consider a process model to be distributed iff its interprocess
communication can be implemented using message passing.](https://image.slidesharecdn.com/94575-250509155417-a47a89df/85/Structural-Failure-Models-for-Fault-Tolerant-Distributed-Computing-1st-Edition-Timo-Warns-Auth-28-320.jpg)
![2.1 Interprocess Communication 11
(a) Complete graph (b) Directed Ring (c) Cube (d) Tree
Figure 2.1: Examples for network topologies. Each node represents a process. Each directed
edge represents a channel. Each undirected edge between two nodes represents two directed
edges with opposite directions between the two nodes.
Channels In a message passing model, processes communicate by sending and
receiving messages over channels. The channels model entities that support com-
munication among the entities modelled by processes. For example, a channel
may represent a twisted pair cable, a TCP/IP connection, or a UNIX named pipe.
We adopt a message passing model, in which pairs of processes communicate
via unidirectional point-to-point channels. A process can directly send a mes-
sage to a process iff there is a channel from to . If there is no channel
from to , cannot directly send a message to (but may be able to do so
indirectly with the help of other processes). We denote the set of channels in a
distributed system by Ξ = {1,..., }. In particular, we assume that at least one
channel exists (i.e., Ξ = /
0) and that channels and processes are different entities
(i.e., Ξ∩Π = /
0). As we will often treat processes and channels alike, we speak of
a component if we mean a process or a channel.
A distributed system that consists of processes and unidirectional channels can
be represented by a directed graph, where the nodes of the graph represent pro-
cesses and the edges represent channels. In principle, different topologies of pro-
cesses and channels are possible. For example, Fig. 2.1a shows a complete graph:
In the system, each process can directly send a message to each other process. Fig-
ure 2.1b to Figure 2.1d show systems, in which the processes and channels form
a ring, a cube, and a tree, respectively. In the following, however, we will restrict
ourselves to complete graphs and only consider reliable channels.
Intuitively, a reliable channel does not lose messages, does not change them,
and does not invent new ones on its own (cf., for example, Lynch [1996]). More
precisely, a reliable channel from a process to a process is characterised by
three properties:](https://image.slidesharecdn.com/94575-250509155417-a47a89df/85/Structural-Failure-Models-for-Fault-Tolerant-Distributed-Computing-1st-Edition-Timo-Warns-Auth-29-320.jpg)
![2.2 States, Traces, Properties 13
Traces The behaviour of a system is represented by an infinite sequence of sys-
tem states. For an arbitrary set , we denote the set of all infinite sequences over
elements in by ∞
. A trace is an infinite sequence of system states in Σ∞
. It
represents a possible execution of a system. For a terminating system, an infinite
sequence of states is obtained by repeating the final state of the system forever.
We extend the projection π from states to traces: For τ = 0,1,... ∈ Σ∞, we
define π (τ) as the sequence π (0),π (1),... ∈ Σ∞ . We call a sequence in
Σ∞ an external trace1.
Note that it may be impossible for an external observer to differentiate some
system states. For example, consider a system that cycles among three different
system states 0,1, and 2 starting with 0. Such a behaviour is represented by the
trace τ = 0,1,2, 0,1,2,.... If 0 = 0,0,1 = 1,1, and 2 = 1,2,
then the external trace π (τ) is 0, 1, 1, 0, 1, 1, .... Hence, an external
observer can only distinguish two different externally visible states, although the
system cycles among three different system states.
Traces can be classified into interleaving and noninterleaving traces. Informally,
an interleaving trace only allows steps (i.e., state transitions) that change the com-
ponent state of at most one component. Noninterleaving traces allow steps that
change the component state of more than one component. Such steps represent
simultaneous (i.e., parallel) operations of different components. Formally, a step
is a pair of system states , and is called a stuttering step iff = . For a com-
ponent ∈ (Π∪Ξ), a step , is an -step iff π(,) = π(, ); that is,
an -step changes the component state of . A trace 0,1,... is called inter-
leaving iff, for each ≥ 0, ,+1 is a stuttering step or is an -step for exactly
one component ∈ (Π∪Ξ). A trace is noninterleaving iff it is not interleaving.
Stuttering Stuttering steps are an important concept for the verification of dis-
tributed systems as they allow to refine a system model [Lamport, 1983a]. For
an arbitrary set , a sequence 0,1,... ∈ ∞
is called stutter-free iff, for each
≥ 0, either = +1 or, to allow terminating systems, = +1 for all ≥ . For
τ = 0,1,... ∈ ∞, the stutter-free form of τ is defined as the sequence that is
obtained from τ by replacing every maximal finite subsequence ,+1,..., of
identical elements by . Two sequences τ,ζ ∈ ∞ are equivalent up to stuttering
iff their stutter-free forms are equal.
1We accept the nuisance that, formally, an external trace is not a trace. Abadi and Lamport [1991], for
example, avoid this by additionally considering external system states as system states. However,
such an approach would complicate the presentation and does not yield relevant advantages for our
purposes.](https://image.slidesharecdn.com/94575-250509155417-a47a89df/85/Structural-Failure-Models-for-Fault-Tolerant-Distributed-Computing-1st-Edition-Timo-Warns-Auth-31-320.jpg)
![14 2 Modelling Fault-Tolerant Distributed Systems
For a subset È of Ë∞, we define the stuttering closure κ (È) as the set of all
infinite sequences over Ë that are equivalent up to stuttering to some sequence in
È. Formally,
κ (È) = {ζ ∈ Ë∞
: ∃τ ∈ È : ζ and τ are equivalent up to stuttering}.
È is called closed under stuttering iff È = κ (È).
Properties Distributed systems are represented by properties being sets of traces
that are closed under stuttering. Intuitively, a property represents all possible ex-
ecutions of a system. An external property is defined as a set of external traces
that is closed under stuttering and represents all executions that can be seen by an
external observer.
We extend the projection π from traces to sets of traces by defining π(È) =
{π(τ) ∈ Σ∞
: τ ∈ È} for a set of traces È ⊆ Σ∞. Note that π(È) may not be
closed under stuttering even if È is a property2. The external property induced by
a property È is defined by κ (π(È)).
We extend the notion of interleaving to properties: A property is interleaving iff
it only contains interleaving traces. Otherwise, it is noninterleaving. As interleav-
ing properties are more easy to reason about, we represent distributed systems only
by interleaving properties. A property is a distributed system property iff it is an
interleaving property. We do not consider the restriction to interleaving properties
a severe limitation, because the choice between interleaving and noninterleaving
is merely one of convenience [Lamport, 2002, Sect. 10.5.2]. On a sufficiently de-
tailed level of abstraction, any system can be represented by interleaving traces
[Abadi and Lamport, 1995].
Describing distributed systems by writing down sets of infinite sequences is
inconvenient. More abstract formalisms are required to describe a property more
concisely. Well-known and useful examples of such formalisms are Guarded Com-
mands [Dijkstra, 1975], temporal logic [Pnueli, 1977], and CSP [Hoare, 1978,
1985]. We do not care which particular formalism is used to describe a system,
but assume that each system description defines a distributed system property. For
illustration purposes, we will rely on TLA and pseudo-code when describing dis-
tributed systems.
2The underlying reason is that the projection yields “new” stuttering steps: Consider a trace τ =
0,0, 0,1, 1,2,... with 0 = 1 and a property with τ ∈ , but without any trace of
the form 0,3, 1,4,.... Then, π (τ) = 0, 0, 1,... is in π (), but 0, 1,... is not
in π () although both are equivalent up to stuttering.](https://image.slidesharecdn.com/94575-250509155417-a47a89df/85/Structural-Failure-Models-for-Fault-Tolerant-Distributed-Computing-1st-Edition-Timo-Warns-Auth-32-320.jpg)
![2.2 States, Traces, Properties 15
Correctness Verifying the correctness of a distributed system means to show that
the system implements a given specification that describes the properties required
from the system. As a system itself may also serve as a specification for another
system, we do not formally distinguish systems and specifications and represent
both by properties. In abuse of terminology, we use the term specification also
for the property that is defined by a specification. For example, we say that a
specification is a distributed system specification if the property defined by the
specification is a distributed system property.
For correctness, only the behaviour that is externally visible is relevant. Internal
states are only used for convenience. Formally, we say that a property 1 im-
plements a property 2 iff the external property induced by 1 is a subset of the
external property induced by 2. Hence, if 1 implements 2, all behaviour per-
mitted by 1 – as can be seen by an external observer – is also permitted by 2.
If 1 implements 2 and 2 implements 1, we call them equivalent, denoted by
1 ≡ 2.
Safety and Liveness Lamport [1977] has identified two important classes of
properties, safety and liveness. Their relevance stems from requiring different
verification techniques and from the result of Alpern and Schneider [1985] that
every property equals an intersection of a safety property and a liveness property.3
In particular, a criterion for considering a distributed system specification being
well-written is that safety and liveness are explicitly separated.
Informally, a safety property states that something “bad” never happens. For-
mally, a property is a safety property iff, for each trace τ that is not in , there
is a prefix of τ such that all traces with the same prefix are not in [Alpern and
Schneider, 1985]. Intuitively, the end of the prefix marks the point in time when
something bad happens that is not permitted by .
A liveness property states that something “good” must eventually happen. For-
mally, a property is a liveness property iff, for each finite sequence τ of states,
there is a trace in that has τ as a prefix. Intuitively, the end of the finite sequence
marks the point when something good happens that is required by .
For example, consider the problem of reaching consensus, which we will ad-
dress in detail in Chapter 5. To reach consensus, each process initially proposes
a value and is supposed to eventually decide for a commonly agreed value. The
problem is defined by the intersection of two safety properties, uniform agreement
and validity, and a liveness property, uniform termination. These properties are
defined as follows:
3Less well known, they have also shown that each property that is defined over more than one state
equals an intersection of two liveness properties.](https://image.slidesharecdn.com/94575-250509155417-a47a89df/85/Structural-Failure-Models-for-Fault-Tolerant-Distributed-Computing-1st-Edition-Timo-Warns-Auth-33-320.jpg)
![16 2 Modelling Fault-Tolerant Distributed Systems
Every process eventually decides on some value.
No two processes decide differently.
If a process decides a value , then was proposed by some process.
The “bad thing” for uniform agreement happens when two processes decide dif-
ferently. For validity, it happens when a process decides a value that has not been
proposed. The “good thing” for uniform termination happens when each process
decides on a value.
Safety properties are generally proved using assertional methods [Keller, 1976,
Ashcroft, 1975, Owicki and Gries, 1976, Lamport, 1977]. For such methods, a
state invariant is found that implies the safety property to be proven. This invariant
is proven by induction on the number of performed steps. For the base case, the
invariant is shown to hold in the initial state (no step has been performed yet).
For the induction step, it is shown that, if the invariant holds in a state, then the
invariant holds after any step performed by the system. By induction, the invariant
holds in any state.
A classical method for proving liveness properties relies on well-founded sets
and convergence functions. A well-founded set is a set with a partial order
on the set such that any sequence 0,1,... with +1, ∈ , and ≥ 0 is
finite. For example, the set N of natural numbers with the common “greater than”
relation is a well-founded set. The convergence function (also called “ranking
function” or “progress function” [Manna and Pnueli, 1982, Lynch, 1996]) maps
from states to the well-founded set. Proving that something “good” eventually
happens means to show that (a) something “good” happens when the value of the
convergence function reaches a minimum of the well-founded set and that (b) the
value of the progress function monotonically decreases with each performed step.
Liveness and safety properties can also be proven by using temporal logic, a
modal logic with symbols for temporal notions like “always” and “eventually”
[Pnueli, 1977, Owicki and Lamport, 1982, Lamport, 1994]. We look into temporal
logic in more detail in Sect. 2.3 when we give an overview of TLA.
Component Properties Analogously to modelling distributed systems, we also
represent the components of a distributed system by properties. To represent a
component and only , a property must describe the behaviour of , but must
not restrict the behaviour of the environment of (e.g., other processes and chan-
nels). Informally, such a property allows the environment of to perform an
arbitrary step whenever does not perform a step.](https://image.slidesharecdn.com/94575-250509155417-a47a89df/85/Structural-Failure-Models-for-Fault-Tolerant-Distributed-Computing-1st-Edition-Timo-Warns-Auth-34-320.jpg)
![2.3 Temporal Logic of Actions 19
PROOF:
31. CASE: , +1 is a stuttering step
PROOF: By the induction hypothesis and as is closed under stuttering.
32. CASE: , +1 is an -step for a component ∈ (Π∪Ξ)
PROOF: By the induction hypothesis and as, otherwise, there would be no
trace with the prefix 0,..., +1 in by the construction of ().
33. Q.E.D.
PROOF: By 31 and 32.
23. Q.E.D.
PROOF: By 21, 22, and mathematical induction.
14. Q.E.D.
PROOF: 11 – 13
2.3 Temporal Logic of Actions
Our results do not depend on a particular formalism to express a property. Never-
theless, we exemplarily show how such an formalism, namely the Temporal Logic
of Actions (TLA) from Lamport [1994], allows to present properties concisely.
Temporal logic in general provides useful means to specify and reason about con-
current systems [Lamport, 1983a]. TLA is a variant of temporal logic for spec-
ifying and verifying concurrent systems in terms of their actions. The main dif-
ferences to the well-known temporal logic of Pnueli [1977] are invariance under
stuttering, support for temporal existential quantification, and allowing action for-
mulae as atomic formulae.
Unlike other formalisms, both, systems and their specifications, are represented
in the same logic. A system implements its specification iff the logic formula that
describes the system implies the formula that describes its specification, where
“implies” means logical implication. Structural relations among components are
also represented by logical operators. For example, logical conjunction is parallel
composition.
We now briefly summarise the syntax and the semantics of TLA. Refer to Lam-
port [1994], Abadi and Merz [1996], and Lamport [2002] for more detailed pre-
sentations.
Syntax Syntactically, TLA formulae are built from constant symbols, variable
symbols, and the special symbols ¬,∧,2,∃,∃
∃
∃
∃
∃
∃, ,(,), and =. The variables are
partitioned into sets of rigid variables, whose values are state-independent, and
flexible variables, whose values are state-dependent.](https://image.slidesharecdn.com/94575-250509155417-a47a89df/85/Structural-Failure-Models-for-Fault-Tolerant-Distributed-Computing-1st-Edition-Timo-Warns-Auth-37-320.jpg)
![20 2 Modelling Fault-Tolerant Distributed Systems
An action is a first-order predicate over the constant and variable symbols, for
example, = + 1. With free variables defined as in first-order logic, a state
function is a first-order expression over the constant and variable symbols without
free primed variables, for example, + −1. Analogously, a predicate is an action
without free primed variables, for example, = +1.
We define substitution for actions, state functions, and predicates as in first-
order logic and write, for example, {/} for the results of substituting the first-
order expression (over constant and variable symbols) for the free occurrences
of variable in action . For any state function (or predicate) , is defined
as the state function (or predicate) obtained by priming the free flexible variables.
For example, ( + − 3) equals ( + − 3).
A TLA formula is built from predicates and formula of the form 2[] , where
is an action, is a state function, and [] is an abbreviation for (∨( = ).
Additionally, 2 is a formula if is one. For example, 2( = +1) and 2[ =
+1] are formulae.
In addition to common abbreviations such as ∨,⇒, and ≡ or the temporal oper-
ator 3, the following abbreviations are defined:
•
Δ
= ∧( = ),
• UNCHANGED
Δ
= = ,
• if 1,..., are the free variables of , then ENABLED
Δ
= ∃1,..., : ,
• if is the tuple 1,..., , then ∃
∃
∃
∃
∃
∃ :
Δ
= ∃
∃
∃
∃
∃
∃1 : ...∃
∃
∃
∃
∃
∃ :
• a weak fairness operator WF ()
Δ
= 32ENABLED ⇒ 23 , and
• a strong fairness operator SF ()
Δ
= 23ENABLED ⇒ 23 ,
where ,1,..., are variables, is an action, and a state function.
Semantics The meaning of TLA formulae is defined over a set of values. We
assume a fixed set of values that includes all required values. A state is a mapping
from the set of flexible variables to the set of values. The internal state is given
by the mapping from the set of hidden flexible variables (see below) to the set of
values; the external state by the mapping from free flexible variables to values. As
in the previous section, a trace is an infinite sequence of states.
The meaning of a state function is a mapping from the set of states to the set of
values. Analogously, a predicate is either true or false for a state. A state satisfies
a predicate iff the meaning of the predicate is true for the state. A predicate is](https://image.slidesharecdn.com/94575-250509155417-a47a89df/85/Structural-Failure-Models-for-Fault-Tolerant-Distributed-Computing-1st-Edition-Timo-Warns-Auth-38-320.jpg)
![2.3 Temporal Logic of Actions 21
called valid, denoted by |= , iff every state satisfies the predicate. The meanings
of common operators such as ∧,¬, and ∃ : are standard.
The semantics of actions is defined for pairs of states, where the unprimed vari-
ables refer to the first state and the primed variables to the second one. For exam-
ple, = +1 is true at a pair of states , iff the value of in equals the value
of plus 1 in . A pair of states satisfies an action iff the meaning of the action is
true for the pair of states. A pair of states that satisfies an action is called an
step. An action is called valid, denoted by |= , iff every step is an step.
Analogously to state predicates, a formula is either true or false for a trace. A
trace satisfies a formula iff the meaning of the formula is true for the trace. A
formula is called valid, denoted by |= , iff every trace satisfies the formula.
Each formula represents a set of traces, namely the set of all traces that satisfy
the formula. As such, a formula can describe a property and, therefore, specify a
distributed system. The meaning of a formula is defined inductively as follows. A
state predicate is true for a trace iff it is true of its first state. Derived from temporal
logic, 2 denotes the always operator; 2 means that is always true in the future.
The operator ∃
∃
∃
∃
∃
∃ denotes temporal existential quantification; the formula ∃
∃
∃
∃
∃
∃ :
means that there exists a sequence of values for such that holds. Intuitively, ∃
∃
∃
∃
∃
∃
hides variables from external observers: is called an internal or hidden variable
of ∃
∃
∃
∃
∃
∃ : . Such a variable is part of the domain for the mapping that defines the
internal state.
Specifications In TLA, a specification is expressed by a formula that has the
“canonical form”
∃
∃
∃
∃
∃
∃ : ∧2[ ]Ú ∧ ,
where is a state predicate, is an action describing possible steps of the
system, is a conjunction of fairness conditions, is a variable, and a state
function. The formula is true for all those traces, whose initial state satisfies ,
where every step is a step or leaves unchanged, and where holds. Note
that each TLA formula defines a property, namely the set of all traces that satisfy
the formula, as each TLA formula is invariant under stuttering [Lamport, 1994,
Abadi and Merz, 1996].
TLA is an expressive formalism that allows to describe different kinds of speci-
fications. For example, it supports interleaving and noninterleaving specifications
and allows to represent closed as well as open systems. Intuitively, an open system
interacts with its environment (being beyond the control of the system). A closed
system is self-contained: it does not interact with its environment, but its external
state may be inspected by an external observer. We restrict ourselves to closed sys-
tem specifications as these are more easy to reason about and as, for our purposes,](https://image.slidesharecdn.com/94575-250509155417-a47a89df/85/Structural-Failure-Models-for-Fault-Tolerant-Distributed-Computing-1st-Edition-Timo-Warns-Auth-39-320.jpg)
![22 2 Modelling Fault-Tolerant Distributed Systems
it suffices to model inputs to a system as being nondeterministically generated by
the system itself.
Decomposed System Specifications Semantically, a state is a mapping from
flexible variables to values in TLA. We assume that the flexible variables of a dis-
tributed system specification can be partitioned among the components. A com-
ponent can only change its own variables, which cannot be changed by any other
component. The internal local state of a component is then given by the map-
ping from the set of hidden flexible variables of to the set of values. The external
local state of is given by the mapping from the set of free flexible variables of
to the set of values.
Component specifications arise when composing a system from reusable com-
ponents or decomposing a given system into its components. We only deal with
system specifications that are decomposed into component specifications such that
the environment of each component is known.4 For a component ∈ (Π∪Ξ), we
assume that the tuple of output variables is the tuple of the free flexible variables
of and that the tuple of environment variables is the tuple of the flexible vari-
ables of all other components. A component specification then has the canonical
form
∃
∃
∃
∃
∃
∃ : ∧2[ ]Ü,Ó ∧
with:
• is the tuple of all internal variables of the component. The set of variables
in are precisely the domain of the mapping that defines the internal local
state of the component.
• The state predicate specifies the initial values of the component’s inter-
nal variables and output variables .
• The action allows steps of the component that change the value of an
output variable of with ⇒ = .
• is a conjunction of fairness conditions. Each fairness condition must be of
the form Ü,Ó( ) or Ü,Ó( ), where is an action.
The formula 2[ ]Ü,Ó allows the environment to do anything but change the
component’s internal or output variables. Due to ⇒ = , the values of the
4In particular, we do not consider assume/guarantee specifications of the form +
− , which as-
sert that a component satisfies a guarantee as long as the environment satisfies an assumption
[Abadi and Lamport, 1995]. Such specifications allow to describe reusable components for
unknown environments.](https://image.slidesharecdn.com/94575-250509155417-a47a89df/85/Structural-Failure-Models-for-Fault-Tolerant-Distributed-Computing-1st-Edition-Timo-Warns-Auth-40-320.jpg)
![2.3 Temporal Logic of Actions 23
Δ
= , ,
Δ
= ∧ ∈ BOOLEAN ∧ =
∧ ∈ BOOLEAN ∧ =
∧ = /
0
Δ
= ∧ =
∧ = ∪{ }
∧ = ¬
∧ UNCHANGED , ,
( )
Δ
= ∧ ∈ ∧ =
∧ = ¬
∧ =
∧ = { }
∧ UNCHANGED ,
Δ
= ∨∃ : ( )
Δ
= ,()
Δ
= ∃
∃
∃
∃
∃
∃ : ∧2[] , ∧
Figure 2.2: A TLA specification of a reliable channel (cf. Abadi and Lamport [1995]).
component’s and its environment variables cannot change simultaneously. Hence,
the specification defines a component property.
Example 2.1 (TLA Component Specification)
An example of a TLA component specification is given in Fig. 2.2. The formula spec-
ifies a reliable channel in terms of three output variables, , , and ,
and one internal variable, . Additionally, the specification refers to three environment
variables, , , and . These variables belong to the processes that are
connected by the channel: and to the sender process and to the
receiver process. Figure 2.3 shows a schematic view on the specification.
As an abbreviation the output variables are joined in the sequence . The internal variable
(hidden by temporal existential quantification) holds the set of messages that are
currently buffered by the channel, that is, messages sent by the sender process, but not yet
delivered to the receiver process. Initially, is empty.
The Boolean variables and are used to implement a simple handshake
protocol between the channel and the sender process. Initially, = , which](https://image.slidesharecdn.com/94575-250509155417-a47a89df/85/Structural-Failure-Models-for-Fault-Tolerant-Distributed-Computing-1st-Edition-Timo-Warns-Auth-41-320.jpg)
![24 2 Modelling Fault-Tolerant Distributed Systems
Figure 2.3: A schematic view on a reliable channel.
models that no new message is to be sent. The sender process sends a message as
follows: It puts into and signals the new message by negating . The
channel handles by acknowledging the sending with the action . A precondition
(or “guard”) for is that = , that is, the sender process must have
signalled a new message. The message is added to and is negated to signal
that the channel is ready for the next message.
Delivering a message to the receiving process is performed similarly by the
action: The message to deliver must be in . The channel removes from
and puts it into . The channel signals the new message by negating . The
receiving process can take the message from and acknowledges that it is ready for
further messages by negating .
Both actions, and , are subsumed by the action for the final
specification. To ensure that a sent message is indeed delivered, the specification includes a
liveness formula defined by weak fairness over the actions.
Verification Using TLA Being a logic, TLA includes axioms and proof rules for
proving formulae. If a formula is provable by the axioms and rules of the logic,
denoted by , then is also valid as TLA is a sound logic. For verification, a
specification 1 implements another specification 2 iff 1 ⇒ 2; that is, logical
implication is implementation. As we use TLA just for presenting specifications,
but not for verification, we do not further discuss the axioms and proof rules of
TLA. Refer to Lamport [1994] for a detailed description and to Lamport and Merz
[1994] for an elaborate example of how to verify a fault-tolerant distributed system
using TLA.](https://image.slidesharecdn.com/94575-250509155417-a47a89df/85/Structural-Failure-Models-for-Fault-Tolerant-Distributed-Computing-1st-Edition-Timo-Warns-Auth-42-320.jpg)
![2.4 Fault Model 25
2.4 Fault Model
Despite interprocess communication, the fault model, the formalisation of a fault
assumption, is another important concern of modelling fault-tolerant distributed
systems. No system can tolerate arbitrary severe faults. Any consideration of
fault tolerance requires a decision, which faults to tolerate, resulting in a fault
assumption.
For a rigorous treatment of fault tolerance, a fault assumption must be for-
malised in terms of the respective system model. The common approach has been
to represent a fault assumption as a mapping (or “transformation”) that augments
a specification by faults. We follow this approach and define a fault model as a
mapping of distributed system properties. Refer to Gärtner [1999] for a survey
of the wide range of approaches that rely on mappings for the specification and
verification of fault-tolerant systems.
Fault Actions and Variables The key for formalising fault assumptions is the
observation of Cristian [1985] that a system changes its state either due to a normal
system step or due to a fault step. This observation allows a simple representation
of faults: syntactically, they are represented in the same way as normal system
steps. For example, if a system is specified by a TLA formula, which describes
steps by actions, then faults can be modelled by augmenting the formula with
additional fault actions.
While not strictly necessary [Gärtner, 2002], it is often convenient to augment
a specification by fault variables. Fault variables indicate the manifestation of
faults and help to structure a specification. They make a specification easier to
understand. For example, with a Boolean fault variable , the manifestation of a
fault is indicated by setting from FALSE to TRUE. If ¬ is added as a conjunct to
each normal action and to each fault action, normal actions are disabled and fault
actions are enabled by the manifestation of a fault. These additional conjuncts help
to easily separate normal from fault actions.
Consider the TLA specification in Fig. 2.4a. The specification contains a sin-
gle variable , which initially equals 0. Each step either leaves unchanged or
increments by 1. Now consider a fault that prevents any further increase of ,
but decreases . Such a fault is formalised in Fig. 2.4b. The manifestation of a
fault is represented by the action that sets the fault variable , which is ini-
tially FALSE, to TRUE. If is TRUE, the normal action is disabled, whereas
the fault action is enabled. While the original specification is only true for
traces, where the value of increases with every step that changes , the trans-
formed specification is also true for traces, in which the value of increases for
some time, but then permanently decreases.](https://image.slidesharecdn.com/94575-250509155417-a47a89df/85/Structural-Failure-Models-for-Fault-Tolerant-Distributed-Computing-1st-Edition-Timo-Warns-Auth-43-320.jpg)
![26 2 Modelling Fault-Tolerant Distributed Systems
Δ
= = 0
Δ
= = + 1
Δ
=
Δ
= Ú ( )
Δ
= ∧2[ ]Ú ∧
(a) Example specification
Δ
= = 0∧ = FALSE
Δ
= ∧¬
∧ = + 1∧ UNCHANGED
Δ
= ∧
∧ = − 1∧ UNCHANGED
Δ
= = TRUE ∧ UNCHANGED
Δ
= ∨ ∨
Δ
= Ú ( )
Δ
= ∧2[ ]Ú ∧
(b) Specification with fault actions and variables
Figure 2.4: Example of augmenting a TLA specification with fault actions and variables.
The approach of modelling faults by additional actions and variables (or equiv-
alent approaches) is common in the literature [Liu and Joseph, 1992, Arora and
Gouda, 1993, Arora and Kulkarni, 1998a, Gärtner, 1998, 1999] across different
system models (e.g., CSP [Nordahl, 1992, 1993] or Petri nets [Völzer, 1998]).
Closely related to our TLA example, Liu and Joseph [1992, 1995, 1996, 1999,
2006] also use TLA for specifications and model faults by fault actions and vari-
ables.
Fault Models Due to faults, an external observer may see behaviour that would
not have been possible without faults. A fault model allows a system to show
additional externally visible behaviour. Semantically, we define a fault model as a
mapping that weakens a distributed system property.
Definition 2.1 (Fault Model)
A fault model α is a mapping from a distributed system property to a distributed system
property α() such that implements α().
If a distributed system property implements α(), then any externally visible
trace permitted by is also permitted by α(). This reflects that a fault model
only describes possible manifestations of faults, but does not make them obliga-
tory.
Defining fault models as mappings of properties bears advantages for comparing
the severeness of different fault models. Consider a distributed system property
and two fault models α and α . If α () implements α (), then α allows at](https://image.slidesharecdn.com/94575-250509155417-a47a89df/85/Structural-Failure-Models-for-Fault-Tolerant-Distributed-Computing-1st-Edition-Timo-Warns-Auth-44-320.jpg)
![30 2 Modelling Fault-Tolerant Distributed Systems
assumption that is formalised by a fault model α. Furthermore, we assume a high-
level specification that captures the desired behaviour of the system such that
implements . We adopt the definitions of Arora and Gouda [1993], Arora
and Kulkarni [1998a,b], and Gärtner [1999] and distinguish three different forms
of fault tolerance: masking, nonmasking, and fail-safe fault tolerance.
Ideally, a system should be able to implement its problem specification despite
any assumed fault. This is the most strict form of fault tolerance and called mask-
ing fault tolerance.
Definition 2.2 (Masking Fault Tolerance)
A distributed system given by a property is masking fault-tolerant for a specification
and a fault model α iff α( ) implements .
Masking fault tolerance is the most desirable form of fault tolerance as faults do
not affect at all whether a system implements its specification or not. In this sense,
the system “masks” the manifestation of faults.
Unfortunately, it may not be possible or too costly to achieve masking fault
tolerance. In such cases, it may be feasible and acceptable to implement a weaker
variant of the original problem specification: is weakened to a specification
( implements
) and only
is required to be implemented. Weakenings for
nonmasking and fail-safe fault tolerance rely on the result that each property can
be written as the intersection of a safety and a liveness property (see Sect. 2.2).
While fail-safe fault tolerance requires to implement the safety part of a problem
specification and allows to violate liveness, nonmasking fault tolerance requires to
implement the liveness part and allows to violate safety requirements for a finite
amount of time.
Definition 2.3 (Fail-Safe Fault Tolerance)
A distributed system given by a property is fail-safe fault-tolerant for a specification
≡ Ë ∩ Ä and a fault model α iff α(È ) implements Ë, where Ë is a safety property and
Ä is a liveness property.
For example, consider the problem of reaching consensus. If a system only de-
cides agreed upon values that were indeed proposed despite faults, then it imple-
ments the safety properties of consensus (i.e., agreement and validity). If the faults
prevent processes from deciding, the system violates liveness (i.e., termination).
Hence, the system may fail, but fails safely; it is fail-safe fault-tolerant.
From a theoretical point of view, another form of fault tolerance could be de-
fined that only requires to implement the liveness part of a problem specification.
However, such a fail-live variant has been basically neglected in the literature as
it seems to lack reasonable applications [Gärtner, 2001]. Other variants, for ex-
ample, implementing some liveness properties of a problem that is given as the
intersection of liveness properties have not been addressed yet as well. A stricter
variant of fail-live, namely nonmasking fault tolerance, is more popular and has
found various applications.](https://image.slidesharecdn.com/94575-250509155417-a47a89df/85/Structural-Failure-Models-for-Fault-Tolerant-Distributed-Computing-1st-Edition-Timo-Warns-Auth-48-320.jpg)
![This would mean that after the weapon is thrown it might be
drawn back again with a leather thong. Possibly the cateia of
Isidore (cateia, to cut or mangle, and catan, to fight; the Irish caꞇ̇
and the Welsh kad, a fight or a corps of fighters, Latin caterva),
survives in the tip-cat. In the Keltic dialect of Wales catai is a
weapon.
[116] See his learned note (p. 410) on the weapon and on Isidore
(Orig. xviii. 7): ‘Hæc est cateia quam Horatius cajam dicit.’ The
disputed word probably derives from the Keltic katten, to cast, to
throw.
[117] Nile Tributaries, by Sir Samuel W. Baker, p. 51. The word
has a curious likeness to the ‘tombat,’ a similar weapon in
Australia (Col. A Lane-Fox, Anthrop. Coll. p. 31).
[118] The ‘Fans’ of M. du Chaillu, a corruption unfortunately
adopted by popular works. In Gorilla-Land (i. 207) I have noticed
the Náyin, or Mpangwe crossbow (with poisoned ebe, or dwarf
bolt), which probably travelled up-Nile like the throw-stick. The
détente and method of releasing the string from its notch are
those of the toy forms of the European weapon. The Museum at
Scarborough contains a crossbow from the Bight of Benin. The
people of Bornu (North-West Africa) also use a crossbow rat-trap.
[119] It is called chakarani in the Coasts of East Africa and
Malabar Coast, by Duarte Barbosa or Magellan (?). The Jibba
negroes of Central Africa wear a similar weapon as a bracelet,
sheathed in a strip of hide.
[120] Col. A. Lane-Fox, Anthrop. Coll., p. 33. For a comparative
anatomy of the boomerang the reader will consult that volume,
pp. 28–61. I have here noticed only the most remarkable points.
[121] The Sword stood in Case 2 of the Salle du Centre,
numbered 695; and was described in p. 225 of the late Mariette
Pasha’s catalogue. I cannot quite free myself from a suspicion
that it was also a boomerang of unusual size. Some of the South
African tribes still use throw-sticks a yard to a yard and a half
long. ‘They are double as thick at one end as they are at the
other,’ says Herr Holub (ii. 340), ‘the lighter extremity being in the
usual way about as thick as one’s finger.’
[122] This meaningless word (cartuccia, a scrap of paper) was
applied by Champollion to the elliptical oval containing a group of](https://image.slidesharecdn.com/94575-250509155417-a47a89df/85/Structural-Failure-Models-for-Fault-Tolerant-Distributed-Computing-1st-Edition-Timo-Warns-Auth-50-320.jpg)
![hieroglyphics. It is simply an Egyptian shield (Wilkinson, loc. cit. i.
chap. 5), and the horizontal line below shows the ground upon
which it rested. The old Nile-dwellers, like the classics of Europe
and the modern Chinese, use the shield for their characteristics,
their heraldic badges, c. The same was the case with our formal
heraldry, which originated about the time of the Crusades,
personal symbolism being its base. As Mr. Hardwick shows, the
horse, raven, and dragon were old familiar badges; many of our
sheep-marks are identical with ‘ordinaries,’ and the tribes of
Australia used signs to serve as kobongs, or crests. Thus, too, in
fortification the shield became the crenelle and the battlement,
and it served to ‘iron-clad’ the war-galleys of the piratical
Norsemen.
[123] So there are two ways of swimming. The civilised man
imitates the action of the frog, the savage the dog, throwing out
the arms and drawing the hands towards his chest.
[124] Journ. Anthrop. Inst. vol. iii. pp. 7–29, April, 1873.
[125] An illustration is given in Mr. J. G. Wood’s Natural History of
Man. He also quotes Mr. F. Baines, who describes the paddles of
the North Australians with barbed and pointed looms.
[126] Capt. James Mackenzie, in a paper read before the Ethno.
Soc. by Mr. G. M. Atkinson (Journal, vol. ii. No. 2, of July 18,
1870. The paddle is figured pl. xiv. 2).
[127] Translated for the Hakluyt Society (1874) by Mr. Albert
Tootal, of Rio de Janeiro, who wisely preserved the plain and
simple style of the unlettered and superstition-haunted gunner.
[128] In Bacon’s day (Aphorisms, book ii.) gummy woods were
supposed to be rather a Northern growth, ‘more pitchy and
resinous than in warm climates, as the fir, pine, and the like.’ They
are as abundant near the Equator, where the viscidity preserves
them from the alternate action of burning suns and torrential
rains; moreover, they are harder and heavier than the pines and
firs of the Temperates.
[129] Historia Geral do Brazil, by F. Adolpho de Varnhagen, vol. i.
p. 112 (Laemmert, Rio de Janeiro, 1854).
[130] M. Paul Bataillard (p. 409, Sur le Mot Pagaie, Soc. Anthrop.
de Paris, 1874) is in error, both when he calls the people of
Paraguay ‘Pagayas,’ or ‘carriers of lances,’ and when he identifies](https://image.slidesharecdn.com/94575-250509155417-a47a89df/85/Structural-Failure-Models-for-Fault-Tolerant-Distributed-Computing-1st-Edition-Timo-Warns-Auth-51-320.jpg)
![Pagaya (not a spear, but a paddle-sword) with the ‘sagaia or
assagai.’ The latter word is of disputed origin, and it is
meaningless in the tongues of South Africa. Space forbids me to
touch its history, except superficially. ‘Azagay,’ a lance, or rather
javelin, appears in Spanish history as far back as the days of
Ojeda (1509); and in 1497 the Portuguese of Vasco da Gama’s
expedition use the term ‘azagayas’ (p. 12, Roteiro or Ruttier,
before alluded to). I believe both to be derived from the Arabic el-
khazúk, a spit—in fact, the Italian spiedo, lance.
[131] Markham (p. 203, Cieça de Leon) makes ‘Macaná’ a
Quichua word; it also belongs to the great Tupi-Guarani family.
[132] Antiquarian Researches, quoted by Markham, loc. cit. p.
181.
[133] The Godeffroy Collection has produced a huge Catalogue of
687 pages (Die ethnographisch-anthropologische Abtheilung des
Museum Godeffroy in Hamburg, vol. i. 8vo (L. Friederichsen u. Co.
1881). It was shown to me by Dr. Graeffe, the naturalist often
mentioned in ‘South Sea Bubbles, by the Earl and the Doctor.’ As a
rule the Samoans had clubs and spears, but few Swords.
[134] This part of Melanesia has been familiar to the home reader
by the life, labours, and death of Bishop Patterson.
[135] Case 21, Petrie, No. 142.
[136] The village of Abu Rawásh, north of the Pyramids of Jízah,
still works this material in large quantities; and its caillouteurs, or
flint-knappers, have produced excellent imitations of the so-called
prehistoric weapons. I have described the flint finds of Egypt in
the Journ. Anthrop. Instit. (Feb. 1879), and shall have something
more to say about them. A Mr. R. P. Greg, who writes in the same
Journal (May 1881) on the ‘Flint Implements of the Nile Valley,’ is
not aware of the fact that I found worked flints near the larger
petrified forest (Cairo). Since that time General Pitt-Rivers made
his grand discovery of ‘Chert Implements in stratified Gravel in the
Nile Valley’ (Journ. Anthrop. Inst. May 1882). In March 1881,
when visiting the Wady, near Elwat El-Díbán (Hill of Flies)
amongst the cliffs of Thebes, he came upon palæolithic flints,
flakes worked with bulbs and facets embedded in the hardened
grit, six and a half to ten feet below the surface. In the same
strata tombs had been cut, flat-topped chambers with
quadrangular pillars. The fragments of pottery enabled Dr. Birch](https://image.slidesharecdn.com/94575-250509155417-a47a89df/85/Structural-Failure-Models-for-Fault-Tolerant-Distributed-Computing-1st-Edition-Timo-Warns-Auth-52-320.jpg)
![to pronounce these excavations ‘not later than the eighteenth
dynasty, and perhaps earlier.’ The New Empire in question was
founded by Amosis (Mah-mes, or Moon-child) circ. b.c. 1700; it
included the three great Tothmes, and lasted about three hundred
years, ending with the heretic Amun-hotep IV., slave of Amun,
circ. b.c. 1400, and Horemhib, the Horus of Manetho. The worked
flints may evidently date thousands of years before that period.
This is a discovery of the highest importance, and we may expect,
with Mr. Campbell, that the ‘works of men’s hands will be found
abundantly underlying the oldest history in the world, in the hard
gravel which underlies the mud of the Nile-hollow from Cairo to
Assouan.’ At any rate, this find disposes of the scientific paradox
that Art has no infancy in Nile-land. The strange fancy has been
made popular by the Egyptologist, who threatens to become as
troublesome as the Sanskritist.
[137] It is figured (p. 8) by Dr. John Evans (Ancient Stone
Implements, c.), who offers another ‘poniard’ (perhaps a
scraper) on p. 292. On p. 308 he notes the large thin flat heads
called ‘Pechs’’ (Picts’?) knives.’
[138] Nephrite is so called because once held a sovereign cure for
kidney disease. Jade is found in various parts of Europe (Page); in
the Hartz (or Resin) Mountains; in Corsica (Bristowe), and about
Schweinsal and Potsdam (Rudler). Saussurite, the ‘Jade of the
Alps,’ appears about the Lake of Geneva and on Monte Rosa. Mr.
Dawkins limits Jade proper in the Old World to Turkestan and
China. Jade, the Chinese you, is popularly derived from the
Persian jádú = (the) magic (stone).
[139] I need hardly notice that the mussel-shell was the original
spoon, still a favourite with savages.
[140] Humboldt (Pers. Narr. vol. i. p. 100) makes the Guanches
call obsidian ‘tabona’; most authors apply the word to the
Guanche knife of obsidian.
[141] Neuhoff, Travels, c. xiv. 874.
[142] Our word ‘glass’ derives from glese (gless, glessaria),
applied by the old Germans to amber (Tacit. De Mor. Germ. cap.
45). Pliny (xxxvii. chap. 11) also notices glæsum (amber) and
Glæsaria Island, by the natives called Austeravia.
[143] Stephens, Yucatan, i. 100.](https://image.slidesharecdn.com/94575-250509155417-a47a89df/85/Structural-Failure-Models-for-Fault-Tolerant-Distributed-Computing-1st-Edition-Timo-Warns-Auth-53-320.jpg)
![[144] The curious and artistic rock inscriptions and engravings of
the South African Bushmen were traced in outline by triangular
flint-flakes mounted on sticks to act as chisels. The subjects were
either simple figures; cows, gnus, and antelopes, a man’s bust
and a woman carrying a load; or compositions, as ostrich and
rider, a jackal chasing a gazelle, or a rhinoceros hunting an
ostrich.
[145] See Chap. I.
[146] Voyage Pittoresque autour du Monde, par M. Louis Choris,
Peintre, 1822.
[147] Trans. Ethno. Soc. vols. i. and ii. p. 290.
[148] Quoted by Col. Lane Fox, Prim. War. i. 25.
[149] Prehistoric Man, by Daniel Wilson (vol. i. pp. 216–17).
[150] Incidents of Travel in Central America, c., p. 51; by J.
Lloyd Stephens. The work is highly interesting, because it shows
Egypt in Central America. Compare the Copan Pyramid with that
of Sakkarah; the Cynocephalus head (i. 135) with those of
Thebes; the beard, a tuft on the chin; the statue and its
headdress (ii. 349); the geese-breeding at the palace (ii. 316);
the central cross (ii. 346) which denotes the position of the
solstices and the equinoxes and the winged globe at Ocosingo (ii.
259). In Yucatan the Agave Americana took the place of the
papyrus for paper-making. Indo-China also appears in the
elephant-trunk ornaments (i. 156).
[151] Prim. War. ii. p. 25.
[152] The two latter are in Demmin, p. 84.
[153] A specimen is in the British Museum, Department of
Meteorolites. (Prim. War. p. 25.)
[154] The distinguished physicist, Prof. Huxley, extends on purely
anthropological grounds, the name ‘Australioids’ to the Dravidians
of India, the Egyptians, ancient and modern, and the dark-
coloured races of Southern Europe. I have ventured to oppose
this theory in Chap. VIII. Mr. Thomas, curious to say, would make
letters (alphabet, c.) arise amongst the Dravidian quasi-savages.
[155] Trans. Anthrop. Inst. May 1881. Mr. Milne brought home
some fine specimens of worked stones, one of which (No. 17, pl.](https://image.slidesharecdn.com/94575-250509155417-a47a89df/85/Structural-Failure-Models-for-Fault-Tolerant-Distributed-Computing-1st-Edition-Timo-Warns-Auth-54-320.jpg)
![xviii.) is a chopper in the shape of the Egyptian flint-knives.
[156] Mr. Heath (who directed the Indian Iron and Steel
Company) opined that the tools with which the Egyptians
engraved hieroglyphics on syenite and porphyry were made of
Indian steel. The theory is, as we shall see, quite uncalled for.
[157] For instance, the magnificent life-sized statue of Khafra
(Cephren or Khabryes) in the Bulak Museum, dated b.c. 3700–
3300 (Brugsch, History, vol. i. p. 78). Scarabæi of diorite can be
safely bought in Egypt, the substance being too hard for cheap
imitation work. Dr. Henry Schliemann constantly mentions diorite
in his Troy and its Remains (1875); for instance, ‘wedges’ (i.e.
axes) large and small, (pp. 21, 28, 154): he speaks of an
immense quantity of diorite implements (p. 75); of a Priapus of
diorite twelve inches high (p. 169); of ‘curious little sling bullets’
(p. 236), and of hammers (p. 285). At Mycenæ he found ‘two
well-polished axes of diorite.’ But as he also calls it ‘hard black
stone,’ I suspect it to be basalt, as his ‘green stone’ (Troy, p. 21)
may be jade or jadeite.
[158] Casting the cannon called after the late General Uchatius is
still kept a secret; and I have been unable to see the process at
the I. R. Arsenal, Vienna.
[159] Stahl-bronce = steel (i.e. hardened) bronze. The
misunderstanding caused some ludicrous errors to the English
press.
[160] I reported to the Athenæum (August 16, 1879) this
‘recovery’ of the lost Egyptian (and Peruvian) secret for tempering
copper and bronze, which had long been denied by metallurgists.
Copper hardened by alloy is described in the Archæologia, by
Governor Pownall. Mr. Assay-Master Alchorn found in it particles
of iron, which may, however, have been in the ore, and some
admixture of zinc, but neither silver nor gold.
[161] Of this I shall have more to say in Chap. V.
[162] This was the weight of the statue of ‘Sesostris,’ Ramses II.,
and his father Pharaoh Seti I.; see Chap. IX. The overseer
standing upon its knee appears about two-thirds the length of the
lower leg (Wilkinson, Frontisp. vol. ii.). Pliny treats of colossal
statues, xxxiv. 18.](https://image.slidesharecdn.com/94575-250509155417-a47a89df/85/Structural-Failure-Models-for-Fault-Tolerant-Distributed-Computing-1st-Edition-Timo-Warns-Auth-55-320.jpg)
![[163] Les Métaux dans l’Antiquité, par J. P. Rossignol. Paris:
Durand, 1863.
[164] So Professor F. Max Müller, Lectures on the Science of
Language, asserted, with a carelessness rare in so learned a
writer (vol. ii. p. 255. London: Longmans, 1873), that ‘the
ancients knew a process of hardening that pliant metal (copper),
most likely by repeated smelting (heating?) and immersion in
water.’ This latter is the common process for softening the metal.
[165] Cieza de Leon (Introd. p. xxviii.): ‘Humboldt mentions a
cutting instrument found near Cuzco (‘the City’) which was
composed of 0·94 parts of copper and 0·06 of tin. The latter
metal is scarcely ever found in South America, but I believe there
are traces of it in parts of Bolivia. In some of the instruments
silica was substituted for tin.’ The South American tin is mostly
impure; still it was and can be used.
[166] Apparently there are two forms of ‘Núb’ (gold), the necklace
and the washing-bowl. See Chapter VIII.
[167] Pliny, xxxvi. 65.
[168] Here Elton, like others of his age, mistranslates Chalcos by
‘brass’:
Their mansions, implements, and armour shine
In brass,—dark iron slept within the mine.
[169] Engraving on copper-plates is popularly attributed to Maso
Finiguerra, of Florence, in 1460; but the Romans engraved maps
and plans, and the ancient Hindus grants, deeds, c. on copper-
plates.
[170] I regret the necessity of troubling the learned reader with
these stock quotations, but they are essential to the symmetry
and uniformity of the subject.
[171] Sophocles and Ovid make Medea, and Virgil makes Elissa,
use a sickle of chalcos. Homer, as will be seen, uses the same
material for his arms, axes, and adzes. Pausanias follows him,
quoting his description of Pisander’s axe and Meriones’ arrow; he
also cites Achilles’ spear in the temple of Athene at Phaselis, with
its point and ferrule of chalcos, and the similar sword of Memnon
in the temple of Æsculapius at Nicomedia. Plutarch tells us that](https://image.slidesharecdn.com/94575-250509155417-a47a89df/85/Structural-Failure-Models-for-Fault-Tolerant-Distributed-Computing-1st-Edition-Timo-Warns-Auth-56-320.jpg)
![the sword and spear-head of Theseus, disinterred by Cymon in
Scyros, were of copper. Empedocles, who (b.c. 444)—
ardentem frigidus Ætnam
Insiluit—
was betrayed by his sandal shoon with chalcos soles.
[172] See Macrob. Sat. vi. 3.
[173] Or ‘a furbisher (whetter, sharpener = acuens) of every
cutting tool of copper and iron.’ See Chap. IX.
[174] I can hardly understand why Dr. Evans (p. 5) insists upon
these sockets being bronze, as they could ‘hardly have been done
from a metal so difficult to cast as unalloyed copper.’ He greatly
undervalues the metallurgy of the Exodist Hebrews, who would
have borrowed their science from Egypt.
[175] Lead is also mentioned, but not tin.
[176] A certain Herr Dromir patented in Germany a process for
making malleable bronze. He added one per cent. of mercury to
the tin, and then mixed it with the molten copper.
[177] For Irish copper swords see the Archéologie, vol. iii. p. 555.
They will be exhaustively described in Part II.
[178] So Chalcis in Mela (ii. 7), now Egripos (Negroponte).
[179] The confusion with iron appears in the Sanskrit (Pali?) ayas;
Latin æs for ahes (as we find in aheneus); the Persian áhan
(;)آهن the Gothic ais, or aiz; the High German er (which is the
Assyrian eru and the Akkadian hurud), and the English iron. J.
Grimm (Die Naturvölker) connects Ἄρης with æs. That æs and
æris metalla in Pliny mean copper, we learn from his tale of
Telephus (xxv. 19), which, by the by, is told by Camoens (Sonnet
lxix.) in a very different way.
[180] χαλκεύειν δὲ καὶ τὸ σίδηρεύειν ἔλγον, καὶ χαλκέας τοὺς τὸν
σίδηρον ἐργαζομένους. Jul. Pollux, Onomasticon, viii. c. 10.
[181] The full term was æs cyprium, which Pliny apparently
applies to the finer kind; then it became cyprium, the adjective,
which expressed only locality; and lastly cuprum. The third is first
used by Spartianus in the biography of Caracalla (No. 5), Cancelli
ex ære vel cupro (doors of æs or copper). Ælius Spartianus dates
from the days of Diocletian and Constantine (Smith, sub voc.).](https://image.slidesharecdn.com/94575-250509155417-a47a89df/85/Structural-Failure-Models-for-Fault-Tolerant-Distributed-Computing-1st-Edition-Timo-Warns-Auth-57-320.jpg)
![When Pliny writes in Cypro prima fuit æris inventio, he leaves it
doubtful if æs be copper or bronze; but we should prefer the
former. So he makes the best ‘Missy’ (native yellow copperas)
proceed from the Cyprus manufactories (xxxiii., iv. 25, and xxxiv.,
xii. 31). The word misí or missí is still used in India for a vitriolic
powder to stain the teeth. Cypros, the wife of Agrippa, was
possibly named from Kafar = the henna plant: the Cyprus of Pliny
(xii. 51) is also the Lawsonia inermis.
[182] Frag. tom. i. p. 226. Edit. Bipont.
[183] The island will be further noticed in Chap. VIII.
[184] Cyprus, c., by General Louis Palma (di Cesnola). London:
Murray, 1877. The author excavated from 1866 to 1876, and
opened some 15,000 tombs, mostly Phœnician.
[185] Quoted in the Kypros of W. H. Engel (vol. i. p. 14). The two
volumes are a mine of information; much of it now antiquated,
but useful to later students who have less leisure to accumulate
learning.
[186] ‘In Cyprus, where the manufacturers of the stone called
chalcitis (copper-smelters) burn it for many days in fire, a winged
creature, something larger than a great fly, is seen walking and
leaping in the fire.’ A brother of the salamander!
[187] Some commentators (Strabo, vi. 1) confound this place with
Ausonian Temĕsa, or Tempsa, in the land of the Brutii, with
Temése of Cyprus.
[188] Herodotus (iii. 23) tells us that, copper being of all metals
the most scarce and valuable in Æthiopia, prisoners were there
bound with golden fetters. As will be seen, copper has lately been
found in Abyssinia.
[189] An awful list of his works is given in Diogenes Laertius.
[190] This ærugo was artificially made by the Ancients with acetic
acid, converting copper to a green salt (Beckmann, sub v.
‘Verdigris or Spanish Green’). The green rust of the carbonate of
copper is still erroneously termed verdigris (acetate of copper).
[191] Ample information is given by Brugsch (Egypt under the
Pharaohs, vol. i. p. 64) of Senoferu; of the valiant Khufu or Suphis
(Cheops); of the Pharaoh Sahura, or Sephris; of Menkauhor
(Mencheres) and Tat-ka-ra (Fifth Dynasty); of the bas-reliefs at](https://image.slidesharecdn.com/94575-250509155417-a47a89df/85/Structural-Failure-Models-for-Fault-Tolerant-Distributed-Computing-1st-Edition-Timo-Warns-Auth-58-320.jpg)
![Wady Magharah dating from King Pepi (Sixth Dynasty); of Thut-
mes III. or the Great, and his sister Hashop (Eighteenth Dynasty
before b.c. 1600), one of whose expeditions produced among
other things ninety-seven Swords (Brugsch, i. 327), and who
mentions ‘gilt copper’; of Amon-hotep III., also ‘the Great’
(Eighteenth Dynasty, about b.c. 1500); and of other Pharaohs who
worked these diggings.
[192] Pottery has lately been found embedded in the bricks of the
Maydúm Pyramid.
[193] The Souphis I. of Manetho is the second king of the Fourth
Dynasty following Soris. Souphis II. is the Khafra of the Tables
and the Cephren of the Greeks.
[194] The hieroglyphic is of several forms; may serve
as a specimen.
[195] ‘Malachite’ is the Greek molochotis, from the molokhe, or
marsh-mallow; whence the Arabic mulukhíyeh. In Poland,
malachite and turquoise preside over the month of December.
[196] Meaning the Beloved of Ptah, the Opener, the Artificer God.
The word is found in the Arabic fath. It is a better derivation for
Hephæstus than ‘Vaishravana’; but Sanskrit is so copious that any
given word can be derived from it.
[197] O Muata Cazembe, by Monteiro and Gamitto, describes the
copper works in South-East Africa long known to the natives. I am
told by Mr. Hooker, C.E., that he has lately seen (pace Herodotus)
‘magnificent specimens of native copper sent from Abyssinia.’
[198] R.N., C.B., c., Across Africa, vol. i. pp. 134, 319; and vol. ii.
pp. 149, 329.
[199] Viagens dos Portuguezes, Colecção de Documentos, c.
[200] Layard’s Nineveh, i. 224, ii. 415; 6th edit. 1854.
[201] Hence our packfong, or German silver, of China, an alloy of
copper (50 per cent.), nickel, and zinc (25 per cent. each).
[202] The Chinese Repository gives a hundred illustrations of the
implements in use by the Chinese and the Japanese.
[203] Fir or fear (vir, a man), and bolg (Bolgi, Belgæ), a belly,
bag, budget, or quiver. They occupied Southern Britain, and](https://image.slidesharecdn.com/94575-250509155417-a47a89df/85/Structural-Failure-Models-for-Fault-Tolerant-Distributed-Computing-1st-Edition-Timo-Warns-Auth-59-320.jpg)
![formed the third immigrant colony preceding the ‘Milesians,’ sons
of Milidh or Miledh (Senchus Mor), evidently Miles, the soldier. He
had two sons, Emer and Airem, from whom the Irish race is
descended. Emer, says Prof. Rhys, may represent the Ivernii or
pre-Celtic population mentioned by Ptolemy; and Airem, which
means ‘a farmer,’ the Iranian race which introduced agriculture
amongst a horde of hunters. The fourth colony was the Tuatha
(people, e.g. Tuatha-Eireann = people of Erin), named from
Danair, a stranger, foreigner, and properly a Dane. We have lately
been shown how much true history may be obtained from these
names, which had become bye-words, almost ridiculous to use.
[204] Bán (our corrupted ‘bawn,’ as in ‘Molly Bawn’), white, is the
Latin canus. It is also a noun substantive, meaning ‘copper.’
[205] Wilde, Catalogue, pp. 58, 356.
[206] Meaning Tectetan = ‘I don’t know.’ So the M’adri on an old
English chart of the Euphrates.
[207] Select Letters of Columbus, c. p. 201. Translated by R. H.
Major, Hakluyt Society, 1870.
[208] Humboldt, Travels, iii. 194.
[209] Commentaries of the Yncas. Translated by Clements R.
Markham, C.B. Hakluyt Society, 1871.
[210] Daniel Wilson’s Prehistoric Man, vol. i. chap. viii.; The
Metallurgic Arts, Copper (pp. 231–79). Prof. Brush, of Yale
College, calculated that 6,000 tons were yielded in 1858.
[211] R.E., Spanish America, c. (Philadelphia: Abraham Small,
1819), p. 49.
[212] It was divided, like the Greek and Roman, into centuries
(pachacas), chiliarchies (hurangos), and inspectorships
(tokrikrok), generally under royalties. The organisation was due to
the Ynka Inti-Kapak (the Great), b.c. 1500–1600. There was a
large fleet (‘magna colcharum classis’) of ships not smaller than
the contemporary European, ‘navigiis velificantur nihili vestris
minoribus,’ says P. Martyr (Decad. ii. lib. 3). Neither traveller nor
historian has explained how this mighty organisation crumbled to
pieces at the touch of a few European adventurers.
I have read with interest the able work of M. Vicente F. Lopez,
Les Races Aryennes du Pérou (Paris: Franck, 1871): he derives](https://image.slidesharecdn.com/94575-250509155417-a47a89df/85/Structural-Failure-Models-for-Fault-Tolerant-Distributed-Computing-1st-Edition-Timo-Warns-Auth-60-320.jpg)
![Indo-Chinese. The geese-breeding (ii. 179) is Egyptian. See also
the Toltec legend of the House of Israel (ii. 172).
[213] The ‘lovely valley, Andahualas,’ is from Anta and Huaylla,
pasture—i.e. ‘copper-coloured meadow.’ Anta in Cieza de Leon
appears to be copper, whereas other writers make it bronze.
[214] Peruvian Antiquities, by Don M. E. de Rivero and J. J. von
Tschudi.
[215] They abandoned the native silver mines when the ore
became too hard, and they smelted it in small portable stoves.
They knew also the chemical combinations, sulphate, antimonial,
and others; and they worked quicksilver. They had mines of
Quella (Khellay, or iron), but they found difficulty in extracting it.
Besides smelting, they could use the tacana (hammer), cast in
moulds, inlay, and solder.
[216] Ewbank, of whom more presently, sketches a well-cast axe
(p. 455). He translates anta by bronze (p. 455).
[217] Doubtless copied from Old-World articles. On the west side
of Palenque the Sword is distinctly Egyptian (Stephens, Yucatan).
I have attempted to show how easily castaway mariners could be
swept by currents from Europe, Asia, Africa, and America. See
‘Ostreiras of the Brazil’ in Anthropologia, No. 1, October 1873.
[218] Antiquarian, Ethnological, and other Researches. By William
Bollaert. London: Trübner, 1860. We must probably change ‘brass’
into ‘bronze’ when he says (p. 90) that ‘the Peruvians used tools
of brass.’
[219] Appendix to Life in Brazil (Sampson Low, 1856).
[220] This white copperas was detected by Scacchi on the
fumaroles after the Vesuvian eruption of 1855.
[221] Gold was shown by yellow, and silver by white. Dr. Evans
(Bronze, c. p. 7) suggests that the round blue bar used by
butchers (Wilkinson, iii. 247) was not of steel; but his reasons are
peculiarly unsatisfactory. The file is a common implement
amongst savages, doubtless derived from the practice of cross-
hatching wooden grips and handles. Mr. A. H. Rhind (Thebes, c.)
attributes little weight to the diversity of colours employed by
ancient Egyptians to depict metallic objects, and he finds red and
green confused.](https://image.slidesharecdn.com/94575-250509155417-a47a89df/85/Structural-Failure-Models-for-Fault-Tolerant-Distributed-Computing-1st-Edition-Timo-Warns-Auth-62-320.jpg)
![[222] Thus we have a blue war-helmet of ring-mail (Lepsius,
Denkmäler, iii. 115 c.), a blue war-hatchet with wooden handle,
and spears pointed with brown-red and blue (copper and iron) in
the tomb of Ramses III. The war-car of an Æthiopian king, in the
days of Tutankamun, has blue wheels and a body of yellow
(gold). Lepsius, however, adds: ‘It is very remarkable that in all
the representations of the old empire, blue-painted instruments
can scarcely be traced.’ This simply proves that iron and steel
were rare.
[223] Prehistoric Man, chap. viii.
[224] It was analysed by Mr. E. Tookey, with the following results:
Copper 97·12
Arsenic 2·29
Iron 0·43
Tin, with traces of gold 0·24
100·08
The presence of the tin may have been accidental. The proportion
of arsenic (2¼ per cent.) might have been expected to harden
the metal, yet it was so soft as to be almost useless.
[225] See chap. ix.
[226] It is equivalent to the Roman’s ‘Aliud clausum in pectore,
aliud in lingua promptum habere.’
[227] So amongst the Jews the sharp knives for circumcision
(Josh. v. 2–3) were of the silex which they learned from the
Egyptians; and the custom continued long after the invention of
metal blades.
[228] It was opened by Herr Ramsauer, and carefully described in
Das Grabfeld von Hallstatt, by Baron E. von Sacken. I shall have
more to say of it in chap. xiii.
[229] Prinseps’ Essays (London, 1858), vol. i. p. 222, pl. xliv. fig.
12, and Journ. R. As. Soc. Bengal, vol. vii. pl. xxxii. fig. 12. Long
descriptions of copper smelting in India are found in Science
Gleanings, pp. 380 et seq., No. 36, Dec. 1831, Calcutta, and in
Percy (Metall. p. 387); the latter by Mr. H. F. Blanford, of the Geol.
Survey, who made especial studies in Himalayan Sikkim and the
Nepaulese Tirhai. The workmen, who are of low caste, win the
stone in small blast-furnaces about three feet high, burning](https://image.slidesharecdn.com/94575-250509155417-a47a89df/85/Structural-Failure-Models-for-Fault-Tolerant-Distributed-Computing-1st-Edition-Timo-Warns-Auth-63-320.jpg)
![charcoal and cow-chips. They work not only the easily reducible
carbonates, but sulphuretted ores, copper pyrites, with a mixture
of mundic (iron pyrites).
[230] Scales are apparently implied by kaskassin (1 Sam. xvii.),
which in Leviticus and Ezekiel applies to fish-scales.
[231] The shekel is usually estimated at 220 grs. (Troy), which
would reduce the weights to 22·91 and 190·97 lbs. respectively;
but Maimonides makes it = 320 grains of barley = as many grains
Troy. See Parkhurst (Lex., s.v. ‘Amat’). Either figure would form a
fair burden for a horse; and the spear would have been a most
unhandy article, unless used by a man ten feet tall. I shall notice
the Gathite’s Sword in chap. ix.
[232] Ethnology of the British Islands. We also read: ‘Copper
Swords have been found in Ireland; iron among the Britons and
Gauls; bronze was used by the Romans, and probably by the
Egyptians; and steel of varying degrees of hardness is now the
only weapon employed.’ (J. Latham: see chap. vii.)
[233] Trans. Edinb. Philos. Soc. Feb. 1822.
[234] J. A. Phillips, F.C.S. Memoirs of the Chemical Soc. vol. iv.
[235] Archæology and Prehistoric Annals of Scotland, p. 246.
[236] See Sir W. Wilde’s Cat. Metallic Materials—Celts, Museum of
Royal Irish Academy.
[237] History of Kerry, p. 125.
[238] Yet Æschylus (Agamem.) uses both chalcos and sideros
generically for a weapon.
[239] Ilios, c. (London, Murray, 1880).
[240] Some small objects are reported as wheel-made; but this
requires confirmation, according to a writer in the Athenæum
(Dec. 18, 1880).
[241] The copper bracelet (Troy, p. 150, No. 88) with its terminal
knobs is the modern trade ‘manilla’ of the West African coast. This
survival will again be noticed in chap. ix.
[242] The word in its older form was written ‘allay.’ Johnson
derives it from à la loi, allier, allocare: it appears to me the
Spanish el ley, the legal quality of coinable metal. We have now](https://image.slidesharecdn.com/94575-250509155417-a47a89df/85/Structural-Failure-Models-for-Fault-Tolerant-Distributed-Computing-1st-Edition-Timo-Warns-Auth-64-320.jpg)
![naturalised in English ley, meaning a standard of metals. (Sub
voc. Dict. of Obsolete and Provincial English, by Thomas Wright;
London, Bell and Daldy, 1869.)
[243] Recherches sur les Mystères; and Mémoire pour servir à la
religion secrète, c. c.
[244] The ‘Aglaophemus,’ so called from the initiator of
Pythagoras. I see symptoms of a revival in assertions concerning
a ‘highly cultivated beginning, with the arts well known and
practised to an extent which, in subsequent ages, has never been
approached; and from which there has not anywhere been
discovered a gradual advancement; but, on the contrary, an
immediate and decidedly progressive declension.’ This, however,
is a mere question of dates. Man’s civilisation began long before
the Mosaic Creation; and science has agreed to believe that
savage life generally is not a decadence from higher types, not a
degeneracy, but a gradual development.
[245] We now divide language into three periods: 1st, intonative,
like the cries of children and lower animals; 2nd, imitative, or on
onomatopoetic; and 3rd, conventional, the civilised form.
[246] Axieros (the earth-goddess), Axiokersa (Proserpine of the
Greeks), Axiokersos (Hades), and Casmilos (Hermes or Mercury).
Ennemoser may be right in making the Kabeiroi pygmies (i.e.
gnomes), but not in rendering Dactyloi by ‘finger-size.’
[247] The lame and deformed ‘artificer of the universe,’ who
became Hephæstos (Vulcan) in Greece, and Vishvakarma in India.
Sokar has left his name in the modern ‘Sakkárah.’
[248] The Assyrian cuneiforms allude to ‘the (Great) Bear making
its crownship,’ that is, circling round the North Pole.
[249] The temples of the Cabiri have lately been explored by Prof.
Conze for the Austrian Government at Samothrace, and we may
expect to learn something less vague concerning these mysterious
ancients.
[250] The Rev. Basil H. Cooper believes that the Phrygian was the
original Ida, which gradually passed to Crete; and here the Idæi
were priests of Cybele. He is disposed to connect with it the
Greek Σίδ(ηρο); the German Eisen (and our iron), and the Ida
feldt and Asi of the Norse myths (Day, p. 133).]](https://image.slidesharecdn.com/94575-250509155417-a47a89df/85/Structural-Failure-Models-for-Fault-Tolerant-Distributed-Computing-1st-Edition-Timo-Warns-Auth-65-320.jpg)
![[251] The name is derived by Bochart from Heb. Lub or Lelub,
חיקלוב, chiefs of the Libu or Ribu, as the old Egyptians called the
Libyans. Hence the Prom. Lilybæum (Li-Lúb) and the Sinus ad
Libyam or Lilybatanus.
[252] We have satisfactory details concerning the Chalybes, who
border on Armenia, in the Anabasis (iv. 5, c.). They dwell two
days from Cotyora, the colony planted by Sinope; they are subject
to the Mossynœci, and they subsist by iron-working (v. 5).
Though few, they are a most warlike people, full of fight. Their
armour consists of helmets, greaves, and cuirasses of twisted
linen cords, reaching to the groin. They carry spears about fifteen
cubits long, ‘having one spike’ (i.e. without ferule); and at their
girdles a short faulchion, as large as a Spartan crooked dagger,
with which they cut the throats of all whom they can master; and
then, lopping off their heads, bear them away (iv. 7). Strabo
makes the Chalybes the same as their neighbours the Chaldæi.
[253] The well-known inscription on the tomb of Midas, and
another given by Texier (Asie Mineure, ii. 57) show the Phrygian
tongue to have been a congener of Greek. Even the Békos of
Herodotus (ii. 2) is allied to our ‘bake,’ and Bédu to our ‘water.’ We
are greatly in want of further information about Phrygia, and it is
to be hoped that Colonel Wilson and Mr. W. M. Ramsay will
complete the labours of Texier and Hamilton.
[254] The Aryans of Herodotus, about the Arius river (Heri-rúd),
are an undistinguished tribe, a mere satrapy. Strabo’s Aria (xi. 9)
is a tract about 250 by 40 miles. In Pliny (vi. 23) Ariana includes
only the lands of the Gedrosi (Mekran), the Arachoti (Kandahár),
the Arii proper (Herat), and the Parapomisadæ (Kabul). It has
been truly said that even if Aryan and Turanian man (first)
centred in and emerged from these areas (the table-lands of
Asia), the so-called history is entirely based on the philological
discoveries of the Sanskritist school.
[255] Therasia and Therassia, now Santorin. Here have been
found ruins of prehistoric cities buried by the great central
volcano. According to most geologists the latter was exhausted in
b.c. 1800–1700.
[256] I have personally noticed this, and described it in Midian
Revisited, vol. i. p. 143.](https://image.slidesharecdn.com/94575-250509155417-a47a89df/85/Structural-Failure-Models-for-Fault-Tolerant-Distributed-Computing-1st-Edition-Timo-Warns-Auth-66-320.jpg)
![[257] Beckmann (s.v. ‘Tin’) tells us that the metal ‘never occurs in
a native state.’ He forgets stream-tin. He also denies that the
oldest ‘cassiteron’ and ‘stannum’ were tin; and considers them to
mean the German Werk, a regulus of silver and lead. His vasa
stannea are vessels covered with tin in the inside. In the fourth
century ‘plumbum candidum’ or ‘album’ was superseded by
‘stannum.’ Speaking of electrum, Beckmann asserts that ‘the
ancients were not acquainted with the art of separating gold and
silver.’ ‘Britain,’ Ynis Prydhain Island, where the god Prydhain was
worshipped, or rather ‘Isle of the Brythons,’ has been fancifully
derived by the energetic Semitiser from Barrat-et-Tanuk = Land of
Tin.
[258] Ezekiel tells us that the Tyrians received tin, as well as other
metals, from Tarshish, or Western Tartessus, in the Bay of
Gibraltar.
[259] M. Emile Burnouf, ‘L’Age de Bronze,’ Revue des Deux
Mondes, July 15, 1877, also brings tin from Banca. The island is
about 150 miles long by 36 broad; it has no mountain backbone,
but the peak of Goonong Maras rises some 3,000 feet above the
sea-level. Chinese coolies still work the mines of Mintok, and in
1852 the yearly yield was some 50,000 piculs (each = 133⅓ lbs.)
at the cost of nine rupees per picul.
[260] Beckmann (loc. cit.), like Michaelis, is surprised at the
Midianites possessing tin in the days of Moses. These were the
views of the last century. I have suggested (Athenæum, Nov. 24,
1880) that the old Nile-dwellers extended through Midian to El-
Hejáz and El-Yemen, where they worked the mines which became
known to the Hebrews.
[261] In 1866 De Rougemont made Phœnicia supply bronze to
Europe, the copper being brought from Cyprus. Besides the
Mediterranean, we find a Uralian and a Danubian branch of the
industry. Before 1877 France had supplied 650 bronze Swords and
daggers, Sweden 480, and Switzerland 86.
[262] Alias the Œstrymnides. Borlase was of opinion that the
group formed one block, with several headlands, of which ‘Scilly’
was the highest, outermost, and most conspicuous. He
conjectures the original name to be Syllé, Sulla, or Sulleh, a flat
rock dedicated to the sun; hence the Lat. Siliræ, Silures, and
Sigdeles; the Engl. Sylley, Scilley, and lately Scilly; the Fr.](https://image.slidesharecdn.com/94575-250509155417-a47a89df/85/Structural-Failure-Models-for-Fault-Tolerant-Distributed-Computing-1st-Edition-Timo-Warns-Auth-67-320.jpg)
![Sorlingues; and the Span. Sorlingas. The Keltic name of the chief
feature was Inis Caer.
[263] Archæology and Prehistoric Annals of Scotland, Part II. ‘The
Archaic or Bronze Period.’ Daniel Wilson.
[264] Pliny represents the Cassiterides as fronting Celtiberia. He
considers it a ‘fabulous story’ that the Greeks fetched ‘white lead’
from the islands of the Adriatic.
[265] Prehistoric Times, by Sir John Lubbock, 4th edit. (London:
Williams and Norgate, 1878.)
[266] The identification is not settled; some propose the Isle of
Thanet.
[267] Beckmann, sub voce ‘Tin.’
[268] According to Messrs. Wibel, Fellemberg, and Damour, who
investigated even 10/1000 parts, the average proportions were ⅒
tin to 9 copper; and ¼ tin for hard metal, as chisels, c. M. E.
Chauntre, Age de Bronze. 3 vols. (Paris: Baudry.)
[269] The late General Uchatius, who ‘trusted in princes,’ and
whose tragical death was greatly lamented by his friends, always
declared that he had rediscovered (not discovered) the hardening
of copper and bronze; and that he hoped to arrive at other
secrets. His career was cut short before he learned to make the
metal and the alloy resilient.
[270] Thut, Tuth, Toth, Thoth, c., the moon-god who became
Hermes Trismegistus.
[271] Phosphor-bronze, for whose manufacture companies are
now established in London and elsewhere, has the ordinary
composition with the addition of red or amorphous phosphorus
dropped upon the melted metal in the crucible. Berthier (Traité
des Essais, ii. 410) states that a very small quantity of phosphorus
renders copper extremely hard and suitable for cutting
instruments. Percy (Metallurgy) found that copper will take up 11
per cent. of phosphorus; the metal, which assumes a grey tint, is
quite homogeneous, and so hard that it can scarcely be touched
by the file. The addition of phosphorus promotes the reduction of
the oxides, and enables an exceedingly sound and durable casting
to be made; but if it exceed ½ per cent. the metal becomes very
brittle. Dr. Percy has described phosphor-silver, phosphor-lead,](https://image.slidesharecdn.com/94575-250509155417-a47a89df/85/Structural-Failure-Models-for-Fault-Tolerant-Distributed-Computing-1st-Edition-Timo-Warns-Auth-68-320.jpg)
![[272] So called from Cape Emeri in Naxos.
[273] Appendix to Layard’s Nineveh and Babylon (London:
Murray). The proportions are nearly those of our day. We may
assume our common bronze at 11:100 for large, and 10:100 for
small objects. Cymbals and sounding instruments, however,
contain tin 22:copper 78.
[274] Analysed by Mr. Robinson of Pimlico (Day, p. 110).
[275] Schliemann’s Troy, p. 361 (London: Murray, 1875).
[276] Sir W. Gell found the bronze nails in the ‘Treasury of Atreus’
composed of 12 tin to 88 copper. The Trojan battle-axes,
according to Dr. Schliemann, yielded only 4, 8, and 9 per cent. of
the former metal.
[277] According to Helbig, the Palafittes and Terramare villagers
had spears but not Swords.
[278] For the tin-ore of Peru see Ethnolog. Journal, vol. lxx. pp.
258–261. Rivero, p. 230, and Garcilasso, vol. i. p. 202.
[279] Amer. Journ. of Science, c. v. 42; July 1866.
[280] From descriptions and drawings by Mr. J. H. Godfrey, Mining
Engineer-in-Chief to the Imperial Government of Japan.
[281] M.D., F.R.S., ‘Observations on some Metallic Arms and
Utensils, with Experiments to determine their Composition.’ Royal
Soc. London, June 9, 1796. Philosophical Transactions.
[282] Taken from Dr. Evans (Bronze Impl. c. chap. xxi.). He
compiled it from Martineau Smith’s Hardware Trade Journal
(April 30, 1879).
[283] Wilkinson remarked that the Egyptian proportions of half tin
and half copper were whitish.
[284] Lord Rosse, in casting specula, preferred using copper and
tin in their atomic proportions, or 68·21 per cent. copper to 31·79
per cent. tin.
[285] Speltrum was introduced by Boyle. During the last century
much zinc was imported from India (possibly supplied by China),
and was called tutenag.](https://image.slidesharecdn.com/94575-250509155417-a47a89df/85/Structural-Failure-Models-for-Fault-Tolerant-Distributed-Computing-1st-Edition-Timo-Warns-Auth-70-320.jpg)
![[286] Bohn’s Trans. ii. 32–45. The learned German begins by
stating that zinc was not known to the Greeks, Romans, and
Arabs, and then proceeds to prove that it was. The word ‘zinc’
(from zenken or zacken, nails, spikes?) first occurs in the works of
the Iatro-chemist, Paracelsus, who died in a.d. 1541.
[287] Blende is a generic word, from blenden, to dazzle.
[288] Mongez, Mém. de l’Institut.
[289] At Goslar, however, according to Lohnriss, brass was made
in a.d. 1617.
[290] Pliny, xxxiii. 27. The solder (χρυσός and κόλλα, glue, or
κόλλησις) is attributed by Herod. (i. 25) to Glaucus of Chios, a
contemporary of Alyattes. The word kóllesis is variously rendered
‘soldering,’ ‘brazing,’ ‘welding,’ and ‘inlaying.’ Kóllesis was used to
agglutinate metals, and treated with a peculiar alkali (Pliny, xxxiii.
24). The ‘gold glue’ (chrysocolla) is usually understood to be a
hydrosilicate of copper; not to be confounded with the
χρυσόκολλα or borax. The Mycenian goldsmiths soldered with the
help of borax (borate of soda): Professor Landerer, of Athens,
found this salt on an old medal from Ægina. It was called in the
Middle Ages, Borax Venetus, because imported by the Venetians
from Persia; and it is the Tinkal of modern India. According to
Pliny, lead cannot be soldered without tin, or tin without lead, and
oil invariably must be used. Later usage substituted for the latter
colophonium and other resins: we now solder by means of
electricity. The same writer makes Nero use chrysocolla-powder (a
siliceous carbonate of copper, a kind of blue-stone which would
turn green by exposure to damp) for strewing the circus, to give
the course the colour of his favourite faction, the Prasine (green).
[291] The Germans, who delight in German derivatives for
European words, would find leiton, c., not in luteum, but in
löthen = to unite. There is little doubt, however, that the first
English manufactory of calamine brass at Esher, in Surrey, was set
up in the seventeenth century by Demetrius, a German. In
Grimm’s Dictionary, as noticed by Demmin (chap. i), bronze is
erroneously called messing (brass).
[292] Derived from ὄρος, οὖρος (mountain), or from Ὀρείος, the
discoverer. Metallic names in Greek are mostly masculine; in Latin
and modern usage, neutral. Oreichalcum or aurichalcum, a hybrid](https://image.slidesharecdn.com/94575-250509155417-a47a89df/85/Structural-Failure-Models-for-Fault-Tolerant-Distributed-Computing-1st-Edition-Timo-Warns-Auth-71-320.jpg)
![word, became aurochalcum in the ninth century: the last
corruption (middle of the sixteenth century) was archal.
[293] De l’Orichalque. J. P. Rossignol (loc. cit.).
[294] Some translate this word ‘yellow frankincense’ (λίβανος)
colour; others derive it from Λίβανος, the Lebanon, and make it
male, argurolibanus, while leucolibanus (white) was female.
Finally, the word was explained by the old interpreters to be =
ὀρείχαλκος = brass of Mount (Lebanon).
[295] The tradition of Atlantis, a middle-land in the Atlantic, has
strong claims to our acceptance. The identity of the site with the
‘Dolphin’s Ridge,’ a volcanic formation, and the shallows noted by
H.M.S. ‘Challenger,’ have been ably pleaded in Atlantis (Ignatius
Donnelly; London: Sampson Low, 1882). Perhaps we may trace
the vestiges in Saint Paul’s Rocks, the remarkable group of rocky
islets situate in the equatorial mid-Atlantic. Mr. Darwin supposed
the group to be an isolated example of non-volcanic oceanic
insularity; but Prof. Renard finds the ‘balance of proof decidedly in
favour of the volcanic origin of the rock.’ It will be remembered
that Atlantis was dismembered by earthquakes, eruptions, and
subsidence.
[296] Quoted by Percy from Watson’s Chemical Essays (iv. p. 85,
1786).
[297] The artificial mixture of copper (four fifths) and gold (one-
fifth) was called pyropus (Pliny, xxxiv. 2), from its fiery red tint; it
was also made of gold and bronze, and termed chrysochalcos,
‘the king of metals.’ Æs corinthiacum (Pliny, xxxiv. 3), or
Corinthian brass, used for mirrors, composed of copper, silver
(steel? zinc?), and gold, was more valuable than gold. According
to Pausanias (ii. 3, § 3), this malleable and ductile metal was
tempered in the Fountain of Pyrene. The vulgar legend, refuted
by Pliny, who tells the tale (xxxiv. 6), dates it from the days of
Mummius (b.c. 146). A medal of Corinthian brass was analysed by
the Duc de Luynes. Pliny (xxxiv. 3) mentions three kinds,
candidum, luteum, and hepatizon (liver-colour), of equal
quantities of metal; this probably resembled our own alloys.
Beckmann (sub voc. ‘Zinc’ and ‘Tin’) gives a list of these and other
compositions, Mannheim gold, Dutch gold, Prince’s metal, Bristol
brass, c.](https://image.slidesharecdn.com/94575-250509155417-a47a89df/85/Structural-Failure-Models-for-Fault-Tolerant-Distributed-Computing-1st-Edition-Timo-Warns-Auth-72-320.jpg)
![[298] Possibly the Armenian bole (Bol-i-Armani), used in the East
as a flux from time immemorial. The ‘dropping’ or ‘distilling’ (per
descensum) must allude to a distillatory or condensing apparatus,
and the ‘false silver’ cannot be mercury, lead, or tin.
[299] Hence tutaneg and tutanego, which sometimes meant an
alloy of tin and bismuth. M. Polo (i. 21) describes ‘tutia’ as very
good for the eyes; and his notice of it, and of spodium, reads,
according to Colonel Yule, almost like a condensed translation of
Galen’s pompholyx, produced from cadmia or carbonate of zinc;
and spodos, the residue of the former, which falls on the hearth
(De Simp. Med. p. ix.). Matthioli makes pompholyx commonly
known in the laboratories by the Arabic name ‘tutia.’ The ‘tutia’
imported into Bombay from the Gulf is made from an argillaceous
ore of zinc, moulded into tubular cakes, and baked to a moderate
hardness.
[300] Masc. and fem.; the neut. ἤλεκτρον is the purest form. Dr.
Schliemann, noticing that it also means ‘amber’ (Mycenæ, p.
204), derives it from ‘elek, signifying resin in Arabic (?), and
probably also in Phœnician (?).’ He found earrings of electrum in
the so-called ‘Trojan Stratum,’ 30½ feet below the surface (Troy,
p. 164). The guanin or gianin of the Chiriquis was an aururet
(electrum) of 19·3 per cent. of pure gold, with specific gravity
11·55. The tombac or tombag of New Granada, used for
statuettes, was also a gold of low standard: 63 gold, 24 silver, 9
copper. Usually ‘tombac’ applies to an alloy like Mannheim gold;
the manufacture was introduced into Birmingham, still its chief
seat, by the Turner family, a.d. 1740.
[301] ‘Elektron,’ however, is generally translated ‘amber’; and it
may be the harpax, or drawer, for it occurs in the same verse with
ivory. Amber beads and weapon-handles were amongst Dr.
Schliemann’s finds. Rossignol (p. 347) supposes that electrum,
the pale-yellow or amber-coloured alloy of gold and silver, gave a
name to the gum amber.
[302] This text, stating a truth concerning native gold, suggests
amongst many that the ancients knew the départ, or separation,
of metals. It has been vehemently doubted whether they could
mineralise the white metal; that is, convert it to sulphide and
allow the gold to subside.
[303] Rossignol quotes Zonaras, Suidas, and John Pediasimus to
prove this position.](https://image.slidesharecdn.com/94575-250509155417-a47a89df/85/Structural-Failure-Models-for-Fault-Tolerant-Distributed-Computing-1st-Edition-Timo-Warns-Auth-73-320.jpg)
![[304] We now lacquer with shell-lac dissolved in proof-spirit and
coloured with ‘dragon’s blood.’
[305] The lead was found in even larger proportions. See chap.
xiii.
[306] In my commentary on Camoens (Camoens: his Life and his
Lusiads), and again in To the Gold Coast for Gold (i. 17), I have
attempted to identify Western Tarshish or Tartessus with Carteia
in the Bay of Gibraltar. Newton makes Melcarth ‘King of Carteia’;
but the word may mean either ‘city-king’ (Malik-el-Karyat), or
‘earth-king’ (Malik-el-Arz).
[307] The well-known anthropologist, M. G. de Mortillet, holds
that the oldest type of bronze celt in France, Switzerland, and
Belgium, is that with straight flanges at the sides. This was
followed by the celt with transverse stop-ridge, by the true
winged tool, by the socketed adaptation, and, lastly, by the simple
flat tool wanting rib or flange, wing or socket, and formed of pure
copper as well as of bronze. Archæologists usually determine the
last form to be the earliest; but M. de Mortillet judges otherwise
from the conditions under which the finds occur.
[308] This weapon (gladius) is a Sword-blade, double-edged or
single-edged, straight or curved, and 4–9 inches long, much used
in the fourteenth and fifteenth centuries. It originated from the
old practice of binding the sickle, scythe, axe, hatchet, or Sword
to the end of a pole and thus forming a pike.
[309] The Amazons of the Mausoleum (Newton, Halicarnassus, p.
235) are armed with axe, bow, and Sword; the Greeks with
javelins and Swords.
[310] The Massagetæ (greater Jats or Goths) are opposed to the
Thyssa (or lesser) Getæ, and both used the sagaris. But while
some authors translate the word securis, others call it a ‘kind of
Sword,’ and others confuse it with the ἀκινάκης, the acinaces
which the Greek mentions separately (iv. 62, viii. 67). Strabo (xi.
8) connects the Massagetæ (Goths) with the Sacæ (Saxons), and
Major Jähn derives Sacæ (the Shaka of the Hindus) from
Saighead = Sagitta. The term ‘Saxones’ was later than the age of
Tacitus, and we first find it in the days of Antoninus Pius. ‘Brevis
gladius apud illos (Saxones) Saxo vocatur’ suggests that the Seax
was connected with the race of old (Trans. Anthrop. Instit. May
1880).](https://image.slidesharecdn.com/94575-250509155417-a47a89df/85/Structural-Failure-Models-for-Fault-Tolerant-Distributed-Computing-1st-Edition-Timo-Warns-Auth-74-320.jpg)
![[311] Loc. cit. p. 43.
[312] Egypt. akhu, Lat. ascia, Germ. Axt. The oldest form is ‘aks’
(securis), the bipennis, ‘dversahs,’ and the dolabrum ‘barte.’ In
Lower Saxon axt is ‘exe,’ a congener of our ‘axe.’
[313] The word is variously written and explained.
[314] A silepe from the armoury of King Mosesh was shown at the
National Exhibition amongst objects from Natal (Col. A. Lane Fox,
Cat. p. 145).
[315] Par Lacombe (Paris, Hachette, 1868).
[316] I have again noticed the sahs, seax, sax, and scramasax in
chap. xiii.
[317] Our ‘bill’ is the German Beil, the securis, or axe. Both words
appear to me congeners of the Greek βέλος, Sword or dart,
showing a missile-age, from βάλλειν, to throw; not, as Jähn
thinks, from the Sanskrit bhil. Robert Barret (1598) preferred the
pike, although owning that the bill had done good service. Even
of late years Messrs. John Mitchel and Meagher (‘of the Sword’)
advised the wretched Irish peasants to make pikes out of reaping-
hooks.
[318] Prehistoric Times, p. 20. The Dublin Museum contains 1,283
articles of the Bronze Age.
[319] I assume as a type, the bronze Sword (Tafel iv.) in Die
Alterthümer von Hallstätten, Salzburg, c. by Friedrich Simony
(Wien, 1851).
[320] Pliny, xxxiv. 39.
[321] The word comes from the root which gave the Persian
áhan; the Irish iaran or yarann; the Welsh hiarn; the Armorican
uarn; the Gothic eisarn; the Danish iern; the Swedish iarn; the
Cimbric jara; the German Eisen, and the Latin ferrum, with the
neo-Latin ferro, hierro (Span.), c. From iaran also we derive
Harnisch, harness.
[322] The unfortunate Cretans gained the name of ‘ever liars’ (ἀεὶ
ψεῦσται) for telling what was probably the truth. They showed in
their island the grave of Jupiter, who must have been originally
some hero or chief deified after his death—evidently one of the
origins of worship. The evil report began with Callimachus (Hymn.](https://image.slidesharecdn.com/94575-250509155417-a47a89df/85/Structural-Failure-Models-for-Fault-Tolerant-Distributed-Computing-1st-Edition-Timo-Warns-Auth-75-320.jpg)
![in Jov. 8); and was continued in the proverbial τρία κάππα κάκιστα
(Krete, Kappadocia, and Kilikia). Hence the syllogistic puzzle of
Eubulides: ‘Epimenides said that the Cretans are liars: Epimenides
is a Cretan: ergo, Epimenides is a liar: ergo, the Cretans are not
liars: ergo, Epimenides is not a liar.’
[323] Chap. iv. The Chalybs of Justin (xliv. 3) is a river between
the Ana (Guadiana) and the Tagus; called by Ptolemy and
Martianus, Κάλιπους or Κάλιπος. Æschylus alludes to the original
Chalybes when he personifies the Sword as the ‘Chalybian
stranger,’ and in the same tragedy (Seven against Thebes) he
entitles it ‘the hammer-wrought Scythian steel.’
[324] ‘To the abundance of iron we may attribute the fact that the
Africans appear to have passed direct from the stone implements,
that are now found in the soil, to those of iron, without passing
through the intermediate bronze period which, in Egypt and other
countries, intervened between the ages of stone and iron.’—
Anthropol. Coll. pp. 128–134.
[325] ‘The High Antiquity of Iron and Steel,’ a valuable paper read
before the Philos. Soc. Glasgow, printed in Iron (1875–76), and
kindly sent to me by the editor, Mr. Nursey; also The Prehistoric
Use of Iron and Steel (Trübner, London, 1877), from which Mr.
Day has allowed me to make extracts.
[326] The question is to be determined by facts, not theories.
Hitherto we are justified in believing, from the skeletons dug up
at great depths, or found in caves associated with the mammals
which they destroyed, that Man in prehistoric times was of a low
physical, and therefore mental type. We shall believe the opposite
view when we are shown ancient crania equal, if not superior, to
those of the present day—relics that will revive the faded glories
of ‘Father Adam’ and ‘Mother Eve.’ But, meanwhile, we cannot be
expected to believe in ipse dixits, inspired or uninspired.
[327] For instance, in North-Western Europe, the early iron age
began about a.d. 250, according to Konrad Englehardt (Denmark
in the early Iron Age, p. 4, London, 1866), quoted by Mr. Day.
[328] Egypt’s Place in Universal History, vol. v.; London,
Longmans, 1867, with additions by Samuel Birch, LL.D.
[329] When Laplace made meteorolites ejections from lunar
volcanoes, Chladni suggested that they were masses of metallic](https://image.slidesharecdn.com/94575-250509155417-a47a89df/85/Structural-Failure-Models-for-Fault-Tolerant-Distributed-Computing-1st-Edition-Timo-Warns-Auth-76-320.jpg)
![matter, moving in irregular orbits through interplanetary, and
possibly interstellar, space.
[330] This word is tortured by non-Orientalists into various ill-
forms. The Arabs write it جيزة (Jízeh), and the Egyptians
pronounce it Gízeh, not Ghizeh.
[331] A full-sized drawing appeared in vol. vii. of Proceedings of
the Phil. Soc. Glasgow; and was repeated by Mr. Day in his book,
Pl. II. he also gives Belzoni’s sickle, Pl. I.
[332] When visiting the ‘Tombs of the Soldans,’ Cairo, I found a
slab of blue basalt bearing the cartouche of Khufu, used as a
threshold for one of the buildings. The characters had been partly
erased; but the material was too hard for the barbarians who had
misused it.
[333] I have elsewhere noticed (chap. iv.) the colours of metals in
the painted tombs of Thebes, and the blue (cyanus-colour) of the
butcher’s steel. The history of this homely article is instructive. For
hundreds of years it retained, in England and elsewhere, its
original shape, an elongated cone. At last some ‘cute citizen had
the idea of breaking the surface into four edges, and of hardening
it with nickel. The simple improvement now fits it for sharpening
everything from a needle to a razor: it thus frees us from the
‘needy knife-grinder,’ who right well deserved to be needy, as he
disadorned everything he touched.
[334] Antiquity of the Use of Metals, especially Iron, among the
Egyptians, p. 18 (London, 1868). Also Ueber die Priorität des
Eisens oder der Bronze in Ostasien, by Dr. M. Müller (Trans.
Vienna Anthrop. Soc. vol. ix.).
[335] I assume this date because it marks when the spring
equinox (vernal colure) occurred in the Taurus-sign. The earliest
of the six epochs proposed by Egyptologists is b.c. 5702 (Böckh),
and the latest is b.c. 3623 (Bunsen); the mean being b.c. 4573,
and the difference a matter of 2079 years (Brugsch, i. 30).
[336] The Table of Sakkarah (Memphis), found about the end of
1864 by the late Mariette Pasha, dates from Ramses the Great
(thirteenth century b.c.), and makes Mibampes the first of his
fifty-six ancestors. No. 2 is the new tablet of Abydos, discovered,
also in 1864, by Herr Dümmichen; it enabled scholars to supply
the illegible name in No. 3, the priceless Turin Papyrus, the](https://image.slidesharecdn.com/94575-250509155417-a47a89df/85/Structural-Failure-Models-for-Fault-Tolerant-Distributed-Computing-1st-Edition-Timo-Warns-Auth-77-320.jpg)
![hieratic Canon of the Ptolemies. Mirbampes, Mirbapen, or Mi-ba of
the monuments is, called in Manetho ‘Miebides, son of
Usarphædus’ (Cory’s Fragments, p. 112).
[337] Of Ramses II., who, with his father Seti, represents the
Greek Sesostris, the Sesesu-Ra of the monuments. (Brugsch, Hist.
ii. 53–62: see my chap. viii.) Prof. G. Ebers has made this
Egyptian proto-Homerid the hero of his romance, Uarda (i.e.
Wardah, ‘the Rose’).
[338] De Iside et Osiride. He quotes Manetho the Priest, who
wrote during the reign of the first Ptolemy, and who told
unpleasant truths concerning Moses, the Hebrews, and the
Exodus.
[339] The limestones of Carniola produce heaps of pisoliths,
which require only smelting; and hence, probably, the early Iron
Age of Noricum and its neighbourhood.
[340] They suggest the magnetic and titaniferous iron sands of
Wicklow, of New Zealand, of Australia, and of a variety of sites
mentioned in To the Gold Coast for Gold, ii. 111.
[341] The Naphtuhim of Scripture.
[342] Percy’s Metallurgy, p. 874, first edit.
[343] Proc. Soc. Antiq. second series, vol. v., June 1873. Mr.
Hartland added rubbings of various Pharaohnic stones, hoping to
‘show how little the mind of civilised man has developed during
3,000 years.’ A pleasant lesson to humanity! But after all thirty
centuries are a mere section of the civilisation which began in
Egypt.
[344] The Corsican is simply a blacksmith’s forge. The Catalan has
a heavy hammer and blowing-machine; if the trompe be used, a
fall of water is required for draught. The Stückofen is a Catalan
extended upwards in the form of a quadrangular or circular shaft,
10–16 feet high.
[345] It is to be noted that flint implements were found all about
these works: Mr. Hartland brought home from them silex arrow-
heads. The late lamented Professor Palmer observed them in
other parts of the Pharan peninsula, and I made a small collection
in Midian. In the Journ. of the Anthrop. Soc. 1879, I showed,
following Mr. Ouvry, Sir John Lubbock, and others, that Cairo is](https://image.slidesharecdn.com/94575-250509155417-a47a89df/85/Structural-Failure-Models-for-Fault-Tolerant-Distributed-Computing-1st-Edition-Timo-Warns-Auth-78-320.jpg)
![surrounded by ancient flint-ateliers. M. Lartet explored them in
Southern Palestine; I picked them up near Bethlehem
(Unexplored Syria, ii. 289). The Abbé Richard and others traced
them at Elbireh (in the Tiberiad); between Tabor and the Lake;
and, lastly, at Galgal, where Joshua circumcised. Lastly, my late
friend Charles F. Tyrwhitt-Drake, when travelling with me, came
upon an atelier east of Damascus. I have noticed General Pitt-
Rivers’ great Egyptian discovery in chap. ii.
[346] Hek or hak (chief) has a suspicious resemblance to Shaykh
and sos to sús, the mare, characteristically ridden by the
Bedawin. In old Egyptian sos is a buffalo.
[347] Movers (Phönicier, ii. 3), quoted by Dr. Evans (Bronze, c.
5), finds bronze (copper?) 44 and iron 13 times in the Pentateuch,
and he theorises upon the later introduction of the latter. But
when was the Pentateuch written in its present form?
[348] Rougemont, L’Age du Bronze, pp. 188 et seq.
[349] Volney, Travels, ii. 438.
[350] Much of it, however, was the amygdaloid greenstone, called
in English ‘toad-stone,’ a corruption of the Germ. Todstein.
[351] Speaker’s Commentary, i. 831.
[352] This term seems first to have been used by Orosius (i. 2) in
our fourth century.
[353] In chap. ix. I shall attempt to show that Naharayn (the dual
of Nahr, a river) is also applied to Palestine in such phrases as
‘Tunipe (Daphne-town) of Naharayn.’
[354] Dr. Percy found that certain Assyrian bronzes had been cast
round a support of the more tenacious metal, thus combining
strength with lightness.
[355] M. F. Lenormant (‘Les Noms d’Airain et du Cuivre dans les
deux Langues ... de la Chaldée et de l’Assyrie, Trans. Soc. Bibl.
Archæology, vi. part 2) renders parzillu, iron; abar, lead; shiparru
(Arab. صفر, brass), bronze; anaku, tin; eru or erudu, copper or
bronze (Arab. ايار, copper or brass); kashpu, silver; and kurashu,
gold. The learned author discovers in the cuneiforms repeated
mention of the ‘ships of Mákan’ and the Kur Makannata (mountain
of Makná), which he translates ‘Pays de Mákan’: finding it a great
centre of copper, he is inclined to confound it with the so-called](https://image.slidesharecdn.com/94575-250509155417-a47a89df/85/Structural-Failure-Models-for-Fault-Tolerant-Distributed-Computing-1st-Edition-Timo-Warns-Auth-79-320.jpg)
![Sinaitic Peninsula. I have only to refer readers to ‘Makná’ in my
three volumes on the Land of Midian.
[356] Akkad is upper, Sumir lower Babylonia.
[357] The Five Great Monarchies of the Ancient Eastern World,
vol. i. p. 62. London, 1871.
[358] The first period extended from b.c. 1500 to 909. The second
from b.c. 909 to 745: the most marking names being
Assurnazirpal = ‘Ashur (arbiter of the gods) protects his son,’ who
built the north-west palace of Nimrúd, b.c. 884; and his son
Shalmanezer II. of the Black Obelisk (Brit. Museum), b.c. 850. The
third period (b.c. 745–555) numbered Tiglath-Pileser II., b.c. 745–
727 (a single generation before the first Olympic, b.c. 776, when
the mythic age of Greece emerges into the historical);
Sennacherib (705–681); Esarhaddon (680–668), Assur-bani-pal
(668–640); Nebuchadnezzar in 604–561, a contemporary of Solon
(b.c. 594); Nergalsharuzur (b.c. 557); and the last Nabonidus (b.c.
555). Herodotus (b.c. 450) wrote about a century after the end of
the third period, Ctesias in b.c. 395, and Berosus in b.c. 280. We
have, it is clear, absolutely no historic proof that ‘the patriarchal
system of communities first locally developed itself at the mouth
of the Euphrates Valley,’ or began in any part of the great
Mesopotamian plain.
[359] Rev. B. H. Cooper (loc. cit.) would derive ‘Ida’ from the
Semitic יר (yad, hand), and make the Daktyls, or fingers, its
peaks.
[360] I shall reserve for chap. xi. notices of iron by the classic and
sacred poets of Greece.
[361] Troy and its Remains, p. 362; the analysis by M. Damour of
Lyons.
[362] The theory of Stephani, Schulze, and others concerning the
Byzantine date and Herulian origin of the Mycenæan graves, has
been treated in England with some respect by Mr. A. S. Murray
and Mr. Perry.
[363] According to Pausanias, Alyattes, the Lydian king (ob. b.c.
570), dedicated to his god, amongst other offerings, an inlaid iron
saucer.](https://image.slidesharecdn.com/94575-250509155417-a47a89df/85/Structural-Failure-Models-for-Fault-Tolerant-Distributed-Computing-1st-Edition-Timo-Warns-Auth-80-320.jpg)
Structural Failure Models for Fault Tolerant Distributed Computing 1st Edition Timo Warns (Auth.)
- 1. Structural Failure Models for Fault Tolerant Distributed Computing 1st Edition Timo Warns (Auth.) download https://ebookgate.com/product/structural-failure-models-for- fault-tolerant-distributed-computing-1st-edition-timo-warns-auth/ Get Instant Ebook Downloads – Browse at https://ebookgate.com
- 2. Get Your Digital Files Instantly: PDF, ePub, MOBI and More Quick Digital Downloads: PDF, ePub, MOBI and Other Formats A pattern language for distributed computing Buschmann https://ebookgate.com/product/a-pattern-language-for-distributed- computing-buschmann/ Parallel Computing for Bioinformatics and Computational Biology Models Enabling Technologies and Case Studies Wiley Series on Parallel and Distributed Computing 1st Edition Albert Y. Zomaya https://ebookgate.com/product/parallel-computing-for- bioinformatics-and-computational-biology-models-enabling- technologies-and-case-studies-wiley-series-on-parallel-and- distributed-computing-1st-edition-albert-y-zomaya/ Distributed data management for grid computing 1st Edition Michael Di Stefano https://ebookgate.com/product/distributed-data-management-for- grid-computing-1st-edition-michael-di-stefano/ Tools and Environments for Parallel and Distributed Computing 1st Edition Salim Hariri https://ebookgate.com/product/tools-and-environments-for- parallel-and-distributed-computing-1st-edition-salim-hariri/
- 3. Cloud Computing Principles and Paradigms Wiley Series on Parallel and Distributed Computing 1st Edition Rajkumar Buyya https://ebookgate.com/product/cloud-computing-principles-and- paradigms-wiley-series-on-parallel-and-distributed-computing-1st- edition-rajkumar-buyya/ Distributed computing principles algorithms and systems 1st Edition Ajay D. Kshemkalyani https://ebookgate.com/product/distributed-computing-principles- algorithms-and-systems-1st-edition-ajay-d-kshemkalyani/ ESD Failure Mechanisms and Models 1st Edition Steven H. Voldman https://ebookgate.com/product/esd-failure-mechanisms-and- models-1st-edition-steven-h-voldman/ Statistical Inference for Models with Multivariate t Distributed Errors 1st Edition A. K. Md. Ehsanes Saleh https://ebookgate.com/product/statistical-inference-for-models- with-multivariate-t-distributed-errors-1st-edition-a-k-md- ehsanes-saleh/ Architecting the Cloud Design Decisions for Cloud Computing Service Models 1st Edition Michael J. Kavis https://ebookgate.com/product/architecting-the-cloud-design- decisions-for-cloud-computing-service-models-1st-edition-michael- j-kavis/
- 5. Timo Warns Structural Failure Models for Fault-Tolerant Distributed Computing
- 6. VIEWEG+TEUBNER RESEARCH Software Engineering Research Herausgeber/Editor: Prof. Dr. Wilhelm Hasselbring Im Software Engineering wird traditionell ein Fokus auf den Prozess der Konstruktion von Softwaresystemen gelegt. Der Betrieb von Systemen, die kontinuierlich Dienste mit einer geforderten Qualität bieten müssen, stellt eine ebenso große Herausforderung dar. Ziel der Reihe Software Engineering Research ist es, innovative Techniken und Methoden für die Entwicklung und den Betrieb von nachhaltigen Softwaresystemen vor- zustellen. Traditionally, software engineering focuses on the process of constructing and evolving software systems. The operation of systems that are expected to continuously provide services with required quality properties is another great challenge. It is the goal of the Series Software Engineering Research to present innovative techniques and methods for engineering and operating sustainable software systems.
- 7. Timo Warns Structural Failure Models for Fault-Tolerant Distributed Computing With a foreword by Prof. Wilhelm Hasselbring VIEWEG+TEUBNER RESEARCH
- 8. Bibliographic information published by the Deutsche Nationalbibliothek The Deutsche Nationalbibliothek lists this publication in the Deutsche Nationalbibliografie; detailed bibliographic data are available in the Internet at http://dnb.d-nb.de. Dissertation Universität Oldenburg, 2009 1st Edition 2010 All rights reserved © Vieweg+Teubner Verlag | Springer Fachmedien Wiesbaden GmbH 2010 Editorial Office: Ute Wrasmann | Anita Wilke Vieweg+Teubner Verlag is a brand of Springer Fachmedien. Springer Fachmedien is part of Springer Science+Business Media. www.viewegteubner.de No part of this publication may be reproduced, stored in a retrieval system or transmitted, in any form or by any means, electronic, mechanical, pho- tocopying, recording, or otherwise, without the prior written permission of the copyright holder. Registered and/or industrial names, trade names, trade descriptions etc.cited in this publica- tion are part of the law for trade-mark protection and may not be used free in any form or by any means even if this is not specifically marked. Cover design: KünkelLopka Medienentwicklung, Heidelberg Printing company: STRAUSS GMBH, Mörlenbach Printed on acid-free paper Printed in Germany ISBN 978-3-8348-1287-2
- 9. Foreword Despite means of fault prevention such as extensive testing or formal verification, errors inevitably occur during system operation. To avoid subsequent system fail- ures, critical distributed systems, therefore, require engineering of means for fault tolerance. Achieving fault tolerance requires some redundancy, which, unfortu- nately, is bound to limitations. Appropriate fault models are needed to describe which types of faults and how many faults are tolerable in a certain context. Previ- ous research on distributed systems has often introduced fault models that abstract too many relevant system properties such as dependent and propagating compo- nent failures. In this research work, Timo Warns introduces new structural failure models that are both accurate (to cover relevant properties) and tractable (to be an- alyzable). These new failure models cover dependent failures (for instance, failure correlation by geographic proximity) and propagating failures (for instance, prop- agation by service utilization). To evaluate the new failure models, Timo Warns shows how some seminal problems in distributed systems can be solved with im- proved resilience and efficiency, as compared to existing solutions. Particularly, the textbook-style introduction to distributed systems and the rig- orous presentation of the new failure models and their evaluation may serve as an example for other software engineering research projects – which is why this book is a valuable addition to both a researcher’s and a student’s library. Wilhelm Hasselbring
- 10. Acknowledgments A PhD thesis – as every result of research – is embedded in a scientific and in a personal context. Let me express my gratitude to a few people who have con- tributed in these contexts to my work in one or another way. While I do not name everyone who would deserve it I will remember each of them and their support. First of all, I would like to thank Willi Hasselbring for the open and friendly environment I was able to work in and for his confidence in me. His support – both professional and personal – and the freedom to develop and elaborate my own ideas have been more than what a PhD student can ask for. During the last year of being a PhD student, I worked in the group of Oliver Theel, whom I would like to thank for cordially accepting me in his group. He taught me a lot about research and has considerably improved the quality of my scientific work. His diverting stories have always been an incredible moral support. Very special thanks go to Felix Freiling for his advice and insights not limited to distributed computing. Working with him was a substantial source of motivation and has significantly shaped different contributions of the thesis. His friendliness, perspicacity, and balance in academia have been a role model. Working in different contexts with different people has been a particular priv- ilege that I appreciate. I would like to thank the members of the Software Engi- neering Group, of the System Software and Distributed Systems Group, and of the graduate school TrustSoft of the University of Oldenburg. They made the time more instructive and more enjoyable by numerous and multifaceted discussions. While there are too many people to mention them individually, I would like to thank some of them in particular. Special thanks go to Christian Storm, Matthias Rohr, Marko Bošković, Jens Happe, Heiko Koziolek, Roland Meyer, and Henrik Lipskoch for more than working on joint papers, co-organizing workshops, shar- ing an office room, a lot of humour, and becoming or remaining best friends. I sincerely appreciate the assistance and continuous good will of Ira Wempe and Manuela Wüstefeld who have relieved me of many disturbances. Particular thanks also go to Christian Storm, André van Hoorn, and Kinga Kiss-Jakab for proofread- ing the thesis.
- 11. Very sincere thanks go to my mum, Elisabeth Warns, who has always had con- fidence in me and has supported me all along the way. The contribution of my girlfriend, Alexandra Kroll, to finishing the thesis can hardly be overestimated. I would like to thank her for sharing the good times and for helping me through the bad times in the last couple of years. Timo Warns VIII Acknowledgments
- 12. Abstract The dependability of distributed systems strongly depends on the occurrence of faults and on the ability of a system to cope with them. A fault-tolerant system is capable of providing service as expected even if some components have failed. Unfortunately, no system can tolerate arbitrary severe and arbitrary many faults. Engineering a fault-tolerant system, therefore, requires a fault model that describes the faults to tolerate. A good fault model must be accurate for the relevant aspects of faults, but abstract away irrelevant details. There is empirical evidence that, in particular, dependences and propagation of faults are relevant in real-world sys- tems. In this thesis, we address the questions of how to model such faults and how to tolerate them. For a fault model, we distinguish functional from structural failure models. A functional failure model describes how a component that is failed may behave. A structural failure model describes the extent of component failures. We investigate different classes of nonprobabilistic structural failure models and, in particular, in- troduce two new ones: set-based models for dependent faults and sequence-based models for dependent and propagating faults. Both classes close a gap between probabilistic models that cover dependent and propagating faults and previous nonprobabilistic models that do not. The new classes and several previous ones are compared with respect to their expressiveness resulting in a comprehensive hierarchy of nonprobabilistic structural failure models. All of the considered pre- vious classes are strictly less expressive than the new set-based class, which is strictly less expressive than the new sequence-based class. For many problems of distributed computing, there exist solutions that rely on quorums and, in particular, on highly available coteries to achieve fault tolerance. We illustrate how to solve distributed computing problems under the new model classes using highly available coteries and probing quorums. More precisely, we give characterisations of highly available coteries that show how to construct such a coterie from a set-based model if a highly-available coterie exists. Considering sequence-based models, we introduce the quality measure refined probe complex- ity that gives a tight bound on the number of required probes to find a quorum of noncrashed processes or to reveal that no such quorum exists. Additionally,
- 13. The considerations of quorums are independent of a particular fault tolerance problem. As a concrete problem, we show how to reach consensus in the presence of faults. In particular, we demonstrate that the new model classes do not require solutions developed from scratch: Adapting and transforming previous solutions for previous model classes suffice to reach consensus. Using the new model classes turns out to be beneficial as it allows more resilient and/or more efficient solutions. we present a new probe strategy that is defined for all quorum sets and is more efficient in the number of required probes than previous strategies. X Abstract
- 14. Zusammenfassung Die Verlässlichkeit verteilter Systeme wird stark vom Auftreten von Fehlern und von der Fähigkeit eines Systems, mit ihnen umzugehen, bestimmt. Ein fehlerto- lerantes System ist auch dann noch in der Lage seinen Dienst wie gewünscht zu erbringen, wenn einige Komponenten ausfallen. Leider kann kein System belie- big schwerwiegende und beliebig häufige Fehler tolerieren. Für die Entwicklung eines fehlertoleranten Systems wird daher ein Fehlermodell benötigt, das die zu tolerierenden Fehler beschreibt. Ein gutes Fehlermodell muss in den relevanten Aspekten von Fehlern genau sein, aber von unwichtigen Details abstrahieren. Em- pirische Studien haben gezeigt, dass Abhängigkeiten und die Fortpflanzung von Fehlern relevante Aspekte in realen Systemen sind. Diese Arbeit beschäftigt sich damit, wie diese Aspekte modelliert werden und die dadurch ausgedrückten Fehler toleriert werden können. In der Arbeit werden funktionale und strukturale Fehlermodelle unterschieden. Ein funktionales Fehlermodell beschreibt, wie sich eine ausgefallene Komponente verhalten kann. Ein strukturales Fehlermodell beschreibt das Ausmaß von Kom- ponentenausfällen (z.B. wie viele Komponenten ausfallen können). Die Arbeit un- tersucht verschiedene Klassen von nicht-probabilistischen strukturalen Modellen und stellt insbesondere zwei neue Klassen vor: mengenbasierte Modelle für ab- hängige Fehler und folgenbasierte Modelle für abhängige und sich fortpflanzende Fehler. Beide Klassen schließen eine Lücke zwischen probabilistischen Modellen, die abhängige und sich fortpflanzende Fehler abdecken, und bereits existierenden nicht-probabilistischen Modellen, die das nicht tun. Die neuen und verschiedene bereits existierende Klassen werden miteinander bzgl. ihrer Ausdrucksmächtigkeit verglichen. Dabei entsteht eine umfassende Hierarchie von nicht-probabilistischen strukturalen Fehlermodellen. Alle der betrachteten bereits existierenden Klassen sind echt weniger ausdrucksmächtig als die neue mengenbasierte Klasse, die wie- derum echt weniger ausdrucksmächtig ist als die neue folgenbasierte Klasse. Für viele Probleme verteilter Systeme gibt es Lösungen, die sich auf Quoren und insbesondere hoch verfügbare Coterien verlassen, um Fehlertoleranz zu erreichen. Die Arbeit zeigt Lösungen für die neuen Modellklassen, die hoch verfügbaren
- 15. XII Zusammenfassung Quorenmengen definiert und effizienter in der Anzahl der benötigten Prüfungen als bisherige Strategien ist. Die Untersuchung von Quoren in der Arbeit ist unabhängig von einem be- stimmten Fehlertoleranz-Problem. Als ein konkretes Problem wird gezeigt, wie man Konsensus erreicht, wenn Fehler auftreten können. Insbesondere machen die neuen Modellklassen es nicht notwendig, vollständig neue Lösungen zu ent- wickeln: Die Anpassung und Transformation bestehender Lösungen für bisherige Modellklassen reichen aus, um das Problem zu lösen. Die Nutzung der neuen Modellklassen stellt sich dabei als vorteilhaft heraus, da sie widerstandsfähigere und/oder effizientere Lösungen ermöglichen. Coterien und Prüfungen von Quoren verwenden. Genauer werden Charakterisie- rungen von hoch verfügbaren Coterien und deren Konstruktion aus mengenbasier- ten Modellen vorgestellt. Bzgl. der folgenbasierten Modelle wird die Qualitäts- metrik der verfeinerten Prüfkomplexität vorgestellt. Sie gibt eine enge Schranke für die Anzahl der Prüfungen an, die notwendig sind, um ein Quorum von nicht ausgefallenen Prozessen zu finden (bzw. um herauszufinden, dass es kein solches Quorum gibt). Zusätzlich wird eine neue Prüfstrategie präsentiert, die für alle
- 16. Contents 1 Introduction 1 1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.2 Objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 1.3 Outline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 1.4 Remarks on Notation . . . . . . . . . . . . . . . . . . . . . . . . 8 2 Modelling Fault-Tolerant Distributed Systems 9 2.1 Interprocess Communication . . . . . . . . . . . . . . . . . . . . 10 2.2 States, Traces, Properties . . . . . . . . . . . . . . . . . . . . . . 12 2.3 Temporal Logic of Actions . . . . . . . . . . . . . . . . . . . . . 19 2.4 Fault Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 2.5 Fault Tolerance . . . . . . . . . . . . . . . . . . . . . . . . . . . 29 2.6 Timing Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 2.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37 3 Modelling Fault Assumptions with Structural Failure Models 39 3.1 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 3.2 Functional Failure Models . . . . . . . . . . . . . . . . . . . . . 42 3.3 Structural Failure Models . . . . . . . . . . . . . . . . . . . . . . 47 3.4 Component Failure Models . . . . . . . . . . . . . . . . . . . . . 50 3.5 Set-Based Structural Failure Models . . . . . . . . . . . . . . . . 60 3.6 Sequence-Based Structural Failure Models . . . . . . . . . . . . . 89 3.7 Stochastics, Sets, and Sequences . . . . . . . . . . . . . . . . . . 92 3.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94 4 Constructing Coteries 97 4.1 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 4.2 Introduction to Quorums . . . . . . . . . . . . . . . . . . . . . . 101 4.3 Highly Available Static Coteries . . . . . . . . . . . . . . . . . . 109 4.4 Highly Available Dynamic Coteries . . . . . . . . . . . . . . . . 118
- 17. XIV Contents 4.5 Reducing Probe Complexity . . . . . . . . . . . . . . . . . . . . 126 4.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137 5 Reaching Consensus 139 5.1 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141 5.2 Introduction to Consensus . . . . . . . . . . . . . . . . . . . . . 143 5.3 Consensus in Asynchronous Systems with Unreliable Failure De- tectors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154 5.4 Consensus in Partially Synchronous Systems . . . . . . . . . . . 169 5.5 Consensus in Synchronous Systems . . . . . . . . . . . . . . . . 174 5.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186 6 Conclusion and Future Work 189 Bibliography 195 Index 213
- 18. List of Figures and Tables Fig. 2.1 Network topologies . . . . . . . . . . . . . . . . . . . . . . 11 Fig. 2.2 TLA specification of a reliable channel . . . . . . . . . . . . 23 Fig. 2.3 Schematic view on a reliable channel . . . . . . . . . . . . . 24 Fig. 2.4 TLA specification with fault actions and variables . . . . . . 26 Fig. 2.5 Correct, faulty, and nonfaulty components . . . . . . . . . . 28 Fig. 2.6 Unreliable failure detector . . . . . . . . . . . . . . . . . . . 35 Tab. 2.1 Classes of Failure Detectors . . . . . . . . . . . . . . . . . . 36 Fig. 3.1 Hierarchy of functional failure model classes . . . . . . . . . 46 Fig. 3.2 Hierarchy of structural failure model classes . . . . . . . . . 85 Fig. 4.1 Grid construction of coteries . . . . . . . . . . . . . . . . . . 108 Fig. 4.2 Simple tree construction of coteries . . . . . . . . . . . . . . 108 Fig. 4.3 Dynamic grid construction of coteries . . . . . . . . . . . . . 109 Fig. 4.4 Probe strategy tree . . . . . . . . . . . . . . . . . . . . . . . 127 Fig. 4.5 Probe strategy tree for a highly available coterie . . . . . . . 130 Fig. 4.6 Probe strategy tree for a dominated coterie . . . . . . . . . . 133 Fig. 5.1 Trace of consensus . . . . . . . . . . . . . . . . . . . . . . . 144 Fig. 5.2 TLA specification of consensus . . . . . . . . . . . . . . . . 145 Fig. 5.3 Trace of uniform consensus . . . . . . . . . . . . . . . . . . 146 Fig. 5.4 Message exchange patterns . . . . . . . . . . . . . . . . . . 150 Fig. 5.5 Trace of the transformed consensus algorithm . . . . . . . . . 164 Fig. 5.6 EIG tree for a threshold model . . . . . . . . . . . . . . . . . 175 Fig. 5.7 EIG tree for a Didep model . . . . . . . . . . . . . . . . . . 176 Fig. 5.8 Example 1 of an annotated EIG tree . . . . . . . . . . . . . . 180 Fig. 5.9 Example 2 of an annotated EIG tree . . . . . . . . . . . . . . 181
- 19. 1 Introduction 1.1 Motivation Dependable Distributed Systems Distributed systems have become crucial in most application domains of computing systems. Their success stems from, for example, being more cost-efficient, more powerful, and more scalable than stand- alone systems. Distributed systems are characterised by consisting of active com- ponents that are spatially distributed and share information via some means of communication. The components are active in the sense that they perform com- putations that are of interest for the user of the system. The range of the compo- nents’ spatial distribution is wide, from micro-scale embedded systems-on-a-chip to globally – and possibly further – distributed systems. As distributed systems more and more pervade our daily life, we increasingly depend on their correct service. Many distributed systems deliver critical services, whose failures are not acceptable. Prominent examples include air traffic control systems, power grid systems, and patient monitoring systems. Failures of such systems may entail excessive costs or even cause loss of life. Hence, the depend- ability of distributed systems is essential for their acceptance in the future. Dependability is a fundamental property of computing systems besides, for ex- ample, performance and costs. Avižienis et al. [2004] have spent significant effort on a comprehensive dependability taxonomy over the past two decades. They describe dependability as the ability of a system to deliver service that can jus- tifiably be trusted. Dependability is a general concept that subsumes different more specific attributes such as availability, reliability, and integrity. For exam- ple, availability deals with the “readiness for correct service,” reliability with the “continuity of correct service,” and integrity with the “absence of improper system alterations.” These attributes are threatened by faults, errors, and failures. A fail- ure is an event that occurs when the delivered service of a system deviates from correct service. An error is a part of the system state that may lead to a failure. A fault is the cause of an error. Means of Fault Tolerance With raising needs for dependable distributed sys- tems, the demand for means to attain dependability increases. Avižienis et al. [2004] divide such means into four classes: fault forecasting, fault prevention, fault T. Warns, Structural Failure Models for Fault-Tolerant Distributed Computing, DOI 10.1007/978-3-8348-9707-7_1, © Vieweg+Teubner Verlag | Springer Fachmedien Wiesbaden GmbH 2010
- 20. 2 1 Introduction removal, and fault tolerance. Despite all efforts of prevention and removal, each distributed system consists of components that are bound to fail eventually. Means of fault tolerance allow to avoid that component failures – being system faults – lead to system failures; a dependable system can be built from undependable com- ponents. Ideally, a system should completely mask the occurrence of faults from external observers of the system. However, other forms (e.g., remaining in a safe state despite faults) may be acceptable while being more cost-efficient. A prerequisite for any type of fault tolerance is some form of redundancy [Gärt- ner, 1999]: A fault-tolerant design must incorporate some entities that are not required per se to deliver the desired service. These entities are only added for the sake of fault tolerance. A simple example is a checksum that is added to some data for purposes of error detection and correction. Another example is to have an algorithm executed by different processors and vote on the result to mask a failed processor. The extent of such redundancy in a system determines the costs of the system and the resilience of the system to faults. The Need for Models While designing and evaluating a distributed system are already complex tasks, fault tolerance makes the tasks even more complex. If these tasks are not done properly, the fault tolerance mechanisms themselves can become the source of failures. For example, Mackall [1988] has reported that the fault detection logic in a flight-crucial control system caused failures of an aircraft: Each of overall three communication channels declared the other two channels as failed although no actual hardware failure occurred. Only a manual selection of a backup system allowed to land the aircraft safely. Such examples illustrate that designing and evaluating fault tolerance mechanisms are complex as well as critical tasks and, therefore, require a rigorous treatment. Models as means of abstraction are crucial for a rigorous treatment and are the key to master the complexity of fault-tolerant distributed systems. For example, Bolosky et al. [2007] have reported that formal modelling was essential for de- signing the distributed directory service of a distributed file system. While this task took several months and 19 design iterations, it would have required even more time without formal modelling. A model is an abstract representation of an object of interest. A good model describes the relevant aspects of the object but abstracts away the irrelevant ones. Schneider [1993a] explains that the challenge in finding a good model lies in find- ing a model that is both, accurate and tractable. A model is accurate if evaluations based on the model yield results that do not only hold for the model, but also for the actual object. A model is tractable if evaluations are possible at all. A model
- 21. 1.1 Motivation 3 that is not accurate or not tractable is useless, because it yields invalid results or does not allow to obtain any results. Fault Assumptions No matter to which extent a fault-tolerant design incorpo- rates redundancy, no system can tolerate arbitrary faults. If faults are too severe or occur too frequently, then the redundancy required for fault tolerance will get exhausted: The system fails. For example, if all components of a system fail with an arbitrary behaviour, the system cannot behave as desired anymore. Therefore, it is essential for the design and the evaluation of a fault-tolerant system to make a fault assumption on the faults to tolerate. Fault assumptions are represented by fault models that – for the sake of tractabil- ity – rely on simplifying assumptions. For example, such assumptions include: • At most components may fail. • Component failures are identically distributed for all components. • Component failures are stochastically independent. • There is no propagation of failures among components. Of course, such assumptions must be justified for the sake of accuracy. If a sys- tem is designed or evaluated under invalid assumptions, the system may fail even if its correctness has been formally verified. For example, Mackall [1988] has reported that, for the flight-crucial control system mentioned above, dual simulta- neous component failures had been ruled out as impossible. Hence, the design of the system did not account for them. However, such faults did occur during a test- ing phase. The assumption that at most one component fails was not valid leading to a failure of the overall system. While simplifications as listed above are frequently found in the literature, there is empirical evidence that they are not valid in many real-world systems. For ex- ample, Tang and Iyer [1992] evaluated two DEC VAX-cluster systems and found correlated failures due to errors in shared resources. Dobson et al. [2004, 2005] argue that large-scale blackouts of power grids are typically caused by propagating failures. It is likely that propagating failures in power grids manifest as propagat- ing failures in distributed systems that are connected to these grids. Dependent and Propagating Faults Correlated and, therefore, dependent com- ponent failures occur over the whole range of spatial distribution. For example, failure correlation coefficients up to 0.92 have been found in the globally dis- tributed PlanetLab system [Warns et al., 2008]. If the failures had been stochas- tically independent, the coefficients would have been approximately equal to 0.
- 22. 4 1 Introduction Likewise, Bakkaloglu et al. [2002] found correlated failures when measuring the availability of globally distributed web servers. Amir and Wool [1996] have found strongly correlated failures when evaluating quorum systems on 14 computers lo- cated in two geographical sites with a 50 km distance between both sites. With an increasing integration density of embedded systems, faults such as electric dis- charges are more likely to affect several neighboured components at the same time and cause dependent failures [Limbourg et al., 2007]. The impact of correlated failures is significant. Although the average failure cor- relation coefficient in the study on PlanetLab is rather low with 0.06, a prediction underestimates the probability that exactly one node fails (in the next 5 minutes) by four orders of magnitude under the assumption that failures are independent. Likewise, Yalagandula et al. [2004] have provided empirical evidence that corre- lated failures significantly hurt the availability of a system. Tang and Iyer [1993] have shown that this is the case even if the correlation coefficients are low. Fault Models for Dependence and Propagation The relevance of dependent and propagating component failures has raised the interest for suitable fault mod- els. Fault models that address the extent of faults are called structural failure models. Most of these models are probabilistic and, for example, rely on corre- lation coefficients as input parameters. Some examples are the models of Tang and Iyer [1992] and Bakkaloglu et al. [2002]. Other models cover dependences by explicitly considering the causes for dependent faults. For example, Limbourg et al. [2007] make the assumption that the spatial arrangement of components has an impact on fault dependences. They construct a probabilistic fault model by explicitly considering the spatial arrangement. Junqueira [2006] has presented a model for dependent faults that associates a set of attributes to each process of a distributed system. Intuitively, these attributes capture causes for process failures: If an attribute “is activated”, all processes that have this attribute fail. Despite such efforts, threshold models are predominant in the literature on fun- damentals of fault-tolerant distributed computing. Describing the extent of faults by a simple threshold eases the design and the evaluation of a system and allows to refrain from a probabilistic system model. Considering probabilistic behaviour introduces additional complexities. As Keidar and Marzullo [2002] have criticised, threshold models only allow to model identically distributed and independent faults. Due to the relevance of de- pendence and propagation in practice, such phenomena deserve to be considered when studying the fundamentals of fault-tolerant distributed computing. Hence, there is a need for simple fault models that allow to describe dependent and prop- agating faults.
- 23. 1.2 Objectives 5 1.2 Objectives The objectives of this thesis are twofold: (i) to identify tractable structural failure models that cover relevant aspects of the real world and (ii) to show how to design fault tolerance mechanisms under these models. More precisely, we are looking for classes of nonprobabilistic structural failure models that allow to accurately describe dependent and propagating component failures for the domain of fault- tolerant distributed computing. We take threshold models – being prevalent in this domain – and other nonprobabilistic models as references for our models. It may be suspected that designing and evaluating fault-tolerant systems under the new models become intractable due to the increased accuracy. We demonstrate that, on the contrary, these tasks are hardly more complex. With these objectives, the thesis contributes to a more comprehensive understanding of the fundamentals of fault-tolerant distributed computing. 1.3 Outline The thesis is organised into six chapters as follows. Chapter 2 – Modelling Fault-Tolerant Distributed Systems After the intro- ductory Chapter 1, Chapter 2 presents the system model that will be used through- out the thesis. Distributed systems are formalised by process models in terms of states, traces, and properties. The chapter gives elementary definitions for these terms and also addresses the fundamental aspects of interprocess communication, faults, and timing. The contribution of this chapter lies in providing the foundation for the rest of the thesis. Chapter 3 – Modelling Fault Assumptions with Structural Failure Models Chapter 3 addresses the question of how to formalise fault assumptions in terms of functional and structural failure models. We give formal definitions for these mod- els and show how to describe process, channel, and hybrid failure models in terms of functional and structural failure models. In the literature, it has been suspected that process failure models are incomplete, that is, some fault assumptions cannot be modelled using process failure models. We show that this suspicion is wrong under reasonable assumptions: Process failure models are complete if channels do not have externally visible states. In this case, any fault assumption for our system model can be given by a process failure model. While functional failure models have been investigated in detail in the literature, structural failure models have received less attention so far. We formalise different
- 24. 6 1 Introduction fault assumptions taken from the literature by structural failure models and com- pare the resulting model classes with respect to their expressiveness. The resulting hierarchy is the most comprehensive classification of structural failure models so far. We introduce two new classes, namely the class of Didep models and the class of sequence-based structural failure models. The class of Didep models is strictly more expressive than the class of threshold models and covers dependent faults. The class of sequence-based models is strictly more expressive than the class of Didep models and additionally covers propagating faults. We close the chapter with showing how to map probabilistic fault models to Didep models and sequence-based models. Chapter 4 – Constructing Coteries Static and dynamic coteries are fundamen- tal means to implement coordination and agreement in fault-tolerant distributed systems. Chapter 4 demonstrates that Didep models and sequence-based models are tractable by relating them to static and dynamic coteries. Additionally, the chapter shows that these models allow more resilient and/or more efficient solu- tions. More precisely, we give constructive characterisations of highly available static coteries in terms of Didep models. The results cover both, crash and Byzan- tine failures (i.e., prematurely halting and arbitrary failures). In particular, Didep models allow to achieve high availability when being impossible with threshold models. For crash failures, we additionally give a constructive characterisation of highly available dynamic coteries. For Byzantine failures, we show that dynamic coteries do not bear advantages over static coteries with respect to high availability. Besides high availability, we address the quality measure of probe complexity. We refine the notion of probe complexity by explicitly considering sequence-based structural failure models. In contrast to the original probe complexity, the refined one gives a tight bound for a quorum set. It is significantly smaller than the original probe complexity under many failure models. Additionally, we give a universal probe strategy that meets the refined probe complexity. The probe strategy requires a number of probes equal to the refined probe complexity in the worst case due to exploiting the knowledge provided by a failure model. Chapter 5 – Reaching Consensus In Chapter 5, we use the problem of reach- ing consensus as a “benchmark problem” to demonstrate that Didep models are tractable. We show how to reach consensus under Didep models by reusing and transforming existing algorithms. This approach illustrates that using Didep mod- els does not require completely new solutions. Some of the algorithms are quorum-
- 25. 1.3 Outline 7 based allowing us to reuse the construction of highly available coteries from the previous chapter. Another aspect to demonstrate is that Didep models can be combined with dif- ferent functional failure models and different synchrony assumptions. Covering a wide range of possible assumptions, we exemplarily consider benign as well as malicious failures, transient and permanent failures, and synchrony assumptions ranging from asynchronous systems to synchronous systems. If a less expressive model (e.g., a threshold model) “under-approximates” a fault assumption, using a Didep model allows to obtain a higher assumption cover- age and, therefore, a more resilient solution. If a less expressive model “over- approximates” a fault assumption, using a Didep model allows to obtain a more efficient solution. We assess these aspects by evaluating the resilience and the efficiency of our solutions to consensus. Chapter 6 – Conclusion and Future Work Chapter 6 summarises the results of the thesis and gives an outlook on topics for future work that arised from the thesis.
- 26. 8 1 Introduction 1.4 Remarks on Notation Structured Proofs We formulate our proofs as structured proofs to ease their readability. As introduced by Lamport [1993], each structured proof is a sequence of hierarchically numbered steps. Each step has a proof on its own that may include additional steps on a lower level. The notation for the number of a step gives its level and its rank within its level. For example, a step numbered 12 is the second step on the first level. The hierarchical structure provides the general outline of the proof on high levels; the details are on low levels. Readers who are not interested in the details of a proof can skip the lower levels. Bulleted Formulae List We use the bulleted-list notation of TLA [Lamport, 2002] for conjunctions and disjunctions to make complicated formulae more read- able. A list of formulae that are bulleted with ∧ or ∨ equals the conjunction or disjunction of the formulae. The ∧ or ∨ symbols in a bulleted-list must line up exactly and indentation is used to eliminate (some) parentheses. For example, the list ∨ = 42 ∨∧ = 3 ∧ = 3 ∨ ≤ 200 denotes the formulae = 42 ∨ ( = 3 ∧ = 3) ∨ ≤ 200.
- 27. 2 Modelling Fault-Tolerant Distributed Systems A distributed system can be modelled on different levels of abstraction, from high- level specifications of its properties to low-level descriptions of its implementa- tion. We describe distributed systems by process models that represent a system by concurrent executions of sequential processes. Sequential processes represent the active entities that perform the computations in a distributed system, for ex- ample, processors, operating system processes, or threads. The most fundamental concerns on such a level of abstraction are the method of interprocess communica- tion and the timing model. The method of interprocess communication determines how different processes in a system communicate. Prominent examples include communication via message passing or by accessing shared memory. The timing model relates events in a system to the passage of time. For example, a timing model may state upper bounds on the period it takes to deliver a message. Semantically, we model distributed systems and their components by describ- ing their possible behaviour. Such behaviour is represented by traces, that is, se- quences of states. A property is a set of such traces. It represents a component or a distributed system as a whole. Component properties can be composed to a distributed system property with set intersection as (parallel) composition. As explicitly writing down sequences of states is inconvenient, we exemplarily show how a variant of temporal logic, namely the Temporal Logic of Actions (TLA), al- lows to describe properties concisely. We use TLA and pseudo-code as abstract notations for describing systems and their components. Modelling a fault-tolerant system raises the question of how to represent faults. A fault model describes the faults that may occur in a system. Usually, it describes faults in the same terms as normal system behaviour is described. A system is fault-tolerant if it is able to cope with the modelled faults and implements some high-level specification even if some of its components become faulty. Different forms of fault tolerance can be distinguished depending the specification that is implemented. For example, if faults are completely hidden from an external ob- server, a system is masking fault-tolerant. A weaker form is nonmasking fault tolerance: A system is nonmasking fault-tolerant if it does not hide the occurrence of faults, but eventually behaves permanently as desired if faults are transient. T. Warns, Structural Failure Models for Fault-Tolerant Distributed Computing, DOI 10.1007/978-3-8348-9707-7_2, © Vieweg+Teubner Verlag | Springer Fachmedien Wiesbaden GmbH 2010
- 28. 10 2 Modelling Fault-Tolerant Distributed Systems Contribution In this chapter, we introduce our system model that forms the foundation for the rest of the thesis. We give a brief overview of the different concerns of modelling fault-tolerant distributed systems. Basic terms (e.g., state, trace, and property) are formally defined for distributed systems and their compo- nents. Note that our system model is not particularly new: Variants of this model are common in the literature on fault-tolerant distributed computing. Therefore, the results that are obtained under our model also hold in settings addressed by previous models (or can be easily adapted). In particular, our system model is de- rived from the models of Abadi and Lamport [1991, 1995], and Lamport [1994], which also serve as the semantic model for TLA. Restricting their models, we limit ourselves to interleaving properties and closed systems, which are sufficient for our purposes. Overview In Section 2.1, we describe how processes communicate by passing messages over channels. Section 2.2 gives formal definitions for states, traces, and properties that are used to describe the behaviour of distributed systems. TLA is briefly summarised in Section 2.3. Fault models and fault tolerance are formally defined in Section 2.4 and Section 2.5, respectively. Section 2.6 gives an overview of different timing models, before we summarise this chapter in Section 2.7. 2.1 Interprocess Communication Processes Distributed systems are generally described by process models that represent a system by concurrent executions of sequential processes [Lamport and Lynch, 1990]. Sequential processes model “active” entities of a system that per- form computations. For example, a process may represent a computing system, a processor, or an operating system thread depending on the distributed system to be modelled. For our system model, we assume that a distributed system consists of a finite set of processes, denoted by Π = {1,...,}, 1. Interprocess Communication Different process models can be distinguished by their style of interprocess communication. For example, processes may com- municate via shared memory, remote procedure calls, or message passing. For our system model, we assume communication via message passing. This style of communication is fundamental for distributed system. For example, Lamport and Lynch [1990] consider a process model to be distributed iff its interprocess communication can be implemented using message passing.
- 29. 2.1 Interprocess Communication 11 (a) Complete graph (b) Directed Ring (c) Cube (d) Tree Figure 2.1: Examples for network topologies. Each node represents a process. Each directed edge represents a channel. Each undirected edge between two nodes represents two directed edges with opposite directions between the two nodes. Channels In a message passing model, processes communicate by sending and receiving messages over channels. The channels model entities that support com- munication among the entities modelled by processes. For example, a channel may represent a twisted pair cable, a TCP/IP connection, or a UNIX named pipe. We adopt a message passing model, in which pairs of processes communicate via unidirectional point-to-point channels. A process can directly send a mes- sage to a process iff there is a channel from to . If there is no channel from to , cannot directly send a message to (but may be able to do so indirectly with the help of other processes). We denote the set of channels in a distributed system by Ξ = {1,..., }. In particular, we assume that at least one channel exists (i.e., Ξ = / 0) and that channels and processes are different entities (i.e., Ξ∩Π = / 0). As we will often treat processes and channels alike, we speak of a component if we mean a process or a channel. A distributed system that consists of processes and unidirectional channels can be represented by a directed graph, where the nodes of the graph represent pro- cesses and the edges represent channels. In principle, different topologies of pro- cesses and channels are possible. For example, Fig. 2.1a shows a complete graph: In the system, each process can directly send a message to each other process. Fig- ure 2.1b to Figure 2.1d show systems, in which the processes and channels form a ring, a cube, and a tree, respectively. In the following, however, we will restrict ourselves to complete graphs and only consider reliable channels. Intuitively, a reliable channel does not lose messages, does not change them, and does not invent new ones on its own (cf., for example, Lynch [1996]). More precisely, a reliable channel from a process to a process is characterised by three properties:
- 30. 12 2 Modelling Fault-Tolerant Distributed Systems If sends a message to over , then eventually delivers to . Every message is delivered to by at most once. If a message is delivered to by , then was sent by . Note that reliable channels do not necessarily preserve the order in which message are sent. A channel that delivers messages in the order they were sent is called a first-in, first-out (FIFO) channel. 2.2 States, Traces, Properties Distributed systems often show unanticipated behaviour due to subtle complica- tions. Dealing with such systems, therefore, requires a rigorous treatment. We continue with introducing a formal system model that represents the behaviour of a system in terms of states, traces, and properties. States The state of a distributed system is composed from the states of the in- dividual components (i.e., the processes and the channels). Each component state consists of an external part and an internal part. The external part is the part that can be observed by an external observer of the system. It is the relevant part for describing behaviour. The internal part is hidden from external observers. It is not strictly necessary for describing a system, but often allows to formulate and understand a system description more easily. Formally, we assume that all components are defined over the same fixed sets Σ of external component states and Σ of internal component states. A com- ponent state is a pair of an external and an internal component state. An internal system state is a finite sequence of internal component states, one for each com- ponent. We denote the set of all finite sequences of length over elements in a set by . The set of all internal system states Σ equals Σ |(Π∪Ξ)| . Analogously, an external system state is a finite sequence of external component states, one for each component, with Σ = Σ |(Π∪Ξ)| denoting the set of all external system states. A system state is a pair of an external and an internal system state with Σ = Σ ×Σ denoting the set of all system states. We define a projection π that maps a system state to its external system state with π (,) = for any , ∈ Σ and a projection π that maps a component ∈ (Π∪Ξ) and a system state ∈ Σ to the component state of in . If clear from context or irrelevant, we omit the terms “system” and “component” and just write “state” instead of “system state” or “component state.”
- 31. 2.2 States, Traces, Properties 13 Traces The behaviour of a system is represented by an infinite sequence of sys- tem states. For an arbitrary set , we denote the set of all infinite sequences over elements in by ∞ . A trace is an infinite sequence of system states in Σ∞ . It represents a possible execution of a system. For a terminating system, an infinite sequence of states is obtained by repeating the final state of the system forever. We extend the projection π from states to traces: For τ = 0,1,... ∈ Σ∞, we define π (τ) as the sequence π (0),π (1),... ∈ Σ∞ . We call a sequence in Σ∞ an external trace1. Note that it may be impossible for an external observer to differentiate some system states. For example, consider a system that cycles among three different system states 0,1, and 2 starting with 0. Such a behaviour is represented by the trace τ = 0,1,2, 0,1,2,.... If 0 = 0,0,1 = 1,1, and 2 = 1,2, then the external trace π (τ) is 0, 1, 1, 0, 1, 1, .... Hence, an external observer can only distinguish two different externally visible states, although the system cycles among three different system states. Traces can be classified into interleaving and noninterleaving traces. Informally, an interleaving trace only allows steps (i.e., state transitions) that change the com- ponent state of at most one component. Noninterleaving traces allow steps that change the component state of more than one component. Such steps represent simultaneous (i.e., parallel) operations of different components. Formally, a step is a pair of system states , and is called a stuttering step iff = . For a com- ponent ∈ (Π∪Ξ), a step , is an -step iff π(,) = π(, ); that is, an -step changes the component state of . A trace 0,1,... is called inter- leaving iff, for each ≥ 0, ,+1 is a stuttering step or is an -step for exactly one component ∈ (Π∪Ξ). A trace is noninterleaving iff it is not interleaving. Stuttering Stuttering steps are an important concept for the verification of dis- tributed systems as they allow to refine a system model [Lamport, 1983a]. For an arbitrary set , a sequence 0,1,... ∈ ∞ is called stutter-free iff, for each ≥ 0, either = +1 or, to allow terminating systems, = +1 for all ≥ . For τ = 0,1,... ∈ ∞, the stutter-free form of τ is defined as the sequence that is obtained from τ by replacing every maximal finite subsequence ,+1,..., of identical elements by . Two sequences τ,ζ ∈ ∞ are equivalent up to stuttering iff their stutter-free forms are equal. 1We accept the nuisance that, formally, an external trace is not a trace. Abadi and Lamport [1991], for example, avoid this by additionally considering external system states as system states. However, such an approach would complicate the presentation and does not yield relevant advantages for our purposes.
- 32. 14 2 Modelling Fault-Tolerant Distributed Systems For a subset È of Ë∞, we define the stuttering closure κ (È) as the set of all infinite sequences over Ë that are equivalent up to stuttering to some sequence in È. Formally, κ (È) = {ζ ∈ Ë∞ : ∃τ ∈ È : ζ and τ are equivalent up to stuttering}. È is called closed under stuttering iff È = κ (È). Properties Distributed systems are represented by properties being sets of traces that are closed under stuttering. Intuitively, a property represents all possible ex- ecutions of a system. An external property is defined as a set of external traces that is closed under stuttering and represents all executions that can be seen by an external observer. We extend the projection π from traces to sets of traces by defining π(È) = {π(τ) ∈ Σ∞ : τ ∈ È} for a set of traces È ⊆ Σ∞. Note that π(È) may not be closed under stuttering even if È is a property2. The external property induced by a property È is defined by κ (π(È)). We extend the notion of interleaving to properties: A property is interleaving iff it only contains interleaving traces. Otherwise, it is noninterleaving. As interleav- ing properties are more easy to reason about, we represent distributed systems only by interleaving properties. A property is a distributed system property iff it is an interleaving property. We do not consider the restriction to interleaving properties a severe limitation, because the choice between interleaving and noninterleaving is merely one of convenience [Lamport, 2002, Sect. 10.5.2]. On a sufficiently de- tailed level of abstraction, any system can be represented by interleaving traces [Abadi and Lamport, 1995]. Describing distributed systems by writing down sets of infinite sequences is inconvenient. More abstract formalisms are required to describe a property more concisely. Well-known and useful examples of such formalisms are Guarded Com- mands [Dijkstra, 1975], temporal logic [Pnueli, 1977], and CSP [Hoare, 1978, 1985]. We do not care which particular formalism is used to describe a system, but assume that each system description defines a distributed system property. For illustration purposes, we will rely on TLA and pseudo-code when describing dis- tributed systems. 2The underlying reason is that the projection yields “new” stuttering steps: Consider a trace τ = 0,0, 0,1, 1,2,... with 0 = 1 and a property with τ ∈ , but without any trace of the form 0,3, 1,4,.... Then, π (τ) = 0, 0, 1,... is in π (), but 0, 1,... is not in π () although both are equivalent up to stuttering.
- 33. 2.2 States, Traces, Properties 15 Correctness Verifying the correctness of a distributed system means to show that the system implements a given specification that describes the properties required from the system. As a system itself may also serve as a specification for another system, we do not formally distinguish systems and specifications and represent both by properties. In abuse of terminology, we use the term specification also for the property that is defined by a specification. For example, we say that a specification is a distributed system specification if the property defined by the specification is a distributed system property. For correctness, only the behaviour that is externally visible is relevant. Internal states are only used for convenience. Formally, we say that a property 1 im- plements a property 2 iff the external property induced by 1 is a subset of the external property induced by 2. Hence, if 1 implements 2, all behaviour per- mitted by 1 – as can be seen by an external observer – is also permitted by 2. If 1 implements 2 and 2 implements 1, we call them equivalent, denoted by 1 ≡ 2. Safety and Liveness Lamport [1977] has identified two important classes of properties, safety and liveness. Their relevance stems from requiring different verification techniques and from the result of Alpern and Schneider [1985] that every property equals an intersection of a safety property and a liveness property.3 In particular, a criterion for considering a distributed system specification being well-written is that safety and liveness are explicitly separated. Informally, a safety property states that something “bad” never happens. For- mally, a property is a safety property iff, for each trace τ that is not in , there is a prefix of τ such that all traces with the same prefix are not in [Alpern and Schneider, 1985]. Intuitively, the end of the prefix marks the point in time when something bad happens that is not permitted by . A liveness property states that something “good” must eventually happen. For- mally, a property is a liveness property iff, for each finite sequence τ of states, there is a trace in that has τ as a prefix. Intuitively, the end of the finite sequence marks the point when something good happens that is required by . For example, consider the problem of reaching consensus, which we will ad- dress in detail in Chapter 5. To reach consensus, each process initially proposes a value and is supposed to eventually decide for a commonly agreed value. The problem is defined by the intersection of two safety properties, uniform agreement and validity, and a liveness property, uniform termination. These properties are defined as follows: 3Less well known, they have also shown that each property that is defined over more than one state equals an intersection of two liveness properties.
- 34. 16 2 Modelling Fault-Tolerant Distributed Systems Every process eventually decides on some value. No two processes decide differently. If a process decides a value , then was proposed by some process. The “bad thing” for uniform agreement happens when two processes decide dif- ferently. For validity, it happens when a process decides a value that has not been proposed. The “good thing” for uniform termination happens when each process decides on a value. Safety properties are generally proved using assertional methods [Keller, 1976, Ashcroft, 1975, Owicki and Gries, 1976, Lamport, 1977]. For such methods, a state invariant is found that implies the safety property to be proven. This invariant is proven by induction on the number of performed steps. For the base case, the invariant is shown to hold in the initial state (no step has been performed yet). For the induction step, it is shown that, if the invariant holds in a state, then the invariant holds after any step performed by the system. By induction, the invariant holds in any state. A classical method for proving liveness properties relies on well-founded sets and convergence functions. A well-founded set is a set with a partial order on the set such that any sequence 0,1,... with +1, ∈ , and ≥ 0 is finite. For example, the set N of natural numbers with the common “greater than” relation is a well-founded set. The convergence function (also called “ranking function” or “progress function” [Manna and Pnueli, 1982, Lynch, 1996]) maps from states to the well-founded set. Proving that something “good” eventually happens means to show that (a) something “good” happens when the value of the convergence function reaches a minimum of the well-founded set and that (b) the value of the progress function monotonically decreases with each performed step. Liveness and safety properties can also be proven by using temporal logic, a modal logic with symbols for temporal notions like “always” and “eventually” [Pnueli, 1977, Owicki and Lamport, 1982, Lamport, 1994]. We look into temporal logic in more detail in Sect. 2.3 when we give an overview of TLA. Component Properties Analogously to modelling distributed systems, we also represent the components of a distributed system by properties. To represent a component and only , a property must describe the behaviour of , but must not restrict the behaviour of the environment of (e.g., other processes and chan- nels). Informally, such a property allows the environment of to perform an arbitrary step whenever does not perform a step.
- 35. 2.2 States, Traces, Properties 17 Formally, a step , is called an -environment step iff π (, ) = π (, ) for a component ∈ (Π∪Ξ). That is, an -environment step does not change the state of . A property is a component property for iff, for each trace 0, 1,... in , each state ∈ Σ, and each ≥ 0, there is a trace 0,1,... ∈ with 0,..., = 0,..., and +1 = if , +1 and , are -environ- ment steps. That is, a component property allows an arbitrary -environment step , whenever it allows an -environmentstep , +1. Note that a component property contains noninterleaving traces. However, each step either changes the state of , changes the state of ’s environment, or is a stuttering step. Such traces are interleaving in the sense that each step is either an -step or an - environment step. Decomposed System Specifications Although a component property contains noninterleaving traces, a composition of properties for all components of a sys- tem must result in a distributed system property, which only contains interleaving traces. The following lemma shows that, with set intersection as (parallel) com- position, the composition of component properties indeed results in a distributed system property. Lemma 2.1 If is a component property for component ∈ (Π∪Ξ), the property = ∈ (Π∪Ξ) is a distributed system property. PROOF: We need to show that is a set of interleaving traces and is closed under stuttering. We prove this separately. 11. is a set of interleaving traces. PROOF: is a set of traces as each ⊆ Σ∞. For each holds that each step permitted by is either an -step or an -environment step. Hence, each step permitted by is a stuttering step or an -step for a single component ∈ (Π∪Ξ). 12. is closed under stuttering. PROOF: By 11, is a set of traces. If a trace τ is in , then τ is in each , ∈ (Π∪Ξ). As each component property is closed under stuttering, all traces that are equivalent to τ up to stuttering are also in each and, therefore, also in .
- 36. 18 2 Modelling Fault-Tolerant Distributed Systems 13. Q.E.D. PROOF: By 11 and 12. Decomposing a system specification into component specifications often eases the understanding and the verification of a specification. We now show that such decompositions are, in principle, possible for any distributed system in our sys- tem model: The model allows to compose every distributed system property from component properties. Lemma 2.2 If is a distributed system property, there exist component properties for each compo- nent ∈ (Π∪Ξ) such that = ∈ (Π∪Ξ) . PROOF: We obtain a component property for ∈ (Π∪Ξ) from by adding all traces that have the same prefix as a trace τ ∈ and only continue with - environment steps if τ continues with an -environment step. ASSUME: For ∈ (Π∪Ξ), let = ∪( ) with ( ) =κ({0,1,... ∈ Σ∞ : ∃0,1,... ∈ , ∈ N : ∧0,..., = 0,... ∧,+1 is an -environment step ∧∀ ≥ : ,+1 is an -environment step}). PROVE: is a component property for and = ∈ (Π∪Ξ) . 11. is a component property for . PROOF: is a property as and ( ) are sets of traces that are closed under stuttering: If two sets of traces are closed under stuttering, their union is closed under stuttering as well. is a component property for as allows an arbitrary -environment step whenever it allows an -environment step. 12. ⊆ ∈ (Π∪Ξ) PROOF: ⊆ for each component ∈ (Π∪Ξ) by the construction of . 13. ∈ (Π∪Ξ) ⊆ PROOF: The proof is by induction over prefixes of a trace, with the base case proved in step 1 and the induction step in step 2. ASSUME: 0,1,... ∈ ∈ (Π∪Ξ) PROVE: 0,1,... ∈ 21. There is a trace in with the prefix 0. PROOF: Otherwise, there would be no 0,1,... in ∈ (Π∪Ξ) . 22. If there is a trace in with the prefix 0,...,, then there is a trace in with the prefix 0,...,+1.
- 37. 2.3 Temporal Logic of Actions 19 PROOF: 31. CASE: , +1 is a stuttering step PROOF: By the induction hypothesis and as is closed under stuttering. 32. CASE: , +1 is an -step for a component ∈ (Π∪Ξ) PROOF: By the induction hypothesis and as, otherwise, there would be no trace with the prefix 0,..., +1 in by the construction of (). 33. Q.E.D. PROOF: By 31 and 32. 23. Q.E.D. PROOF: By 21, 22, and mathematical induction. 14. Q.E.D. PROOF: 11 – 13 2.3 Temporal Logic of Actions Our results do not depend on a particular formalism to express a property. Never- theless, we exemplarily show how such an formalism, namely the Temporal Logic of Actions (TLA) from Lamport [1994], allows to present properties concisely. Temporal logic in general provides useful means to specify and reason about con- current systems [Lamport, 1983a]. TLA is a variant of temporal logic for spec- ifying and verifying concurrent systems in terms of their actions. The main dif- ferences to the well-known temporal logic of Pnueli [1977] are invariance under stuttering, support for temporal existential quantification, and allowing action for- mulae as atomic formulae. Unlike other formalisms, both, systems and their specifications, are represented in the same logic. A system implements its specification iff the logic formula that describes the system implies the formula that describes its specification, where “implies” means logical implication. Structural relations among components are also represented by logical operators. For example, logical conjunction is parallel composition. We now briefly summarise the syntax and the semantics of TLA. Refer to Lam- port [1994], Abadi and Merz [1996], and Lamport [2002] for more detailed pre- sentations. Syntax Syntactically, TLA formulae are built from constant symbols, variable symbols, and the special symbols ¬,∧,2,∃,∃ ∃ ∃ ∃ ∃ ∃, ,(,), and =. The variables are partitioned into sets of rigid variables, whose values are state-independent, and flexible variables, whose values are state-dependent.
- 38. 20 2 Modelling Fault-Tolerant Distributed Systems An action is a first-order predicate over the constant and variable symbols, for example, = + 1. With free variables defined as in first-order logic, a state function is a first-order expression over the constant and variable symbols without free primed variables, for example, + −1. Analogously, a predicate is an action without free primed variables, for example, = +1. We define substitution for actions, state functions, and predicates as in first- order logic and write, for example, {/} for the results of substituting the first- order expression (over constant and variable symbols) for the free occurrences of variable in action . For any state function (or predicate) , is defined as the state function (or predicate) obtained by priming the free flexible variables. For example, ( + − 3) equals ( + − 3). A TLA formula is built from predicates and formula of the form 2[] , where is an action, is a state function, and [] is an abbreviation for (∨( = ). Additionally, 2 is a formula if is one. For example, 2( = +1) and 2[ = +1] are formulae. In addition to common abbreviations such as ∨,⇒, and ≡ or the temporal oper- ator 3, the following abbreviations are defined: • Δ = ∧( = ), • UNCHANGED Δ = = , • if 1,..., are the free variables of , then ENABLED Δ = ∃1,..., : , • if is the tuple 1,..., , then ∃ ∃ ∃ ∃ ∃ ∃ : Δ = ∃ ∃ ∃ ∃ ∃ ∃1 : ...∃ ∃ ∃ ∃ ∃ ∃ : • a weak fairness operator WF () Δ = 32ENABLED ⇒ 23 , and • a strong fairness operator SF () Δ = 23ENABLED ⇒ 23 , where ,1,..., are variables, is an action, and a state function. Semantics The meaning of TLA formulae is defined over a set of values. We assume a fixed set of values that includes all required values. A state is a mapping from the set of flexible variables to the set of values. The internal state is given by the mapping from the set of hidden flexible variables (see below) to the set of values; the external state by the mapping from free flexible variables to values. As in the previous section, a trace is an infinite sequence of states. The meaning of a state function is a mapping from the set of states to the set of values. Analogously, a predicate is either true or false for a state. A state satisfies a predicate iff the meaning of the predicate is true for the state. A predicate is
- 39. 2.3 Temporal Logic of Actions 21 called valid, denoted by |= , iff every state satisfies the predicate. The meanings of common operators such as ∧,¬, and ∃ : are standard. The semantics of actions is defined for pairs of states, where the unprimed vari- ables refer to the first state and the primed variables to the second one. For exam- ple, = +1 is true at a pair of states , iff the value of in equals the value of plus 1 in . A pair of states satisfies an action iff the meaning of the action is true for the pair of states. A pair of states that satisfies an action is called an step. An action is called valid, denoted by |= , iff every step is an step. Analogously to state predicates, a formula is either true or false for a trace. A trace satisfies a formula iff the meaning of the formula is true for the trace. A formula is called valid, denoted by |= , iff every trace satisfies the formula. Each formula represents a set of traces, namely the set of all traces that satisfy the formula. As such, a formula can describe a property and, therefore, specify a distributed system. The meaning of a formula is defined inductively as follows. A state predicate is true for a trace iff it is true of its first state. Derived from temporal logic, 2 denotes the always operator; 2 means that is always true in the future. The operator ∃ ∃ ∃ ∃ ∃ ∃ denotes temporal existential quantification; the formula ∃ ∃ ∃ ∃ ∃ ∃ : means that there exists a sequence of values for such that holds. Intuitively, ∃ ∃ ∃ ∃ ∃ ∃ hides variables from external observers: is called an internal or hidden variable of ∃ ∃ ∃ ∃ ∃ ∃ : . Such a variable is part of the domain for the mapping that defines the internal state. Specifications In TLA, a specification is expressed by a formula that has the “canonical form” ∃ ∃ ∃ ∃ ∃ ∃ : ∧2[ ]Ú ∧ , where is a state predicate, is an action describing possible steps of the system, is a conjunction of fairness conditions, is a variable, and a state function. The formula is true for all those traces, whose initial state satisfies , where every step is a step or leaves unchanged, and where holds. Note that each TLA formula defines a property, namely the set of all traces that satisfy the formula, as each TLA formula is invariant under stuttering [Lamport, 1994, Abadi and Merz, 1996]. TLA is an expressive formalism that allows to describe different kinds of speci- fications. For example, it supports interleaving and noninterleaving specifications and allows to represent closed as well as open systems. Intuitively, an open system interacts with its environment (being beyond the control of the system). A closed system is self-contained: it does not interact with its environment, but its external state may be inspected by an external observer. We restrict ourselves to closed sys- tem specifications as these are more easy to reason about and as, for our purposes,
- 40. 22 2 Modelling Fault-Tolerant Distributed Systems it suffices to model inputs to a system as being nondeterministically generated by the system itself. Decomposed System Specifications Semantically, a state is a mapping from flexible variables to values in TLA. We assume that the flexible variables of a dis- tributed system specification can be partitioned among the components. A com- ponent can only change its own variables, which cannot be changed by any other component. The internal local state of a component is then given by the map- ping from the set of hidden flexible variables of to the set of values. The external local state of is given by the mapping from the set of free flexible variables of to the set of values. Component specifications arise when composing a system from reusable com- ponents or decomposing a given system into its components. We only deal with system specifications that are decomposed into component specifications such that the environment of each component is known.4 For a component ∈ (Π∪Ξ), we assume that the tuple of output variables is the tuple of the free flexible variables of and that the tuple of environment variables is the tuple of the flexible vari- ables of all other components. A component specification then has the canonical form ∃ ∃ ∃ ∃ ∃ ∃ : ∧2[ ]Ü,Ó ∧ with: • is the tuple of all internal variables of the component. The set of variables in are precisely the domain of the mapping that defines the internal local state of the component. • The state predicate specifies the initial values of the component’s inter- nal variables and output variables . • The action allows steps of the component that change the value of an output variable of with ⇒ = . • is a conjunction of fairness conditions. Each fairness condition must be of the form Ü,Ó( ) or Ü,Ó( ), where is an action. The formula 2[ ]Ü,Ó allows the environment to do anything but change the component’s internal or output variables. Due to ⇒ = , the values of the 4In particular, we do not consider assume/guarantee specifications of the form + − , which as- sert that a component satisfies a guarantee as long as the environment satisfies an assumption [Abadi and Lamport, 1995]. Such specifications allow to describe reusable components for unknown environments.
- 41. 2.3 Temporal Logic of Actions 23 Δ = , , Δ = ∧ ∈ BOOLEAN ∧ = ∧ ∈ BOOLEAN ∧ = ∧ = / 0 Δ = ∧ = ∧ = ∪{ } ∧ = ¬ ∧ UNCHANGED , , ( ) Δ = ∧ ∈ ∧ = ∧ = ¬ ∧ = ∧ = { } ∧ UNCHANGED , Δ = ∨∃ : ( ) Δ = ,() Δ = ∃ ∃ ∃ ∃ ∃ ∃ : ∧2[] , ∧ Figure 2.2: A TLA specification of a reliable channel (cf. Abadi and Lamport [1995]). component’s and its environment variables cannot change simultaneously. Hence, the specification defines a component property. Example 2.1 (TLA Component Specification) An example of a TLA component specification is given in Fig. 2.2. The formula spec- ifies a reliable channel in terms of three output variables, , , and , and one internal variable, . Additionally, the specification refers to three environment variables, , , and . These variables belong to the processes that are connected by the channel: and to the sender process and to the receiver process. Figure 2.3 shows a schematic view on the specification. As an abbreviation the output variables are joined in the sequence . The internal variable (hidden by temporal existential quantification) holds the set of messages that are currently buffered by the channel, that is, messages sent by the sender process, but not yet delivered to the receiver process. Initially, is empty. The Boolean variables and are used to implement a simple handshake protocol between the channel and the sender process. Initially, = , which
- 42. 24 2 Modelling Fault-Tolerant Distributed Systems Figure 2.3: A schematic view on a reliable channel. models that no new message is to be sent. The sender process sends a message as follows: It puts into and signals the new message by negating . The channel handles by acknowledging the sending with the action . A precondition (or “guard”) for is that = , that is, the sender process must have signalled a new message. The message is added to and is negated to signal that the channel is ready for the next message. Delivering a message to the receiving process is performed similarly by the action: The message to deliver must be in . The channel removes from and puts it into . The channel signals the new message by negating . The receiving process can take the message from and acknowledges that it is ready for further messages by negating . Both actions, and , are subsumed by the action for the final specification. To ensure that a sent message is indeed delivered, the specification includes a liveness formula defined by weak fairness over the actions. Verification Using TLA Being a logic, TLA includes axioms and proof rules for proving formulae. If a formula is provable by the axioms and rules of the logic, denoted by , then is also valid as TLA is a sound logic. For verification, a specification 1 implements another specification 2 iff 1 ⇒ 2; that is, logical implication is implementation. As we use TLA just for presenting specifications, but not for verification, we do not further discuss the axioms and proof rules of TLA. Refer to Lamport [1994] for a detailed description and to Lamport and Merz [1994] for an elaborate example of how to verify a fault-tolerant distributed system using TLA.
- 43. 2.4 Fault Model 25 2.4 Fault Model Despite interprocess communication, the fault model, the formalisation of a fault assumption, is another important concern of modelling fault-tolerant distributed systems. No system can tolerate arbitrary severe faults. Any consideration of fault tolerance requires a decision, which faults to tolerate, resulting in a fault assumption. For a rigorous treatment of fault tolerance, a fault assumption must be for- malised in terms of the respective system model. The common approach has been to represent a fault assumption as a mapping (or “transformation”) that augments a specification by faults. We follow this approach and define a fault model as a mapping of distributed system properties. Refer to Gärtner [1999] for a survey of the wide range of approaches that rely on mappings for the specification and verification of fault-tolerant systems. Fault Actions and Variables The key for formalising fault assumptions is the observation of Cristian [1985] that a system changes its state either due to a normal system step or due to a fault step. This observation allows a simple representation of faults: syntactically, they are represented in the same way as normal system steps. For example, if a system is specified by a TLA formula, which describes steps by actions, then faults can be modelled by augmenting the formula with additional fault actions. While not strictly necessary [Gärtner, 2002], it is often convenient to augment a specification by fault variables. Fault variables indicate the manifestation of faults and help to structure a specification. They make a specification easier to understand. For example, with a Boolean fault variable , the manifestation of a fault is indicated by setting from FALSE to TRUE. If ¬ is added as a conjunct to each normal action and to each fault action, normal actions are disabled and fault actions are enabled by the manifestation of a fault. These additional conjuncts help to easily separate normal from fault actions. Consider the TLA specification in Fig. 2.4a. The specification contains a sin- gle variable , which initially equals 0. Each step either leaves unchanged or increments by 1. Now consider a fault that prevents any further increase of , but decreases . Such a fault is formalised in Fig. 2.4b. The manifestation of a fault is represented by the action that sets the fault variable , which is ini- tially FALSE, to TRUE. If is TRUE, the normal action is disabled, whereas the fault action is enabled. While the original specification is only true for traces, where the value of increases with every step that changes , the trans- formed specification is also true for traces, in which the value of increases for some time, but then permanently decreases.
- 44. 26 2 Modelling Fault-Tolerant Distributed Systems Δ = = 0 Δ = = + 1 Δ = Δ = Ú ( ) Δ = ∧2[ ]Ú ∧ (a) Example specification Δ = = 0∧ = FALSE Δ = ∧¬ ∧ = + 1∧ UNCHANGED Δ = ∧ ∧ = − 1∧ UNCHANGED Δ = = TRUE ∧ UNCHANGED Δ = ∨ ∨ Δ = Ú ( ) Δ = ∧2[ ]Ú ∧ (b) Specification with fault actions and variables Figure 2.4: Example of augmenting a TLA specification with fault actions and variables. The approach of modelling faults by additional actions and variables (or equiv- alent approaches) is common in the literature [Liu and Joseph, 1992, Arora and Gouda, 1993, Arora and Kulkarni, 1998a, Gärtner, 1998, 1999] across different system models (e.g., CSP [Nordahl, 1992, 1993] or Petri nets [Völzer, 1998]). Closely related to our TLA example, Liu and Joseph [1992, 1995, 1996, 1999, 2006] also use TLA for specifications and model faults by fault actions and vari- ables. Fault Models Due to faults, an external observer may see behaviour that would not have been possible without faults. A fault model allows a system to show additional externally visible behaviour. Semantically, we define a fault model as a mapping that weakens a distributed system property. Definition 2.1 (Fault Model) A fault model α is a mapping from a distributed system property to a distributed system property α() such that implements α(). If a distributed system property implements α(), then any externally visible trace permitted by is also permitted by α(). This reflects that a fault model only describes possible manifestations of faults, but does not make them obliga- tory. Defining fault models as mappings of properties bears advantages for comparing the severeness of different fault models. Consider a distributed system property and two fault models α and α . If α () implements α (), then α allows at
- 45. 2.4 Fault Model 27 least the same additional behaviour as α and possibly more behaviour with respect to . If α () implements α () for any distributed system property, then α can be considered to describe “more severe” faults as α . Faults as Component Failures We do not consider stuttering steps as fault steps, because, otherwise, each distributed system would include fault steps. As our system model allows to attribute each nonstuttering step to a single compo- nent, each fault step can be assigned to a single component as well. We, therefore, call a fault step that is an -step for a component ∈ (Π∪Ξ), an -failure step. This terminology reflects that we change our view on faults from system- to component-level: A system contains (system) faults that must be tolerated; system faults are component failures. We partition the states into failed and nonfailed states for each component . Intuitively, a state is a failed state for a component iff suffered from a failure and cannot perform “useful” work at the moment. Formally, we assume a state predicate that is invariant under -environment steps and is true for any state that immediately results from an -failure step. If is true in a state, we call failed in this state and, otherwise, nonfailed. A component may recover from a failed state and continue to perform useful work. A component recovery of is an -step that leads from a state, where is true, to a state, where is false. Let τ = 0,1,... ∈ Σ∞ be a trace. A component ∈ (Π∪Ξ) is considered correct in τ iff is false for each state , ≥ 0. Hence, an -failure- step invalidates the correctness of . Even if a component is not correct, it may perform useful work if it recovers and remains nonfailed. We call such components nonfaulty. More precisely, a component is nonfaulty in τ iff there is an ≥ 0 such that ( ) is false for each ≥ . As a special case any correct component is nonfaulty. If a component is not nonfaulty, we call the component faulty. Note that is faulty iff is infinitely often true such that may not be able to perform any useful work. Figure 2.5 illustrates the terms correct, faulty, and nonfaulty. We assume that each -failure step is externally visible, that is, changes the external system state of component . Such a requirement is useful for high-level specifications that refer to the occurrence of component failures. For example, some properties give requirements that have to be satisfied by nonfaulty compo- nents. Having externally visible -failure steps eases the specification of such requirements.
- 46. 28 2 Modelling Fault-Tolerant Distributed Systems (a) A correct component. (b) A nonfaulty component. (c) A faulty component (without recovery). (d) A faulty component (with intermittent recov- ery). Figure 2.5: Correct, faulty, and nonfaulty components. A correct component is always non- failed. A nonfaulty component is eventually permanently nonfailed. A faulty component fails infinitely often. We denote the faulty components of a trace τ by (τ); More precisely, we define (τ) = { ∈ (Π∪Ξ) : ∀ ∈ N : ∃ ≥ : ( ) = TRUE} for a trace τ = 0, 1,.... We extend from traces to properties to denote all components that are faulty in some trace of a property: ( ) = τ ∈ (τ) for a property . In particular, allows to construct a necessary condition for the equivalence of distributed system properties: If the sets of faulty components for two properties are not equal, then the properties are not equivalent. This result will be used later for comparing fault models: If two fault models are applied to the same distributed system property and result in different sets of faulty components, the fault mod-
- 47. 2.5 Fault Tolerance 29 els express different fault assumptions. Let and be two distributed system properties. Lemma 2.3 If ≡ , then ( ) = . PROOF: We assume that ≡ . As ( ) = τ ∈ (τ) and ( ) = τ ∈ (τ) by the definition of , it suffices to prove τ ∈ (τ) = τ ∈ (τ). 11. τ ∈ (τ) ⊆ τ ∈ (τ) PROOF: 21. ∀τ ∈ : ∃τ ∈ : π (τ ) = π (τ ) PROOF: As ≡ . 22. ∀, ∈ Σ, ∈ (Π∪Ξ) : π () = π () ⇒ () = () PROOF: As fault steps are externally visible. 23. ∀τ ,τ ∈ Σ∞ : π (τ ) = π (τ ) ⇒ (τ ) = (τ ) PROOF: If π (τ ) = π (τ ), then π (0) = π (0),π (1) = π (1),... for τ = 0,1,... and τ = 0,1,.... Then, by 22, (0) = (0), (1) = (1),... for each ∈ (Π∪Ξ). (τ ) = (τ ) follows from the definition of . 24. Q.E.D. PROOF: By 21 and 23. 12. τ ∈ (τ ) ⊇ τ ∈ (τ ) PROOF: Analogously to the proof of 11. 13. Q.E.D. PROOF: By 11 and 12. The definitions of “correct” and “nonfaulty” may seem too strong for real-world systems as no realistic component will be nonfailed forever. We do not consider this a severe limitation, but just a matter of convenience: Any realistic system has an upper bound to fulfill its purpose (e.g., to provide a service). Hence, a component must only be nonfailed up to this upper bound. For convenience, we do not give these bounds explicitly. If necessary, they can be determined for a given system. 2.5 Fault Tolerance While the notion of fault tolerance was only treated informally so far, we are now ready to formally define what it means for a system to be fault-tolerant. We as- sume a distributed system that is given by a low-level specification and a fault
- 48. 30 2 Modelling Fault-Tolerant Distributed Systems assumption that is formalised by a fault model α. Furthermore, we assume a high- level specification that captures the desired behaviour of the system such that implements . We adopt the definitions of Arora and Gouda [1993], Arora and Kulkarni [1998a,b], and Gärtner [1999] and distinguish three different forms of fault tolerance: masking, nonmasking, and fail-safe fault tolerance. Ideally, a system should be able to implement its problem specification despite any assumed fault. This is the most strict form of fault tolerance and called mask- ing fault tolerance. Definition 2.2 (Masking Fault Tolerance) A distributed system given by a property is masking fault-tolerant for a specification and a fault model α iff α( ) implements . Masking fault tolerance is the most desirable form of fault tolerance as faults do not affect at all whether a system implements its specification or not. In this sense, the system “masks” the manifestation of faults. Unfortunately, it may not be possible or too costly to achieve masking fault tolerance. In such cases, it may be feasible and acceptable to implement a weaker variant of the original problem specification: is weakened to a specification ( implements ) and only is required to be implemented. Weakenings for nonmasking and fail-safe fault tolerance rely on the result that each property can be written as the intersection of a safety and a liveness property (see Sect. 2.2). While fail-safe fault tolerance requires to implement the safety part of a problem specification and allows to violate liveness, nonmasking fault tolerance requires to implement the liveness part and allows to violate safety requirements for a finite amount of time. Definition 2.3 (Fail-Safe Fault Tolerance) A distributed system given by a property is fail-safe fault-tolerant for a specification ≡ Ë ∩ Ä and a fault model α iff α(È ) implements Ë, where Ë is a safety property and Ä is a liveness property. For example, consider the problem of reaching consensus. If a system only de- cides agreed upon values that were indeed proposed despite faults, then it imple- ments the safety properties of consensus (i.e., agreement and validity). If the faults prevent processes from deciding, the system violates liveness (i.e., termination). Hence, the system may fail, but fails safely; it is fail-safe fault-tolerant. From a theoretical point of view, another form of fault tolerance could be de- fined that only requires to implement the liveness part of a problem specification. However, such a fail-live variant has been basically neglected in the literature as it seems to lack reasonable applications [Gärtner, 2001]. Other variants, for ex- ample, implementing some liveness properties of a problem that is given as the intersection of liveness properties have not been addressed yet as well. A stricter variant of fail-live, namely nonmasking fault tolerance, is more popular and has found various applications.
- 49. Another Random Scribd Document with Unrelated Content
- 50. This would mean that after the weapon is thrown it might be drawn back again with a leather thong. Possibly the cateia of Isidore (cateia, to cut or mangle, and catan, to fight; the Irish caꞇ̇ and the Welsh kad, a fight or a corps of fighters, Latin caterva), survives in the tip-cat. In the Keltic dialect of Wales catai is a weapon. [116] See his learned note (p. 410) on the weapon and on Isidore (Orig. xviii. 7): ‘Hæc est cateia quam Horatius cajam dicit.’ The disputed word probably derives from the Keltic katten, to cast, to throw. [117] Nile Tributaries, by Sir Samuel W. Baker, p. 51. The word has a curious likeness to the ‘tombat,’ a similar weapon in Australia (Col. A Lane-Fox, Anthrop. Coll. p. 31). [118] The ‘Fans’ of M. du Chaillu, a corruption unfortunately adopted by popular works. In Gorilla-Land (i. 207) I have noticed the Náyin, or Mpangwe crossbow (with poisoned ebe, or dwarf bolt), which probably travelled up-Nile like the throw-stick. The détente and method of releasing the string from its notch are those of the toy forms of the European weapon. The Museum at Scarborough contains a crossbow from the Bight of Benin. The people of Bornu (North-West Africa) also use a crossbow rat-trap. [119] It is called chakarani in the Coasts of East Africa and Malabar Coast, by Duarte Barbosa or Magellan (?). The Jibba negroes of Central Africa wear a similar weapon as a bracelet, sheathed in a strip of hide. [120] Col. A. Lane-Fox, Anthrop. Coll., p. 33. For a comparative anatomy of the boomerang the reader will consult that volume, pp. 28–61. I have here noticed only the most remarkable points. [121] The Sword stood in Case 2 of the Salle du Centre, numbered 695; and was described in p. 225 of the late Mariette Pasha’s catalogue. I cannot quite free myself from a suspicion that it was also a boomerang of unusual size. Some of the South African tribes still use throw-sticks a yard to a yard and a half long. ‘They are double as thick at one end as they are at the other,’ says Herr Holub (ii. 340), ‘the lighter extremity being in the usual way about as thick as one’s finger.’ [122] This meaningless word (cartuccia, a scrap of paper) was applied by Champollion to the elliptical oval containing a group of
- 51. hieroglyphics. It is simply an Egyptian shield (Wilkinson, loc. cit. i. chap. 5), and the horizontal line below shows the ground upon which it rested. The old Nile-dwellers, like the classics of Europe and the modern Chinese, use the shield for their characteristics, their heraldic badges, c. The same was the case with our formal heraldry, which originated about the time of the Crusades, personal symbolism being its base. As Mr. Hardwick shows, the horse, raven, and dragon were old familiar badges; many of our sheep-marks are identical with ‘ordinaries,’ and the tribes of Australia used signs to serve as kobongs, or crests. Thus, too, in fortification the shield became the crenelle and the battlement, and it served to ‘iron-clad’ the war-galleys of the piratical Norsemen. [123] So there are two ways of swimming. The civilised man imitates the action of the frog, the savage the dog, throwing out the arms and drawing the hands towards his chest. [124] Journ. Anthrop. Inst. vol. iii. pp. 7–29, April, 1873. [125] An illustration is given in Mr. J. G. Wood’s Natural History of Man. He also quotes Mr. F. Baines, who describes the paddles of the North Australians with barbed and pointed looms. [126] Capt. James Mackenzie, in a paper read before the Ethno. Soc. by Mr. G. M. Atkinson (Journal, vol. ii. No. 2, of July 18, 1870. The paddle is figured pl. xiv. 2). [127] Translated for the Hakluyt Society (1874) by Mr. Albert Tootal, of Rio de Janeiro, who wisely preserved the plain and simple style of the unlettered and superstition-haunted gunner. [128] In Bacon’s day (Aphorisms, book ii.) gummy woods were supposed to be rather a Northern growth, ‘more pitchy and resinous than in warm climates, as the fir, pine, and the like.’ They are as abundant near the Equator, where the viscidity preserves them from the alternate action of burning suns and torrential rains; moreover, they are harder and heavier than the pines and firs of the Temperates. [129] Historia Geral do Brazil, by F. Adolpho de Varnhagen, vol. i. p. 112 (Laemmert, Rio de Janeiro, 1854). [130] M. Paul Bataillard (p. 409, Sur le Mot Pagaie, Soc. Anthrop. de Paris, 1874) is in error, both when he calls the people of Paraguay ‘Pagayas,’ or ‘carriers of lances,’ and when he identifies
- 52. Pagaya (not a spear, but a paddle-sword) with the ‘sagaia or assagai.’ The latter word is of disputed origin, and it is meaningless in the tongues of South Africa. Space forbids me to touch its history, except superficially. ‘Azagay,’ a lance, or rather javelin, appears in Spanish history as far back as the days of Ojeda (1509); and in 1497 the Portuguese of Vasco da Gama’s expedition use the term ‘azagayas’ (p. 12, Roteiro or Ruttier, before alluded to). I believe both to be derived from the Arabic el- khazúk, a spit—in fact, the Italian spiedo, lance. [131] Markham (p. 203, Cieça de Leon) makes ‘Macaná’ a Quichua word; it also belongs to the great Tupi-Guarani family. [132] Antiquarian Researches, quoted by Markham, loc. cit. p. 181. [133] The Godeffroy Collection has produced a huge Catalogue of 687 pages (Die ethnographisch-anthropologische Abtheilung des Museum Godeffroy in Hamburg, vol. i. 8vo (L. Friederichsen u. Co. 1881). It was shown to me by Dr. Graeffe, the naturalist often mentioned in ‘South Sea Bubbles, by the Earl and the Doctor.’ As a rule the Samoans had clubs and spears, but few Swords. [134] This part of Melanesia has been familiar to the home reader by the life, labours, and death of Bishop Patterson. [135] Case 21, Petrie, No. 142. [136] The village of Abu Rawásh, north of the Pyramids of Jízah, still works this material in large quantities; and its caillouteurs, or flint-knappers, have produced excellent imitations of the so-called prehistoric weapons. I have described the flint finds of Egypt in the Journ. Anthrop. Instit. (Feb. 1879), and shall have something more to say about them. A Mr. R. P. Greg, who writes in the same Journal (May 1881) on the ‘Flint Implements of the Nile Valley,’ is not aware of the fact that I found worked flints near the larger petrified forest (Cairo). Since that time General Pitt-Rivers made his grand discovery of ‘Chert Implements in stratified Gravel in the Nile Valley’ (Journ. Anthrop. Inst. May 1882). In March 1881, when visiting the Wady, near Elwat El-Díbán (Hill of Flies) amongst the cliffs of Thebes, he came upon palæolithic flints, flakes worked with bulbs and facets embedded in the hardened grit, six and a half to ten feet below the surface. In the same strata tombs had been cut, flat-topped chambers with quadrangular pillars. The fragments of pottery enabled Dr. Birch
- 53. to pronounce these excavations ‘not later than the eighteenth dynasty, and perhaps earlier.’ The New Empire in question was founded by Amosis (Mah-mes, or Moon-child) circ. b.c. 1700; it included the three great Tothmes, and lasted about three hundred years, ending with the heretic Amun-hotep IV., slave of Amun, circ. b.c. 1400, and Horemhib, the Horus of Manetho. The worked flints may evidently date thousands of years before that period. This is a discovery of the highest importance, and we may expect, with Mr. Campbell, that the ‘works of men’s hands will be found abundantly underlying the oldest history in the world, in the hard gravel which underlies the mud of the Nile-hollow from Cairo to Assouan.’ At any rate, this find disposes of the scientific paradox that Art has no infancy in Nile-land. The strange fancy has been made popular by the Egyptologist, who threatens to become as troublesome as the Sanskritist. [137] It is figured (p. 8) by Dr. John Evans (Ancient Stone Implements, c.), who offers another ‘poniard’ (perhaps a scraper) on p. 292. On p. 308 he notes the large thin flat heads called ‘Pechs’’ (Picts’?) knives.’ [138] Nephrite is so called because once held a sovereign cure for kidney disease. Jade is found in various parts of Europe (Page); in the Hartz (or Resin) Mountains; in Corsica (Bristowe), and about Schweinsal and Potsdam (Rudler). Saussurite, the ‘Jade of the Alps,’ appears about the Lake of Geneva and on Monte Rosa. Mr. Dawkins limits Jade proper in the Old World to Turkestan and China. Jade, the Chinese you, is popularly derived from the Persian jádú = (the) magic (stone). [139] I need hardly notice that the mussel-shell was the original spoon, still a favourite with savages. [140] Humboldt (Pers. Narr. vol. i. p. 100) makes the Guanches call obsidian ‘tabona’; most authors apply the word to the Guanche knife of obsidian. [141] Neuhoff, Travels, c. xiv. 874. [142] Our word ‘glass’ derives from glese (gless, glessaria), applied by the old Germans to amber (Tacit. De Mor. Germ. cap. 45). Pliny (xxxvii. chap. 11) also notices glæsum (amber) and Glæsaria Island, by the natives called Austeravia. [143] Stephens, Yucatan, i. 100.
- 54. [144] The curious and artistic rock inscriptions and engravings of the South African Bushmen were traced in outline by triangular flint-flakes mounted on sticks to act as chisels. The subjects were either simple figures; cows, gnus, and antelopes, a man’s bust and a woman carrying a load; or compositions, as ostrich and rider, a jackal chasing a gazelle, or a rhinoceros hunting an ostrich. [145] See Chap. I. [146] Voyage Pittoresque autour du Monde, par M. Louis Choris, Peintre, 1822. [147] Trans. Ethno. Soc. vols. i. and ii. p. 290. [148] Quoted by Col. Lane Fox, Prim. War. i. 25. [149] Prehistoric Man, by Daniel Wilson (vol. i. pp. 216–17). [150] Incidents of Travel in Central America, c., p. 51; by J. Lloyd Stephens. The work is highly interesting, because it shows Egypt in Central America. Compare the Copan Pyramid with that of Sakkarah; the Cynocephalus head (i. 135) with those of Thebes; the beard, a tuft on the chin; the statue and its headdress (ii. 349); the geese-breeding at the palace (ii. 316); the central cross (ii. 346) which denotes the position of the solstices and the equinoxes and the winged globe at Ocosingo (ii. 259). In Yucatan the Agave Americana took the place of the papyrus for paper-making. Indo-China also appears in the elephant-trunk ornaments (i. 156). [151] Prim. War. ii. p. 25. [152] The two latter are in Demmin, p. 84. [153] A specimen is in the British Museum, Department of Meteorolites. (Prim. War. p. 25.) [154] The distinguished physicist, Prof. Huxley, extends on purely anthropological grounds, the name ‘Australioids’ to the Dravidians of India, the Egyptians, ancient and modern, and the dark- coloured races of Southern Europe. I have ventured to oppose this theory in Chap. VIII. Mr. Thomas, curious to say, would make letters (alphabet, c.) arise amongst the Dravidian quasi-savages. [155] Trans. Anthrop. Inst. May 1881. Mr. Milne brought home some fine specimens of worked stones, one of which (No. 17, pl.
- 55. xviii.) is a chopper in the shape of the Egyptian flint-knives. [156] Mr. Heath (who directed the Indian Iron and Steel Company) opined that the tools with which the Egyptians engraved hieroglyphics on syenite and porphyry were made of Indian steel. The theory is, as we shall see, quite uncalled for. [157] For instance, the magnificent life-sized statue of Khafra (Cephren or Khabryes) in the Bulak Museum, dated b.c. 3700– 3300 (Brugsch, History, vol. i. p. 78). Scarabæi of diorite can be safely bought in Egypt, the substance being too hard for cheap imitation work. Dr. Henry Schliemann constantly mentions diorite in his Troy and its Remains (1875); for instance, ‘wedges’ (i.e. axes) large and small, (pp. 21, 28, 154): he speaks of an immense quantity of diorite implements (p. 75); of a Priapus of diorite twelve inches high (p. 169); of ‘curious little sling bullets’ (p. 236), and of hammers (p. 285). At Mycenæ he found ‘two well-polished axes of diorite.’ But as he also calls it ‘hard black stone,’ I suspect it to be basalt, as his ‘green stone’ (Troy, p. 21) may be jade or jadeite. [158] Casting the cannon called after the late General Uchatius is still kept a secret; and I have been unable to see the process at the I. R. Arsenal, Vienna. [159] Stahl-bronce = steel (i.e. hardened) bronze. The misunderstanding caused some ludicrous errors to the English press. [160] I reported to the Athenæum (August 16, 1879) this ‘recovery’ of the lost Egyptian (and Peruvian) secret for tempering copper and bronze, which had long been denied by metallurgists. Copper hardened by alloy is described in the Archæologia, by Governor Pownall. Mr. Assay-Master Alchorn found in it particles of iron, which may, however, have been in the ore, and some admixture of zinc, but neither silver nor gold. [161] Of this I shall have more to say in Chap. V. [162] This was the weight of the statue of ‘Sesostris,’ Ramses II., and his father Pharaoh Seti I.; see Chap. IX. The overseer standing upon its knee appears about two-thirds the length of the lower leg (Wilkinson, Frontisp. vol. ii.). Pliny treats of colossal statues, xxxiv. 18.
- 56. [163] Les Métaux dans l’Antiquité, par J. P. Rossignol. Paris: Durand, 1863. [164] So Professor F. Max Müller, Lectures on the Science of Language, asserted, with a carelessness rare in so learned a writer (vol. ii. p. 255. London: Longmans, 1873), that ‘the ancients knew a process of hardening that pliant metal (copper), most likely by repeated smelting (heating?) and immersion in water.’ This latter is the common process for softening the metal. [165] Cieza de Leon (Introd. p. xxviii.): ‘Humboldt mentions a cutting instrument found near Cuzco (‘the City’) which was composed of 0·94 parts of copper and 0·06 of tin. The latter metal is scarcely ever found in South America, but I believe there are traces of it in parts of Bolivia. In some of the instruments silica was substituted for tin.’ The South American tin is mostly impure; still it was and can be used. [166] Apparently there are two forms of ‘Núb’ (gold), the necklace and the washing-bowl. See Chapter VIII. [167] Pliny, xxxvi. 65. [168] Here Elton, like others of his age, mistranslates Chalcos by ‘brass’: Their mansions, implements, and armour shine In brass,—dark iron slept within the mine. [169] Engraving on copper-plates is popularly attributed to Maso Finiguerra, of Florence, in 1460; but the Romans engraved maps and plans, and the ancient Hindus grants, deeds, c. on copper- plates. [170] I regret the necessity of troubling the learned reader with these stock quotations, but they are essential to the symmetry and uniformity of the subject. [171] Sophocles and Ovid make Medea, and Virgil makes Elissa, use a sickle of chalcos. Homer, as will be seen, uses the same material for his arms, axes, and adzes. Pausanias follows him, quoting his description of Pisander’s axe and Meriones’ arrow; he also cites Achilles’ spear in the temple of Athene at Phaselis, with its point and ferrule of chalcos, and the similar sword of Memnon in the temple of Æsculapius at Nicomedia. Plutarch tells us that
- 57. the sword and spear-head of Theseus, disinterred by Cymon in Scyros, were of copper. Empedocles, who (b.c. 444)— ardentem frigidus Ætnam Insiluit— was betrayed by his sandal shoon with chalcos soles. [172] See Macrob. Sat. vi. 3. [173] Or ‘a furbisher (whetter, sharpener = acuens) of every cutting tool of copper and iron.’ See Chap. IX. [174] I can hardly understand why Dr. Evans (p. 5) insists upon these sockets being bronze, as they could ‘hardly have been done from a metal so difficult to cast as unalloyed copper.’ He greatly undervalues the metallurgy of the Exodist Hebrews, who would have borrowed their science from Egypt. [175] Lead is also mentioned, but not tin. [176] A certain Herr Dromir patented in Germany a process for making malleable bronze. He added one per cent. of mercury to the tin, and then mixed it with the molten copper. [177] For Irish copper swords see the Archéologie, vol. iii. p. 555. They will be exhaustively described in Part II. [178] So Chalcis in Mela (ii. 7), now Egripos (Negroponte). [179] The confusion with iron appears in the Sanskrit (Pali?) ayas; Latin æs for ahes (as we find in aheneus); the Persian áhan (;)آهن the Gothic ais, or aiz; the High German er (which is the Assyrian eru and the Akkadian hurud), and the English iron. J. Grimm (Die Naturvölker) connects Ἄρης with æs. That æs and æris metalla in Pliny mean copper, we learn from his tale of Telephus (xxv. 19), which, by the by, is told by Camoens (Sonnet lxix.) in a very different way. [180] χαλκεύειν δὲ καὶ τὸ σίδηρεύειν ἔλγον, καὶ χαλκέας τοὺς τὸν σίδηρον ἐργαζομένους. Jul. Pollux, Onomasticon, viii. c. 10. [181] The full term was æs cyprium, which Pliny apparently applies to the finer kind; then it became cyprium, the adjective, which expressed only locality; and lastly cuprum. The third is first used by Spartianus in the biography of Caracalla (No. 5), Cancelli ex ære vel cupro (doors of æs or copper). Ælius Spartianus dates from the days of Diocletian and Constantine (Smith, sub voc.).
- 58. When Pliny writes in Cypro prima fuit æris inventio, he leaves it doubtful if æs be copper or bronze; but we should prefer the former. So he makes the best ‘Missy’ (native yellow copperas) proceed from the Cyprus manufactories (xxxiii., iv. 25, and xxxiv., xii. 31). The word misí or missí is still used in India for a vitriolic powder to stain the teeth. Cypros, the wife of Agrippa, was possibly named from Kafar = the henna plant: the Cyprus of Pliny (xii. 51) is also the Lawsonia inermis. [182] Frag. tom. i. p. 226. Edit. Bipont. [183] The island will be further noticed in Chap. VIII. [184] Cyprus, c., by General Louis Palma (di Cesnola). London: Murray, 1877. The author excavated from 1866 to 1876, and opened some 15,000 tombs, mostly Phœnician. [185] Quoted in the Kypros of W. H. Engel (vol. i. p. 14). The two volumes are a mine of information; much of it now antiquated, but useful to later students who have less leisure to accumulate learning. [186] ‘In Cyprus, where the manufacturers of the stone called chalcitis (copper-smelters) burn it for many days in fire, a winged creature, something larger than a great fly, is seen walking and leaping in the fire.’ A brother of the salamander! [187] Some commentators (Strabo, vi. 1) confound this place with Ausonian Temĕsa, or Tempsa, in the land of the Brutii, with Temése of Cyprus. [188] Herodotus (iii. 23) tells us that, copper being of all metals the most scarce and valuable in Æthiopia, prisoners were there bound with golden fetters. As will be seen, copper has lately been found in Abyssinia. [189] An awful list of his works is given in Diogenes Laertius. [190] This ærugo was artificially made by the Ancients with acetic acid, converting copper to a green salt (Beckmann, sub v. ‘Verdigris or Spanish Green’). The green rust of the carbonate of copper is still erroneously termed verdigris (acetate of copper). [191] Ample information is given by Brugsch (Egypt under the Pharaohs, vol. i. p. 64) of Senoferu; of the valiant Khufu or Suphis (Cheops); of the Pharaoh Sahura, or Sephris; of Menkauhor (Mencheres) and Tat-ka-ra (Fifth Dynasty); of the bas-reliefs at
- 59. Wady Magharah dating from King Pepi (Sixth Dynasty); of Thut- mes III. or the Great, and his sister Hashop (Eighteenth Dynasty before b.c. 1600), one of whose expeditions produced among other things ninety-seven Swords (Brugsch, i. 327), and who mentions ‘gilt copper’; of Amon-hotep III., also ‘the Great’ (Eighteenth Dynasty, about b.c. 1500); and of other Pharaohs who worked these diggings. [192] Pottery has lately been found embedded in the bricks of the Maydúm Pyramid. [193] The Souphis I. of Manetho is the second king of the Fourth Dynasty following Soris. Souphis II. is the Khafra of the Tables and the Cephren of the Greeks. [194] The hieroglyphic is of several forms; may serve as a specimen. [195] ‘Malachite’ is the Greek molochotis, from the molokhe, or marsh-mallow; whence the Arabic mulukhíyeh. In Poland, malachite and turquoise preside over the month of December. [196] Meaning the Beloved of Ptah, the Opener, the Artificer God. The word is found in the Arabic fath. It is a better derivation for Hephæstus than ‘Vaishravana’; but Sanskrit is so copious that any given word can be derived from it. [197] O Muata Cazembe, by Monteiro and Gamitto, describes the copper works in South-East Africa long known to the natives. I am told by Mr. Hooker, C.E., that he has lately seen (pace Herodotus) ‘magnificent specimens of native copper sent from Abyssinia.’ [198] R.N., C.B., c., Across Africa, vol. i. pp. 134, 319; and vol. ii. pp. 149, 329. [199] Viagens dos Portuguezes, Colecção de Documentos, c. [200] Layard’s Nineveh, i. 224, ii. 415; 6th edit. 1854. [201] Hence our packfong, or German silver, of China, an alloy of copper (50 per cent.), nickel, and zinc (25 per cent. each). [202] The Chinese Repository gives a hundred illustrations of the implements in use by the Chinese and the Japanese. [203] Fir or fear (vir, a man), and bolg (Bolgi, Belgæ), a belly, bag, budget, or quiver. They occupied Southern Britain, and
- 60. formed the third immigrant colony preceding the ‘Milesians,’ sons of Milidh or Miledh (Senchus Mor), evidently Miles, the soldier. He had two sons, Emer and Airem, from whom the Irish race is descended. Emer, says Prof. Rhys, may represent the Ivernii or pre-Celtic population mentioned by Ptolemy; and Airem, which means ‘a farmer,’ the Iranian race which introduced agriculture amongst a horde of hunters. The fourth colony was the Tuatha (people, e.g. Tuatha-Eireann = people of Erin), named from Danair, a stranger, foreigner, and properly a Dane. We have lately been shown how much true history may be obtained from these names, which had become bye-words, almost ridiculous to use. [204] Bán (our corrupted ‘bawn,’ as in ‘Molly Bawn’), white, is the Latin canus. It is also a noun substantive, meaning ‘copper.’ [205] Wilde, Catalogue, pp. 58, 356. [206] Meaning Tectetan = ‘I don’t know.’ So the M’adri on an old English chart of the Euphrates. [207] Select Letters of Columbus, c. p. 201. Translated by R. H. Major, Hakluyt Society, 1870. [208] Humboldt, Travels, iii. 194. [209] Commentaries of the Yncas. Translated by Clements R. Markham, C.B. Hakluyt Society, 1871. [210] Daniel Wilson’s Prehistoric Man, vol. i. chap. viii.; The Metallurgic Arts, Copper (pp. 231–79). Prof. Brush, of Yale College, calculated that 6,000 tons were yielded in 1858. [211] R.E., Spanish America, c. (Philadelphia: Abraham Small, 1819), p. 49. [212] It was divided, like the Greek and Roman, into centuries (pachacas), chiliarchies (hurangos), and inspectorships (tokrikrok), generally under royalties. The organisation was due to the Ynka Inti-Kapak (the Great), b.c. 1500–1600. There was a large fleet (‘magna colcharum classis’) of ships not smaller than the contemporary European, ‘navigiis velificantur nihili vestris minoribus,’ says P. Martyr (Decad. ii. lib. 3). Neither traveller nor historian has explained how this mighty organisation crumbled to pieces at the touch of a few European adventurers. I have read with interest the able work of M. Vicente F. Lopez, Les Races Aryennes du Pérou (Paris: Franck, 1871): he derives
- 61. the word from Pirhua, the first Ynka deified to a Creator. He adopts (p. 17) against Garcilasso de la Vega, who gave the Ynkarial Empire 400 years, the opinions of the learned Dr. Fernando Montésinos el Visitador, of the later sixteenth century, who is set aside by Markham, Narratives of the Yncas (Hakluyt, 1873). Montésinos derives the Peruvians from Armenia five centuries after ‘the Flood,’ and assigns 4,000 years with 101 emperors to the dynasty; it begins with Manko Kapak, son of Pirhua Manko; and Sinchi Roka (No. xcv. of Montésinos) is Garcilasso’s official founder (p. 25). But I cannot follow M. Lopez in his theories of ‘Aryanism’ (Zend and Sanskrit) or ‘Turanianism’ (Chinese and Tartar). The Quichua wants the peculiar Hindu cerebrals (which linger in English), and lacks the ‘l,’ so common in ‘Indo-European’ speech; ‘Lima,’ for instance, should be ‘Rima.’ It has no dual, and no distinction between masculine and feminine. But with the licence which M. Lopez allows himself, any language might be derived from any other. For instance, chinka from sinha, ‘the lion’ (p. 138); hakchikis = hashish, ‘intoxicating herb’; kekenti, ‘humming-bird,’ from kvan, ‘to hum’; huahua, ‘son,’ from su, ‘to engender,’ sunus, c., (when in Egypt we have su); and mama, ‘mother,’ from mata, μήτηρ, mater, when we have mut and mute in Nile-land. For mara, ‘to kill,’ ‘death,’ the old Coptic preserves mer, meran, ‘to die’; and for mayu, ‘water,’ mu. I thus prefer the monosyllabic Egyptian for Quichua roots, noting the two forms of pronoun, isolated (nyoka = I = anuk) and affixed (huahua-í, ‘my son;’ huahua-ki, ‘thy son;’ huahua-u, ‘his son’). The heliolatry of the Andes was that of the Nile Valley; Kon is the Egyptian Tum, ‘the setting sun.’ The god Papacha wears on his head the scarabæus of Ptah, or Creative Might. The pyramids and megalithic buildings are also Nilotic. The pottery shows three several styles, Egyptian, Etruscan, and Pelasgic. The population was divided into the four Egyptian castes (p. 396), priests (mankos and amautas), soldiers (aucas, aukas), peasants (uyssus), and shepherds or nomads (chakis). According to Cieza de Leon (p. 197) they thought more of the building and adorning of their tombs than of their houses; their mummies were protected by little idols, and the corpse carried the ferryman’s fee. The pyramid of Copan (Yucatan), 122 feet high, with its 6-feet steps, is that of Sakkarah. The Yucatan beard in statues is Pharaohic. The elephant-trunk ornaments (Stephens, ii. 156) are
- 62. Indo-Chinese. The geese-breeding (ii. 179) is Egyptian. See also the Toltec legend of the House of Israel (ii. 172). [213] The ‘lovely valley, Andahualas,’ is from Anta and Huaylla, pasture—i.e. ‘copper-coloured meadow.’ Anta in Cieza de Leon appears to be copper, whereas other writers make it bronze. [214] Peruvian Antiquities, by Don M. E. de Rivero and J. J. von Tschudi. [215] They abandoned the native silver mines when the ore became too hard, and they smelted it in small portable stoves. They knew also the chemical combinations, sulphate, antimonial, and others; and they worked quicksilver. They had mines of Quella (Khellay, or iron), but they found difficulty in extracting it. Besides smelting, they could use the tacana (hammer), cast in moulds, inlay, and solder. [216] Ewbank, of whom more presently, sketches a well-cast axe (p. 455). He translates anta by bronze (p. 455). [217] Doubtless copied from Old-World articles. On the west side of Palenque the Sword is distinctly Egyptian (Stephens, Yucatan). I have attempted to show how easily castaway mariners could be swept by currents from Europe, Asia, Africa, and America. See ‘Ostreiras of the Brazil’ in Anthropologia, No. 1, October 1873. [218] Antiquarian, Ethnological, and other Researches. By William Bollaert. London: Trübner, 1860. We must probably change ‘brass’ into ‘bronze’ when he says (p. 90) that ‘the Peruvians used tools of brass.’ [219] Appendix to Life in Brazil (Sampson Low, 1856). [220] This white copperas was detected by Scacchi on the fumaroles after the Vesuvian eruption of 1855. [221] Gold was shown by yellow, and silver by white. Dr. Evans (Bronze, c. p. 7) suggests that the round blue bar used by butchers (Wilkinson, iii. 247) was not of steel; but his reasons are peculiarly unsatisfactory. The file is a common implement amongst savages, doubtless derived from the practice of cross- hatching wooden grips and handles. Mr. A. H. Rhind (Thebes, c.) attributes little weight to the diversity of colours employed by ancient Egyptians to depict metallic objects, and he finds red and green confused.
- 63. [222] Thus we have a blue war-helmet of ring-mail (Lepsius, Denkmäler, iii. 115 c.), a blue war-hatchet with wooden handle, and spears pointed with brown-red and blue (copper and iron) in the tomb of Ramses III. The war-car of an Æthiopian king, in the days of Tutankamun, has blue wheels and a body of yellow (gold). Lepsius, however, adds: ‘It is very remarkable that in all the representations of the old empire, blue-painted instruments can scarcely be traced.’ This simply proves that iron and steel were rare. [223] Prehistoric Man, chap. viii. [224] It was analysed by Mr. E. Tookey, with the following results: Copper 97·12 Arsenic 2·29 Iron 0·43 Tin, with traces of gold 0·24 100·08 The presence of the tin may have been accidental. The proportion of arsenic (2¼ per cent.) might have been expected to harden the metal, yet it was so soft as to be almost useless. [225] See chap. ix. [226] It is equivalent to the Roman’s ‘Aliud clausum in pectore, aliud in lingua promptum habere.’ [227] So amongst the Jews the sharp knives for circumcision (Josh. v. 2–3) were of the silex which they learned from the Egyptians; and the custom continued long after the invention of metal blades. [228] It was opened by Herr Ramsauer, and carefully described in Das Grabfeld von Hallstatt, by Baron E. von Sacken. I shall have more to say of it in chap. xiii. [229] Prinseps’ Essays (London, 1858), vol. i. p. 222, pl. xliv. fig. 12, and Journ. R. As. Soc. Bengal, vol. vii. pl. xxxii. fig. 12. Long descriptions of copper smelting in India are found in Science Gleanings, pp. 380 et seq., No. 36, Dec. 1831, Calcutta, and in Percy (Metall. p. 387); the latter by Mr. H. F. Blanford, of the Geol. Survey, who made especial studies in Himalayan Sikkim and the Nepaulese Tirhai. The workmen, who are of low caste, win the stone in small blast-furnaces about three feet high, burning
- 64. charcoal and cow-chips. They work not only the easily reducible carbonates, but sulphuretted ores, copper pyrites, with a mixture of mundic (iron pyrites). [230] Scales are apparently implied by kaskassin (1 Sam. xvii.), which in Leviticus and Ezekiel applies to fish-scales. [231] The shekel is usually estimated at 220 grs. (Troy), which would reduce the weights to 22·91 and 190·97 lbs. respectively; but Maimonides makes it = 320 grains of barley = as many grains Troy. See Parkhurst (Lex., s.v. ‘Amat’). Either figure would form a fair burden for a horse; and the spear would have been a most unhandy article, unless used by a man ten feet tall. I shall notice the Gathite’s Sword in chap. ix. [232] Ethnology of the British Islands. We also read: ‘Copper Swords have been found in Ireland; iron among the Britons and Gauls; bronze was used by the Romans, and probably by the Egyptians; and steel of varying degrees of hardness is now the only weapon employed.’ (J. Latham: see chap. vii.) [233] Trans. Edinb. Philos. Soc. Feb. 1822. [234] J. A. Phillips, F.C.S. Memoirs of the Chemical Soc. vol. iv. [235] Archæology and Prehistoric Annals of Scotland, p. 246. [236] See Sir W. Wilde’s Cat. Metallic Materials—Celts, Museum of Royal Irish Academy. [237] History of Kerry, p. 125. [238] Yet Æschylus (Agamem.) uses both chalcos and sideros generically for a weapon. [239] Ilios, c. (London, Murray, 1880). [240] Some small objects are reported as wheel-made; but this requires confirmation, according to a writer in the Athenæum (Dec. 18, 1880). [241] The copper bracelet (Troy, p. 150, No. 88) with its terminal knobs is the modern trade ‘manilla’ of the West African coast. This survival will again be noticed in chap. ix. [242] The word in its older form was written ‘allay.’ Johnson derives it from à la loi, allier, allocare: it appears to me the Spanish el ley, the legal quality of coinable metal. We have now
- 65. naturalised in English ley, meaning a standard of metals. (Sub voc. Dict. of Obsolete and Provincial English, by Thomas Wright; London, Bell and Daldy, 1869.) [243] Recherches sur les Mystères; and Mémoire pour servir à la religion secrète, c. c. [244] The ‘Aglaophemus,’ so called from the initiator of Pythagoras. I see symptoms of a revival in assertions concerning a ‘highly cultivated beginning, with the arts well known and practised to an extent which, in subsequent ages, has never been approached; and from which there has not anywhere been discovered a gradual advancement; but, on the contrary, an immediate and decidedly progressive declension.’ This, however, is a mere question of dates. Man’s civilisation began long before the Mosaic Creation; and science has agreed to believe that savage life generally is not a decadence from higher types, not a degeneracy, but a gradual development. [245] We now divide language into three periods: 1st, intonative, like the cries of children and lower animals; 2nd, imitative, or on onomatopoetic; and 3rd, conventional, the civilised form. [246] Axieros (the earth-goddess), Axiokersa (Proserpine of the Greeks), Axiokersos (Hades), and Casmilos (Hermes or Mercury). Ennemoser may be right in making the Kabeiroi pygmies (i.e. gnomes), but not in rendering Dactyloi by ‘finger-size.’ [247] The lame and deformed ‘artificer of the universe,’ who became Hephæstos (Vulcan) in Greece, and Vishvakarma in India. Sokar has left his name in the modern ‘Sakkárah.’ [248] The Assyrian cuneiforms allude to ‘the (Great) Bear making its crownship,’ that is, circling round the North Pole. [249] The temples of the Cabiri have lately been explored by Prof. Conze for the Austrian Government at Samothrace, and we may expect to learn something less vague concerning these mysterious ancients. [250] The Rev. Basil H. Cooper believes that the Phrygian was the original Ida, which gradually passed to Crete; and here the Idæi were priests of Cybele. He is disposed to connect with it the Greek Σίδ(ηρο); the German Eisen (and our iron), and the Ida feldt and Asi of the Norse myths (Day, p. 133).]
- 66. [251] The name is derived by Bochart from Heb. Lub or Lelub, חיקלוב, chiefs of the Libu or Ribu, as the old Egyptians called the Libyans. Hence the Prom. Lilybæum (Li-Lúb) and the Sinus ad Libyam or Lilybatanus. [252] We have satisfactory details concerning the Chalybes, who border on Armenia, in the Anabasis (iv. 5, c.). They dwell two days from Cotyora, the colony planted by Sinope; they are subject to the Mossynœci, and they subsist by iron-working (v. 5). Though few, they are a most warlike people, full of fight. Their armour consists of helmets, greaves, and cuirasses of twisted linen cords, reaching to the groin. They carry spears about fifteen cubits long, ‘having one spike’ (i.e. without ferule); and at their girdles a short faulchion, as large as a Spartan crooked dagger, with which they cut the throats of all whom they can master; and then, lopping off their heads, bear them away (iv. 7). Strabo makes the Chalybes the same as their neighbours the Chaldæi. [253] The well-known inscription on the tomb of Midas, and another given by Texier (Asie Mineure, ii. 57) show the Phrygian tongue to have been a congener of Greek. Even the Békos of Herodotus (ii. 2) is allied to our ‘bake,’ and Bédu to our ‘water.’ We are greatly in want of further information about Phrygia, and it is to be hoped that Colonel Wilson and Mr. W. M. Ramsay will complete the labours of Texier and Hamilton. [254] The Aryans of Herodotus, about the Arius river (Heri-rúd), are an undistinguished tribe, a mere satrapy. Strabo’s Aria (xi. 9) is a tract about 250 by 40 miles. In Pliny (vi. 23) Ariana includes only the lands of the Gedrosi (Mekran), the Arachoti (Kandahár), the Arii proper (Herat), and the Parapomisadæ (Kabul). It has been truly said that even if Aryan and Turanian man (first) centred in and emerged from these areas (the table-lands of Asia), the so-called history is entirely based on the philological discoveries of the Sanskritist school. [255] Therasia and Therassia, now Santorin. Here have been found ruins of prehistoric cities buried by the great central volcano. According to most geologists the latter was exhausted in b.c. 1800–1700. [256] I have personally noticed this, and described it in Midian Revisited, vol. i. p. 143.
- 67. [257] Beckmann (s.v. ‘Tin’) tells us that the metal ‘never occurs in a native state.’ He forgets stream-tin. He also denies that the oldest ‘cassiteron’ and ‘stannum’ were tin; and considers them to mean the German Werk, a regulus of silver and lead. His vasa stannea are vessels covered with tin in the inside. In the fourth century ‘plumbum candidum’ or ‘album’ was superseded by ‘stannum.’ Speaking of electrum, Beckmann asserts that ‘the ancients were not acquainted with the art of separating gold and silver.’ ‘Britain,’ Ynis Prydhain Island, where the god Prydhain was worshipped, or rather ‘Isle of the Brythons,’ has been fancifully derived by the energetic Semitiser from Barrat-et-Tanuk = Land of Tin. [258] Ezekiel tells us that the Tyrians received tin, as well as other metals, from Tarshish, or Western Tartessus, in the Bay of Gibraltar. [259] M. Emile Burnouf, ‘L’Age de Bronze,’ Revue des Deux Mondes, July 15, 1877, also brings tin from Banca. The island is about 150 miles long by 36 broad; it has no mountain backbone, but the peak of Goonong Maras rises some 3,000 feet above the sea-level. Chinese coolies still work the mines of Mintok, and in 1852 the yearly yield was some 50,000 piculs (each = 133⅓ lbs.) at the cost of nine rupees per picul. [260] Beckmann (loc. cit.), like Michaelis, is surprised at the Midianites possessing tin in the days of Moses. These were the views of the last century. I have suggested (Athenæum, Nov. 24, 1880) that the old Nile-dwellers extended through Midian to El- Hejáz and El-Yemen, where they worked the mines which became known to the Hebrews. [261] In 1866 De Rougemont made Phœnicia supply bronze to Europe, the copper being brought from Cyprus. Besides the Mediterranean, we find a Uralian and a Danubian branch of the industry. Before 1877 France had supplied 650 bronze Swords and daggers, Sweden 480, and Switzerland 86. [262] Alias the Œstrymnides. Borlase was of opinion that the group formed one block, with several headlands, of which ‘Scilly’ was the highest, outermost, and most conspicuous. He conjectures the original name to be Syllé, Sulla, or Sulleh, a flat rock dedicated to the sun; hence the Lat. Siliræ, Silures, and Sigdeles; the Engl. Sylley, Scilley, and lately Scilly; the Fr.
- 68. Sorlingues; and the Span. Sorlingas. The Keltic name of the chief feature was Inis Caer. [263] Archæology and Prehistoric Annals of Scotland, Part II. ‘The Archaic or Bronze Period.’ Daniel Wilson. [264] Pliny represents the Cassiterides as fronting Celtiberia. He considers it a ‘fabulous story’ that the Greeks fetched ‘white lead’ from the islands of the Adriatic. [265] Prehistoric Times, by Sir John Lubbock, 4th edit. (London: Williams and Norgate, 1878.) [266] The identification is not settled; some propose the Isle of Thanet. [267] Beckmann, sub voce ‘Tin.’ [268] According to Messrs. Wibel, Fellemberg, and Damour, who investigated even 10/1000 parts, the average proportions were ⅒ tin to 9 copper; and ¼ tin for hard metal, as chisels, c. M. E. Chauntre, Age de Bronze. 3 vols. (Paris: Baudry.) [269] The late General Uchatius, who ‘trusted in princes,’ and whose tragical death was greatly lamented by his friends, always declared that he had rediscovered (not discovered) the hardening of copper and bronze; and that he hoped to arrive at other secrets. His career was cut short before he learned to make the metal and the alloy resilient. [270] Thut, Tuth, Toth, Thoth, c., the moon-god who became Hermes Trismegistus. [271] Phosphor-bronze, for whose manufacture companies are now established in London and elsewhere, has the ordinary composition with the addition of red or amorphous phosphorus dropped upon the melted metal in the crucible. Berthier (Traité des Essais, ii. 410) states that a very small quantity of phosphorus renders copper extremely hard and suitable for cutting instruments. Percy (Metallurgy) found that copper will take up 11 per cent. of phosphorus; the metal, which assumes a grey tint, is quite homogeneous, and so hard that it can scarcely be touched by the file. The addition of phosphorus promotes the reduction of the oxides, and enables an exceedingly sound and durable casting to be made; but if it exceed ½ per cent. the metal becomes very brittle. Dr. Percy has described phosphor-silver, phosphor-lead,
- 69. and phosphor-iron. The phosphorus is, according to some authorities, apt to volatilise with time. At present a new form of bronze, the antimonial, in proportions of 1–2 per cent., is coming into fashion: it is said to be malleable and ductile, and to resist torsion in a high degree. Another new bronze is the aluminium, whose price has been reduced from 1,000l. to 100l. per ton by Mr. Webster, of Hollywood, near Birmingham.
- 70. [272] So called from Cape Emeri in Naxos. [273] Appendix to Layard’s Nineveh and Babylon (London: Murray). The proportions are nearly those of our day. We may assume our common bronze at 11:100 for large, and 10:100 for small objects. Cymbals and sounding instruments, however, contain tin 22:copper 78. [274] Analysed by Mr. Robinson of Pimlico (Day, p. 110). [275] Schliemann’s Troy, p. 361 (London: Murray, 1875). [276] Sir W. Gell found the bronze nails in the ‘Treasury of Atreus’ composed of 12 tin to 88 copper. The Trojan battle-axes, according to Dr. Schliemann, yielded only 4, 8, and 9 per cent. of the former metal. [277] According to Helbig, the Palafittes and Terramare villagers had spears but not Swords. [278] For the tin-ore of Peru see Ethnolog. Journal, vol. lxx. pp. 258–261. Rivero, p. 230, and Garcilasso, vol. i. p. 202. [279] Amer. Journ. of Science, c. v. 42; July 1866. [280] From descriptions and drawings by Mr. J. H. Godfrey, Mining Engineer-in-Chief to the Imperial Government of Japan. [281] M.D., F.R.S., ‘Observations on some Metallic Arms and Utensils, with Experiments to determine their Composition.’ Royal Soc. London, June 9, 1796. Philosophical Transactions. [282] Taken from Dr. Evans (Bronze Impl. c. chap. xxi.). He compiled it from Martineau Smith’s Hardware Trade Journal (April 30, 1879). [283] Wilkinson remarked that the Egyptian proportions of half tin and half copper were whitish. [284] Lord Rosse, in casting specula, preferred using copper and tin in their atomic proportions, or 68·21 per cent. copper to 31·79 per cent. tin. [285] Speltrum was introduced by Boyle. During the last century much zinc was imported from India (possibly supplied by China), and was called tutenag.
- 71. [286] Bohn’s Trans. ii. 32–45. The learned German begins by stating that zinc was not known to the Greeks, Romans, and Arabs, and then proceeds to prove that it was. The word ‘zinc’ (from zenken or zacken, nails, spikes?) first occurs in the works of the Iatro-chemist, Paracelsus, who died in a.d. 1541. [287] Blende is a generic word, from blenden, to dazzle. [288] Mongez, Mém. de l’Institut. [289] At Goslar, however, according to Lohnriss, brass was made in a.d. 1617. [290] Pliny, xxxiii. 27. The solder (χρυσός and κόλλα, glue, or κόλλησις) is attributed by Herod. (i. 25) to Glaucus of Chios, a contemporary of Alyattes. The word kóllesis is variously rendered ‘soldering,’ ‘brazing,’ ‘welding,’ and ‘inlaying.’ Kóllesis was used to agglutinate metals, and treated with a peculiar alkali (Pliny, xxxiii. 24). The ‘gold glue’ (chrysocolla) is usually understood to be a hydrosilicate of copper; not to be confounded with the χρυσόκολλα or borax. The Mycenian goldsmiths soldered with the help of borax (borate of soda): Professor Landerer, of Athens, found this salt on an old medal from Ægina. It was called in the Middle Ages, Borax Venetus, because imported by the Venetians from Persia; and it is the Tinkal of modern India. According to Pliny, lead cannot be soldered without tin, or tin without lead, and oil invariably must be used. Later usage substituted for the latter colophonium and other resins: we now solder by means of electricity. The same writer makes Nero use chrysocolla-powder (a siliceous carbonate of copper, a kind of blue-stone which would turn green by exposure to damp) for strewing the circus, to give the course the colour of his favourite faction, the Prasine (green). [291] The Germans, who delight in German derivatives for European words, would find leiton, c., not in luteum, but in löthen = to unite. There is little doubt, however, that the first English manufactory of calamine brass at Esher, in Surrey, was set up in the seventeenth century by Demetrius, a German. In Grimm’s Dictionary, as noticed by Demmin (chap. i), bronze is erroneously called messing (brass). [292] Derived from ὄρος, οὖρος (mountain), or from Ὀρείος, the discoverer. Metallic names in Greek are mostly masculine; in Latin and modern usage, neutral. Oreichalcum or aurichalcum, a hybrid
- 72. word, became aurochalcum in the ninth century: the last corruption (middle of the sixteenth century) was archal. [293] De l’Orichalque. J. P. Rossignol (loc. cit.). [294] Some translate this word ‘yellow frankincense’ (λίβανος) colour; others derive it from Λίβανος, the Lebanon, and make it male, argurolibanus, while leucolibanus (white) was female. Finally, the word was explained by the old interpreters to be = ὀρείχαλκος = brass of Mount (Lebanon). [295] The tradition of Atlantis, a middle-land in the Atlantic, has strong claims to our acceptance. The identity of the site with the ‘Dolphin’s Ridge,’ a volcanic formation, and the shallows noted by H.M.S. ‘Challenger,’ have been ably pleaded in Atlantis (Ignatius Donnelly; London: Sampson Low, 1882). Perhaps we may trace the vestiges in Saint Paul’s Rocks, the remarkable group of rocky islets situate in the equatorial mid-Atlantic. Mr. Darwin supposed the group to be an isolated example of non-volcanic oceanic insularity; but Prof. Renard finds the ‘balance of proof decidedly in favour of the volcanic origin of the rock.’ It will be remembered that Atlantis was dismembered by earthquakes, eruptions, and subsidence. [296] Quoted by Percy from Watson’s Chemical Essays (iv. p. 85, 1786). [297] The artificial mixture of copper (four fifths) and gold (one- fifth) was called pyropus (Pliny, xxxiv. 2), from its fiery red tint; it was also made of gold and bronze, and termed chrysochalcos, ‘the king of metals.’ Æs corinthiacum (Pliny, xxxiv. 3), or Corinthian brass, used for mirrors, composed of copper, silver (steel? zinc?), and gold, was more valuable than gold. According to Pausanias (ii. 3, § 3), this malleable and ductile metal was tempered in the Fountain of Pyrene. The vulgar legend, refuted by Pliny, who tells the tale (xxxiv. 6), dates it from the days of Mummius (b.c. 146). A medal of Corinthian brass was analysed by the Duc de Luynes. Pliny (xxxiv. 3) mentions three kinds, candidum, luteum, and hepatizon (liver-colour), of equal quantities of metal; this probably resembled our own alloys. Beckmann (sub voc. ‘Zinc’ and ‘Tin’) gives a list of these and other compositions, Mannheim gold, Dutch gold, Prince’s metal, Bristol brass, c.
- 73. [298] Possibly the Armenian bole (Bol-i-Armani), used in the East as a flux from time immemorial. The ‘dropping’ or ‘distilling’ (per descensum) must allude to a distillatory or condensing apparatus, and the ‘false silver’ cannot be mercury, lead, or tin. [299] Hence tutaneg and tutanego, which sometimes meant an alloy of tin and bismuth. M. Polo (i. 21) describes ‘tutia’ as very good for the eyes; and his notice of it, and of spodium, reads, according to Colonel Yule, almost like a condensed translation of Galen’s pompholyx, produced from cadmia or carbonate of zinc; and spodos, the residue of the former, which falls on the hearth (De Simp. Med. p. ix.). Matthioli makes pompholyx commonly known in the laboratories by the Arabic name ‘tutia.’ The ‘tutia’ imported into Bombay from the Gulf is made from an argillaceous ore of zinc, moulded into tubular cakes, and baked to a moderate hardness. [300] Masc. and fem.; the neut. ἤλεκτρον is the purest form. Dr. Schliemann, noticing that it also means ‘amber’ (Mycenæ, p. 204), derives it from ‘elek, signifying resin in Arabic (?), and probably also in Phœnician (?).’ He found earrings of electrum in the so-called ‘Trojan Stratum,’ 30½ feet below the surface (Troy, p. 164). The guanin or gianin of the Chiriquis was an aururet (electrum) of 19·3 per cent. of pure gold, with specific gravity 11·55. The tombac or tombag of New Granada, used for statuettes, was also a gold of low standard: 63 gold, 24 silver, 9 copper. Usually ‘tombac’ applies to an alloy like Mannheim gold; the manufacture was introduced into Birmingham, still its chief seat, by the Turner family, a.d. 1740. [301] ‘Elektron,’ however, is generally translated ‘amber’; and it may be the harpax, or drawer, for it occurs in the same verse with ivory. Amber beads and weapon-handles were amongst Dr. Schliemann’s finds. Rossignol (p. 347) supposes that electrum, the pale-yellow or amber-coloured alloy of gold and silver, gave a name to the gum amber. [302] This text, stating a truth concerning native gold, suggests amongst many that the ancients knew the départ, or separation, of metals. It has been vehemently doubted whether they could mineralise the white metal; that is, convert it to sulphide and allow the gold to subside. [303] Rossignol quotes Zonaras, Suidas, and John Pediasimus to prove this position.
- 74. [304] We now lacquer with shell-lac dissolved in proof-spirit and coloured with ‘dragon’s blood.’ [305] The lead was found in even larger proportions. See chap. xiii. [306] In my commentary on Camoens (Camoens: his Life and his Lusiads), and again in To the Gold Coast for Gold (i. 17), I have attempted to identify Western Tarshish or Tartessus with Carteia in the Bay of Gibraltar. Newton makes Melcarth ‘King of Carteia’; but the word may mean either ‘city-king’ (Malik-el-Karyat), or ‘earth-king’ (Malik-el-Arz). [307] The well-known anthropologist, M. G. de Mortillet, holds that the oldest type of bronze celt in France, Switzerland, and Belgium, is that with straight flanges at the sides. This was followed by the celt with transverse stop-ridge, by the true winged tool, by the socketed adaptation, and, lastly, by the simple flat tool wanting rib or flange, wing or socket, and formed of pure copper as well as of bronze. Archæologists usually determine the last form to be the earliest; but M. de Mortillet judges otherwise from the conditions under which the finds occur. [308] This weapon (gladius) is a Sword-blade, double-edged or single-edged, straight or curved, and 4–9 inches long, much used in the fourteenth and fifteenth centuries. It originated from the old practice of binding the sickle, scythe, axe, hatchet, or Sword to the end of a pole and thus forming a pike. [309] The Amazons of the Mausoleum (Newton, Halicarnassus, p. 235) are armed with axe, bow, and Sword; the Greeks with javelins and Swords. [310] The Massagetæ (greater Jats or Goths) are opposed to the Thyssa (or lesser) Getæ, and both used the sagaris. But while some authors translate the word securis, others call it a ‘kind of Sword,’ and others confuse it with the ἀκινάκης, the acinaces which the Greek mentions separately (iv. 62, viii. 67). Strabo (xi. 8) connects the Massagetæ (Goths) with the Sacæ (Saxons), and Major Jähn derives Sacæ (the Shaka of the Hindus) from Saighead = Sagitta. The term ‘Saxones’ was later than the age of Tacitus, and we first find it in the days of Antoninus Pius. ‘Brevis gladius apud illos (Saxones) Saxo vocatur’ suggests that the Seax was connected with the race of old (Trans. Anthrop. Instit. May 1880).
- 75. [311] Loc. cit. p. 43. [312] Egypt. akhu, Lat. ascia, Germ. Axt. The oldest form is ‘aks’ (securis), the bipennis, ‘dversahs,’ and the dolabrum ‘barte.’ In Lower Saxon axt is ‘exe,’ a congener of our ‘axe.’ [313] The word is variously written and explained. [314] A silepe from the armoury of King Mosesh was shown at the National Exhibition amongst objects from Natal (Col. A. Lane Fox, Cat. p. 145). [315] Par Lacombe (Paris, Hachette, 1868). [316] I have again noticed the sahs, seax, sax, and scramasax in chap. xiii. [317] Our ‘bill’ is the German Beil, the securis, or axe. Both words appear to me congeners of the Greek βέλος, Sword or dart, showing a missile-age, from βάλλειν, to throw; not, as Jähn thinks, from the Sanskrit bhil. Robert Barret (1598) preferred the pike, although owning that the bill had done good service. Even of late years Messrs. John Mitchel and Meagher (‘of the Sword’) advised the wretched Irish peasants to make pikes out of reaping- hooks. [318] Prehistoric Times, p. 20. The Dublin Museum contains 1,283 articles of the Bronze Age. [319] I assume as a type, the bronze Sword (Tafel iv.) in Die Alterthümer von Hallstätten, Salzburg, c. by Friedrich Simony (Wien, 1851). [320] Pliny, xxxiv. 39. [321] The word comes from the root which gave the Persian áhan; the Irish iaran or yarann; the Welsh hiarn; the Armorican uarn; the Gothic eisarn; the Danish iern; the Swedish iarn; the Cimbric jara; the German Eisen, and the Latin ferrum, with the neo-Latin ferro, hierro (Span.), c. From iaran also we derive Harnisch, harness. [322] The unfortunate Cretans gained the name of ‘ever liars’ (ἀεὶ ψεῦσται) for telling what was probably the truth. They showed in their island the grave of Jupiter, who must have been originally some hero or chief deified after his death—evidently one of the origins of worship. The evil report began with Callimachus (Hymn.
- 76. in Jov. 8); and was continued in the proverbial τρία κάππα κάκιστα (Krete, Kappadocia, and Kilikia). Hence the syllogistic puzzle of Eubulides: ‘Epimenides said that the Cretans are liars: Epimenides is a Cretan: ergo, Epimenides is a liar: ergo, the Cretans are not liars: ergo, Epimenides is not a liar.’ [323] Chap. iv. The Chalybs of Justin (xliv. 3) is a river between the Ana (Guadiana) and the Tagus; called by Ptolemy and Martianus, Κάλιπους or Κάλιπος. Æschylus alludes to the original Chalybes when he personifies the Sword as the ‘Chalybian stranger,’ and in the same tragedy (Seven against Thebes) he entitles it ‘the hammer-wrought Scythian steel.’ [324] ‘To the abundance of iron we may attribute the fact that the Africans appear to have passed direct from the stone implements, that are now found in the soil, to those of iron, without passing through the intermediate bronze period which, in Egypt and other countries, intervened between the ages of stone and iron.’— Anthropol. Coll. pp. 128–134. [325] ‘The High Antiquity of Iron and Steel,’ a valuable paper read before the Philos. Soc. Glasgow, printed in Iron (1875–76), and kindly sent to me by the editor, Mr. Nursey; also The Prehistoric Use of Iron and Steel (Trübner, London, 1877), from which Mr. Day has allowed me to make extracts. [326] The question is to be determined by facts, not theories. Hitherto we are justified in believing, from the skeletons dug up at great depths, or found in caves associated with the mammals which they destroyed, that Man in prehistoric times was of a low physical, and therefore mental type. We shall believe the opposite view when we are shown ancient crania equal, if not superior, to those of the present day—relics that will revive the faded glories of ‘Father Adam’ and ‘Mother Eve.’ But, meanwhile, we cannot be expected to believe in ipse dixits, inspired or uninspired. [327] For instance, in North-Western Europe, the early iron age began about a.d. 250, according to Konrad Englehardt (Denmark in the early Iron Age, p. 4, London, 1866), quoted by Mr. Day. [328] Egypt’s Place in Universal History, vol. v.; London, Longmans, 1867, with additions by Samuel Birch, LL.D. [329] When Laplace made meteorolites ejections from lunar volcanoes, Chladni suggested that they were masses of metallic
- 77. matter, moving in irregular orbits through interplanetary, and possibly interstellar, space. [330] This word is tortured by non-Orientalists into various ill- forms. The Arabs write it جيزة (Jízeh), and the Egyptians pronounce it Gízeh, not Ghizeh. [331] A full-sized drawing appeared in vol. vii. of Proceedings of the Phil. Soc. Glasgow; and was repeated by Mr. Day in his book, Pl. II. he also gives Belzoni’s sickle, Pl. I. [332] When visiting the ‘Tombs of the Soldans,’ Cairo, I found a slab of blue basalt bearing the cartouche of Khufu, used as a threshold for one of the buildings. The characters had been partly erased; but the material was too hard for the barbarians who had misused it. [333] I have elsewhere noticed (chap. iv.) the colours of metals in the painted tombs of Thebes, and the blue (cyanus-colour) of the butcher’s steel. The history of this homely article is instructive. For hundreds of years it retained, in England and elsewhere, its original shape, an elongated cone. At last some ‘cute citizen had the idea of breaking the surface into four edges, and of hardening it with nickel. The simple improvement now fits it for sharpening everything from a needle to a razor: it thus frees us from the ‘needy knife-grinder,’ who right well deserved to be needy, as he disadorned everything he touched. [334] Antiquity of the Use of Metals, especially Iron, among the Egyptians, p. 18 (London, 1868). Also Ueber die Priorität des Eisens oder der Bronze in Ostasien, by Dr. M. Müller (Trans. Vienna Anthrop. Soc. vol. ix.). [335] I assume this date because it marks when the spring equinox (vernal colure) occurred in the Taurus-sign. The earliest of the six epochs proposed by Egyptologists is b.c. 5702 (Böckh), and the latest is b.c. 3623 (Bunsen); the mean being b.c. 4573, and the difference a matter of 2079 years (Brugsch, i. 30). [336] The Table of Sakkarah (Memphis), found about the end of 1864 by the late Mariette Pasha, dates from Ramses the Great (thirteenth century b.c.), and makes Mibampes the first of his fifty-six ancestors. No. 2 is the new tablet of Abydos, discovered, also in 1864, by Herr Dümmichen; it enabled scholars to supply the illegible name in No. 3, the priceless Turin Papyrus, the
- 78. hieratic Canon of the Ptolemies. Mirbampes, Mirbapen, or Mi-ba of the monuments is, called in Manetho ‘Miebides, son of Usarphædus’ (Cory’s Fragments, p. 112). [337] Of Ramses II., who, with his father Seti, represents the Greek Sesostris, the Sesesu-Ra of the monuments. (Brugsch, Hist. ii. 53–62: see my chap. viii.) Prof. G. Ebers has made this Egyptian proto-Homerid the hero of his romance, Uarda (i.e. Wardah, ‘the Rose’). [338] De Iside et Osiride. He quotes Manetho the Priest, who wrote during the reign of the first Ptolemy, and who told unpleasant truths concerning Moses, the Hebrews, and the Exodus. [339] The limestones of Carniola produce heaps of pisoliths, which require only smelting; and hence, probably, the early Iron Age of Noricum and its neighbourhood. [340] They suggest the magnetic and titaniferous iron sands of Wicklow, of New Zealand, of Australia, and of a variety of sites mentioned in To the Gold Coast for Gold, ii. 111. [341] The Naphtuhim of Scripture. [342] Percy’s Metallurgy, p. 874, first edit. [343] Proc. Soc. Antiq. second series, vol. v., June 1873. Mr. Hartland added rubbings of various Pharaohnic stones, hoping to ‘show how little the mind of civilised man has developed during 3,000 years.’ A pleasant lesson to humanity! But after all thirty centuries are a mere section of the civilisation which began in Egypt. [344] The Corsican is simply a blacksmith’s forge. The Catalan has a heavy hammer and blowing-machine; if the trompe be used, a fall of water is required for draught. The Stückofen is a Catalan extended upwards in the form of a quadrangular or circular shaft, 10–16 feet high. [345] It is to be noted that flint implements were found all about these works: Mr. Hartland brought home from them silex arrow- heads. The late lamented Professor Palmer observed them in other parts of the Pharan peninsula, and I made a small collection in Midian. In the Journ. of the Anthrop. Soc. 1879, I showed, following Mr. Ouvry, Sir John Lubbock, and others, that Cairo is
- 79. surrounded by ancient flint-ateliers. M. Lartet explored them in Southern Palestine; I picked them up near Bethlehem (Unexplored Syria, ii. 289). The Abbé Richard and others traced them at Elbireh (in the Tiberiad); between Tabor and the Lake; and, lastly, at Galgal, where Joshua circumcised. Lastly, my late friend Charles F. Tyrwhitt-Drake, when travelling with me, came upon an atelier east of Damascus. I have noticed General Pitt- Rivers’ great Egyptian discovery in chap. ii. [346] Hek or hak (chief) has a suspicious resemblance to Shaykh and sos to sús, the mare, characteristically ridden by the Bedawin. In old Egyptian sos is a buffalo. [347] Movers (Phönicier, ii. 3), quoted by Dr. Evans (Bronze, c. 5), finds bronze (copper?) 44 and iron 13 times in the Pentateuch, and he theorises upon the later introduction of the latter. But when was the Pentateuch written in its present form? [348] Rougemont, L’Age du Bronze, pp. 188 et seq. [349] Volney, Travels, ii. 438. [350] Much of it, however, was the amygdaloid greenstone, called in English ‘toad-stone,’ a corruption of the Germ. Todstein. [351] Speaker’s Commentary, i. 831. [352] This term seems first to have been used by Orosius (i. 2) in our fourth century. [353] In chap. ix. I shall attempt to show that Naharayn (the dual of Nahr, a river) is also applied to Palestine in such phrases as ‘Tunipe (Daphne-town) of Naharayn.’ [354] Dr. Percy found that certain Assyrian bronzes had been cast round a support of the more tenacious metal, thus combining strength with lightness. [355] M. F. Lenormant (‘Les Noms d’Airain et du Cuivre dans les deux Langues ... de la Chaldée et de l’Assyrie, Trans. Soc. Bibl. Archæology, vi. part 2) renders parzillu, iron; abar, lead; shiparru (Arab. صفر, brass), bronze; anaku, tin; eru or erudu, copper or bronze (Arab. ايار, copper or brass); kashpu, silver; and kurashu, gold. The learned author discovers in the cuneiforms repeated mention of the ‘ships of Mákan’ and the Kur Makannata (mountain of Makná), which he translates ‘Pays de Mákan’: finding it a great centre of copper, he is inclined to confound it with the so-called
- 80. Sinaitic Peninsula. I have only to refer readers to ‘Makná’ in my three volumes on the Land of Midian. [356] Akkad is upper, Sumir lower Babylonia. [357] The Five Great Monarchies of the Ancient Eastern World, vol. i. p. 62. London, 1871. [358] The first period extended from b.c. 1500 to 909. The second from b.c. 909 to 745: the most marking names being Assurnazirpal = ‘Ashur (arbiter of the gods) protects his son,’ who built the north-west palace of Nimrúd, b.c. 884; and his son Shalmanezer II. of the Black Obelisk (Brit. Museum), b.c. 850. The third period (b.c. 745–555) numbered Tiglath-Pileser II., b.c. 745– 727 (a single generation before the first Olympic, b.c. 776, when the mythic age of Greece emerges into the historical); Sennacherib (705–681); Esarhaddon (680–668), Assur-bani-pal (668–640); Nebuchadnezzar in 604–561, a contemporary of Solon (b.c. 594); Nergalsharuzur (b.c. 557); and the last Nabonidus (b.c. 555). Herodotus (b.c. 450) wrote about a century after the end of the third period, Ctesias in b.c. 395, and Berosus in b.c. 280. We have, it is clear, absolutely no historic proof that ‘the patriarchal system of communities first locally developed itself at the mouth of the Euphrates Valley,’ or began in any part of the great Mesopotamian plain. [359] Rev. B. H. Cooper (loc. cit.) would derive ‘Ida’ from the Semitic יר (yad, hand), and make the Daktyls, or fingers, its peaks. [360] I shall reserve for chap. xi. notices of iron by the classic and sacred poets of Greece. [361] Troy and its Remains, p. 362; the analysis by M. Damour of Lyons. [362] The theory of Stephani, Schulze, and others concerning the Byzantine date and Herulian origin of the Mycenæan graves, has been treated in England with some respect by Mr. A. S. Murray and Mr. Perry. [363] According to Pausanias, Alyattes, the Lydian king (ob. b.c. 570), dedicated to his god, amongst other offerings, an inlaid iron saucer.
- 81. Welcome to Our Bookstore - The Ultimate Destination for Book Lovers Are you passionate about books and eager to explore new worlds of knowledge? At our website, we offer a vast collection of books that cater to every interest and age group. From classic literature to specialized publications, self-help books, and children’s stories, we have it all! Each book is a gateway to new adventures, helping you expand your knowledge and nourish your soul Experience Convenient and Enjoyable Book Shopping Our website is more than just an online bookstore—it’s a bridge connecting readers to the timeless values of culture and wisdom. With a sleek and user-friendly interface and a smart search system, you can find your favorite books quickly and easily. Enjoy special promotions, fast home delivery, and a seamless shopping experience that saves you time and enhances your love for reading. Let us accompany you on the journey of exploring knowledge and personal growth! ebookgate.com