![ZeroMQ
by Pieter Hintjens
Copyright © 2013 Pieter Hintjens. All rights reserved.
Printed in the United States of America.
Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472.
O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are
alsoavailableformosttitles(http://my.safaribooksonline.com).Formoreinformation,contactourcorporate/
institutional sales department: 800-998-9938 or corporate@oreilly.com.
Editors: Andy Oram and Maria Gulick
Production Editor: Christopher Hearse
Copyeditor: Gillian McGarvey
Proofreader: Rachel Head
Indexer: Angela Howard
Cover Designer: Randy Comer
Interior Designer: David Futato
Illustrator: Rebecca Demarest and Kara Ebrahim
March 2013: First Edition
Revision History for the First Edition:
2013-03-11: First release
See http://oreilly.com/catalog/errata.csp?isbn=9781449334062 for release details.
Nutshell Handbook, the Nutshell Handbook logo, and the O’Reilly logo are registered trademarks of O’Reilly
Media, Inc. ZeroMQ, the image of a fourhorn sculpin, and related trade dress are trademarks of O’Reilly
Media, Inc.
Many of the designations used by manufacturers and sellers to distinguish their products are claimed as
trademarks. Where those designations appear in this book, and O’Reilly Media, Inc., was aware of a trade‐
mark claim, the designations have been printed in caps or initial caps.
While every precaution has been taken in the preparation of this book, the publisher and authors assume
no responsibility for errors or omissions, or for damages resulting from the use of the information contained
herein.
ISBN: 978-1-449-33406-2
[LSI]](https://image.slidesharecdn.com/2157176-250609215532-c1f30a76/85/Zeromq-Messaging-For-Many-Applications-Pieter-Hintjens-6-320.jpg)
![$responder = new ZMQSocket($context, ZMQ::SOCKET_REP);
$responder->bind("tcp://*:5555");
while (true) {
// Wait for next request from client
$request = $responder->recv();
printf ("Received request: [%s]n", $request);
// Do some 'work'
sleep (1);
// Send reply back to client
$responder->send("World");
}
Example 1-4 shows the client code.
Example 1-4. Hello World client (hwclient.c)
//
// Hello World client
// Connects REQ socket to tcp://localhost:5555
// Sends "Hello" to server, expects "World" back
//
#include <zmq.h>
#include <string.h>
#include <stdio.h>
#include <unistd.h>
int main (void)
{
void *context = zmq_ctx_new ();
// Socket to talk to server
printf ("Connecting to hello world server...n");
void *requester = zmq_socket (context, ZMQ_REQ);
zmq_connect (requester, "tcp://localhost:5555");
// int request_nbr;
// for (request_nbr = 0; request_nbr != 10; request_nbr++) {
// zmq_msg_t request;
// zmq_msg_init_size (&request, 5);
// memcpy (zmq_msg_data (&request), "Hello", 5);
// printf ("Sending Hello %d...n", request_nbr);
// zmq_msg_send (&request, requester, 0);
// zmq_msg_close (&request);
//
// zmq_msg_t reply;
// zmq_msg_init (&reply);
// zmq_msg_recv (&reply, requester, 0);
// printf ("Received World %dn", request_nbr);
// zmq_msg_close (&reply);
// }
8 | Chapter 1: Basics](https://image.slidesharecdn.com/2157176-250609215532-c1f30a76/85/Zeromq-Messaging-For-Many-Applications-Pieter-Hintjens-30-320.jpg)
![if (size == -1)
return NULL;
char *string = malloc (size + 1);
memcpy (string, zmq_msg_data (&message), size);
zmq_msg_close (&message);
string [size] = 0;
return (string);
}
This makes a very handy helper function. In the spirit of making things we can reuse
profitably, we can write a similar s_send() function that sends strings in the correct
ØMQ format and package this into a header file we can reuse.
The result is zhelpers.h, which lets us write sweeter and shorter ØMQ applications in
C. It is a fairly long source, and only fun for C developers, so read it at your leisure.
Version Reporting
ØMQ does come in several versions, and quite often if you hit a problem, it’ll be some‐
thing that’s been fixed in a later version. So it’s a useful trick to know exactly what version
of ØMQ you’re actually linking with. Example 1-5 is a tiny program that lets you do just
that.
Example 1-5. ØMQ version reporting (version.c)
//
// Report 0MQ version
//
#include "zhelpers.h"
int main (void)
{
int major, minor, patch;
zmq_version (&major, &minor, &patch);
printf ("Current 0MQ version is %d.%d.%dn", major, minor, patch);
return EXIT_SUCCESS;
}
Getting the Message Out
Thesecondclassicpatternisone-waydatadistribution,inwhichaserverpushesupdates
to a set of clients. Let’s look at an example that pushes out weather updates consisting
of a zip code, temperature, and relative humidity. We’ll generate random values, just
like the real weather stations do.
Version Reporting | 11](https://image.slidesharecdn.com/2157176-250609215532-c1f30a76/85/Zeromq-Messaging-For-Many-Applications-Pieter-Hintjens-33-320.jpg)
![Example 1-6 shows the code for the server. We’ll use port 5556 for this application.
Example 1-6. Weather update server (wuserver.c)
//
// Weather update server
// Binds PUB socket to tcp://*:5556
// Publishes random weather updates
//
#include "zhelpers.h"
int main (void)
{
// Prepare our context and publisher
void *context = zmq_ctx_new ();
void *publisher = zmq_socket (context, ZMQ_PUB);
int rc = zmq_bind (publisher, "tcp://*:5556");
assert (rc == 0);
rc = zmq_bind (publisher, "ipc://weather.ipc");
assert (rc == 0);
// Initialize random number generator
srandom ((unsigned) time (NULL));
while (1) {
// Get values that will fool the boss
int zipcode, temperature, relhumidity;
zipcode = randof (100000);
temperature = randof (215) - 80;
relhumidity = randof (50) + 10;
// Send message to all subscribers
char update [20];
sprintf (update, "%05d %d %d", zipcode, temperature, relhumidity);
s_send (publisher, update);
}
zmq_close (publisher);
zmq_ctx_destroy (context);
return 0;
}
There’s no start and no end to this stream of updates; it’s like a never-ending broadcast
(Figure 1-4).
12 | Chapter 1: Basics](https://image.slidesharecdn.com/2157176-250609215532-c1f30a76/85/Zeromq-Messaging-For-Many-Applications-Pieter-Hintjens-34-320.jpg)
![Figure 1-4. Publish-subscribe
Example 1-7 shows the client application, which listens to the stream of updates and
grabs anything to do with a specified zip code (by default, New York City, because that’s
a great place to start any adventure).
Example 1-7. Weather update client (wuclient.c)
//
// Weather update client
// Connects SUB socket to tcp://localhost:5556
// Collects weather updates and finds avg temp in zipcode
//
#include "zhelpers.h"
int main (int argc, char *argv [])
{
void *context = zmq_ctx_new ();
// Socket to talk to server
printf ("Collecting updates from weather server...n");
void *subscriber = zmq_socket (context, ZMQ_SUB);
int rc = zmq_connect (subscriber, "tcp://localhost:5556");
assert (rc == 0);
// Subscribe to zipcode, default is NYC, 10001
char *filter = (argc > 1)? argv [1]: "10001 ";
rc = zmq_setsockopt (subscriber, ZMQ_SUBSCRIBE, filter, strlen (filter));
assert (rc == 0);
// Process 100 updates
int update_nbr;
long total_temp = 0;
for (update_nbr = 0; update_nbr < 100; update_nbr++) {
char *string = s_recv (subscriber);
int zipcode, temperature, relhumidity;
Getting the Message Out | 13](https://image.slidesharecdn.com/2157176-250609215532-c1f30a76/85/Zeromq-Messaging-For-Many-Applications-Pieter-Hintjens-35-320.jpg)
![Example 1-8. Parallel task ventilator (taskvent.c)
//
// Task ventilator
// Binds PUSH socket to tcp://localhost:5557
// Sends batch of tasks to workers via that socket
//
#include "zhelpers.h"
int main (void)
{
void *context = zmq_ctx_new ();
// Socket to send messages on
void *sender = zmq_socket (context, ZMQ_PUSH);
zmq_bind (sender, "tcp://*:5557");
// Socket to send start of batch message on
void *sink = zmq_socket (context, ZMQ_PUSH);
zmq_connect (sink, "tcp://localhost:5558");
printf ("Press Enter when the workers are ready: ");
getchar ();
printf ("Sending tasks to workers...n");
// The first message is "0" and signals start of batch
s_send (sink, "0");
// Initialize random number generator
srandom ((unsigned) time (NULL));
// Send 100 tasks
int task_nbr;
int total_msec = 0; // Total expected cost in msec
for (task_nbr = 0; task_nbr < 100; task_nbr++) {
int workload;
// Random workload from 1 to 100 msec
workload = randof (100) + 1;
total_msec += workload;
char string [10];
sprintf (string, "%d", workload);
s_send (sender, string);
}
printf ("Total expected cost: %d msecn", total_msec);
sleep (1); // Give 0MQ time to deliver
zmq_close (sink);
zmq_close (sender);
zmq_ctx_destroy (context);
return 0;
}
Divide and Conquer | 17](https://image.slidesharecdn.com/2157176-250609215532-c1f30a76/85/Zeromq-Messaging-For-Many-Applications-Pieter-Hintjens-39-320.jpg)
![proper garnished,” being a perpetuation of the triskelion or
triquetrum of Sicily.[234]
The arms of the Isle of Man afford an excellent illustration of the
migration of symbols as maintained in the work of Count Goblet
d’Alviella; but the attempt made by others to show it to be an
evolution from and migration of the Swastika is a failure.
Punch marks on Corinthian coins mistaken for Swastikas.—But is the
Swastika really found on ancient coins? The use of precious metals
as money dates to an unknown time in antiquity. Gold was used in
early Bible times (1500 B. C.) among nearly every people as money,
but it was by weight as a talent, and not as minted coin. The
coinage of money began about 700 B. C. in Lydia. Lydia was a
province on the western side of the peninsula of Asia Minor looking
out toward Greece, while Lycia, its neighbor, was a province on the
southern side looking toward the island of Rhodes. The Lydians
began coinage by stamping with a punch each ingot or nugget of
gold or silver, or a mixture of them called “Electrum.” In the
beginning these ingots were marked upon but one side, the reverse
showing plainly the fiber of the anvil on which the ingot was laid
when struck with the punch. But in a short time, it may have been
two hundred years, this system was changed so as to use a die
which would be reproduced on the coin when it was struck with a
punch. The lion, bull, boar, dolphin, and many other figures were
employed as designs for these dies. Athens used an owl; Corinth,
Pegasus; Metapontine, a sheaf of wheat; Naples, a human-headed
bull. The head and, occasionally, the entire form of the gods were
employed. During almost the entire first period of nigh three
hundred years the punch was used, and the punch marks show on
the reverse side of the coins. These punch marks were as various as
the dies for the obverse of the coins, but most of them took a
variety of the square, as it would present the greatest surface of
resistance to the punch. Even the triskelion of the Lycian coins is
within an indented square (figs. 225 and 226). A series of these
punch marks is given for demonstration on pl. 9. A favorite design](https://image.slidesharecdn.com/2157176-250609215532-c1f30a76/85/Zeromq-Messaging-For-Many-Applications-Pieter-Hintjens-55-320.jpg)
![was a square punch with a cross of two arms passing through the
center, dividing the field into four quarters. Most of the punch marks
on the coins of that period were of this kind. These punch marks
and the method and machinery with which they were made are
described in standard numismatic works.[235]
Fig. 229.
CORINTHIAN COINS.
Obverse and reverse. Punch mark resembling Swastika.
It is believed by the author that the assertions as to the presence of
the Swastika on these ancient coins is based upon an erroneous
interpretation of these punch marks. Fig. 229 shows the obverse and
reverse of a coin from Corinth. It belonged to the first half of the
sixth century B. C. The obverse represents a Pegasus standing, while
the reverse is a punch mark, said to have been a Swastika; but,
examining closely, we will find there is no Swastika in this punch
mark. The arms of the normal Swastika consist of straight lines
crossing each other. In this case they do not cross. The design
consists of four gammas, and each gamma is separated from its
fellows, all forming together very nearly the same design as
hundreds of other punch marks of the same period. If each outer
arm of this mark is made slightly longer, the Swastika form
disappears and the entire design resolves itself into the square
habitually employed for that purpose. If the punch mark on this
Corinthian coin be a Swastika, it depends upon the failure to make
the extreme end of the bent arm an eighth of an inch longer. This is](https://image.slidesharecdn.com/2157176-250609215532-c1f30a76/85/Zeromq-Messaging-For-Many-Applications-Pieter-Hintjens-56-320.jpg)
![too fine a point to be relied upon. If this punch mark had these arms
lengthened an eighth of an inch, it would confessedly become a
square.
EXPLANATION OF PLATE 9.
1 2 3
4 5 6
7 8 9
10 11 12
Punch Marks on Reverse of Ancient Coins.
Fig. 1. Coin of Lydia. Electrum. Oblong sinking between two
squares. Babylonian stater.
The earliest known coinage. Circa 700 B. C.
2. Phenician Half Stater. Electrum. Incuse square with
cruciform ornament.
3. Silver Coin of Teos. Incuse square. Circa 544 B. C.
4. Silver Coin of Acanthus. Incuse square.
5. Silver Coin of Mende. Incuse triangles.
6. Silver Coin of Terone. Incuse square.
7. Coin of Bisaltæ.[236] Incuse square. Octadrachm.
8. Silver Coin of Orrescii.[236] Incuse square. Octadrachm.
9. Corinthian Silver Coin. Incuse square divided into eight
triangular compartments.
The earliest coin of Corinth, dating B. C. 625 to 585.
10. Silver Coin of Abdera. Incuse square.
11. Silver Coin of Byzantium. Incuse square, granulated.
12. Silver Coin of Thrasos (Thrace). Incuse square.](https://image.slidesharecdn.com/2157176-250609215532-c1f30a76/85/Zeromq-Messaging-For-Many-Applications-Pieter-Hintjens-57-320.jpg)
![Fig. 230.
ANCIENT HINDU COIN IN THE FORM
OF A CROSS WITH A SWASTIKA ON
THE EXTREMITY OF EACH ARM.[237]
Waring, “Ceramic Art in Remote
Ages,” pl. 41, fig. 13.
field of each of these circles bears
a Swastika of normal form. Other
coins are cited of the same style,
with small center dots and
concentric circles in the stead of
the Swastika. What meaning the
Swastika has here, beyond the
possible one of being a lucky
penny, is not suggested.
Other ancient Hindu coins bearing
the Swastika (figs. 231-234) are
attributed to Cunningham by
Waring.[238] These are said by
Waring to be Buddhist coins found
at Behat near Scharaupur. Mr. E.
Thomas, in his article on the “Earliest Indian Coinage,”[239] ascribes
them to the reign of Krananda, a Buddhist Indian king contemporary
with or prior to Alexander, about 330 B. C.
The coins of Krananda,[240] contemporary of Alexander the Great,
[241] bear the Swastika mark, associated with the principal Buddhist
marks, the trisula, the stupha, sacred tree, sacred cone, etc. Waring
says[242] that according to Prinsep’s “Engravings of Hindu Coins,” the
Swastika seems to disappear from them about 200 B. C., nor is it
found on the Indo-Bactrian, the Indo-Sassanian, or the later Hindu
or subsequent Mohammedan, and he gives in a note the
approximate dates of these dynasties: Early native Buddhist
monarchs from about 500 B. C. to the conquest of Alexander, about
330 B. C.; the Indo-Bactrian or Greek successors of Alexander from
about 300 to 126 B. C.; the Indo-Parthian or Scythic from about 126
B. C.; the second Hindu dynasty from about 56 B. C.; the Indo-
Sassanian from A. D. 200 to 636, and subsequent to that the Indo-
Mohammedan from the eleventh to the close of the thirteenth
century; the Afghan dynasty from A. D. 1290 to 1526, and the](https://image.slidesharecdn.com/2157176-250609215532-c1f30a76/85/Zeromq-Messaging-For-Many-Applications-Pieter-Hintjens-59-320.jpg)
![Fig. 235.
ANCIENT COIN WITH
SWASTIKA.
Gaza, Palestine. Waring,
“Ceramic Art in Remote
Ages,” pl. 42, fig. 6.
Mongol dynasty to the eighteenth century, when it was destroyed by
Nadir Shah. (See p. 772.)
Figs. 231, 232, 233, and 234.
ANCIENT HINDU COINS WITH SWASTIKAS, NORMAL AND OGEE.
Waring, “Ceramic Art in Remote Ages,” pl. 41, figs. 20-24.
Swastika on coins in Mesembria and Gaza.
—Mr. Percy Gardner, in his article, “Ares as
a Sun-god,”[243] finds the Swastika on a
coin of Mesembria in Thrace. He explains
that “Mesembria is simply the Greek word
for noon, midday (μεσημβρία).” The coins
of this city bear the inscription ΜΕΣ ,
which Greg[244] believes refers by a kind
of pun to the name of the city, and so to
noon, or the sun or solar light. The
answer to this is the same given
throughout this paper, that it may be true,
but there is no evidence in support of it.
Max Müller[245] argues that this specimen is decisive of the meaning
of the sign Swastika. Both these gentlemen place great stress upon
the position which the Swastika held in the field relative to other
objects, and so determine it to have represented the sun or sunlight;
but all this seems non sequitur. A coin from Gaza, Palestine, ancient,
but date not given, is attributed to R. Rochette, and by him to](https://image.slidesharecdn.com/2157176-250609215532-c1f30a76/85/Zeromq-Messaging-For-Many-Applications-Pieter-Hintjens-60-320.jpg)
![Fig. 236.
GOLD BRACTEATE
WITH JAIN SWASTIKA.
Denmark. Waring,
“Ceramic Art in Remote
Ages,” pl. 1, fig. 9.
Munter (fig. 235). The Swastika sign is not perfect, only two arms of
the cross being turned, and not all four.
Swastika on Danish gold bracteates.—Fig.
236 represents a Danish gold bracteate
with a portrait head, two serpents, and a
Swastika with the outer ends finished with
a curve or flourish similar to that of the
Jains (fig. 33).
There are other bracteates with the
Swastika mark, which belong to the
Scandinavian countries.[246] Some of them
bear signs referring to Christian civilization,
such as raising hands in prayer; and from a
determination of the dates afforded by the
coins and other objects the Swastika can
be identified as having continued into the Christian era.
The coinage of the ancient world is not a prolific field for the
discovery of the Swastika. Other specimens may possibly be found
than those here given. This search is not intended to be exhaustive.
Their negative information is, however, valuable. It shows, first, that
some of the early stamps or designs on coins which have been
claimed as Swastikas were naught but the usual punch marks;
second, it shows a limited use of the Swastika on the coinage and
that it came to an end in very early times. Numismatics afford great
aid to archæology from the facility and certainty with which it fixes
dates. Using the dates furnished by the coinage of antiquity, it is
gravely to be questioned whether the prolific use of the Swastika in
Asia Minor (of which we have such notable examples on specimens
of pottery from the hill of Hissarlik, in Greece) did not terminate
before coinage began, or before 480 B. C., when the period of finer
engraving began, and it became the custom to employ on coins the
figures of gods, of tutelary deities, and of sacred animals. Thus the
use of the Swastika became relegated to objects of commoner use,
or those having greater relation to superstition and folklore wherein](https://image.slidesharecdn.com/2157176-250609215532-c1f30a76/85/Zeromq-Messaging-For-Many-Applications-Pieter-Hintjens-61-320.jpg)
![the possible value of the Swastika as an amulet or sign with power
to bring good luck could be better employed; or, as suggested by Mr.
Greg, that the great gods which, according to him, had the Swastika
for a symbol, fell into disrepute and it became changed to represent
something else.
UNITED STATES OF AMERICA.
PRE-COLUMBIAN TIMES.
Fains Island and Toco Mounds, Tennessee.—That the Swastika found
its way to the Western Hemisphere in prehistoric times can not be
doubted. A specimen (fig. 237) was taken by Dr. Edward Palmer in
the year 1881 from an ancient mound opened by him on Fains
Island, 3 miles from Bainbridge, Jefferson County, Tenn. It is figured
and described in the Third Annual Report of the Bureau of
Ethnology,[247] as follows:
A shell ornament, on the convex surface of which a very
curious ornamental design has been engraved. The
design, inclosed by a circle, represents a cross such as
would be formed by two rectangular tablets or slips slit
longitudinally and interlaced at right angles to each other.
The lines are neatly and deeply incised. The edge of the
ornament has been broken away nearly all around.
The incised lines of this design (fig. 237) represent the Swastika
turned to the left (though the description does not recognize it as
such). It has small circles with dots in the center, a style of work that
may become of peculiar value on further investigation, but not to be
confounded with the dots or points in what M. Zmigrodzki calls the
Croix swasticale. The mound from which this specimen came, and
the objects associated with it, show its antiquity and its manufacture
by the aborigines untainted by contact with the whites. The mound](https://image.slidesharecdn.com/2157176-250609215532-c1f30a76/85/Zeromq-Messaging-For-Many-Applications-Pieter-Hintjens-62-320.jpg)
![Fig. 238.
ENGRAVED SHELL WITH SWASTIKA, CIRCLES, AND DOTS.
Toco Mound, Monroe County, Tenn. Cat. No. 115624, U. S. N. M.
The peculiar form of this Swastika is duplicated by a Runic Swastika
in Sweden, cited by Ludwig Müller and by Count d’Alviella.[248]
The following objects were found in the mound on Fains Island
associated with the Swastika shell (fig. 237) and described, and
many of them figured:[249] A gorget of the same Fulgur shell (fig.
239); a second gorget of Fulgur shell with an engraved spider (fig.
278); a pottery vase with a figure of a frog; three rude axes from
four to seven inches in length, of diorite and quartzite; a pierced
tablet of slate; a disk of translucent quartz 1¾ inches in diameter
and three-quarters of an inch in thickness; a mass of pottery, much
of it in fragments, and a number of bone implements, including
needles and paddle-shaped objects. The shell objects (in addition to
the disks and gorgets mentioned) were pins made from the
columellæ of Fulgur (Busycon perversum?) of the usual form and
about four inches in length. There were also found shell beads,
cylindrical in form, an inch in length and upward of an inch in
diameter, with other beads of various sizes and shapes made from
marine shells, and natural specimens of Io spinosa, Unio probatus.](https://image.slidesharecdn.com/2157176-250609215532-c1f30a76/85/Zeromq-Messaging-For-Many-Applications-Pieter-Hintjens-64-320.jpg)
![not; but to all appearances it represents one of the Buddhist
divinities. Its material, similar to the hundred others found in the
neighborhood, shows it to have been indigenous, yet parts of its
style are different from other aboriginal North American images.
Attention is called to the slim waist, the winged arms, the crossed
legs, the long feet, breadth of toes, the many dots and circles shown
over the body, with triple lines of garters or anklets. All these show a
different dress from the ancient North American. The girdle about
the waist, and the triangular dress which, with its decorations and
arrangement of dots and circles, cover the lower part of the body,
are to be remarked. While there are several specimens of aboriginal
art from this part of the country which bear these peculiarities of
costumes, positions, appearance, and manner of work, showing
them to have been in use among a portion of the people, yet they
are not part of the usual art products. There is a manifest difference
between this and the ordinary statue of the Indian or of the mound
builder of that neighborhood or epoch.
It is not claimed that this shell proves the migration of Buddhism
from Asia, nor its presence among North American Indians. “One
swallow does not make a summer.” But this figure, taken in
connection with the Swastika, presents a set of circumstances
corresponding with that possibility which goes a long distance in
forming circumstantial evidence in its favor.
M. Gustave d’Eichthal wrote a series of essays in the Revue
Archæologique, 1864-65, in which he collated the evidence and
favored the theory of Buddhist influence in ancient America. Other
writers have taken the same or similar views and have attributed all
manner of foreign influence, like the Lost Tribes of Israel, etc., to the
North American Indian,[250] but all these theories have properly had
but slight influence in turning public opinion in their direction. Mr. V.
R. Gandhi, in a recent letter to the author, says of this specimen (pl.
10):
While Swastika technically means the cross with the arms
bent to the right, later on it came to signify anything](https://image.slidesharecdn.com/2157176-250609215532-c1f30a76/85/Zeromq-Messaging-For-Many-Applications-Pieter-Hintjens-67-320.jpg)
![which had the form of a cross; for instance, the posture in
which a persons sits with his legs crossed is called the
Swastika posture;[251] also when a person keeps his arms
crosswise over his chest, or a woman covers her breast
with her arms crossed, that particular attitude, is called
the Swastika attitude, which has no connection, however,
with the symbolic meaning of the Swastika with four arms.
The figure [pl. 10], a photograph of which you gave me
the other day, has the same Swastika posture. In matters
of concentration and meditation, Swastika posture is
oftentimes prescribed, which is also called Sukhasana,
meaning a posture of ease and comfort. In higher forms
of concentration, the posture is changed from Sukhasana
to Padmasana, the posture which is generally found in
Jain and Buddhist images. The band around the waist,
which goes from the navel lower on till it reaches the back
part, has a peculiar significance in the Jain philosophy.
The Shvetamber division of the Jain community have
always this kind of band in their images. The object is
twofold: The first is that the generative parts ought not to
be visible; the second is that this band is considered a
symbol of perfect chastity.
There can be no doubt of the authenticity of these objects, nor any
suspicion against their having been found as stated in the labels
attached. They are in the Museum collection, as are other
specimens. They come unheralded and with their peculiar character
unknown. They were obtained by excavations made by a competent
and reliable investigator who had been engaged in mound
exploration, a regular employé of the Bureau of Ethnology, under the
direction of Prof. Cyrus Thomas during several years, and always of
good reputation and unblemished integrity. They come with other
objects, labeled in the same way and forming one of a series of
numbers among thousands. Its resemblance to Buddhist statues was
apparently undiscovered or unrecognized, at least unmentioned, by
all those having charge of it, and in its mutilated condition it was laid](https://image.slidesharecdn.com/2157176-250609215532-c1f30a76/85/Zeromq-Messaging-For-Many-Applications-Pieter-Hintjens-68-320.jpg)
![had been engraved on the concave side. The design represents two
human figures plumed and winged, armed with eagles’ talons and
engaged in mortal combat. The design apparently covered the entire
shell, leaving no space for encircling lines. The two figures are in
profile and face each other in a fierce onset. Of the right-hand
figure, only the body, one arm, and one leg remain. The left-hand
figure is almost complete. The outline of the face, one arm, and one
foot is all that is affected. The right hand is raised above the head in
the act of brandishing a long knife pointed at both ends. The other
combatant, clutching in his right hand a savage-looking blade with
its point curved, seems delivering a blow in the face of his
antagonist. Of the visible portions of the figures, the hands are
vigorously drawn, the thumbs press down upon the outside of the
forefingers in a natural effort to tighten the grasp. The body, arms,
and legs are well defined and in proper proportion, the joints are
correctly placed, the left knee is bent forward, and the foot planted
firmly on the ground, while the right is thrown gracefully back
against the rim at the left, and the legs terminate in well-drawn
eagles’ feet armed with curved talons. The head is decorated with a
single plume which springs from a circular ornament placed over the
ear; an angular figure extends forward from the base of this plume,
and probably represents what is left of the headdress proper. In
front of this—on the very edge of the crumbling shell—is one-half of
the lozenge-shaped eye, the dot representing the pupil being almost
obliterated. The ankles and legs just below the knee and the wrists
each have three lines representing bracelets or anklets. It is
uncertain whether the leg is covered or naked; but between the
waistband and the leggings, over the abdomen, is represented on
both figures a highly decorated triangular garment, or, possibly coat
of mail, to which particular attention is called.[252] In the center, at
the top, just under the waistband, are four circles with dots in the
center arranged in a square; outside of this, still at the top, are two
triangular pieces, and outside of them are two more circles and dots;
while the lower part of the triangle, with certain decorations of
incised lines, completes the garment. This decoration is the same on
both figures, and corresponds exactly with the Buddha figure. An](https://image.slidesharecdn.com/2157176-250609215532-c1f30a76/85/Zeromq-Messaging-For-Many-Applications-Pieter-Hintjens-73-320.jpg)
![ornament is suspended on the breast which shows three more of the
circles and dots. The earring is still another. The right-hand figure, so
far as it can be seen, is a duplicate of the left, and in the drawing it
has, where destroyed, been indicated by dotted lines. It is
remarkable that the peculiar clothing or decoration of these two
figures should be almost an exact reproduction of the Buddha figure
(pl. 10). Another interesting feature of the design is the highly
conventionalized wing which fills the space beneath the uplifted arm.
This wing is unlike the usual specimens of aboriginal art which have
been found in such profusion in that neighborhood. But it is again
remarkable that this conventionalized wing and the bracelets,
anklets, and garters should correspond in all their peculiarities of
construction and design with the wings on the copper and shell
figures from the Etowah mound, Georgia (figs. 240, 241, and 242)
[253]. Behind the left-hand figure is an ornament resembling the
spreading tail of an eagle which, with its feather arrangement and
the detail of their mechanism, correspond to a high degree with the
eagle effigies in repoussé copper (fig. 243) from the mound in Union
County, Ill., shown in the Fifth Annual Report of the Bureau of
Ethnology (p. 105) and in the Twelfth Annual Report (p. 309).](https://image.slidesharecdn.com/2157176-250609215532-c1f30a76/85/Zeromq-Messaging-For-Many-Applications-Pieter-Hintjens-74-320.jpg)
![Fig. 241.
COPPER PLATE.
Repoussé work. Entowah Mound, Georgia.
Cat. No. 91117, U. S. N. M.
Hopewell Mound, Chillicothe, Ross County, Ohio.—A later discovery
of the Swastika belonging to the same period and the same general
locality—that is, to the Ohio Valley—was that of Prof. Warren K.
Moorehead, in the fall and winter of 1891-92, in his excavations of
the Hopewell mound, seven miles northwest of Chillicothe, Ross
County, Ohio.[254] The locality of this mound is well shown in Squier
and Davis’s work on the “Monuments of the Mississippi Valley” (pl.
10, p. 26), under the name of “Clark’s Works,” here reproduced as](https://image.slidesharecdn.com/2157176-250609215532-c1f30a76/85/Zeromq-Messaging-For-Many-Applications-Pieter-Hintjens-76-320.jpg)
Zeromq Messaging For Many Applications Pieter Hintjens
- 1. Zeromq Messaging For Many Applications Pieter Hintjens download https://ebookbell.com/product/zeromq-messaging-for-many- applications-pieter-hintjens-4314352 Explore and download more ebooks at ebookbell.com
- 2. Here are some recommended products that we believe you will be interested in. You can click the link to download. Zeromq Use Zeromq And Learn To Apply Different Message Patterns New Edition Faruk Akgul https://ebookbell.com/product/zeromq-use-zeromq-and-learn-to-apply- different-message-patterns-new-edition-faruk-akgul-23358066 Zeromq Pieter Hintjens https://ebookbell.com/product/zeromq-pieter-hintjens-52974522 Code Connected Volume 1 Learning Zeromq 1st Edition Pieter Hintjens https://ebookbell.com/product/code-connected-volume-1-learning- zeromq-1st-edition-pieter-hintjens-48793592
- 6. ZeroMQ by Pieter Hintjens Copyright © 2013 Pieter Hintjens. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are alsoavailableformosttitles(http://my.safaribooksonline.com).Formoreinformation,contactourcorporate/ institutional sales department: 800-998-9938 or corporate@oreilly.com. Editors: Andy Oram and Maria Gulick Production Editor: Christopher Hearse Copyeditor: Gillian McGarvey Proofreader: Rachel Head Indexer: Angela Howard Cover Designer: Randy Comer Interior Designer: David Futato Illustrator: Rebecca Demarest and Kara Ebrahim March 2013: First Edition Revision History for the First Edition: 2013-03-11: First release See http://oreilly.com/catalog/errata.csp?isbn=9781449334062 for release details. Nutshell Handbook, the Nutshell Handbook logo, and the O’Reilly logo are registered trademarks of O’Reilly Media, Inc. ZeroMQ, the image of a fourhorn sculpin, and related trade dress are trademarks of O’Reilly Media, Inc. Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in this book, and O’Reilly Media, Inc., was aware of a trade‐ mark claim, the designations have been printed in caps or initial caps. While every precaution has been taken in the preparation of this book, the publisher and authors assume no responsibility for errors or omissions, or for damages resulting from the use of the information contained herein. ISBN: 978-1-449-33406-2 [LSI]
- 7. To Noémie, Freeman, and Gregor.
- 9. Table of Contents Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii Part I. Learning to Work with ØMQ 1. Basics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Fixing the World 3 Audience for This Book 5 Getting the Examples 5 Ask and Ye Shall Receive 5 A Minor Note on Strings 10 Version Reporting 11 Getting the Message Out 11 Divide and Conquer 16 Programming with ØMQ 21 Getting the Context Right 21 Making a Clean Exit 22 Why We Needed ØMQ 23 Socket Scalability 27 Upgrading from ØMQ v2.2 to ØMQ v3.2 27 Warning: Unstable Paradigms! 28 2. Sockets and Patterns. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 The Socket API 32 Plugging Sockets into the Topology 32 Using Sockets to Carry Data 34 Unicast Transports 35 ØMQ Is Not a Neutral Carrier 35 I/O Threads 36 Messaging Patterns 37 v
- 10. High-Level Messaging Patterns 38 Working with Messages 39 Handling Multiple Sockets 41 Multipart Messages 44 Intermediaries and Proxies 45 The Dynamic Discovery Problem 45 Shared Queue (DEALER and ROUTER Sockets) 48 ØMQ’s Built-in Proxy Function 53 Transport Bridging 54 Handling Errors and ETERM 56 Handling Interrupt Signals 61 Detecting Memory Leaks 62 Multithreading with ØMQ 63 Signaling Between Threads (PAIR Sockets) 68 Node Coordination 70 Zero-Copy 74 Pub-Sub Message Envelopes 75 High-Water Marks 77 Missing Message Problem Solver 78 3. Advanced Request-Reply Patterns. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81 The Request-Reply Mechanisms 81 The Simple Reply Envelope 82 The Extended Reply Envelope 82 What’s This Good For? 85 Recap of Request-Reply Sockets 85 Request-Reply Combinations 86 The REQ to REP Combination 87 The DEALER to REP Combination 87 The REQ to ROUTER Combination 87 The DEALER to ROUTER Combination 88 The DEALER to DEALER Combination 88 The ROUTER to ROUTER Combination 88 Invalid Combinations 88 Exploring ROUTER Sockets 89 Identities and Addresses 89 ROUTER Error Handling 91 The Load-Balancing Pattern 91 ROUTER Broker and REQ Workers 92 ROUTER Broker and DEALER Workers 94 A Load-Balancing Message Broker 96 A High-Level API for ØMQ 102 vi | Table of Contents
- 11. Features of a Higher-Level API 104 The CZMQ High-Level API 105 The Asynchronous Client/Server Pattern 111 Worked Example: Inter-Broker Routing 116 Establishing the Details 116 Architecture of a Single Cluster 117 Scaling to Multiple Clusters 118 Federation Versus Peering 121 The Naming Ceremony 122 Prototyping the State Flow 123 Prototyping the Local and Cloud Flows 126 Putting It All Together 133 4. Reliable Request-Reply Patterns. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141 What Is “Reliability”? 141 Designing Reliability 142 Client-Side Reliability (Lazy Pirate Pattern) 144 Basic Reliable Queuing (Simple Pirate Pattern) 148 Robust Reliable Queuing (Paranoid Pirate Pattern) 151 Heartbeating 159 Shrugging It Off 160 One-Way Heartbeats 160 Ping-Pong Heartbeats 161 Heartbeating for Paranoid Pirate 161 Contracts and Protocols 163 Service-Oriented Reliable Queuing (Majordomo Pattern) 164 Asynchronous Majordomo Pattern 186 Service Discovery 191 Idempotent Services 193 Disconnected Reliability (Titanic Pattern) 194 High-Availability Pair (Binary Star Pattern) 206 Detailed Requirements 208 Preventing Split-Brain Syndrome 211 Binary Star Implementation 211 Binary Star Reactor 218 Brokerless Reliability (Freelance Pattern) 223 Model One: Simple Retry and Failover 225 Model Two: Brutal Shotgun Massacre 228 Model Three: Complex and Nasty 233 Conclusion 244 5. Advanced Publish-Subscribe Patterns. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245 Table of Contents | vii
- 12. Pros and Cons of Publish-Subscribe 245 Pub-Sub Tracing (Espresso Pattern) 247 Last Value Caching 250 Slow Subscriber Detection (Suicidal Snail Pattern) 254 High-Speed Subscribers (Black Box Pattern) 258 Reliable Publish-Subscribe (Clone Pattern) 260 Centralized Versus Decentralized 261 Representing State as Key-Value Pairs 261 Getting an Out-of-Band Snapshot 271 Republishing Updates from Clients 276 Working with Subtrees 281 Ephemeral Values 284 Using a Reactor 292 Adding the Binary Star Pattern for Reliability 296 The Clustered Hashmap Protocol 306 Building a Multithreaded Stack and API 310 Part II. Software Engineering Using ØMQ 6. The ØMQ Community. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 325 Architecture of the ØMQ Community 326 How to Make Really Large Architectures 327 Psychology of Software Architecture 328 The Contract 330 The Process 332 Crazy, Beautiful, and Easy 332 Stranger, Meet Stranger 333 Infinite Property 333 Care and Feeding 334 The ØMQ Process: C4 335 Language 335 Goals 336 Preliminaries 338 Licensing and Ownership 339 Patch Requirements 340 Development Process 342 Creating Stable Releases 345 Evolution of Public Contracts 347 A Real-Life Example 349 Git Branches Considered Harmful 352 Simplicity Versus Complexity 353 viii | Table of Contents
- 13. Change Latency 353 Learning Curve 353 Cost of Failure 353 Up-Front Coordination 354 Scalability 354 Surprise and Expectations 354 Economics of Participation 354 Robustness in Conflict 355 Guarantees of Isolation 355 Visibility 355 Conclusions 355 Designing for Innovation 356 The Tale of Two Bridges 356 How ØMQ Lost Its Road Map 356 Trash-Oriented Design 359 Complexity-Oriented Design 361 Simplicity-Oriented Design 362 Burnout 364 Patterns for Success 366 The Lazy Perfectionist 366 The Benevolent Tyrant 366 The Earth and Sky 366 The Open Door 367 The Laughing Clown 367 The Mindful General 367 The Social Engineer 367 The Constant Gardener 367 The Rolling Stone 368 The Pirate Gang 368 The Flash Mob 368 The Canary Watcher 368 The Hangman 369 The Historian 369 The Provocateur 369 The Mystic 369 7. Advanced Architecture Using ØMQ. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 371 Message-Oriented Pattern for Elastic Design 372 Step 1: Internalize the Semantics 373 Step 2: Draw a Rough Architecture 373 Step 3: Decide on the Contracts 374 Step 4: Write a Minimal End-to-End Solution 374 Table of Contents | ix
- 14. Step 5: Solve One Problem and Repeat 375 Unprotocols 375 Contracts Are Hard 376 How to Write Unprotocols 377 Why Use the GPLv3 for Public Specifications? 378 Using ABNF 379 The Cheap or Nasty Pattern 380 Serializing Your Data 382 ØMQ Framing 382 Serialization Languages 383 Serialization Libraries 384 Handwritten Binary Serialization 385 Code Generation 386 Transferring Files 392 State Machines 403 Authentication Using SASL 410 Large-Scale File Publishing: FileMQ 411 Why Make FileMQ? 412 Initial Design Cut: The API 412 Initial Design Cut: The Protocol 413 Building and Trying FileMQ 414 Internal Architecture 415 Public API 416 Design Notes 417 Configuration 418 File Stability 419 Delivery Notifications 420 Symbolic Links 420 Recovery and Late Joiners 421 Test Use Case: The Track Tool 423 Getting an Official Port Number 424 8. A Framework for Distributed Computing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 425 Design for the Real World 426 The Secret Life of WiFi 427 Why Mesh Isn’t Here Yet 428 Some Physics 429 What’s the Current Status? 430 Conclusions 432 Discovery 432 Preemptive Discovery over Raw Sockets 432 Cooperative Discovery Using UDP Broadcasts 434 x | Table of Contents
- 15. Multiple Nodes on One Device 439 Designing the API 439 More About UDP 448 Spinning Off a Library Project 448 Point-to-Point Messaging 450 UDP Beacon Framing 450 True Peer Connectivity (Harmony Pattern) 452 Detecting Disappearances 454 Group Messaging 455 Testing and Simulation 457 On Assertions 457 On Up-Front Testing 458 The Zyre Tester 459 Test Results 461 Tracing Activity 463 Dealing with Blocked Peers 464 Distributed Logging and Monitoring 467 A Plausible Minimal Implementation 468 Protocol Assertions 470 Binary Logging Protocol 471 Content Distribution 473 Writing the Unprotocol 475 Conclusions 476 9. Postface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 479 Tales from Out There 479 Rob Gagnon’s Story 479 Tom van Leeuwen’s Story 479 Michael Jakl’s Story 480 Vadim Shalts’s Story 480 How This Book Happened 481 Removing Friction 482 Licensing 484 Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 485 Table of Contents | xi
- 17. Preface ØMQ in a Hundred Words ØMQ (also known as ZeroMQ, 0MQ, or zmq) looks like an embeddable networking library, but acts like a concurrency framework. It gives you sockets that carry atomic messages across various transports, like in-process, inter-process, TCP, and multicast. You can connect sockets N-to-N with patterns like fan-out, pub-sub, task distribution, and request-reply. It’s fast enough to be the fabric for clustered products. Its asynchro‐ nous I/O model gives you scalable multicore applications, built as asynchronous message-processing tasks. It has a score of language APIs and runs on most operating systems. ØMQ is from iMatix and is LGPLv3 open source. The Zen of Zero TheØinØMQisallabouttrade-offs.Ontheonehand,thisstrangenamelowersØMQ’s visibility on Google and Twitter. On the other hand, it annoys the heck out of some Danish folk who write us things like “ØMG røtfl”, and “Ø is not a funny-looking zero!” and “Rødgrød med Fløde!” (which is apparently an insult that means “May your neigh‐ bours be the direct descendants of Grendel!”). Seems like a fair trade. Originally, the zero in ØMQ was meant to signify “zero broker” and (as close to) “zero latency” (as possible). Since then, it has come to encompass different goals: zero ad‐ ministration, zero cost, zero waste. More generally, “zero” refers to the culture of min‐ imalism that permeates the project. We add power by removing complexity rather than by exposing new functionality. How This Book Came to Be In the summer of 2010, ØMQ was still a little-known niche library described by its rather terse reference manual and a living but sparse wiki. Martin Sustrik and I were sitting in the bar of the Hotel Kyjev in Bratislava plotting how to make ØMQ more xiii
- 18. widely popular. Martin had written most of the ØMQ code, and I’d put up the funding and organized the community. Over some Zlatý Bažant, we agreed that ØMQ needed a new, simpler website and a basic guide for new users. Martin collected some ideas for topics to explain. I’d never written a line of ØMQ code before this, so it became a live learning documentary. As I worked through simple examples to more complex ones, I tried to answer many of the questions I’d seen on the mailing list. Because I’d been building large-scale architectures for 30 years, there were alotofproblemsIwaskeentothrowØMQat.Amazingly,theresultsweremostlysimple and elegant, even when working in C. I felt a pure joy learning ØMQ and using it to solve real problems, which brought me back to programming after a few years’ pause. And often, not knowing how it was “supposed” to be done, we improved ØMQ as we went along. From the start, I wanted the guide to be a community project, so I put it onto GitHub and let others contribute with pull requests. This was considered a radical, even vulgar approach by some. We came to a division of labor: I’d do the writing and make the original C examples, and others would help fix the text and translate the examples into other languages. This worked better than I dared hope. You can now find all the examples in several languages, and many in a dozen languages. It’s a kind of programming language Rosetta Stone, and a valuable outcome in itself. We set up a high score: reach 80% translation and your language gets its own guide. PHP, Python, Lua, and Haxe reached this goal. People asked for PDFs, and we created those. People asked for ebooks, and got those. About a hundred people have contributed to the guide to date. The guide achieved its goal of popularizing ØMQ. The style pleases most and annoys some, which is how it should be. In December 2010, my work on ØMQ and the guide stopped, as I found myself going through late-stage cancer, heavy surgery, and six months of chemotherapy. When I picked up work again in mid-2011, it was to start using ØMQ in anger for one of the largest use-cases imagineable: on the mobile phones and tablets of the world’s biggest electronics company. But the goal of the guide was, from the start, a printed book. So it was exciting to get an email from Bill Lubanovic in January 2012, introducing me to his editor, Andy Oram, at O’Reilly, suggesting a ØMQ book. “Of course!” I said. Where do I sign? How much do I have to pay? Oh, I get money for this? All I have to do is finish it?” Ofcourse,assoonasO’ReillyannouncedaØMQbook,otherpublishersstartedsending out emails to potential authors. You’ll probably see a rash of ØMQ books coming out next year. That’s good. Our niche library has hit the mainstream and deserves its six inches of shelf space. My apologies to the other ØMQ authors. We’ve set the bar horribly high, and my advice is to make your books complementary. Perhaps focus on a specific language, platform, or pattern. xiv | Preface
- 19. This is the magic and power of communities: be the first community in a space, stay healthy, and you own that space for ever. Audience This book is written for professional programmers who want to learn how to make the massively distributed software that will dominate the future of computing. We assume you can read C code, because most of the examples here are in C (even though ØMQ is used in many languages). We assume you care about scale, because ØMQ solves that problem above all others. We assume you need the best possible results with the least possible cost, because otherwise you won’t appreciate the trade-offs that ØMQ makes. Other than that basic background, we try to present all the concepts in networking and distributed computing you will need to use ØMQ. Conventions Used in This Book We used the following typographical conventions in this book: Italic Indicates new terms, commands and command-line options, URLs, email address‐ es, filenames, and file extensions. Constant width Used for program listings, as well as within paragraphs to refer to program elements such as variable or function names, data types, and environment variables. Constant width bold Shows user input at the command line. Constant width italic Shows placeholder user input that you should replace with something that makes sense for you. This icon signifies a tip, suggestion, or general note. Using the Code Examples The code examples are all online in the repository at https://github.com/imatix/zguide/ tree/master/examples/. You’ll find each example translated into several—often a dozen —other languages. The examples are licensed under MIT/X11; see the LICENSE file in that directory. The text of the book explains in each case how to run each example. Preface | xv
- 20. We appreciate, but do not require, attribution. An attribution usually includes the title, author, publisher, and ISBN. For example: “ZeroMQ by Pieter Hintjens (O’Reilly). Copyright 2013 Pieter Hintjens, 978-1-449-33406-2.” If you feel your use of code examples falls outside fair use or the permission given above, feel free to contact us at permissions@oreilly.com. Safari® Books Online Safari Books Online (www.safaribooksonline.com) is an on-demand digital library that delivers expert content in both book and video form from the world’s leading authors in technology and business. Technology professionals, software developers, web designers, and business and crea‐ tive professionals use Safari Books Online as their primary resource for research, prob‐ lem solving, learning, and certification training. Safari Books Online offers a range of product mixes and pricing programs for organi‐ zations, government agencies, and individuals. Subscribers have access to thousands of books, training videos, and prepublication manuscripts in one fully searchable database from publishers like O’Reilly Media, Prentice Hall Professional, Addison-Wesley Pro‐ fessional, Microsoft Press, Sams, Que, Peachpit Press, Focal Press, Cisco Press, John Wiley & Sons, Syngress, Morgan Kaufmann, IBM Redbooks, Packt, Adobe Press, FT Press, Apress, Manning, New Riders, McGraw-Hill, Jones & Bartlett, Course Technol‐ ogy, and dozens more. For more information about Safari Books Online, please visit us online. How to Contact Us Please address comments and questions concerning this book to the publisher: O’Reilly Media, Inc. 1005 Gravenstein Highway North Sebastopol, CA 95472 800-998-9938 (in the United States or Canada) 707-829-0515 (international or local) 707-829-0104 (fax) We have a web page for this book, where we list errata, examples, and any additional information. You can access this page at http://bit.ly/ZeroMQ-OReilly. To comment or ask technical questions about this book, send email to bookques tions@oreilly.com. xvi | Preface
- 21. For more information about our books, courses, conferences, and news, see our website at http://www.oreilly.com. Find us on Facebook: http://facebook.com/oreilly Follow us on Twitter: http://twitter.com/oreillymedia Watch us on YouTube: http://www.youtube.com/oreillymedia Acknowledgments Thanks to Andy Oram for making this happen at O’Reilly and editing the book. Thanks to Bill Desmarais, Brian Dorsey, Daniel Lin, Eric Desgranges, Gonzalo Dieth‐ elm, Guido Goldstein, Hunter Ford, Kamil Shakirov, Martin Sustrik, Mike Castleman, Naveen Chawla, Nicola Peduzzi, Oliver Smith, Olivier Chamoux, Peter Alexander, Pierre Rouleau, Randy Dryburgh, John Unwin, Alex Thomas, Mihail Minkov, Jeremy Avnet, Michael Compton, Kamil Kisiel, Mark Kharitonov, Guillaume Aubert, Ian Bar‐ ber,MikeSheridan,FarukAkgul,OlegSidorov,LevGivon,AllisterMacLeod,Alexander D’Archangel, Andreas Hoelzlwimmer, Han Holl, Robert G. Jakabosky, Felipe Cruz, Marcus McCurdy, Mikhail Kulemin, Dr. Gergö Érdi, Pavel Zhukov, Alexander Else, Giovanni Ruggiero, Rick “Technoweenie”, Daniel Lundin, Dave Hoover, Simon Jefford, Benjamin Peterson, Justin Case, Devon Weller, Richard Smith, Alexander Morland, Wadim Grasza, Michael Jakl, Uwe Dauernheim, Sebastian Nowicki, Simone Deponti, Aaron Raddon, Dan Colish, Markus Schirp, Benoit Larroque, Jonathan Palardy, Isaiah Peng, Arkadiusz Orzechowski, Umut Aydin, Matthew Horsfall, Jeremy W. Sherman, Eric Pugh, Tyler Sellon, John E. Vincent, Pavel Mitin, Min RK, Igor Wiedler, Olof Åkes‐ son, Patrick Lucas, Heow Goodman, Senthil Palanisami, John Gallagher, Tomas Roos, StephenMcQuay,ErikAllik,ArnaudCogoluègnes,RobGagnon,DanWilliams,Edward Smith, James Tucker, Kristian Kristensen, Vadim Shalts, Martin Trojer, Tom van Leeu‐ wen, Hiten Pandya, Harm Aarts, Marc Harter, Iskren Ivov Chernev, Jay Han, Sonia Hamilton, Nathan Stocks, Naveen Palli, and Zed Shaw for their contributions to this work. Thanks to Martin Sustrik for his years of incredible work on ZeroMQ. Thanks to Stathis Sideris for Ditaa. Preface | xvii
- 23. PART I Learning to Work with ØMQ In the first part of this book, you’ll learn how to use ØMQ. We’ll cover the basics, the API, the different socket types and how they work, reliability, and a host of patterns you can use in your applications. You’ll get the best results by working through the examples and text from start to end.
- 25. CHAPTER 1 Basics Fixing the World How to explain ØMQ? Some of us start by saying all the wonderful things it does. It’s sockets on steroids. It’s like mailboxes with routing. It’s fast! Others try to share their moment of enlightenment, that zap-pow-kaboom satori paradigm-shift moment when it all became obvious. Things just become simpler. Complexity goes away. It opens the mind. Others try to explain by comparison. It’s smaller, simpler, but still looks famili‐ ar. Personally, I like to remember why we made ØMQ at all, because that’s most likely where you, the reader, still are today. Programming is a science dressed up as art, because most of us don’t understand the physics of software and it’s rarely, if ever, taught. The physics of software is not algo‐ rithms, data structures, languages, and abstractions. These are just tools we make, use, and throw away. The real physics of software is the physics of people. Specifically, it’s about our limitations when it comes to complexity and our desire to work together to solve large problems in pieces. This is the science of programming: make building blocks that people can understand and use easily, and people will work together to solve the very largest problems. We live in a connected world, and modern software has to navigate this world. So, the building blocks for tomorrow’s very largest solutions are connected and massively par‐ allel. It’s not enough for code to be “strong and silent” any more. Code has to talk to code. Code has to be chatty, sociable, and well-connected. Code has to run like the human brain; trillions of individual neurons firing off messages to each other, a mas‐ sively parallel network with no central control, no single point of failure, yet able to solve immensely difficult problems. And it’s no accident that the future of code looks like the human brain, because the endpoints of every network are, at some level, human brains. 3
- 26. If you’ve done any work with threads, protocols, or networks, you’ll realize this is pretty much impossible. It’s a dream. Even connecting a few programs across a few sockets is plain nasty when you start to handle real-life situations. Trillions? The cost would be unimaginable. Connecting computers is so difficult that creating software and services to do this is a multi-billion dollar business. So we live in a world where the wiring is years ahead of our ability to use it. We had a software crisis in the 1980s, when leading software engineers like Fred Brooks believed there was no “silver bullet” to “promise even one order of magnitude of improvement in productivity, reliability, or simplicity.” Brooks missed free and open source software, which solved that crisis, enabling us to share knowledge efficiently. Today we face another software crisis, but it’s one we don’t talk about much. Only the largest, richest firms can afford to create connected appli‐ cations. There is a cloud, but it’s proprietary. Our data and our knowledge are disap‐ pearing from our personal computers into clouds that we cannot access and with which we cannot compete. Who owns our social networks? It is like the mainframe-PC rev‐ olution in reverse. We can leave the political philosophy for another book. The point is that while the Internet offers the potential of massively connected code, the reality is that this is out of reach for most of us, and so large, interesting problems (in health, education, eco‐ nomics, transport, and so on) remain unsolved because there is no way to connect the code, and thus no way to connect the brains that could work together to solve these problems. There have been many attempts to solve the challenge of connected software. There are thousands of IETF specifications, each solving part of the puzzle. For application de‐ velopers, HTTP is perhaps the one solution to have been simple enough to work, but it arguably makes the problem worse by encouraging developers and architects to think in terms of big servers and thin, stupid clients. So today people are still connecting applications using raw UDP and TCP, proprietary protocols,HTTP,andWebSockets.Itremainspainful,slow,hardtoscale,andessentially centralized. Distributed peer-to-peer architectures are mostly for play, not work. How many applications use Skype or BitTorrent to exchange data? Which brings us back to the science of programming. To fix the world, we needed to do two things. One, to solve the general problem of “how to connect any code to any code, anywhere.” Two, to wrap that up in the simplest possible building blocks that people could understand and use easily. It sounds ridiculously simple. And maybe it is. That’s kind of the whole point. 4 | Chapter 1: Basics
- 27. Audience for This Book We assume you are using the latest 3.2 release of ØMQ. We assume you are using a Linux box or something similar. We assume you can read C code, more or less, as that’s the default language for the examples. We assume that when we write constants like PUSH or SUBSCRIBE, you can imagine they are really called ZMQ_PUSH or ZMQ_SUB SCRIBE if the programming language needs it. Getting the Examples This book’s examples live in the book’s Git repository. The simplest way to get all the examples is to clone this repository: git clone --depth=1 git://github.com/imatix/zguide.git Next, browse the examples subdirectory. You’ll find examples by language. If there are examples missing in a language you use, you’re encouraged to submit a translation. This is how this book became so useful, thanks to the work of many people. All examples are licensed under MIT/X11. Ask and Ye Shall Receive So let’s start with some code. We’ll begin, of course, with a “Hello World” example. We’ll make a client and a server. The client sends “Hello” to the server, which replies with “World” (Figure 1-1). Example 1-1 presents the code for the server in C, which opens a ØMQ socket on port 5555, reads requests on it, and replies with “World” to each request. Example 1-1. Hello World server (hwserver.c) // // Hello World server // Binds REP socket to tcp://*:5555 // Expects "Hello" from client, replies with "World" // #include <zmq.h> #include <stdio.h> #include <unistd.h> #include <string.h> int main (void) { void *context = zmq_ctx_new (); // Socket to talk to clients void *responder = zmq_socket (context, ZMQ_REP); zmq_bind (responder, "tcp://*:5555"); Audience for This Book | 5
- 28. while (1) { // Wait for next request from client zmq_msg_t request; zmq_msg_init (&request); zmq_msg_recv (&request, responder, 0); printf ("Received Hellon"); zmq_msg_close (&request); // Do some 'work' sleep (1); // Send reply back to client zmq_msg_t reply; zmq_msg_init_size (&reply, 5); memcpy (zmq_msg_data (&reply), "World", 5); zmq_msg_send (&reply, responder, 0); zmq_msg_close (&reply); } // We never get here but if we did, this would be how we end zmq_close (responder); zmq_ctx_destroy (context); return 0; } Figure 1-1. Request-reply The REQ-REP socket pair is in lockstep. The client issues zmq_msg_send() and then zmq_msg_recv(), in a loop (or once if that’s all it needs). Any other sequence (e.g., sending two messages in a row) will result in a return code of -1 from the send or recv call. Similarly, the server issues zmq_msg_recv() and then zmq_msg_send(), in that or‐ der, as often as it needs to. ØMQ uses C as its reference language, and this is the main language we’ll use for ex‐ amples.Ifyou’rereadingthisonline,thelinkbelowtheexampletakesyoutotranslations intootherprogramminglanguages.Forprintreaders,Example1-2showswhatthesame server looks like in C++. 6 | Chapter 1: Basics
- 29. Example 1-2. Hello World server (hwserver.cpp) // // Hello World server in C++ // Binds REP socket to tcp://*:5555 // Expects "Hello" from client, replies with "World" // #include <zmq.hpp> #include <string> #include <iostream> #include <unistd.h> int main () { // Prepare our context and socket zmq::context_t context (1); zmq::socket_t socket (context, ZMQ_REP); socket.bind ("tcp://*:5555"); while (true) { zmq::message_t request; // Wait for next request from client socket.recv (&request); std::cout << "Received Hello" << std::endl; // Do some 'work' sleep (1); // Send reply back to client zmq::message_t reply (5); memcpy ((void *) reply.data (), "World", 5); socket.send (reply); } return 0; } You can see that the ØMQ API is similar in C and C++. In a language like PHP, we can hide even more and the code becomes even easier to read, as shown in Example 1-3. Example 1-3. Hello World server (hwserver.php) <?php /* * Hello World server * Binds REP socket to tcp://*:5555 * Expects "Hello" from client, replies with "World" * @author Ian Barber <ian(dot)barber(at)gmail(dot)com> */ $context = new ZMQContext(1); // Socket to talk to clients Ask and Ye Shall Receive | 7
- 30. $responder = new ZMQSocket($context, ZMQ::SOCKET_REP); $responder->bind("tcp://*:5555"); while (true) { // Wait for next request from client $request = $responder->recv(); printf ("Received request: [%s]n", $request); // Do some 'work' sleep (1); // Send reply back to client $responder->send("World"); } Example 1-4 shows the client code. Example 1-4. Hello World client (hwclient.c) // // Hello World client // Connects REQ socket to tcp://localhost:5555 // Sends "Hello" to server, expects "World" back // #include <zmq.h> #include <string.h> #include <stdio.h> #include <unistd.h> int main (void) { void *context = zmq_ctx_new (); // Socket to talk to server printf ("Connecting to hello world server...n"); void *requester = zmq_socket (context, ZMQ_REQ); zmq_connect (requester, "tcp://localhost:5555"); // int request_nbr; // for (request_nbr = 0; request_nbr != 10; request_nbr++) { // zmq_msg_t request; // zmq_msg_init_size (&request, 5); // memcpy (zmq_msg_data (&request), "Hello", 5); // printf ("Sending Hello %d...n", request_nbr); // zmq_msg_send (&request, requester, 0); // zmq_msg_close (&request); // // zmq_msg_t reply; // zmq_msg_init (&reply); // zmq_msg_recv (&reply, requester, 0); // printf ("Received World %dn", request_nbr); // zmq_msg_close (&reply); // } 8 | Chapter 1: Basics
- 31. sleep (2); zmq_close (requester); zmq_ctx_destroy (context); return 0; } Now this looks too simple to be realistic, but a ØMQ socket is what you get when you takeanormalTCPsocket,injectitwithamixofradioactiveisotopesstolenfromasecret Soviet atomic research project, bombard it with 1950s-era cosmic rays, and put it into the hands of a drug-addled comic book author with a badly disguised fetish for bulging muscles clad in spandex (Figure 1-2). Yes, ØMQ sockets are the world-saving super‐ heroes of the networking world. Figure 1-2. There was a terrible accident... You could throw thousands of clients at this server, all at once, and it would continue to work happily and quickly. For fun, try starting the client and then starting the server, see how it all still works, and then think for a second what this means. Let us explain briefly what these two programs are actually doing. They create a ØMQ context to work with, and a socket. Don’t worry what the words mean. You’ll pick it up. The server binds its REP (reply) socket to port 5555. It then waits for a request in a loop, and responds each time with a reply. The client sends a request and reads the reply back from the server. If you kill the server (Ctrl-C) and restart it, the client won’t recover properly. Recovering from crashing processes isn’t quite that easy. Making a reliable request-reply flow is complex enough that we won’t cover it until “Reliable Request-Reply Patterns” in Chap‐ ter 4. There is a lot happening behind the scenes, but what matters to us programmers is how short and sweet the code is and how often it doesn’t crash, even under a heavy load. This is the request-reply pattern, probably the simplest way to use ØMQ. It maps to RPC (remote procedure calls) and the classic client/server model. Ask and Ye Shall Receive | 9
- 32. A Minor Note on Strings ØMQdoesn’tknowanythingaboutthedatayousendexceptitssizeinbytes.Thatmeans you are responsible for formatting it safely so that applications can read it back. Doing this for objects and complex data types is a job for specialized libraries like protocol buffers. But even for strings, you need to take care. In C and some other languages, strings are terminated with a null byte. We could send a string like “HELLO” with that extra null byte: zmq_msg_init_data (&request, "Hello", 6, NULL, NULL); However, if you send a string from another language, it probably will not include that null byte. For example, when we send that same string in Python, we do this: socket.send ("Hello") Then what goes onto the wire is a length (one byte for shorter strings) and the string contents as individual characters (Figure 1-3). Figure 1-3. A ØMQ string And if you read this from a C program, you will get something that looks like a string, and might by accident act like a string (if by luck the five bytes find themselves followed by an innocently lurking null), but isn’t a proper string. When your client and server don’t agree on the string format, you will get weird results. When you receive string data from ØMQ in C, you simply cannot trust that it’s safely terminated. Every single time you read a string, you should allocate a new buffer with space for an extra byte, copy the string, and terminate it properly with a null. So let’s establish the rule that ØMQ strings are length-specified and are sent on the wire without a trailing null. In the simplest case (and we’ll do this in our examples), a ØMQ stringmapsneatlytoaØMQmessageframe,whichlooksliketheabovefigure—alength and some bytes. Hereiswhatweneedtodo,inC,toreceiveaØMQstringanddeliverittotheapplication as a valid C string: // Receive 0MQ string from socket and convert into C string static char * s_recv (void *socket) { zmq_msg_t message; zmq_msg_init (&message); int size = zmq_msg_recv (&message, socket, 0); 10 | Chapter 1: Basics
- 33. if (size == -1) return NULL; char *string = malloc (size + 1); memcpy (string, zmq_msg_data (&message), size); zmq_msg_close (&message); string [size] = 0; return (string); } This makes a very handy helper function. In the spirit of making things we can reuse profitably, we can write a similar s_send() function that sends strings in the correct ØMQ format and package this into a header file we can reuse. The result is zhelpers.h, which lets us write sweeter and shorter ØMQ applications in C. It is a fairly long source, and only fun for C developers, so read it at your leisure. Version Reporting ØMQ does come in several versions, and quite often if you hit a problem, it’ll be some‐ thing that’s been fixed in a later version. So it’s a useful trick to know exactly what version of ØMQ you’re actually linking with. Example 1-5 is a tiny program that lets you do just that. Example 1-5. ØMQ version reporting (version.c) // // Report 0MQ version // #include "zhelpers.h" int main (void) { int major, minor, patch; zmq_version (&major, &minor, &patch); printf ("Current 0MQ version is %d.%d.%dn", major, minor, patch); return EXIT_SUCCESS; } Getting the Message Out Thesecondclassicpatternisone-waydatadistribution,inwhichaserverpushesupdates to a set of clients. Let’s look at an example that pushes out weather updates consisting of a zip code, temperature, and relative humidity. We’ll generate random values, just like the real weather stations do. Version Reporting | 11
- 34. Example 1-6 shows the code for the server. We’ll use port 5556 for this application. Example 1-6. Weather update server (wuserver.c) // // Weather update server // Binds PUB socket to tcp://*:5556 // Publishes random weather updates // #include "zhelpers.h" int main (void) { // Prepare our context and publisher void *context = zmq_ctx_new (); void *publisher = zmq_socket (context, ZMQ_PUB); int rc = zmq_bind (publisher, "tcp://*:5556"); assert (rc == 0); rc = zmq_bind (publisher, "ipc://weather.ipc"); assert (rc == 0); // Initialize random number generator srandom ((unsigned) time (NULL)); while (1) { // Get values that will fool the boss int zipcode, temperature, relhumidity; zipcode = randof (100000); temperature = randof (215) - 80; relhumidity = randof (50) + 10; // Send message to all subscribers char update [20]; sprintf (update, "%05d %d %d", zipcode, temperature, relhumidity); s_send (publisher, update); } zmq_close (publisher); zmq_ctx_destroy (context); return 0; } There’s no start and no end to this stream of updates; it’s like a never-ending broadcast (Figure 1-4). 12 | Chapter 1: Basics
- 35. Figure 1-4. Publish-subscribe Example 1-7 shows the client application, which listens to the stream of updates and grabs anything to do with a specified zip code (by default, New York City, because that’s a great place to start any adventure). Example 1-7. Weather update client (wuclient.c) // // Weather update client // Connects SUB socket to tcp://localhost:5556 // Collects weather updates and finds avg temp in zipcode // #include "zhelpers.h" int main (int argc, char *argv []) { void *context = zmq_ctx_new (); // Socket to talk to server printf ("Collecting updates from weather server...n"); void *subscriber = zmq_socket (context, ZMQ_SUB); int rc = zmq_connect (subscriber, "tcp://localhost:5556"); assert (rc == 0); // Subscribe to zipcode, default is NYC, 10001 char *filter = (argc > 1)? argv [1]: "10001 "; rc = zmq_setsockopt (subscriber, ZMQ_SUBSCRIBE, filter, strlen (filter)); assert (rc == 0); // Process 100 updates int update_nbr; long total_temp = 0; for (update_nbr = 0; update_nbr < 100; update_nbr++) { char *string = s_recv (subscriber); int zipcode, temperature, relhumidity; Getting the Message Out | 13
- 36. sscanf (string, "%d %d %d", &zipcode, &temperature, &relhumidity); total_temp += temperature; free (string); } printf ("Average temperature for zipcode '%s' was %dFn", filter, (int) (total_temp / update_nbr)); zmq_close (subscriber); zmq_ctx_destroy (context); return 0; } Note that when you use a SUB socket you must set a subscription using zmq_setsock opt() and SUBSCRIBE, as in this code. If you don’t set any subscription, you won’t get any messages. It’s a common mistake for beginners. The subscriber can set many sub‐ scriptions, which are added together. That is, if an update matches any subscription, the subscriber receives it. The subscriber can also cancel specific subscriptions. A subscrip‐ tion is often but not necessarily a printable string. See zmq_setsockopt() for how this works. The PUB-SUB socket pair is asynchronous. The client does zmq_msg_recv(), in a loop (or once if that’s all it needs). Trying to send a message to a SUB socket will cause an error. Similarly, the service does zmq_msg_send() as often as it needs to, but must not do zmq_msg_recv() on a PUB socket. In theory, with ØMQ sockets it does not matter which end connects and which end binds. However, in practice there are undocumented differences that I’ll come to later. For now, bind the PUB and connect the SUB, unless your network design makes that impossible. There is one more important thing to know about PUB-SUB sockets: you do not know precisely when a subscriber starts to get messages. Even if you start a subscriber, wait a while, and then start the publisher, the subscriber will always miss the first messages that the publisher sends. This is because as the subscriber connects to the publisher (some‐ thing that takes a small but nonzero amount of time), the publisher may already be sending messages out. This “slow joiner” symptom hits enough people, often enough, that we’re going to ex‐ plain it in detail. Remember that ØMQ does asynchronous I/O (i.e., in the background). Say you have two nodes doing this, in this order: • Subscriber connects to an endpoint and receives and counts messages. • Publisher binds to an endpoint and immediately sends 1,000 messages. The subscriber will most likely not receive anything. You’ll blink, check that you set a correct filter, and try again, and the subscriber will still not receive anything. 14 | Chapter 1: Basics
- 37. Making a TCP connection involves to and from handshaking that can take several mil‐ liseconds (msec), depending on your network and the number of hops between peers. In that time, ØMQ can send very many messages. For the sake of argument, assume it takes 5 msec to establish a connection, and that same link can handle 1M messages per second. During the 5 msec that the subscriber is connecting to the publisher, it takes the publisher only 1 msec to send out those 1K messages. In Chapter 2, we’ll explain how to synchronize a publisher and subscribers so that you don’t start to publish data until the subscribers really are connected and ready. There is a simple (and stupid) way to delay the publisher, which is to sleep. Don’t do this in a real application, though, because it is extremely fragile as well as inelegant and slow. Use sleeps to prove to yourself what’s happening, and then read Chapter 2 to see how to do this right. The alternative to synchronization is to simply assume that the published data stream is infinite and has no start and no end. One also assumes that the subscriber doesn’t care what transpired before it started up. This is how we built our weather client example. So, the client subscribes to its chosen zip code and collects a thousand updates for that zipcode.Thatmeansabout10millionupdatesfromtheserver,ifzipcodesarerandomly distributed.Youcanstarttheclient,andthentheserver,andtheclientwillkeepworking. You can stop and restart the server as often as you like, and the client will keep working. When the client has collected its thousand updates, it calculates the average, prints it, and exits. Some points about the publish-subscribe pattern: • A subscriber can connect to more than one publisher, using one connect call each time. Data will then arrive and be interleaved (“fair queued”) so that no single pub‐ lisher drowns out the others. • If a publisher has no connected subscribers, then it will simply drop all messages. • If you’re using TCP and a subscriber is slow, messages will queue up on the pub‐ lisher. We’ll look at how to protect publishers against this by using the “high-water mark” in the next chapter. • From ØMQ v3.x, filtering happens on the publisher’s side when using a connected protocol (tcp or ipc). Using the epgm protocol, filtering happens on the subscriber’s side. In ØMQ v2.x, all filtering happened on the subscriber’s side. This is how long it takes to receive and filter 10M messages on my laptop, which is a 2011-era Intel I7—fast, but nothing special: ph@nb201103:~/work/git/zguide/examples/c$ time wuclient Collecting updates from weather server... Average temperature for zipcode '10001 ' was 28F Getting the Message Out | 15
- 38. real 0m4.470s user 0m0.000s sys 0m0.008s Divide and Conquer As a final example (you are surely getting tired of juicy code and want to delve back into philological discussions about comparative abstractive norms), let’s do a little super‐ computing. Then, coffee. Our supercomputing application is a fairly typical parallel processing model (Figure 1-5). We have: • A ventilator that produces tasks that can be done in parallel • A set of workers that processes tasks • A sink that collects results back from the worker processes Figure 1-5. Parallel pipeline In reality, workers run on superfast boxes, perhaps using GPUs (graphic processing units) to do the hard math. Example 1-8 shows the code for the ventilator. It generates 100 tasks, each one a message telling the worker to sleep for some number of milliseconds. 16 | Chapter 1: Basics
- 39. Example 1-8. Parallel task ventilator (taskvent.c) // // Task ventilator // Binds PUSH socket to tcp://localhost:5557 // Sends batch of tasks to workers via that socket // #include "zhelpers.h" int main (void) { void *context = zmq_ctx_new (); // Socket to send messages on void *sender = zmq_socket (context, ZMQ_PUSH); zmq_bind (sender, "tcp://*:5557"); // Socket to send start of batch message on void *sink = zmq_socket (context, ZMQ_PUSH); zmq_connect (sink, "tcp://localhost:5558"); printf ("Press Enter when the workers are ready: "); getchar (); printf ("Sending tasks to workers...n"); // The first message is "0" and signals start of batch s_send (sink, "0"); // Initialize random number generator srandom ((unsigned) time (NULL)); // Send 100 tasks int task_nbr; int total_msec = 0; // Total expected cost in msec for (task_nbr = 0; task_nbr < 100; task_nbr++) { int workload; // Random workload from 1 to 100 msec workload = randof (100) + 1; total_msec += workload; char string [10]; sprintf (string, "%d", workload); s_send (sender, string); } printf ("Total expected cost: %d msecn", total_msec); sleep (1); // Give 0MQ time to deliver zmq_close (sink); zmq_close (sender); zmq_ctx_destroy (context); return 0; } Divide and Conquer | 17
- 40. The code for the worker application is in Example 1-9. It receives a message, sleeps for that number of seconds, and then signals that it’s finished. Example 1-9. Parallel task worker (taskwork.c) // // Task worker // Connects PULL socket to tcp://localhost:5557 // Collects workloads from ventilator via that socket // Connects PUSH socket to tcp://localhost:5558 // Sends results to sink via that socket // #include "zhelpers.h" int main (void) { void *context = zmq_ctx_new (); // Socket to receive messages on void *receiver = zmq_socket (context, ZMQ_PULL); zmq_connect (receiver, "tcp://localhost:5557"); // Socket to send messages to void *sender = zmq_socket (context, ZMQ_PUSH); zmq_connect (sender, "tcp://localhost:5558"); // Process tasks forever while (1) { char *string = s_recv (receiver); // Simple progress indicator for the viewer fflush (stdout); printf ("%s.", string); // Do the work s_sleep (atoi (string)); free (string); // Send results to sink s_send (sender, ""); } zmq_close (receiver); zmq_close (sender); zmq_ctx_destroy (context); return 0; } Finally, Example 1-10 shows the sink application. It collects the 100 messages and then calculates how long the overall processing took, so we can confirm that the workers really were running in parallel if there are more than one of them. 18 | Chapter 1: Basics
- 41. Example 1-10. Parallel task sink (tasksink.c) // // Task sink // Binds PULL socket to tcp://localhost:5558 // Collects results from workers via that socket // #include "zhelpers.h" int main (void) { // Prepare our context and socket void *context = zmq_ctx_new (); void *receiver = zmq_socket (context, ZMQ_PULL); zmq_bind (receiver, "tcp://*:5558"); // Wait for start of batch char *string = s_recv (receiver); free (string); // Start our clock now int64_t start_time = s_clock (); // Process 100 confirmations int task_nbr; for (task_nbr = 0; task_nbr < 100; task_nbr++) { char *string = s_recv (receiver); free (string); if ((task_nbr / 10) * 10 == task_nbr) printf (":"); else printf ("."); fflush (stdout); } // Calculate and report duration of batch printf ("Total elapsed time: %d msecn", (int) (s_clock () - start_time)); zmq_close (receiver); zmq_ctx_destroy (context); return 0; } The average cost of a batch is five seconds. When we start one, two, and four workers, we get results like this from the sink: # 1 worker Total elapsed time: 5034 msec # 2 workers Total elapsed time: 2421 msec # 4 workers Total elapsed time: 1018 msec Divide and Conquer | 19
- 42. Let’s look at some aspects of this code in more detail: • The workers connect upstream to the ventilator, and downstream to the sink. This means you can add workers arbitrarily. If the workers bound to their endpoints, you would need (a) more endpoints and (b) to modify the ventilator and/or the sink each time you added a worker. We say that the ventilator and sink are stable parts of our architecture and the workers are dynamic parts of it. • Wehavetosynchronizethestartofthebatchwithallworkersbeingupandrunning. This is a fairly common gotcha in ØMQ, and there is no easy solution. The con nect method takes a certain amount of time, so when a set of workers connect to the ventilator, the first one to successfully connect will get a whole load of messages in that short time while the others are still connecting. If you don’t synchronize the start of the batch somehow, the system won’t run in parallel at all. Try removing the wait in the ventilator, and see what happens. • The ventilator’s PUSH socket distributes tasks to workers (assuming they are all connected before the batch starts going out) evenly. This is called load balancing, and it’s something we’ll look at again in more detail. • The sink’s PULL socket collects results from workers evenly. This is called fair queuing (Figure 1-6). Figure 1-6. Fair queuing The pipeline pattern also exhibits the “slow joiner” syndrome, leading to accusations that PUSH sockets don’t load-balance properly. If you are using PUSH and PULL and one of your workers gets way more messages than the others, it’s because that PULL socket has joined faster than the others, and grabs a lot of messages before the others manage to connect. 20 | Chapter 1: Basics
- 43. Programming with ØMQ Havingseensomeexamples,youmustbeeagertostartusingØMQinsomeapps.Before you start that, take a deep breath, chillax, and reflect on some basic advice that will save you much stress and confusion: • Learn ØMQ step-by-step. It’s just one simple API, but it hides a world of possibil‐ ities. Take the possibilities slowly and master each one. • Write nice code. Ugly code hides problems and makes it hard for others to help you. You might get used to meaningless variable names, but people reading your code won’t. Use names that are real words, that say something other than “I’m too careless to tell you what this variable is really for.” Use consistent indentation and clean layout. Write nice code, and your world will be more comfortable. • Test what you make as you make it. When your program doesn’t work, you should knowwhichfivelinesaretoblame.ThisisespeciallytruewhenyoudoØMQmagic, which just won’t work the first few times you try it. • When you find that things don’t work as expected, break your code into pieces, test eachone,andseewhichoneisnotworking.ØMQletsyoumakeessentiallymodular code; use that to your advantage. • Makeabstractions(classes,methods,whatever)asyouneedthem.Ifyoucopy/paste a lot of code, you’re going to copy/paste errors, too. Getting the Context Right ØMQ applications always start by creating a context, and then using that for creating sockets. In C, it’s the zmq_ctx_new() call. You should create and use exactly one context inyourprocess.Technically,thecontextisthecontainerforallsocketsinasingleprocess, and it acts as the transport for inproc sockets, which are the fastest way to connect threads in one process. If at runtime a process has two contexts, these are like separate ØMQ instances. If that’s explicitly what you want, that’s OK, but otherwise remember: Do one zmq_ctx_new() at the start of your main code, and one zmq_ctx_destroy() at the end. If you’re using the fork() system call, each process needs its own context. If you do zmq_ctx_new() in the main process before calling fork(), the child processes get their own contexts. In general, you want to do the interesting stuff in the child processes and just manage these from the parent process. Programming with ØMQ | 21
- 44. Making a Clean Exit Classy programmers share the same motto as classy hit men: always clean up when you finish the job. When you use ØMQ in a language like Python, stuff gets automatically freed for you. But when using C, you have to carefully free objects when you’re finished with them, or else you get memory leaks, unstable applications, and generally bad kar‐ ma. Memoryleaksareonething,butØMQisquitefinickyabouthowyouexitanapplication. The reasons are technical and painful, but the upshot is that if you leave any sockets open, the zmq_ctx_destroy() function will hang forever. And even if you close all sockets, zmq_ctx_destroy() will by default wait forever if there are pending connects or sends, unless you set the LINGER to zero on those sockets before closing them. The ØMQ objects we need to worry about are messages, sockets, and contexts. Luckily, it’s quite simple, at least in simple programs: • Always close a message the moment you are done with it, using zmq_msg_close(). • If you are opening and closing a lot of sockets, that’s probably a sign that you need to redesign your application. • When you exit the program, close your sockets and then call zmq_ctx_destroy(). This destroys the context. This is at least the case for C development. In a language with automatic object de‐ struction, sockets and contexts will be destroyed as you leave the scope. If you use exceptions you’ll have to do the cleanup in something like a “final” block, the same as for any resource. If you’re doing multithreaded work, it gets rather more complex than this. We’ll get to multithreading in the next chapter, but because some of you will, despite warnings, try to run before you can safely walk, here is a quick and dirty guide to making a clean exit in a multithreaded ØMQ application. First, do not try to use the same socket from multiple threads. Please don’t explain why you think this would be excellent fun; just don’t do it. Next, you need to shut down each socket that has ongoing requests. The proper way is to set a low LINGER value (one second), and then close the socket. If your language binding doesn’t do this for you automatically when you destroy a context, I’d suggest sending a patch. Finally, destroy the context. This will cause any blocking receives or polls or sends in attached threads (i.e., which share the same context) to return with an error. Catch that error, and then set LINGER on and close sockets in that thread, and exit. Do not destroy the same context twice. The zmq_ctx_destroy() call in the main thread will block until all sockets it knows about are safely closed. 22 | Chapter 1: Basics
- 45. Voilà! It’s complex and painful enough that any language binding author worth his or her salt will do this automatically and make the socket closing dance unnecessary. Why We Needed ØMQ Now that you’ve seen ØMQ in action, let’s go back to the “why.” Many applications these days consist of components that stretch across some kind of network, either a LAN or the Internet. So, many application developers end up doing some kind of messaging. Some developers use message queuing products, but most of the time they do it themselves, using TCP or UDP. These protocols are not hard to use, but there is a great difference between sending a few bytes from A to B and doing messaging in any kind of reliable way. Let’s look at the typical questions we face when we start to connect pieces using raw TCP. Any reusable messaging layer would need to address all or most of these: • How do we handle I/O? Does our application block, or do we handle I/O in the background? This is a key design decision. Blocking I/O creates architectures that do not scale well, but background I/O can be very hard to do right. • How do we handle dynamic components (i.e., pieces that go away temporarily)? Do we formally split components into “clients” and “servers” and mandate that servers cannot disappear? What, then, if we want to connect servers to servers? Do we try to reconnect every few seconds? • How do we represent a message on the wire? How do we frame data so it’s easy to write and read, safe from buffer overflows, and efficient for small messages, yet adequate for the very largest videos of dancing cats wearing party hats? • How do we handle messages that we can’t deliver immediately? Particularly if we’re waiting for a component to come back online? Do we discard messages, put them into a database, or put them into a memory queue? • Where do we store message queues? What happens if the component reading from a queue is very slow and causes our queues to build up? What’s our strategy then? • How do we handle lost messages? Do we wait for fresh data, request a resend, or do we build some kind of reliability layer that ensures messages cannot be lost? What if that layer itself crashes? • What if we need to use a different network transport? Say, multicast instead of TCP unicast? Or IPv6? Do we need to rewrite the applications, or is the transport ab‐ stracted in some layer? • How do we route messages? Can we send the same message to multiple peers? Can we send replies back to an original requester? Why We Needed ØMQ | 23
- 46. • How do we write an API for another language? Do we reimplement a wire-level protocol, or do we repackage a library? If the former, how can we guarantee efficient and stable stacks? If the latter, how can we guarantee interoperability? • How do we represent data so that it can be read between different architectures? Do we enforce a particular encoding for data types? To what extent is this the job of the messaging system rather than a higher layer? • How do we handle network errors? Do we wait and retry, ignore them silently, or abort? Take a typical open source project like Apache ZooKeeper. Read the C API code in src/ c/src/zookeeper.c. When I read this code in 2010, it was 3,200 lines of mystery, and in there is an undocumented client/server network communication protocol. I see it’s ef‐ ficientbecauseitusespoll()insteadofselect().Butreally,ZooKeepershouldbeusing a generic messaging layer and an explicitly documented wire-level protocol. It is in‐ credibly wasteful for teams to be building this particular wheel over and over. But how do we make a reusable messaging layer? Why, when so many projects need this technology, are people still doing it the hard way by driving TCP sockets in their code, and solving the problems in that long list over and over (Figure 1-7)? Figure 1-7. Messaging as it starts It turns out that building reusable messaging systems is really difficult, which is why few free and open source (FOSS) projects ever tried, and why commercial messaging products are complex, expensive, inflexible, and brittle. In 2006, iMatix designed the AdvancedMessageQueuingProtocol,orAMQP,whichstartedtogiveFOSSdevelopers perhaps the first reusable recipe for a messaging system. AMQP works better than many other designs, but remains relatively complex, expensive, and brittle. It takes weeks to learn to use it, and months to create stable architectures that don’t crash when things get hairy. Most messaging projects (like AMQP) that try to solve this long list of problems in a reusable way do so by inventing a new concept, the “broker,” that does addressing, routing, and queuing. This results in a client/server protocol or a set of APIs on top of some undocumented protocol that allows applications to speak to this broker. Brokers are an excellent thing in reducing the complexity of large networks. But adding broker- based messaging to a product like ZooKeeper would make it worse, not better. It would 24 | Chapter 1: Basics
- 47. mean adding an additional big box, and a new single point of failure. A broker rapidly becomes a bottleneck and a new risk to manage. If the software supports it, we can add a second, third, and fourth broker and make some failover scheme. People do this. However, it creates more moving pieces, more complexity, more things to break. Also, a broker-centric setup needs its own operations team. You literally need to watch the brokers day and night, and beat them with a stick when they start misbehaving. You need boxes, and you need backup boxes, and you need people to manage those boxes. It is only worth doing for large applications with many moving pieces, built by several teams of people over several years. So, small to medium application developers are trapped. Either they avoid network programming and make monolithic applications that do not scale, or they jump into networkprogrammingandmakebrittle,complexapplicationsthatarehardtomaintain. Or they bet on a messaging product, and end up with scalable applications that depend on expensive, easily broken technology. There has been no really good choice, which may be why messaging is largely stuck in the last century and stirs strong emotions—negative ones for users, gleeful joy for those selling support and licenses (Figure 1-8). Figure 1-8. Messaging as it becomes What we need is something that does the job of messaging, but does it in such a simple and cheap way that it can work in any application, with close to zero cost. It should be a library with which you link without any other dependencies. No additional moving pieces, so no additional risk. It should run on any OS and work with any programming language. Why We Needed ØMQ | 25
- 48. And this is ØMQ: an efficient, embeddable library that solves most of the problems an application needs to become nicely elastic across a network, without much cost. Specifically: • It handles I/O asynchronously, in background threads. These communicate with application threads using lock-free data structures, so concurrent ØMQ applica‐ tions need no locks, semaphores, or other wait states. • Componentscancomeandgodynamically,andØMQwillautomaticallyreconnect. Thismeansyoucanstartcomponentsinanyorder.Youcancreate“service-oriented architectures” (SOAs) where services can join and leave the network at any time. • It queues messages automatically when needed. It does this intelligently, pushing messages as close as possible to the receiver before queuing them. • It has ways of dealing with over-full queues (called the “high-water mark”). When a queue is full, ØMQ automatically blocks senders, or throws away messages, de‐ pending on the kind of messaging you are doing (the so-called “pattern”). • It lets your applications talk to each other over arbitrary transports: TCP, multicast, in-process, inter-process. You don’t need to change your code to use a different transport. • It handles slow/blocked readers safely, using different strategies that depend on the messaging pattern. • It lets you route messages using a variety of patterns, such as request-reply and publish-subscribe. These patterns are how you create the topology, the structure of your network. • It lets you create proxies to queue, forward, or capture messages with a single call. Proxies can reduce the interconnection complexity of a network. • It delivers whole messages exactly as they were sent, using a simple framing on the wire. If you write a 10KB message, you will receive a 10KB message. • Itdoesnotimposeanyformatonmessages.Theyareblobsofzerobytestogigabytes large. When you want to represent data you choose some other product on top, such as Google’s protocol buffers, XDR, and others. • It handles network errors intelligently. Sometimes it retries, sometimes it tells you an operation failed. • It reduces your carbon footprint. Doing more with less CPU means your boxes use less power, and you can keep your old boxes in use for longer. Al Gore would love ØMQ. Actually, ØMQ does rather more than this. It has a subversive effect on how you develop network-capable applications. Superficially, it’s a socket-inspired API on which you do zmq_msg_recv() and zmq_msg_send(). But the message processing loop rapidly be‐ 26 | Chapter 1: Basics
- 49. comes the central loop, and your application soon breaks down into a set of message processing tasks. It is elegant and natural. And it scales: each of these tasks maps to a node, and the nodes talk to each other across arbitrary transports. Two nodes in one process (node is a thread), two nodes on one box (node is a process), or two boxes on one network (node is a box)—it’s all the same, with no application code changes. Socket Scalability Let’s see ØMQ’s scalability in action. Here is a shell script that starts the weather server and then a bunch of clients in parallel: wuserver & wuclient 12345 & wuclient 23456 & wuclient 34567 & wuclient 45678 & wuclient 56789 & As the clients run, we take a look at the active processes using top, and we see something like (on a four-core box): PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 7136 ph 20 0 1040m 959m 1156 R 157 12.0 16:25.47 wuserver 7966 ph 20 0 98608 1804 1372 S 33 0.0 0:03.94 wuclient 7963 ph 20 0 33116 1748 1372 S 14 0.0 0:00.76 wuclient 7965 ph 20 0 33116 1784 1372 S 6 0.0 0:00.47 wuclient 7964 ph 20 0 33116 1788 1372 S 5 0.0 0:00.25 wuclient 7967 ph 20 0 33072 1740 1372 S 5 0.0 0:00.35 wuclient Let’s think for a second about what is happening here. The weather server has a single socket, and yet here we have it sending data to five clients in parallel. We could have thousands of concurrent clients. The server application doesn’t see them and doesn’t talk to them directly. So the ØMQ socket is acting like a little server, silently accepting client requests and shoving data out to them as fast as the network can handle it. And it’s a multithreaded server, squeezing more juice out of your CPU. Upgrading from ØMQ v2.2 to ØMQ v3.2 In early 2012, ØMQ v3.2 became stable enough for live use, and by the time you’re reading this, it’s what you really should be using. If you are still using v2.2, here’s a quick summary of the changes and instructions on how to migrate your code. Pub-sub filtering is now done on the publisher side instead of the subscriber side. This improves performance significantly in many pub-sub use cases. You can mix version 3.2 and 2.1/2.2 publishers and subscribers safely. Most of the API is backward compatible, except a few changes that went into v3.0 with little regard to the cost of breaking existing code. The syntax of zmq_send() and Socket Scalability | 27
- 50. zmq_recv() changed, and ZMQ_NOBLOCK got rebaptized ZMQ_DONTWAIT. So although I’d love to say, “You just recompile your code with the latest libzmq and everything will work,” that’s not how it is. For what it’s worth, we banned such API breakage afterwards. So, the minimal change for C/C++ apps that use the low-level libzmq API is to replace all calls to zmq_send() with zmq_msg_send(), and zmq_recv() with zmq_msg_recv(). In other languages, your binding author may have done the work already. Note that these two functions now return -1 in case of error, and zero or more according to how many bytes were sent or received. Other parts of the libzmq API became more consistent. We deprecated zmq_init() and zmq_term(), replacing them with zmq_ctx_new() and zmq_ctx_destroy(). We added zmq_ctx_set() to let you configure a context before starting to work with it. Finally, we added context monitoring via the zmq_ctx_set_monitor() call, which lets you track connections and disconnections, and other events on sockets. Warning: Unstable Paradigms! Traditional network programming is built on the general assumption that one socket talks to one connection, one peer. There are multicast protocols, but these are exotic. When we assume “one socket = one connection,” we scale our architectures in certain ways. We create threads of logic where each thread works with one socket, one peer. We place intelligence and state in these threads. In the ØMQ universe, sockets are doorways to fast little background communications engines that manage a whole set of connections automagically for you. You can’t see, work with, open, close, or attach state to these connections. Whether you use blocking send or receive or poll, all you can talk to is the socket, not the connections it manages for you. The connections are private and invisible, and this is the key to ØMQ’s scala‐ bility. This is because your code, talking to a socket, can then handle any number of connec‐ tions across whatever network protocols are around, without change. A messaging pat‐ tern sitting in ØMQ can scale more cheaply than a messaging pattern sitting in your application code. So, the general assumption no longer applies. As you read the code examples, your brain will try to map them to what you know. You will read “socket” and think “Ah, that represents a connection to another node.” That is wrong. You will read “thread” and your brain will again think, “Ah, a thread represents a connection to another node,” and again your brain will be wrong. If you’re reading this book for the first time, realize that until you actually spend a day or two writing ØMQ code (or maybe three or four days), you may feel confused, espe‐ cially by how simple ØMQ makes things for you; you may try to impose that general 28 | Chapter 1: Basics
- 51. assumption on ØMQ, and it won’t work. And then you will experience your moment of enlightenment and trust, that zap-pow-kaboom satori paradigm-shift moment when it all becomes clear. Warning: Unstable Paradigms! | 29
- 53. Another Random Document on Scribd Without Any Related Topics
- 54. took the title of King of Sicily and quartered the Sicilian arms with the Royal arms of England. The negotiations between Henry and the Pope progressed for several years (1255 to 1259), when Henry, finding that he could no longer make it an excuse for raising money, allowed it to pass into the limbo of forgotten objects. Alexander III of Scotland had married Margaret, the youngest daughter of Henry III, and thus was brother-in-law to Edmund as well as to Frederick. In 1256, and while these negotiations between Henry and the Pope concerning Sicily were in progress, Alexander visited, at London, his royal father-in-law, the King of England, and his royal brother-in-law, the King of Sicily, and was received with great honors. About that time Haco, the Norse king of the Isle of Man, was defeated by Alexander III of Scotland, and killed, soon after which event (1266) the Isle of Man was ceded to the latter. The Norse coat of arms disappeared from the escutcheon of the Isle of Man, and, being replaced by the three legs of Sicily, Mr. Newton inquires: What more likely than that the King (Alexander III), when he struck the Norwegian flag, should replace it by one bearing the picturesque and striking device of Sicily, an island having so many points of resemblance with that of Man, and over which his sister ruled as Queen and her brother had been appointed as King? However little we may know concerning the method of transfer of the coat of arms from Sicily to the Isle of Man, we are not left at all in doubt as to the fact of its accomplishment; and the triskelion of Sicily became then and has been ever since, and is now, the armorial emblem of the Isle of Man. The Duke of Athol, the last proprietary of the Isle of Man, and who, in 1765, sold his rights to the Crown of England, still bears the arms of Man as the fifth quartering, “The three human legs in armor, conjoined at the upper part of the thigh and flexed in triangle,
- 55. proper garnished,” being a perpetuation of the triskelion or triquetrum of Sicily.[234] The arms of the Isle of Man afford an excellent illustration of the migration of symbols as maintained in the work of Count Goblet d’Alviella; but the attempt made by others to show it to be an evolution from and migration of the Swastika is a failure. Punch marks on Corinthian coins mistaken for Swastikas.—But is the Swastika really found on ancient coins? The use of precious metals as money dates to an unknown time in antiquity. Gold was used in early Bible times (1500 B. C.) among nearly every people as money, but it was by weight as a talent, and not as minted coin. The coinage of money began about 700 B. C. in Lydia. Lydia was a province on the western side of the peninsula of Asia Minor looking out toward Greece, while Lycia, its neighbor, was a province on the southern side looking toward the island of Rhodes. The Lydians began coinage by stamping with a punch each ingot or nugget of gold or silver, or a mixture of them called “Electrum.” In the beginning these ingots were marked upon but one side, the reverse showing plainly the fiber of the anvil on which the ingot was laid when struck with the punch. But in a short time, it may have been two hundred years, this system was changed so as to use a die which would be reproduced on the coin when it was struck with a punch. The lion, bull, boar, dolphin, and many other figures were employed as designs for these dies. Athens used an owl; Corinth, Pegasus; Metapontine, a sheaf of wheat; Naples, a human-headed bull. The head and, occasionally, the entire form of the gods were employed. During almost the entire first period of nigh three hundred years the punch was used, and the punch marks show on the reverse side of the coins. These punch marks were as various as the dies for the obverse of the coins, but most of them took a variety of the square, as it would present the greatest surface of resistance to the punch. Even the triskelion of the Lycian coins is within an indented square (figs. 225 and 226). A series of these punch marks is given for demonstration on pl. 9. A favorite design
- 56. was a square punch with a cross of two arms passing through the center, dividing the field into four quarters. Most of the punch marks on the coins of that period were of this kind. These punch marks and the method and machinery with which they were made are described in standard numismatic works.[235] Fig. 229. CORINTHIAN COINS. Obverse and reverse. Punch mark resembling Swastika. It is believed by the author that the assertions as to the presence of the Swastika on these ancient coins is based upon an erroneous interpretation of these punch marks. Fig. 229 shows the obverse and reverse of a coin from Corinth. It belonged to the first half of the sixth century B. C. The obverse represents a Pegasus standing, while the reverse is a punch mark, said to have been a Swastika; but, examining closely, we will find there is no Swastika in this punch mark. The arms of the normal Swastika consist of straight lines crossing each other. In this case they do not cross. The design consists of four gammas, and each gamma is separated from its fellows, all forming together very nearly the same design as hundreds of other punch marks of the same period. If each outer arm of this mark is made slightly longer, the Swastika form disappears and the entire design resolves itself into the square habitually employed for that purpose. If the punch mark on this Corinthian coin be a Swastika, it depends upon the failure to make the extreme end of the bent arm an eighth of an inch longer. This is
- 57. too fine a point to be relied upon. If this punch mark had these arms lengthened an eighth of an inch, it would confessedly become a square. EXPLANATION OF PLATE 9. 1 2 3 4 5 6 7 8 9 10 11 12 Punch Marks on Reverse of Ancient Coins. Fig. 1. Coin of Lydia. Electrum. Oblong sinking between two squares. Babylonian stater. The earliest known coinage. Circa 700 B. C. 2. Phenician Half Stater. Electrum. Incuse square with cruciform ornament. 3. Silver Coin of Teos. Incuse square. Circa 544 B. C. 4. Silver Coin of Acanthus. Incuse square. 5. Silver Coin of Mende. Incuse triangles. 6. Silver Coin of Terone. Incuse square. 7. Coin of Bisaltæ.[236] Incuse square. Octadrachm. 8. Silver Coin of Orrescii.[236] Incuse square. Octadrachm. 9. Corinthian Silver Coin. Incuse square divided into eight triangular compartments. The earliest coin of Corinth, dating B. C. 625 to 585. 10. Silver Coin of Abdera. Incuse square. 11. Silver Coin of Byzantium. Incuse square, granulated. 12. Silver Coin of Thrasos (Thrace). Incuse square.
- 58. Plate 9. Punch Marks on Reverse of Ancient Coins. Swastika on ancient Hindu coins.—It is not to be inferred from this opposition that the Swastika never appeared on ancient coins. It did appear, but seems to have been of a later date and to have belonged farther east among the Hindus. Fig. 230 shows an ancient (Hindu?) coin reported by Waring, who cites Cunningham as authority for its having been found at Ujain. The design consists of a cross with independent circles on the outer end of each of the four arms, the circles being large enough to intersect each other. The
- 59. Fig. 230. ANCIENT HINDU COIN IN THE FORM OF A CROSS WITH A SWASTIKA ON THE EXTREMITY OF EACH ARM.[237] Waring, “Ceramic Art in Remote Ages,” pl. 41, fig. 13. field of each of these circles bears a Swastika of normal form. Other coins are cited of the same style, with small center dots and concentric circles in the stead of the Swastika. What meaning the Swastika has here, beyond the possible one of being a lucky penny, is not suggested. Other ancient Hindu coins bearing the Swastika (figs. 231-234) are attributed to Cunningham by Waring.[238] These are said by Waring to be Buddhist coins found at Behat near Scharaupur. Mr. E. Thomas, in his article on the “Earliest Indian Coinage,”[239] ascribes them to the reign of Krananda, a Buddhist Indian king contemporary with or prior to Alexander, about 330 B. C. The coins of Krananda,[240] contemporary of Alexander the Great, [241] bear the Swastika mark, associated with the principal Buddhist marks, the trisula, the stupha, sacred tree, sacred cone, etc. Waring says[242] that according to Prinsep’s “Engravings of Hindu Coins,” the Swastika seems to disappear from them about 200 B. C., nor is it found on the Indo-Bactrian, the Indo-Sassanian, or the later Hindu or subsequent Mohammedan, and he gives in a note the approximate dates of these dynasties: Early native Buddhist monarchs from about 500 B. C. to the conquest of Alexander, about 330 B. C.; the Indo-Bactrian or Greek successors of Alexander from about 300 to 126 B. C.; the Indo-Parthian or Scythic from about 126 B. C.; the second Hindu dynasty from about 56 B. C.; the Indo- Sassanian from A. D. 200 to 636, and subsequent to that the Indo- Mohammedan from the eleventh to the close of the thirteenth century; the Afghan dynasty from A. D. 1290 to 1526, and the
- 60. Fig. 235. ANCIENT COIN WITH SWASTIKA. Gaza, Palestine. Waring, “Ceramic Art in Remote Ages,” pl. 42, fig. 6. Mongol dynasty to the eighteenth century, when it was destroyed by Nadir Shah. (See p. 772.) Figs. 231, 232, 233, and 234. ANCIENT HINDU COINS WITH SWASTIKAS, NORMAL AND OGEE. Waring, “Ceramic Art in Remote Ages,” pl. 41, figs. 20-24. Swastika on coins in Mesembria and Gaza. —Mr. Percy Gardner, in his article, “Ares as a Sun-god,”[243] finds the Swastika on a coin of Mesembria in Thrace. He explains that “Mesembria is simply the Greek word for noon, midday (μεσημβρία).” The coins of this city bear the inscription ΜΕΣ , which Greg[244] believes refers by a kind of pun to the name of the city, and so to noon, or the sun or solar light. The answer to this is the same given throughout this paper, that it may be true, but there is no evidence in support of it. Max Müller[245] argues that this specimen is decisive of the meaning of the sign Swastika. Both these gentlemen place great stress upon the position which the Swastika held in the field relative to other objects, and so determine it to have represented the sun or sunlight; but all this seems non sequitur. A coin from Gaza, Palestine, ancient, but date not given, is attributed to R. Rochette, and by him to
- 61. Fig. 236. GOLD BRACTEATE WITH JAIN SWASTIKA. Denmark. Waring, “Ceramic Art in Remote Ages,” pl. 1, fig. 9. Munter (fig. 235). The Swastika sign is not perfect, only two arms of the cross being turned, and not all four. Swastika on Danish gold bracteates.—Fig. 236 represents a Danish gold bracteate with a portrait head, two serpents, and a Swastika with the outer ends finished with a curve or flourish similar to that of the Jains (fig. 33). There are other bracteates with the Swastika mark, which belong to the Scandinavian countries.[246] Some of them bear signs referring to Christian civilization, such as raising hands in prayer; and from a determination of the dates afforded by the coins and other objects the Swastika can be identified as having continued into the Christian era. The coinage of the ancient world is not a prolific field for the discovery of the Swastika. Other specimens may possibly be found than those here given. This search is not intended to be exhaustive. Their negative information is, however, valuable. It shows, first, that some of the early stamps or designs on coins which have been claimed as Swastikas were naught but the usual punch marks; second, it shows a limited use of the Swastika on the coinage and that it came to an end in very early times. Numismatics afford great aid to archæology from the facility and certainty with which it fixes dates. Using the dates furnished by the coinage of antiquity, it is gravely to be questioned whether the prolific use of the Swastika in Asia Minor (of which we have such notable examples on specimens of pottery from the hill of Hissarlik, in Greece) did not terminate before coinage began, or before 480 B. C., when the period of finer engraving began, and it became the custom to employ on coins the figures of gods, of tutelary deities, and of sacred animals. Thus the use of the Swastika became relegated to objects of commoner use, or those having greater relation to superstition and folklore wherein
- 62. the possible value of the Swastika as an amulet or sign with power to bring good luck could be better employed; or, as suggested by Mr. Greg, that the great gods which, according to him, had the Swastika for a symbol, fell into disrepute and it became changed to represent something else. UNITED STATES OF AMERICA. PRE-COLUMBIAN TIMES. Fains Island and Toco Mounds, Tennessee.—That the Swastika found its way to the Western Hemisphere in prehistoric times can not be doubted. A specimen (fig. 237) was taken by Dr. Edward Palmer in the year 1881 from an ancient mound opened by him on Fains Island, 3 miles from Bainbridge, Jefferson County, Tenn. It is figured and described in the Third Annual Report of the Bureau of Ethnology,[247] as follows: A shell ornament, on the convex surface of which a very curious ornamental design has been engraved. The design, inclosed by a circle, represents a cross such as would be formed by two rectangular tablets or slips slit longitudinally and interlaced at right angles to each other. The lines are neatly and deeply incised. The edge of the ornament has been broken away nearly all around. The incised lines of this design (fig. 237) represent the Swastika turned to the left (though the description does not recognize it as such). It has small circles with dots in the center, a style of work that may become of peculiar value on further investigation, but not to be confounded with the dots or points in what M. Zmigrodzki calls the Croix swasticale. The mound from which this specimen came, and the objects associated with it, show its antiquity and its manufacture by the aborigines untainted by contact with the whites. The mound
- 63. is on the east end of Fains Island. It was 10 feet in height and about 100 feet in circumference at the base. In the bed of clay 4 feet beneath the surface were found the remains of 32 human skeletons; of these, only 17 skulls could be preserved. There had been no regularity in placing the bodies. Fig. 237. SHELL GORGET WITH ENGRAVED SWASTIKA, CIRCLES, AND DOTS. Fains Island, Tennessee. Cat. No. 62928, U. S. N. M.
- 64. Fig. 238. ENGRAVED SHELL WITH SWASTIKA, CIRCLES, AND DOTS. Toco Mound, Monroe County, Tenn. Cat. No. 115624, U. S. N. M. The peculiar form of this Swastika is duplicated by a Runic Swastika in Sweden, cited by Ludwig Müller and by Count d’Alviella.[248] The following objects were found in the mound on Fains Island associated with the Swastika shell (fig. 237) and described, and many of them figured:[249] A gorget of the same Fulgur shell (fig. 239); a second gorget of Fulgur shell with an engraved spider (fig. 278); a pottery vase with a figure of a frog; three rude axes from four to seven inches in length, of diorite and quartzite; a pierced tablet of slate; a disk of translucent quartz 1¾ inches in diameter and three-quarters of an inch in thickness; a mass of pottery, much of it in fragments, and a number of bone implements, including needles and paddle-shaped objects. The shell objects (in addition to the disks and gorgets mentioned) were pins made from the columellæ of Fulgur (Busycon perversum?) of the usual form and about four inches in length. There were also found shell beads, cylindrical in form, an inch in length and upward of an inch in diameter, with other beads of various sizes and shapes made from marine shells, and natural specimens of Io spinosa, Unio probatus.
- 65. Plate 10. Engraved Fulgur(?) Shell, Resembling Statue of Buddha. Toco Mound, Tennessee. Cat. No. 115560, U. S. N. M. The specimen represented in fig. 238 is a small shell from the Big Toco mound, Monroe County, Tenn., found by Mr. Emmert with skeleton No. 49 and is fig. 262, Twelfth Annual Report of the Bureau of Ethnology, 1890-91, page 383, although it is not described. This is a circular disk of Fulgur shell, much damaged around the edge, 1½ inches in diameter, on which has been engraved a Swastika. It has a small circle and a dot in the center, around which circle the arms of the Swastika are interlaced. There are also circles and central dots at
- 66. each turn of the four arms. The hatch work in the arc identifies this work with that of other crosses and a triskelion from the same general locality—figs. 302, 305, and 306, the former being part of the same find by Mr. Emmert. Fig. 222, a bronze gilt fibula from Berkshire, England, bears a Swastika of the same style as fig. 238 from Tennessee. The circles and central dots of fig. 238 have a similarity to Peruvian ornamentation. The form and style, the broad arms, the circles and central dots, the lines of engravings, show such similarity of form and work as mark this specimen as a congener of the Swastika from Fains Island (fig. 237). The other objects found in the mound associated with this Swastika will be described farther on. There can be no doubt of these figures being the genuine Swastika, and that they were of aboriginal workmanship. Their discovery immediately suggests investigation as to evidences of communication with the Eastern Hemisphere, and naturally the first question would be, Are there any evidences of Buddhism in the Western Hemisphere? When I found, a few days ago, the two before-described representations of Swastikas, it was my belief that no reliable trace of Buddha or the Buddhist religion had ever been found among the aboriginal or prehistoric Americans. This statement was made, as almost all other statements concerning prehistoric man should be, with reserve, and subject to future discoveries, but without idea that a discovery of evidence on the subject was so near. In searching the U. S. National Museum for the objects described in the Second Annual Report of the Bureau of Ethnology under the title of “Art in Shell among the Ancient Americans,” the writer discovered a neglected specimen of a mutilated and damaged shell (pl. 10), marked as shown on the back, found by Mr. Emmert, an employé of the Bureau of Ethnology, in the year 1882. Its original field number was 267, Professor Thomas’s 6542, the Museum number 115562, and it was found in the Big Toco mound, Monroe County, Tenn. It is not figured nor mentioned in any of the Bureau reports. It is greatly to be regretted that this shell is so mutilated. In its present condition no one can say positively what it is, whether a statue of Buddha or
- 67. not; but to all appearances it represents one of the Buddhist divinities. Its material, similar to the hundred others found in the neighborhood, shows it to have been indigenous, yet parts of its style are different from other aboriginal North American images. Attention is called to the slim waist, the winged arms, the crossed legs, the long feet, breadth of toes, the many dots and circles shown over the body, with triple lines of garters or anklets. All these show a different dress from the ancient North American. The girdle about the waist, and the triangular dress which, with its decorations and arrangement of dots and circles, cover the lower part of the body, are to be remarked. While there are several specimens of aboriginal art from this part of the country which bear these peculiarities of costumes, positions, appearance, and manner of work, showing them to have been in use among a portion of the people, yet they are not part of the usual art products. There is a manifest difference between this and the ordinary statue of the Indian or of the mound builder of that neighborhood or epoch. It is not claimed that this shell proves the migration of Buddhism from Asia, nor its presence among North American Indians. “One swallow does not make a summer.” But this figure, taken in connection with the Swastika, presents a set of circumstances corresponding with that possibility which goes a long distance in forming circumstantial evidence in its favor. M. Gustave d’Eichthal wrote a series of essays in the Revue Archæologique, 1864-65, in which he collated the evidence and favored the theory of Buddhist influence in ancient America. Other writers have taken the same or similar views and have attributed all manner of foreign influence, like the Lost Tribes of Israel, etc., to the North American Indian,[250] but all these theories have properly had but slight influence in turning public opinion in their direction. Mr. V. R. Gandhi, in a recent letter to the author, says of this specimen (pl. 10): While Swastika technically means the cross with the arms bent to the right, later on it came to signify anything
- 68. which had the form of a cross; for instance, the posture in which a persons sits with his legs crossed is called the Swastika posture;[251] also when a person keeps his arms crosswise over his chest, or a woman covers her breast with her arms crossed, that particular attitude, is called the Swastika attitude, which has no connection, however, with the symbolic meaning of the Swastika with four arms. The figure [pl. 10], a photograph of which you gave me the other day, has the same Swastika posture. In matters of concentration and meditation, Swastika posture is oftentimes prescribed, which is also called Sukhasana, meaning a posture of ease and comfort. In higher forms of concentration, the posture is changed from Sukhasana to Padmasana, the posture which is generally found in Jain and Buddhist images. The band around the waist, which goes from the navel lower on till it reaches the back part, has a peculiar significance in the Jain philosophy. The Shvetamber division of the Jain community have always this kind of band in their images. The object is twofold: The first is that the generative parts ought not to be visible; the second is that this band is considered a symbol of perfect chastity. There can be no doubt of the authenticity of these objects, nor any suspicion against their having been found as stated in the labels attached. They are in the Museum collection, as are other specimens. They come unheralded and with their peculiar character unknown. They were obtained by excavations made by a competent and reliable investigator who had been engaged in mound exploration, a regular employé of the Bureau of Ethnology, under the direction of Prof. Cyrus Thomas during several years, and always of good reputation and unblemished integrity. They come with other objects, labeled in the same way and forming one of a series of numbers among thousands. Its resemblance to Buddhist statues was apparently undiscovered or unrecognized, at least unmentioned, by all those having charge of it, and in its mutilated condition it was laid
- 69. away among a score of other specimens of insufficient value to justify notice or publication, and is now brought to light through accident, no one having charge of it recognizing it as being different from any other of the half hundred engraved shells theretofore described. The excavation of Toco mound is described by Professor Thomas in the Twelfth Annual Report of the Bureau of Ethnology, pages 379-384. We can now be governed only by the record as to the objects associated with this shell (pl. 10), which shows it to have been found with skeleton No. 8, in Big Toco mound, Monroe County, Tenn., while the Swastika of figure 238 was found with skeleton No. 49. Toco mound contained fifty-two skeletons, or, rather, it contained buried objects reported as from that many skeletons. Those reported as with skeleton No. 8 were, in addition to this gorget: One polished stone hatchet, one stone pipe, and one bowl with scalloped rim. Toco mound seems to have been exceedingly rich, having furnished 198 objects of considerable importance. Association of discovered objects is one of the important means of furnishing evidence in prehistoric archæology. It is deemed of sufficient importance in the present case to note objects from Toco mound associated with the Buddha statue. They are given in list form, segregated by skeletons: Skeleton No. 4. Two polished stone hatchets, one discoidal stone. 5. One polished stone hatchet. 7. Two large seashells. 8. One stone pipe, one polished stone hatchet, one ornamented shell gorget (the Buddha statue, pl. 10), one ornamented bowl, with scalloped rim. 9. Two polished stone hatchets. 12. A lot of small shell beads.
- 70. 13. Four bone implements (one ornamented), one stone pipe, two shell gorgets (one ornamented), one bear tooth. 17. One polished stone hatchet. 18. Two polished stone hatchets, one stone pipe, one boat-shaped bowl (ornamented), one shell gorget (ornamented), one shell mask, one shell pin, one shell gorget, one bear tooth, lot of shell beads. 22. Two polished stone chisels, one stone disk. 24. One polished stone hatchet. 26. Two polished stone hatchets, one waterworn stone, two hammer stones. 27. One polished stone hatchet. 28. Two polished stone hatchets, one ornamented bowl. 31. One polished stone hatchet, one polished stone chisel. 33. Two polished stone hatchets, one two-eared pot, one small shell gorget, three shell pins, fragments of pottery. 34. Three polished stone hatchets. 36. One discoidal stone. 37. One polished stone chisel, one stone pipe, one shell mask (ornamented). 41. One polished stone hatchet, one stone pipe, pottery vase with ears (ornamented), one shell mask, one shell pin, four arrowheads (two with serrated edges), two stone perforators. 43. Lot of shell beads. 49. One polished stone hatchet, one spade-shaped stone ornament (perforated), one spear-head, one stone pipe, one pottery bowl
- 71. with two handles, two shell masks (ornamented), twenty-seven bone needles, two beaver teeth, one bone implement (raccoon), piece of mica, lot of red paint, two shell gorgets (one ornamented with Swastika, fig. 238), thirty-six arrow-heads, lot of flint chips, fragment of animal jaw and bones, lot of large shells, one image pot. 51. One shell pin, one shell mask, one arrow-head, two small shell beads. 52. One shell mask, one shell gorget, one shell ornament. These objects are now in the U. S. National Museum and in my department. The list is taken from the official catalogue, and they number from 115505 to 115684. I have had the opportunity of comparing the objects with this description and find their general agreement. Dr. Palmer, the finder, was an employé of the Bureau of Ethnology, is a man of the highest character, of great zeal as an archæologist and naturalist, and has been for many years, and is now, in the employ of the Bureau or Museum, always with satisfaction and confidence. Mr. Emmert was also an employé of the Bureau for many years, and equally reliable. The specimens of shell in this and several other mounds, some of which are herein figured, were in an advanced stage of decay, pitted, discolored, and crumbling, requiring to be handled with the utmost care to prevent disintegration. They were dried by the collector, immersed in a weak solution of glue, and forwarded immediately (in 1885), with other relics from the neighborhood, to the Bureau of Ethnology and National Museum at Washington, where they have remained ever since. There is not the slightest suspicion concerning the genuineness or antiquity of this specimen or of those bearing the Swastika as belonging to the mound-building epoch in the valley of the Tennessee.
- 72. Fig. 239. SHELL GORGET. Two fighting figures with triangular breech-clout, garters and anklets, and dots and circles. Fains Island, Tennessee. Third Annual Report of the Bureau of Ethnology, p. 452, fig. 128. Cat. No. 62930, U. S. N. M. Other figures of sufficient similarity to the Swastika have been found among the aborigines of North America to show that these do not stand alone; and there are also other human figures which show a style of work so similar and such resemblance in detail of design as to establish the practical identity of their art. One of these was a remarkable specimen of engraved shell found in the same mound, Fains Island, which contained the first Swastika (fig. 237). It is described in the Second Annual Report of the Bureau of Ethnology, page 301, under the name of McMahon’s mound. It is a large polished Fulgur shell disk which, when entire, has been nearly 5 inches in diameter (fig. 239). A little more than one-third has crumbled away, and the remaining portion has been preserved only by careful handling and immediate immersion in a solution of glue. It
- 73. had been engraved on the concave side. The design represents two human figures plumed and winged, armed with eagles’ talons and engaged in mortal combat. The design apparently covered the entire shell, leaving no space for encircling lines. The two figures are in profile and face each other in a fierce onset. Of the right-hand figure, only the body, one arm, and one leg remain. The left-hand figure is almost complete. The outline of the face, one arm, and one foot is all that is affected. The right hand is raised above the head in the act of brandishing a long knife pointed at both ends. The other combatant, clutching in his right hand a savage-looking blade with its point curved, seems delivering a blow in the face of his antagonist. Of the visible portions of the figures, the hands are vigorously drawn, the thumbs press down upon the outside of the forefingers in a natural effort to tighten the grasp. The body, arms, and legs are well defined and in proper proportion, the joints are correctly placed, the left knee is bent forward, and the foot planted firmly on the ground, while the right is thrown gracefully back against the rim at the left, and the legs terminate in well-drawn eagles’ feet armed with curved talons. The head is decorated with a single plume which springs from a circular ornament placed over the ear; an angular figure extends forward from the base of this plume, and probably represents what is left of the headdress proper. In front of this—on the very edge of the crumbling shell—is one-half of the lozenge-shaped eye, the dot representing the pupil being almost obliterated. The ankles and legs just below the knee and the wrists each have three lines representing bracelets or anklets. It is uncertain whether the leg is covered or naked; but between the waistband and the leggings, over the abdomen, is represented on both figures a highly decorated triangular garment, or, possibly coat of mail, to which particular attention is called.[252] In the center, at the top, just under the waistband, are four circles with dots in the center arranged in a square; outside of this, still at the top, are two triangular pieces, and outside of them are two more circles and dots; while the lower part of the triangle, with certain decorations of incised lines, completes the garment. This decoration is the same on both figures, and corresponds exactly with the Buddha figure. An
- 74. ornament is suspended on the breast which shows three more of the circles and dots. The earring is still another. The right-hand figure, so far as it can be seen, is a duplicate of the left, and in the drawing it has, where destroyed, been indicated by dotted lines. It is remarkable that the peculiar clothing or decoration of these two figures should be almost an exact reproduction of the Buddha figure (pl. 10). Another interesting feature of the design is the highly conventionalized wing which fills the space beneath the uplifted arm. This wing is unlike the usual specimens of aboriginal art which have been found in such profusion in that neighborhood. But it is again remarkable that this conventionalized wing and the bracelets, anklets, and garters should correspond in all their peculiarities of construction and design with the wings on the copper and shell figures from the Etowah mound, Georgia (figs. 240, 241, and 242) [253]. Behind the left-hand figure is an ornament resembling the spreading tail of an eagle which, with its feather arrangement and the detail of their mechanism, correspond to a high degree with the eagle effigies in repoussé copper (fig. 243) from the mound in Union County, Ill., shown in the Fifth Annual Report of the Bureau of Ethnology (p. 105) and in the Twelfth Annual Report (p. 309).
- 75. Fig. 240. COPPER PLATE. Entowah Mound, Georgia. Fifth Annual Report of the Bureau of Ethnology, fig. 42. Cat. No. 91113, U. S. N. M.
- 76. Fig. 241. COPPER PLATE. Repoussé work. Entowah Mound, Georgia. Cat. No. 91117, U. S. N. M. Hopewell Mound, Chillicothe, Ross County, Ohio.—A later discovery of the Swastika belonging to the same period and the same general locality—that is, to the Ohio Valley—was that of Prof. Warren K. Moorehead, in the fall and winter of 1891-92, in his excavations of the Hopewell mound, seven miles northwest of Chillicothe, Ross County, Ohio.[254] The locality of this mound is well shown in Squier and Davis’s work on the “Monuments of the Mississippi Valley” (pl. 10, p. 26), under the name of “Clark’s Works,” here reproduced as
- 77. Fig. 242. ENGRAVED SHELL. Triangular breech-clout with dots and circles. Entowah Mound, Georgia. Cat. No. 91443, U. S. N. M. pl. 11. It is the large irregular unnumbered triple mound just within the arc of the circle shown in the center of the plan. The excavation contemplated the destruction of the mound by cutting it down to the surrounding level and scattering the earth of which it was made over the surface; and this was done. Preparatory to this, a survey and ground plan was made (pl. 12). I assisted at this survey and can vouch for the general correctness. The mound was surrounded by parallel lines laid out at right angles and marked by stakes 50 feet apart. The mound was found to be 530 feet long and 250 feet wide. Squier and Davis reported its height at 32 feet, but the excavation of the trenches required but 18 and 16 feet to the original surface on which the mound was built. It was too large to be cut down as a whole, and for convenience it was decided by Mr. Moorehead to cut it down in trenches, commencing on the northeast. Nothing was found until, in opening trench 3, about five feet above the base of the mound, they struck a mass of thin worked copper objects, laid flat one atop the other, in a rectangular space, say three by four feet square. These objects are unique in American prehistoric archæology. Some of them bore a resemblance in form to the scalloped mica pieces found by Squier and Davis, and described by them in their “Ancient Monuments of the Mississippi Valley” (p. 240), and also those of the same material found by Professor Putnam in the Turner group of
- 78. mounds in the valley of the Little Miami. They had been apparently laid between two layers of bark, whether for preservation or mere convenience of deposit, can only be guessed. Larger Image Plate 11. Plan of North Fork (Hopewell) Works. Ross County, Ohio. Smithsonian Contributions to Knowledge, Vol. I, Pl. x.
- 79. Larger Image Plate 12. Plan of Hopewell Mound, in which Aboriginal Copper Swastikas were Found. Ross County, Ohio. Moorehead, “Primitive Man in Ohio,” Pl. xxxiv. The following list of objects is given, to the end that the reader may see what was associated with these newly found copper Swastikas: Five Swastika crosses (fig. 244); a long mass of copper covered with wood on one side and with squares and five similar designs traceable on the reverse; smaller mass of copper; eighteen single copper rings; a number of double copper rings, one set of three and one set of two; five pan lids or hat-shaped rings; ten circular disks with holes in center, represented in fig. 245, originally placed in a
- 80. Welcome to our website – the perfect destination for book lovers and knowledge seekers. We believe that every book holds a new world, offering opportunities for learning, discovery, and personal growth. That’s why we are dedicated to bringing you a diverse collection of books, ranging from classic literature and specialized publications to self-development guides and children's books. More than just a book-buying platform, we strive to be a bridge connecting you with timeless cultural and intellectual values. With an elegant, user-friendly interface and a smart search system, you can quickly find the books that best suit your interests. Additionally, our special promotions and home delivery services help you save time and fully enjoy the joy of reading. Join us on a journey of knowledge exploration, passion nurturing, and personal growth every day! ebookbell.com