![978-1-492-05701-7
[LSI]
Learning Serverless
by Jason Katzer
Copyright © 2021 Versa Labs LLC. All rights reserved.
Printed in the United States of America.
Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472.
O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are
also available for most titles (http://oreilly.com). For more information, contact our corporate/institutional
sales department: 800-998-9938 or corporate@oreilly.com.
Acquisitions Editor: Jennifer Pollock
Development Editor: Sarah Grey
Tech Editor: Right Touch Editing
Production Editor: Deborah Baker
Copyeditor: Sonia Saruba
Proofreader: Piper Editorial, LLC
Indexer: nSight, Inc.
Interior Designer: David Futato
Cover Designer: Karen Montgomery
Illustrator: Kate Dullea
November 2020: First Edition
Revision History for the First Edition
2020-10-28: First Edition
See http://oreilly.com/catalog/errata.csp?isbn=9781492057017 for release details.
The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Learning Serverless, the cover image,
and related trade dress are trademarks of O’Reilly Media, Inc.
The views expressed in this work are those of the author, and do not represent the publisher’s views.
While the publisher and the author have used good faith efforts to ensure that the information and
instructions contained in this work are accurate, the publisher and the author disclaim all responsibility
for errors or omissions, including without limitation responsibility for damages resulting from the use of
or reliance on this work. Use of the information and instructions contained in this work is at your own
risk. If any code samples or other technology this work contains or describes is subject to open source
licenses or the intellectual property rights of others, it is your responsibility to ensure that your use
thereof complies with such licenses and/or rights.](https://image.slidesharecdn.com/97247-250510175035-d64fe2a2/85/Learning-Serverless-Design-Develop-and-Deploy-with-Confidence-1st-Edition-Jason-Katzer-9-320.jpg)
![underwood as on the northern bank of the Oxus. They, however, occurred in the
same succession and formation as have been there described: they were quite
soft, but the sand was not dusty, and the camels slid down them with their
burthens. Here and there we came upon a sheet of indurated clay, as if the sand-
hills here also rested on a base of that kind. In these hollows, and on the brow of
the hills, we found a shrub like tamarisk, called “kasura,” also a kind of grass, or
bent, called “salun.” There were likewise two thorny shrubs, called “kuzzak” and
“karaghan”[1], neither of them the common camel thorn, but on which the camels
delighted to browse. There was no water throughout the whole march, and no
signs of inhabitants but a ruined fort, that had once served as a look-out from the
Oxus. The Indian deserts of Jaysulmeer and Parkur sank into insignificance before
this vast ocean of sand. No sight is more imposing than a desert; and the eye
rests with a deep interest on the long line of camels, as it winds its crooked course
through the frightful waste. The simile of a ship in the ocean and a camel in the
desert may be hackneyed, but it is just. The objects animate impart a strong
interest to inanimate nature.
In the middle of our march through the desert, we met seven unfortunate
Persians, who had been captured by the Toorkmuns, and were now on their road
to Bokhara, where they would be sold. Five of them were chained together, and
trod their way through the deep sand. There was a general shout of compassion,
as the caravan passed these miserable beings; and the sympathy did not fail to
affect the poor creatures themselves. They cried, and gave a longing look, as the
last camel of the caravan passed to their dear native country. The camel on which
I rode happened to be in the rear, and I stayed to hear their tale of woe. They had
been seized by the Toorkmuns at Ghaeen, near Meshid, a few weeks before, when
the culture of their fields had led them beyond the threshold of their homes. They
were weary and thirsty, and I gave them all I could,—a single melon; a civility,
little as it was, which was received with gratitude. What a frightful notion must not
these unfortunate beings have entertained of the country they were entering, after
their travels in such a desert. The Toorkmuns evince but little compassion for their
Persian slaves; and what other treatment is to be expected from men who pass
their lives in selling human beings. They give them but a scanty supply of food and
water, that they may waste their strength, and prevent their escape; but beyond
this the Toorkmun inflicts no other ills. The tales which have been circulated of
their cutting the sinew of the heel, and of their passing a cord round the collar
bone, are at variance with truth, since these blemishes would diminish the value of
the slave. These unfortunate captives suffer a much heavier calamity,—they lose
their liberty.
As we reached our halting-ground in the morning, we had now an opportunity of
observing the number and composition of the caravan. There were upwards of
eighty camels, and about 150 persons, several of them men of the first
respectability, who accompanied their merchandise to the markets of Persia. Some](https://image.slidesharecdn.com/97247-250510175035-d64fe2a2/85/Learning-Serverless-Design-Develop-and-Deploy-with-Confidence-1st-Edition-Jason-Katzer-50-320.jpg)
![CHAP. XIII.
CONTINUATION OF THE JOURNEY IN THE TOORKMUN
DESERT.
On the morning of the 29th of August we moved at dawn, with buoyant spirits,
and followed the course of the Moorghab, or river of Merve, for twelve miles
before we could cross it. We found it about eighty yards wide and five feet deep,
running within steep clayey banks, at the rate of five miles an hour. We crossed by
an indifferent ford, over a clay bottom with many holes. There was no village; but
the place is called Uleesha. This river rises on the mountains of Huzara, and was
long believed to fall into the Oxus or the Caspian. Both opinions are erroneous,
since it forms a lake, or loses itself in one, about fifty miles N.W. of Merve. This
river was formerly dammed above Merve, which turned the principal part of its
waters to that neighbourhood, and raised that city to the state of richness and
opulence which it once enjoyed. The dam was thrown down about forty-five years
ago, by Shah Moorad, a king of Bokhara, and the river only now irrigates the
country in its immediate vicinity, where it is covered with the tenements, or “obas,”
of the Toorkmuns; for there are no fixed villages. These people cultivate by
irrigation, and every thing grows in rich luxuriance. The Juwaree (holcus sorghum)
has a stalk thicker than a walking-stick, and in the uncultivated parts there is the
richest fodder for cattle and the finest thorny shrubs for the camel, an animal
which is here found in vast herds. Above Merve the country is called Maroochak,
and said to be unhealthy: there is a proverb, at least, which runs thus,—“Before
God gets intelligence, the water of Maroochak has killed the man.”[2] This river is
the Epardus of Arrian, a word which, I observe in one author, is said to mean
irrigator,—nor is it here misapplied. The historian would even appear to have been
acquainted with its course; for we are told that the Epardus “hides its streams in
the sand, as did many other great rivers.”[3]
The transition which we had experienced, from a sandy desert to the verge of a
running stream, was most gratifying; every one seemed delighted, and even the
animals appeared to feel the change. Throughout the day the banks presented a
spectacle of merriment and joy; the Toorkmuns plunging into the water with their
horses, and the greater part of the caravan sporting about in the stream. We hit
upon a contrivance, which contributed not a little to our sport, and produced a
“tunga,” or the third part of a rupee, which was to be the reward of the person
who could first cross the river. The enormous sum was solemnly vested in a
committee; I believe even the blessing was said; and sixteen competitors
appeared on the occasion. It was won by a Toorkmun of Shurukhs, who had the
art of running quickest in the deep water.](https://image.slidesharecdn.com/97247-250510175035-d64fe2a2/85/Learning-Serverless-Design-Develop-and-Deploy-with-Confidence-1st-Edition-Jason-Katzer-60-320.jpg)
![wealth and size of the caravan with which we were travelling. The Orgunje officer,
too, it was now stated, had been bribed to the amount of ten tillas. It appeared
both reasonable and just that we should bear our share of this outlay, and I
therefore offered the usual tax of a tilla on each of our camels, since it would tend
to diminish the general expenditure of the caravan. It was a point that called for
the exercise of discretion and judgment, since a total denial might have converted
a friendly into a hostile party; and, on the other hand, it at all times behoved us to
be most sparing in our expenses. In the present instance, I had the good fortune
to conciliate by my concession the principal merchants of the party. There were
several who still called on us to pay a fourth share of the tax; but as I ascertained
that no additional expenses had been incurred on our account, and the duties
would have been levied whether we had been present or absent, I declined
compliance, and stated to them that we were travellers, and their guests, in a
foreign land, and hoped for their forbearance and justice. The Toorkmun chief, our
friend Ernuzzer, appeared at this stage of our conversation, to enter his protest
against such an outrage to hospitality as the demand which had been made upon
us; but I had already made up my mind, and passed my word. The rights of the
stranger are much respected among these people, and the cry of the many died
away into the feeble vociferations of the poorer traders, whose scanty means
made them feel more heavily the levy that had been made upon them. In one
respect the character of an European in such countries is ill suited for a traveller;
he is believed to possess boundless wealth, though he may be sunk in poverty; an
Asiatic, in his expenses, has nothing in common with the opinions of an European.
We now commenced our march in the desert westward of the Moorghab river, and
made a progress of thirty-seven miles. The tract was entirely different from the
opposite side, and about the middle of the journey the desert changed into a level,
hard, flat surface, which it ever afterwards preserved. The camels moved up in
four strings abreast of each other, and we continued to advance in that order. The
tract put me much in mind of the Run of Cutch, though there were patches of
bushes, which are not to be seen in that most singular region.[4] The country was
destitute of water, but there were many remains of caravansarais and cisterns that
had been built by the philanthropic Abdulla Khan of Bokhara. In this
neighbourhood, and more particularly while on the banks of the river, we
witnessed a constant succession of whirlwinds, that raised the dust to a great
height, and moved over the plain like water-spouts at sea. In India these
phenomena are familiarly known by the name of devils, where they sometimes
unroof a house; but I had not seen them in that country either of such size or
frequency as now prevailed in the Toorkmun desert. They appeared to rise from
gusts of wind, for the air itself was not disturbed but by the usual north wind that
blows steadily in this desert.
As we halted in the morning of the 1st of September, at a ruin which bore the
name of Kalournee, we descried the hills of Persian Khorasan. In the direction](https://image.slidesharecdn.com/97247-250510175035-d64fe2a2/85/Learning-Serverless-Design-Develop-and-Deploy-with-Confidence-1st-Edition-Jason-Katzer-64-320.jpg)
![the ground of their conversion to a true religion, and consequent salvation.
Unhappily for mankind, the history of the world presents us with too many and
similar instances of this mistaken and religious zeal. The Spaniards pursued their
conquests in the New World under the specious pretence of disseminating
Christianity; they sacked the empires of Mexico and Peru, and butchered their
inoffending inhabitants; while their priests impiously blessed their inhuman
outrages.[5] They, too, like the Toorkmuns, propitiated their king by a present of a
fifth of their spoil. Human nature, under king or khan, is the same in all countries,
whether we contemplate the frenzy and avarice of the Spaniards in America, or
the roaming Toorkmun in the Scythian deserts.
The opportunity which was afforded us of seeing these robbers, inspired a good
opinion of their courage, for many of them were indifferently armed. They all had
swords, most had light, long lances, quite different from those used by the
Uzbeks, and a few had small matchlocks. Their horses looked quite done up, and
walked as if on beds of gravel; but they had been thirteen days in motion, with
scanty food and much work. While we admire the courage of these men, what
shall we think of the Persians, who are encamped within two days’ journey of
Meshid, under the heir-apparent of their throne, and numbering an army of twenty
thousand men?
The return of the Orgunje Allamans should have now settled our movements, but
some timid being spread a rumour that half of the robbers yet lay in wait for our
caravan on the Persian frontier. Our departure was therefore still put off, and I
cannot say that I felt comfortable in such quarters. We had no tent or shelter for
ten days but the rotten walls of an old tomb, which were infested with reptiles.
Though our bed had always been the ground, and we had long ceased to feel the
aches which one experiences from an occasional bivouac in civilised life, we could
not now spread a carpet, lest we should appear too rich among the Toorkmuns,
who stated in upon us at all times, and frequently asked us questions. Our bread,
too, had been ten times coarser than “bannocks of barley meal,” not half so
palatable. We could with great difficulty read or write for a single hour during the
day, and the time passed as heavily as possible, exhausting our patience. During
our detention, one of the camels was said to have gone mad, whether from ennui
or some more cogent cause I knew not. The poor creature foamed at the mouth,
groaned, and refused its food. The case was referred to us, as he was pronounced
to be possessed of a devil; but of course without avail. At length they fell on the
expedient of frightening the camel, by clashing a lighted torch before his eyes and
body, and kindling reeds and furze under his nose. They also passed a red-hot iron
over his head; and the animal assuredly improved under this rough treatment, of
burning the devil who had lodged in so ugly a creature.
At length, on the 11th of September, after a detention of ten long days, we joyfully
quitted Shurukhs at sunrise. The Toorkmuns maintained their character to the last.](https://image.slidesharecdn.com/97247-250510175035-d64fe2a2/85/Learning-Serverless-Design-Develop-and-Deploy-with-Confidence-1st-Edition-Jason-Katzer-72-320.jpg)
Learning Serverless Design Develop and Deploy with Confidence 1st Edition Jason Katzer
- 1. Learning Serverless Design Develop and Deploy with Confidence 1st Edition Jason Katzer download https://ebookgate.com/product/learning-serverless-design-develop- and-deploy-with-confidence-1st-edition-jason-katzer/ Get Instant Ebook Downloads – Browse at https://ebookgate.com
- 2. Get Your Digital Files Instantly: PDF, ePub, MOBI and More Quick Digital Downloads: PDF, ePub, MOBI and Other Formats Building Serverless Web Applications Develop scalable web apps using the Serverless Framework on AWS 1st Edition Diego Zanon https://ebookgate.com/product/building-serverless-web- applications-develop-scalable-web-apps-using-the-serverless- framework-on-aws-1st-edition-diego-zanon/ Learning AWS design build and deploy responsive applications using AWS cloud components Sarkar https://ebookgate.com/product/learning-aws-design-build-and- deploy-responsive-applications-using-aws-cloud-components-sarkar/ Machine Learning with Spark Develop intelligent machine learning systems with Spark 2 x 2nd Edition Rajdeep Dua https://ebookgate.com/product/machine-learning-with-spark- develop-intelligent-machine-learning-systems-with-spark-2-x-2nd- edition-rajdeep-dua/ Oracle ADF Enterprise Application Development Made Simple 2nd Edition Successfully plan develop test and deploy enterprise applications with Oracle ADF Sten E. Vesterli https://ebookgate.com/product/oracle-adf-enterprise-application- development-made-simple-2nd-edition-successfully-plan-develop- test-and-deploy-enterprise-applications-with-oracle-adf-sten-e-
- 3. OpenCV with Python blueprints design and develop advanced computer vision projects using OpenCV with Python 1st Edition Michael Beyeler https://ebookgate.com/product/opencv-with-python-blueprints- design-and-develop-advanced-computer-vision-projects-using- opencv-with-python-1st-edition-michael-beyeler/ Calculate with Confidence Deborah Gray Morris https://ebookgate.com/product/calculate-with-confidence-deborah- gray-morris/ The Little League Guide to Tee Ball Helping Beginning Players Develop Coordination and Confidence 2nd Edition Ned Mcintosh https://ebookgate.com/product/the-little-league-guide-to-tee- ball-helping-beginning-players-develop-coordination-and- confidence-2nd-edition-ned-mcintosh/ Collaborative Design and Learning Joao Bento https://ebookgate.com/product/collaborative-design-and-learning- joao-bento/ Straight to Yes Asking with Confidence and Getting What You Want 1st Edition Haider Imam https://ebookgate.com/product/straight-to-yes-asking-with- confidence-and-getting-what-you-want-1st-edition-haider-imam/
- 4. Jason Katzer Learning Serverless Design, Develop, and Deploy with Confidence
- 6. Praise for LearningServerless “In this book, Jason covers every important aspect of serverless architecture. Simply put, if you’re working on or planning to work on serverless—this is the book for you!” —Erez Berkner, CEO at Lumigo “Serverless adoption is growing significantly. Jason Katzer’s book offers an important overview of what developers need to learn, and what challenges they’ll have to overcome, to become productive with serverless.” —Vadym Kazulkin, Head of Technology Strategy at ip.labs “Capital One leverages serverless architecture throughout our organization. Knowing when, when not, and how to manage serverless at scale is a must-have tool for any technology leader in the modern world.” —Jason Valentino, Senior Director and Head of Engineering, Capital One Shopping
- 8. Jason Katzer Learning Serverless Design, Develop, and Deploy with Confidence Boston Farnham Sebastopol Tokyo Beijing Boston Farnham Sebastopol Tokyo Beijing
- 9. 978-1-492-05701-7 [LSI] Learning Serverless by Jason Katzer Copyright © 2021 Versa Labs LLC. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://oreilly.com). For more information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com. Acquisitions Editor: Jennifer Pollock Development Editor: Sarah Grey Tech Editor: Right Touch Editing Production Editor: Deborah Baker Copyeditor: Sonia Saruba Proofreader: Piper Editorial, LLC Indexer: nSight, Inc. Interior Designer: David Futato Cover Designer: Karen Montgomery Illustrator: Kate Dullea November 2020: First Edition Revision History for the First Edition 2020-10-28: First Edition See http://oreilly.com/catalog/errata.csp?isbn=9781492057017 for release details. The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Learning Serverless, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc. The views expressed in this work are those of the author, and do not represent the publisher’s views. While the publisher and the author have used good faith efforts to ensure that the information and instructions contained in this work are accurate, the publisher and the author disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the information and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights.
- 10. Table of Contents Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv Introduction to Serverless. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xxi Part I. The Path to Production 1. Distributed Systems. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 What Is a Distributed System? 1 Why Do We Want a Distributed System? 3 The Harsh Realities of Distributed Systems 3 The Physical World 4 Missing Messages 4 Unreliable Clocks 5 Cascading Failures 6 Unexpected Ordering 7 Idempotency 7 What Am I Responsible For? 8 What Do You Need to Consider When Designing a Distributed System? 8 Loose Coupling (or Decoupling) 9 Fault Tolerance 9 Generating Unique (Primary) Keys 10 Planning for Idempotency 10 Two-Phase Changes 11 Further Reading 11 Conclusion 12 v
- 11. 2. Microservices. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 Why Do You Want to Use Microservices? 14 Improved Developer Velocity 14 Increased Developer Freedom 15 Issues with Microservices 16 Increased Complexity 16 Proper DevOps Practices and Resources Needed 17 Challenges with Local Development and Testing 17 How Do You Use Microservices Effectively? 18 Consistent Interfaces 18 Loosely Coupled 19 How Micro Is a Microservice? 20 Choosing Between Monoliths and Microservices 21 When Should You Use a Monolith? 21 When Do You Want to Use Microservices? 24 Conclusion 24 3. Serverless Architecture and Patterns. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 The Role of an Architect 26 What Do You Need to Know to Be an Architect? 27 Making Decisions 27 What Kinds of Decisions? 28 Documenting Your Decisions 28 How Do We Make Decisions? 29 When Do We Make Decisions? 30 Cloud Provider Components 31 Streams 31 Queues 31 Buckets 32 Compute 33 Datastores 33 Identity Service 34 API Gateways 34 GraphQL 35 Networking 35 State Machines 35 Logging 36 Monitoring and Alerting 36 Events from Your Cloud Provider 37 Periodic Invocations 37 Patterns 38 Example 1: Serverless Monolith 38 vi | Table of Contents
- 12. Example 2: Incoming Webhook 39 Example 3: Using Your Cloud Provider for User Authentication 40 Example 4: Generic Background Task Pattern 41 Example 5: Streaming Extract, Transform, Load 41 Example 6: Create Your Own Polling Integration 41 Example 7: Processing Files and Images 42 Example 8: Migration Service Pattern 43 Example 9: Fanning Out 43 Conclusion 44 4. Interfaces. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 Interfaces: Some Assembly Required 46 The Message 46 The Protocol 46 The Contract 46 Serverless Interfaces 47 Automatic Retries and Dead Letter Queues 47 Finite Versus Infinite Scale 48 Designing Your Interfaces 48 Messages/Payloads 49 Sessions and Users/Auth 50 Avoid Unbounded Requests 50 Interface Versus Implementation 51 Lines with Logic 52 Designing the Unhappy Path 52 Validating Input 53 Failures 53 Strategies for Integrating with Other Services 55 Time-Outs 55 Retries 56 Exponential Backoff 57 Webhooks 57 Evaluating External Services 58 Rate Limits 58 Conclusion 59 Part II. The Tools 5. The Serverless Framework. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63 Why Use the Serverless Framework? 64 When the Serverless Framework Isn’t for You 65 Table of Contents | vii
- 13. AWS Is the Only First-Class Citizen 66 AWS CloudFormation Is Not Perfect 66 Relying on Strangers for Your Infrastructure 66 What to Know Before You Start 67 YAML 68 Node.js 69 Cloud Resources and Permissions 69 Infrastructure Templates 71 Production Secrets 71 .gitignore 71 The Components of a serverless.yml File 72 Provider 72 Environment 74 Functions 74 Resources 75 Package 75 Plug-Ins 76 Custom 77 Namespacing for Sanity and Security 77 Using the serverless Command 78 Installing Serverless 78 Setting Up Serverless with Credentials 79 Pulling in Templates Using serverless install 79 Inspecting the Package of Our Sample Project (What’s Inside) 80 Deployment 80 Invoking the Function, and Viewing Logs 81 Rollbacks 81 Destroying the Service 81 Deployment Packages 82 Real-World serverless.yml 82 Setting Environment Variables 83 Modify Permissions 83 Conclusion 84 6. Monitoring, Observability, and Alerting. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85 What Is Monitoring? 85 Why Do We Need Monitoring? 86 How Does Monitoring Relate to Serverless? 86 The On-Ramp to Automation 87 What Are My Options? 87 Hosted SaaS Offerings 88 Self-Hosted and Open Source 88 viii | Table of Contents
- 14. Components of Monitoring 89 Metrics 89 Charts/Graphs 93 Dashboards 94 Alerts/Alarms 95 A Selection of Advanced Practices 96 Heartbeats 96 Smoke Testing and/or Canaries 97 The Most Important Metric in the World 98 Avoiding Vendor Lock-In 99 Cleaning Up Metrics and Alerts over Time 100 Conclusion 100 7. Logging. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101 What Does It Mean to Log? 102 Why Log? 103 When to Rely on Logs Instead of Metrics 104 What Should You Log? 104 What Shouldn’t You Log? 106 How Does Logging Work? 107 Ensuring Your Logs Scale 108 Structured Logging 109 More Effective Debugging with Logs 109 Searching Logs 110 Exception Logging (Sentry) 110 Collecting Other Logs 111 Compliance 111 Distributed Tracing 112 Encrypting Logs for Privacy and Compliance 112 Encrypt Only the Values of Sensitive Fields 113 Encrypt the Entire Log Statement 113 Conclusion 114 8. Changes, Automation, and Deployment Pipelines. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115 Dealing with Change 116 The Role of Automation 116 What Do We Automate? 117 Getting Your Code Ready for Production 118 Infrastructure as Code 119 Database Changes (Migrations) 120 Configuration Management 121 What Is a Pipeline? 122 Table of Contents | ix
- 15. Decisions to Make Regarding Your Pipeline 123 Canaries and Blue/Green Deployments 123 Pipeline Permissions 124 Why Do You Need a Pipeline? 125 Key Phases of a Deployment Pipeline 125 Step 1. Enforce Standards 126 Step 2. Build and Package 126 Step 3. Test 127 Step 4. Publish the Artifact 127 Step 5. Deploy to the Target Environment 128 Step 6. Validate Deployment 128 Step 7. Roll Back if Necessary (and Possible) 128 Handling Pipeline Failures 129 Conclusion 130 Part III. Concepts 9. Security, Permissions, and Privacy. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133 Everyone Is Responsible, but You Are Especially Responsible 134 Prepare to Be Hacked 135 Understanding Your Threats and Your Attackers 136 Design for Security 137 Limit, Track, and Review All Secrets and Access 139 Be Ready to Roll 140 Defense in Depth 141 Limit Blast Radius 142 Trust but Verify 142 Validate All User Input and Double-Check Those Settings 145 Monitoring Your System for Anomalies 146 Test Your Security 146 Select Dependencies Carefully and Keep Your Software Up to Date 147 Prioritize Privacy for Your Data and Your Customers’ Data 149 Don’t Mess with Production 149 Keep Your Machine Secure 151 Keep Learning 151 Conclusion 151 10. Quality, Testing, and Staging. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153 The Role of Code Quality 154 Code Style 155 Linting 156 x | Table of Contents
- 16. Testing 157 What to Test and What Not to Test 158 Types of Testing 158 Code Coverage 163 Power Up Your Testing 164 Staging 164 Conclusion 168 11. Planning for Failure. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169 Introduction: Understand It, Even if You Don’t Manage It 169 Identify Risks 170 Exercise: Finding Your Failure Points 171 Be Prepared 172 Making a Runbook 173 Planning for Outages 174 On-Call/Escalation Plan 175 Monitor Your Cloud Provider 175 Know Your (Service) Limits 176 Conclusion 176 12. Conclusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177 Deciding among Vendors 178 Community 179 Gather the Advice of Others 179 What to Do When You Get Stuck 180 Taking the Next Step in Your Career 180 Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183 Table of Contents | xi
- 18. This book is dedicated to my loving parents.
- 20. Preface This book will not make you an expert, but it will be an important step of your jour‐ ney—it will put you in the top half of developers. The purpose of this book is to help you understand what’s important, and what you need to learn and improve on in order to level up in your career or ship your next big personal project. About This Book This book is about arming you with the knowledge you need to represent serverless as an important new technology. There are plenty of doubters out there. And there are plenty of zealots. This book is not from any of those, or for any of those. This book is for people who want to write serious software and gain the respect of peers and colleagues by doing it predictably. I didn’t choose to defend serverless. I just chose to ship great software in a way that minimizes maintenance as much as possible. This book will not tell you how to make your use case serverless. Nor should it. This book will tell you how serverless can help you ship amazing software while saving a bunch of time. It will be real and honest with you about where the serverless world stands. If, on the other hand, you already know everything, it will (I hope) reinforce your worldview in a way that you can share with people who “don’t get it.” The goal of this book is to serve as a guide for building maintainable and scalable services through the lens of serverless computing. First, we’ll align ourselves on what it means to build a production system. Then, we’ll discuss knowledge specific to the current world of serverless compute, and the way that you or your team have decided to run your serverless workloads. The underlying ideas and philosophies in the first part of the book are meant to be as timeless as a technical tome can be: plan your software for all edge cases and the real xv
- 21. world. Use creative thinking and prioritization to spend the appropriate amount of time solving issues that are less likely to occur. I have been fortunate enough to spend time teaching programming. I have taught students of all ages, and even ones who were learning English at the same time as learning programming. Hopefully, these lessons will be shaped by that experience so a 12-year-old would be able to pick up this book and build something amazing. The same goes for someone who is 88. So bear with me as I talk about some concepts in this book that you may already be familiar with. I promise to keep the technical knowledge in this book high, while keeping the barrier to understanding it as low as possible. How This Book Is Organized After numerous conversations with engineers getting started in their careers, review‐ ing hundreds of online tutorials, and talking to users of serverless technologies at conferences, I had a revelation. Most of the information produced and consumed about programming is focused around building systems, but not around operating them. Most of the questions I was being asked after giving a talk on serverless were DevOps questions from those who chose serverless to avoid DevOps in the first place. Serverless may abstract away servers, but it does not abstract away DevOps. You will have to make tough decisions when designing, building, and operating your systems. But if done properly, based on the advice of this book and using serverless when applicable, you can meaningfully minimize frustration and time spent, while maxi‐ mizing your confidence in your systems. To achieve that mission, you may need to round out your knowledge of system design as it applies to building an internet application. You have the innate ability to design your systems so they achieve the desired functionality intentionally instead of acci‐ dentally. This will be done in Part I. In Part II, you’ll walk you through the tools at your disposal to achieve your wildest serverless dreams. You are then ready to build. But before you do, you should read Part III if you intend to launch your creation into any form of production. xvi | Preface
- 22. Conventions Used in This Book The following typographical conventions are used in this book: Italic Indicates new terms, URLs, email addresses, filenames, and file extensions. Constant width Used for program listings, as well as within paragraphs to refer to program ele‐ ments such as variable or function names, databases, data types, environment variables, statements, and keywords. Constant width italic Shows text that should be replaced with user-supplied values or by values deter‐ mined by context. This element signifies a tip or suggestion. This element signifies a general note. This element indicates a warning or caution. Preface | xvii
- 23. O’Reilly Online Learning For more than 40 years, O’Reilly Media has provided technol‐ ogy and business training, knowledge, and insight to help companies succeed. Our unique network of experts and innovators share their knowledge and expertise through books, articles, and our online learning platform. O’Reilly’s online learning platform gives you on-demand access to live training courses, in-depth learning paths, interactive coding environments, and a vast collection of text and video from O’Reilly and 200+ other publishers. For more information, visit http://oreilly.com. How to Contact Us Please address comments and questions concerning this book to the publisher: O’Reilly Media, Inc. 1005 Gravenstein Highway North Sebastopol, CA 95472 800-998-9938 (in the United States or Canada) 707-829-0515 (international or local) 707-829-0104 (fax) We have a web page for this book, where we list errata, examples, and any additional information. You can access this page at https://oreil.ly/Learning_Serverless. Email bookquestions@oreilly.com to comment or ask technical questions about this book. For news and information about our books and courses, visit http://oreilly.com. Find us on Facebook: http://facebook.com/oreilly Follow us on Twitter: http://twitter.com/oreillymedia Watch us on YouTube: http://www.youtube.com/oreillymedia Acknowledgments First and foremost, I want to thank everyone with whom I have ever had a conversa‐ tion about technology. No single book can contain all the wisdom on developing cloud native applications and workloads. But with the intent of broadening your scope of what that entails, is not achieved by only one person. This book may contain my interpretations, explanations, and attempts at pith, but most revelations offered xviii | Preface
- 24. come from the current state of knowledge of a larger community. And to that com‐ munity I am endlessly grateful. For personal friends who reviewed portions of this book and provided feedback, please know that it all was taken deeply to heart. For those I have worked with who provided feedback or enabled dialogue about building and evolving real working sys‐ tems, I hope we can continue to share such insights through thoughtful peer review, or as nonengineers refer to it—arguing. To all of the editors involved in this process, I truly could not have done this without you. Thank you to Vadym Kazulkin, who provided crucial technical feedback. Each and every one of you has taught me something truly valuable about who I am as a person in addition to bringing my writing from zero to one. And finally, I want to thank those in my life who had absolutely nothing to do with this book, other than just being a part of my life. Special mention to my partner and the love of my life, for not only putting up with the normal day-to-day version of me, but especially for dealing with my writing alter ego. Preface | xix
- 26. Introduction to Serverless Figure I-1. “Ancient Wisdom”, from the webcomic FaaS and Furious by Forrest Brazeal, 2019 xxi
- 27. To begin...To begin...How to start? I’m hungry. I should get coffee. Coffee would help me think. Maybe I should write something first, then reward myself with coffee. Coffee and a muffin. Okay, so I need to establish the themes. Maybe a banana-nut. That’s a good muffin. —Charlie Kaufman, Adaptation What Is Serverless? Serverless is the idea that you can run a server-based application without having to manage a server. If you have this text in front of you, there is a chance you are already familiar with serverless. But how do you explain it to others? Do you focus on what it is, a new way to run application code, or on what it isn’t, managing servers? Do you next tell people about all of its weaknesses or its strengths? If your path to success with serverless involves others, and it likely does, you might be worried about how to best sell its benefits without scaring anyone away in the process. You see, serverless is still in its early days and it’s on a path of continuous improvement. Some of its weak‐ nesses are here to stay, but they are trade-offs to implementing the benefits and fea‐ tures that you are sure to love. As the serverless community benefits from the rapid improvements of a cutting-edge technology, it can be a struggle to invent and adopt best practices. I hope not only to instill you with the most relevant best practices in serverless at the time of this writ‐ ing, but to also help you create and adopt best practices. I want you to know enough of the rules to break them safely. The first offering from Amazon Web Service (AWS) was the Simple Storage Service (S3). S3 allows you to store as many files as you like without having to provision any infrastructure. It is serverless storage. You may remember how S3 simplified storing arbitrary files: create a bucket (the S3 abstraction for a collection of files), then just give each file a unique name, and that’s it. You don’t have to worry about provisioning drives or backups, or the many other previous issues with storing files. Sure, S3 may not serve all file-storage purposes; it does have a maximum file size of 5 GB. Or it did until 2010, when this maximum was updated to 5 TB. How did they do it? AWS does all the heavy lifting on your behalf, splitting up files that are above 5 GB into multiple chunks in a way that is fully seamless to the user. That is the benefit of a serverless system. The similarities to modern serverless compute are uncanny: a general- purpose solution to a common problem (that may not fit all use cases), seamless improvements made behind the scenes (that usually began as hacks implemented by customers), and a pay-for-usage billing model. xxii | Introduction to Serverless
- 28. 1 Cloud Native Computing Foundation, “CNCF WG-Serverless Whitepaper v1.0,” 2018, https://oreil.ly/A2ehY. 2 Ken Fromm, “Why the Future of Software and Apps Is Serverless,” Read/Write, https://oreil.ly/vh5ck. The term serverless is a misnomer, because there are servers involved. In reality, there is no serverless, just someone else’s container. The Cloud Native Computing Founda‐ tion’s Serverless Working Group best summarizes this in a whitepaper:1 Serverless computing does not mean that we no longer use servers to host and run code; nor does it mean that operations engineers are no longer required. Rather, it refers to the idea that consumers of serverless computing no longer need to spend time and resources on server provisioning, maintenance, updates, scaling, and capacity planning. Instead, all of these tasks and capabilities are handled by a serverless plat‐ form and are completely abstracted away from the developers and IT/operations teams. As a result, developers focus on writing their applications’ business logic. Less time spent on wrangling infrastructure, and more time spent on shipping fea‐ tures. That is why serverless demand is increasing. There are limits to what it can do, but they are fading away as the technology progresses, and we may see functions become the new containers as cloud compute becomes increasingly managed. But how did we get here? History of Serverless Google’s App Engine was one of the first popularized use examples of serverless. Launched in 2008, it was too early for the modern wave of serverless adoption. Many developers viewed App Engine as being too restrictive, and that it was more of a hobby offering from Google. In fact, despite being launched in 2008, it wasn’t out of preview until 2011. But it was so ahead of its time that if you spin up a Google Cloud Function (at least in Python), it wraps your function in an App Engine-compatible package (using Flask) and runs it that way. The term serverless first started swirling around in 2012, in an article written by Ken Fromm of Iron.io.2 Fromm argued that web applications were moving from mono‐ lithic patterns to fully fledged distributed systems with loosely coupled components— which the next chapter will touch on. Fromm made his prediction more than two years before AWS released Lambda. But over six years before Lambda, and four years before this article, the first modern serverless system may have launched. Serverless offerings predate the term. Introduction to Serverless | xxiii
- 29. The Cloud Provider Landscape You may not get to choose the cloud provider for the project you are working on. You may also be so picky that you only work at companies using certain cloud providers. Right now, we live in a time of great competition among the providers. They are con‐ stantly finding new dimensions to compete in, ranging from price, performance, capability, scalability, etc. However, this period will not last forever as business natu‐ rally transitions from high growth in an emerging market into a period where the ROI of developing new features, attracting clients based on price, and even support and engineering resources will no longer look attractive, and those things will get cut out as quickly as you can say “shareholder returns.” Think about the costs of storage, and network, especially egress. And sometimes that can be compounded when using external providers for things like logging or moni‐ toring. Even transferring data between regions or availability zones of a given region may count the same as sending over the public internet, and may be caked into click‐ ing a box such as “multi-AZ availability.” How easy is it to click a button and have a database that is available in data centers across the globe? Is that even something your organization would need or be allowed to use based on data protection laws? You aren’t just renting a commodity offering. Even when it seems like you are, you are paying for a product. Evaluate it as such. Also consider the “cloud services” that may not fit into your mental model of what a cloud service is. For example, Google Meet is considered a Google Cloud product that is even marketed as being ready for telehealth. So is Google Maps. Amazon offers the ability to communicate with satellites, and to bring anywhere from a hardback book- sized device to an entire semi trailer to your site for easier migration of larger datasets to the cloud. Microsoft offers a lot of advanced functionality around its Office suite of products, which could be important for integrating with the software already in use by some in your organization. Reliability, Availability, Disaster Recovery What kind of SLAs/guarantees does the cloud provider offer? What is their track record? What are the remedies provided if they fail to meet their obligations? Google is known for services that stay in the beta phase too long, while AWS generally opts to offer a preview that may be more reliable but may still have some well-documented sharp edges. Also consider how easy it is to build for reliability and availability on the foundation and services provided. The network between the points of presence could be of interest if you plan on run‐ ning a truly global service. This may be outside of your expertise and even your area of comfort, but some of these decisions may be made by taking a leap of faith in the xxiv | Introduction to Serverless
- 30. right direction and realizing that any related issues will be understood. The best way to avoid these issues and to avoid surprises is with well-maintained documentation. Amazon Web Services Amazon Web Services (AWS) is oddly analogous to just being a data center with APIs. This is because of the services mandate at Amazon that states that all teams must build services, opening these very enterprise-y-feeling systems for public usage. This is the only explanation I can come up with that explains why it has so many sharp edges and weird quirks. It’s almost like joining Amazon as an engineer to build a greenfield project, and this is the internal service catalog. In fact, should you choose to use Amazon as your primary cloud provider, this mentality will help you make the most of the vendor lock-in to achieve maximum lift. AWS has the largest service cata‐ log, although sometimes to its own detriment. Google Cloud Platform Google Cloud Platform (GCP) is a powerful cloud contender that is the closest approximation an outsider would have to Google’s own infrastructure. Kubernetes is a recreation based on its internal Borg platform, and follows its infrastructure as a service offering. Or that used to be the case. As the cloud wars heat up, Google has launched more competitive products that are marketing directly to the users of the public cloud, instead of relaunching its internal offerings once they have been in use for a number of years. Microsoft Azure This is a great choice if your organization is already all in on Microsoft. For example, if your organization uses Sharepoint, this would be the most straightforward way to trigger advanced workflows or custom logic in reaction to your company’s shared filesystem. You don’t have to choose one of the big three to go serverless. If your organization is using Kubernetes, there are a number of open source options to run functions as you might run containers. Or even better, you can run containers as if they were functions. Kna‐ tive, one of these options, is actually what powers Google Cloud Run. So don’t feel left out if your organization isn’t in the public cloud, or has gone “all-in” on Kubernetes. Running in Kubernetes may already come with its own sets of pros and cons that you may want to consider when going this route if you have other options. Introduction to Serverless | xxv
- 31. Strengths of Serverless Some of the strengths of serverless come from the change in focus from an applica‐ tion as a unit of deployment to a smaller and more finely grained model of individual functions. You can still choose to deploy an entire monolithic web application as one function and have it execute one API request per invocation, or you can choose to carve up your applications into individual functions to reap the most benefits of serv‐ erless. These are not the only two choices, as you can meet in the middle and use one function per service/microservice. But doing so is the same as utilizing containers: you won’t get all of the benefits of serverless, but you will still get the downsides. Meanwhile, some of the benefits of serverless are ones that you don’t want or need, and therefore become problems. Just as every coin has two sides, some of these bene‐ fits will directly map to a weakness. Increased Scalability, Security, and Reliability This is a core feature of the serverless experience. You don’t have to plan for future capacity, other than service limits from your cloud provider and interacting with nonserverless components or systems. For example, there was a big marketing cam‐ paign for new users on a project where I was using serverless. I found out the next day, which isn’t ideal, but sure enough, Lambda and Amazon DynamoDB took on all the load without any action or knowledge from yours truly. You don’t have to manage security other than the controls provided to you for granting permissions, and your application code and bundled libraries. When you have dedicated teams keeping up the servers that run your application code, you benefit from the economies of scale that provide maximum uptime. You Only Pay for What You Use One of the most attractive features of serverless compute is not paying for idle time. If your system is entirely serverless and isn’t used in a given billing period, the total bill for compute will be $0. Pricing can be more predictable when you are charged for a specific number of tasks instead of instance hours. If you have an application that is used only during regular business hours and utilizes containers or other instances, you can automatically shut it down on the weekends to save money. But what hap‐ pens if people need to use this service on the weekend? You leave it up and running in a minimal state, and wind up paying for every single weekend. What about holidays? What about a company all-hands event? You are paying for servers you don’t need, but if you shut them off, your application has no availability. With serverless, a request automatically spins up the compute it needs if none is available, and you are only charged for that request. Your application is always available (although some‐ times it may suffer from a cold start, which we will address later); if no one uses it, your cost for that time period is zero. Other parts of your application may have an xxvi | Introduction to Serverless
- 32. effect on your cloud bill, such as data storage, monitoring, and other support systems, but the compute will be zero. Saving Time and Money on Managing Servers Of course, you’ll be spending valuable engineering time on optimizing the cost of non-serverless systems. The time spent making those decisions isn’t free. It is meas‐ ured in the pay of engineers and the costs of recruiting and retaining them, as well as not shipping valuable features to users! Tedious tasks such as capacity planning don’t entirely disappear when you use serverless, but you get to zoom out by an order of magnitude, and that has clear benefits. Think of it this way: if you can’t afford to hire a full-time platform-engineering team to run your code, why not rent one from your cloud provider? You may lose the abil‐ ity to handle certain low-level tasks, but this is specialization of labor and economies of scale at their best. Instead of you having to manually configure autoscaling groups to provision and deprovision computing resources based on some abstractions of work that needs to be performed by your system, serverless specifically operates by scaling on the real metric of work that needs to be performed. There is no organiza‐ tion running in the cloud that does not have some amount of idle compute being wasted at any given time. Improved Developer Productivity Some cloud providers suggest using functions as glue to add logic and process to con‐ nect services. You don’t have to reinvent the wheel when it comes to the distributed execution environment, queuing, retrying logic, and so on for modern serverless offerings that continue to increase with time. There is no better example of this than creating an extract, transform, load (ETL) pipeline using serverless. An ETL pipeline takes data from one source, runs some compute over it, and loads it into a new destination. You can connect a data source that will automatically invoke a function for every single write performed on a data‐ base, and that lambda can transform that data without any servers or worrying about how many writes the original database will scale up or down to. It just works! Decreased Management Responsibilities I have already mentioned the idea of renting your DevOps from your cloud provider when your organization can’t afford, or doesn’t need a full-time dedicated team of platform engineers. That benefit cascades into other benefits as well. Serverless pro‐ vides a stable container to target while having someone else manage security updates and patching of underlying infrastructure. It is important to remember the shared model of responsibility when utilizing any such offering, because you still have to take care of the security of your code and the libraries you utilize in your application. Introduction to Serverless | xxvii
- 33. 3 Personally, I like to stay up to date with Hacker News. (I will cover security further in Chapter 9.) But you don’t have to worry about patch‐ ing the operating system, the libraries included on the system, and the version of the programming language itself. Your cloud provider employs a 24/7 staff of engineers who handle those choices and responsibilities. Convenient Integrations The biggest draw to the big three cloud providers when it comes to serverless is the integrations. It all comes down to the events. Publish a message in Google Cloud Pub/ Sub? Why not react to that in real time with code? No need to monitor your worker nodes anymore. Add or update a record in your database, and boom, you can attach something that audits that action. Have a client upload an image directly to S3, and you can process that image into thumbnails without provisioning a single server. Using AWS Cognito to handle user accounts, and you want to send a welcome email after a user registration? Serverless handles all of those use cases and many more. Current offerings provide a way to have your function code glue together actions in different parts of your system without worrying about provisioning queuing resour‐ ces or creating a task execution and background work environment on your own. Some of this leads to opaqueness and comes back as a weakness in debugging. This is especially true as it becomes easier to glue together external services into your appli‐ cation architecture. Weaknesses of Serverless The most interesting part of the weaknesses of serverless is how they become less cumbersome or start to disappear as time progresses. The industry has seen major advances on some such issues while this book was being written, and change will continue to be rapid. To stay up to date on developments, especially in a space as rap‐ idly evolving as serverless, or cloud native as a whole, make sure to follow the blog or get email announcements from your cloud providers, join mailing lists for relevant groups, or even follow developments on a site like Reddit.3 The Cold (Start) War A cold start happens when a function invocation occurs and there is no running func‐ tion available to execute the work. Instead, a new function container will spin up, and your users have to spend time waiting for your application to respond. Some people keep functions warm to prevent this problem, but I believe in using the right tool for the job. People who are faking usage to keep their functions ready for user traffic are not using serverless as it was intended. What they really want is to instantly answer xxviii | Introduction to Serverless
- 34. up to a certain number of concurrent requests without waiting for an additional machine to spin up and be added to a cluster. A serverless function will certainly beat spinning up an entire additional EC2 instance, but for some people that just isn’t enough. I will give these users the benefit of the doubt by saying they are just so exci‐ ted to use serverless that they are willing to use hacks to fix some of the weaknesses. If this form of latency is a deal breaker for your application, then serverless may not be right for your use case. Instead, utilize serverless for workloads that aren’t directly user facing. This cold start issue will continue to fade with time, but that future is already avail‐ able now. Some environments offering compute at the edge or Content Delivery Net‐ work (CDN), such as Cloudflare workers, have increased limitations on the functions they will execute to decrease the cold start time in order to preprocess or post process a web request. Think about that. While most developers are trying to respond to API requests in under 100 ms, they are adding additional compute before or after that 100 ms. A common use case for this concept is injecting personalization into a cached page being served from a CDN. Many companies offering are also alternative environments to solve this issue. It’s an arms race. If you need the performance at this time, it may not be there. But it will get faster until it reaches the minimum overhead. AWS Lambda, for example, greatly improved its start-up time for cold starts by completely reinventing how it connects a function to a private network. Compute Time One agreed-upon weakness of serverless is the limited amount of time in which a particular workload can run. There are some workarounds, but it may make sense not to utilize serverless in some use cases. However, this limitation is arbitrary in many ways. In 2018, Amazon changed the limits on Lambda from 5 minutes to 15 minutes. There was no need to rearchitect Lambda to make this change. As some issues with serverless are solved, the solutions will be available to you without any additional engineering overhead. You may have to spend engineering time to take the most advantage of the changing landscape, but your system will still work without those changes as well. VPC/Network Issues If your application needs to run inside a specific private subnet or cloud network, there are some limitations. You can’t scale to 10,000 concurrent executions in a sub‐ net with room for 254 IP addresses. Depending on your organization, you may be forced to operate in a virtual private cloud (VPC) in order to access private resources, or your application may call for accessing a database that can only be reached in a certain network. You will have to capacity plan to make sure your private networks Introduction to Serverless | xxix
- 35. are large enough. If you want to build a truly serverless system, you will have to avoid certain cloud offerings, persistence layers, or other design choices that will tie you to a specific private network. Application Size Limitations like compute time are also arbitrary, but if your application is too large, the cold start times may become unmanageable, so limiting the bundle size of your application is a good sanity check. How does this limitation affect you? One example is that you may not be able to ship a large Java application into a serverless function— using containers or instances is a better strategy for now, but keep an eye out for changes that could enable this. You may also be limited in the amount and size of dependencies of your application, although with the introduction of layers in AWS, there are advancements in this area as well. Potential to Be More Expensive If your application requires a predictable and stable amount of compute, you will overpay by using serverless. But consider the cost of maintenance and upkeep required for patching systems with security and other updates. You can pay your employees to do this, or you can overpay for your compute to have some of those maintenance costs bundled in. Does it make more sense to spend an extra $200,000 per year on a DevOps engineer or overpay on your cloud bill by $20,000 per year? Spend that money on another engineer who will build functionality with directly attributable revenue. Vendor Lock-In Every technology you select will likely lock you into using a specific vendor in one way or another: which base Docker image you use, which database you use, should you really add that additional package, and so on. You can lessen this by having your organization self-host a function execution environment on top of Kubernetes using open source software. But there is a high likelihood your organization already has some level of vendor lock-in to one of these cloud providers. If your organization has already made a trade-off in a specific direction, it makes sense to piggyback on top of that. This may be the case for you. Vendor lock-in is an interesting concern. Some suggest this is just an overreaction to switching costs, which comes with all technology choices. They liken it to what hap‐ pens if you want to change from Java to Python, or Go to Erlang. This is true only in that every developer has the choice of making optimizations and trade-offs as they see fit. Sure, you can save a lot of money on hosting costs by running your application on an old server under your desk, or on a cluster of Raspberry Pis, but you will likely choose to use virtualized instances from a large cloud provider because you will have xxx | Introduction to Serverless
- 36. to decide how you want to spend your time: writing code, or carrying buckets of die‐ sel fuel up a staircase after a hurricane (see “The Physical World” on page 4). Lock-in is something to be mindful of, but not to spend much time on. I will be focusing on examples primarily from AWS due to the depth of supporting services and integrations, but these examples are for illustration purposes. I am not advocat‐ ing allegiance to any one particular provider, and think the most pragmatic approach is to keep your options open. If your organization has chosen to invest in one of these platforms, take advantage of the deep service catalog you have available to get your job done in the best way with the fewest trade-offs possible. Learn to love your provider, but don’t trust them more than you should. Complex Debugging When you have a dynamic runtime, debugging can be complicated to reproduce errors in order to solve them. The more components or microservices your system is comprised of, the more difficult it can be to trace a user action throughout the entire system. That’s why so many tools and SaaS offerings address these issues. I believe this is generally a symptom of using serverless incorrectly. Used correctly, serverless should give you more understanding of the core functionality of your systems. Some of these tools, however, are evolving into really compelling ways to find and filter issues, as well as providing data helpful in reproducing such errors. Your debugging and introspection are more powerful than ever before. What a time to be alive! When Does It Make Sense to Use Serverless? Many developers are making the move to serverless, or exploring serverless compo‐ nents for parts of their applications. Werner Vogels says: At Amazon, we’re not completely serverless ourselves, but we’re moving in that direc‐ tion. And so are many of our customers. In fact, we anticipate that there will soon be a whole generation of developers who have never touched a server and only write busi‐ ness logic. The reason is simple. Whether you’re building net new applications or migrating legacy, using serverless primitives for compute, data, and integration enables you to benefit from the most agility that the cloud has to offer. He sees serverless primitives (the most basic types of resources), as superior to their server-based equivalents, just as the cloud primitives were superior to the data center primitives were superior to the mainframe equivalents. Use cases vary, but here are some of the most common and best reasons to use serverless. The most important factor to determine your success will be the use case. Have you heard people complain about serverless? What do they talk about? Cold starts. While Introduction to Serverless | xxxi
- 37. cold starts will eventually be optimized as close to zero as possible, you can build a system that is unaffected by cold starts. This pattern is the same for people who com‐ plain about how NoSQL doesn’t have transactions, or how iPads don’t have mouse support. Although these days, things are changing: DynamoDB offers NoSQL with transactions, and the latest iPad Pro has a trackpad. You don’t need a specific reason to use serverless, but here are some examples of the characteristics of the compute work you want to perform that will have the least fric‐ tion and most benefit when utilizing serverless: • Tasks that can be broken up into small independent units of work • Tasks that either have infrequent or unpredictable usage patterns • Background work, or system to system communication that will not be impacted by cold starts Let’s break these down. A task is a unit of work that isn’t blocking, and can be broken up into smaller units of work that would each fit into a function. Serverless is best used for load that is not predictable. This doesn’t mean you can’t use it in this case, it just may not be the most efficient and will cost more than use con‐ tainers (but again, that doesn’t include the overhead of managing the containers). But what about your workload? If you can see your system as a collection of easily separable parts, and you don’t want to deal with the overhead of servers for a lack of resources, it may make sense to use serverless. Some parts of your application will be high velocity, at least when it comes to the rate of change of features and priorities. But then you have the strong and steady work‐ horse components. Imagine some of the problems you have yet to solve. There are some parts of your overall application that will be low velocity once version 1.0 is shipped. They don’t directly serve users, but offload work from the application servers that do. Sending email to users is a perfect asynchronous task to set up to hap‐ pen in the background that won’t need much change to the basic architecture. It has somewhat unpredictable demand. And while you want it to happen in real time, the latency of a cold start is not going to ruin the password reset experience for a user locked out of their account. Another interesting use case of serverless is the nearly infinite scale it brings. Let’s assume it takes 30 seconds to process one minute of high resolution video for stream‐ ing. How long will it take to process a 90-minute film? 30 seconds. Because you can break up and parallelize the work and instantly feed it out to as many Lambda func‐ tions as possible, you can drastically speed up the time it takes to complete a task. This is actually one way Netflix uses serverless. xxxii | Introduction to Serverless
- 38. Another strong use case for serverless is event-driven architecture. Chapter 3 will cover serverless architectural patterns in detail. One of the most helpful uses of serverless compute is that it acts as the glue between services. For example, you can monitor the utilization of a resource to scale a service up or down to save costs. Want to resize uploaded images into thumbnails automatically without setting up a task or queueing service? Do you want to save money on your instances by using spot instances that cost less money than traditional instances? When those spot instances are being taken away (part of the reason they are less expensive), you can have a func‐ tion automatically invoked on that cloud event to spin up a regular instance to take its place. Another spot instance becomes available later? Same thing in reverse: your function can spin up the instance that costs less money and terminate the more expensive one. Want to react to changes in data as they happen without adding brittle analytical code to the main conversion funnel of your application? Serverless can help with all of these use cases. It can be glue, DevOps, automation, out-of-band process‐ ing of data, or fully fledged applications. When Is Serverless Compute Not Right for You? Serverless will not serve you best when you have tasks that are computationally inten‐ sive, when your tasks have a long runtime that can’t be broken up into smaller work‐ loads, or when you need additional functionality not currently supported by the cloud providers, to name a few examples. These tasks might look like reading a large table of data and turning each row into an API request, encoding a feature-length film for streaming, or running a persistent WebSockets connection for a chat func‐ tion. But some of these examples do have ways of being adapted to work. You can run a parallel scan or certain types of datastores such as DynamoDB. You can break up large files into smaller parallelized chucks as Netflix currently does to encode movies. You can use an API Gateway with WebSockets to maintain a real-time connection to clients, while invoking a Lambda for each message passed. Let’s Get Started It is time to start or continue your serverless journey. By the end of this book you will have learned many fundamentals and best practices needed to succeed in any form of cloud computing, servers or not. How “full stack” are you? If you specialize in certain areas, how did you choose those areas? Did you try other things out before deciding not to be an expert? You need to know how to manage servers before you can manage a system that manages them for you. Introduction to Serverless | xxxiii
- 39. The choice to go serverless, is generally made to reduce the complexity in configuring and managing infrastructure, but you must have some basic understanding of the work you are abstracting away to build a reliable system on top of it. That will all be covered in this book. Part I of this book will walk you through what it means to launch a proper produc‐ tion system. There will be servers involved, of course, but you won’t need to know them personally. Part II will cover the tools you will need to be successful with server‐ less. Part III will cover some more advanced topics in depth, such as security. Now let’s talk about production systems. xxxiv | Introduction to Serverless
- 40. PART I The Path to Production “Multicloud”, from the webcomic FaaS and Furious by Forrest Brazeal, 2018
- 42. CHAPTER 1 Distributed Systems We’ll begin our journey through serverless by talking about distributed systems. Before we jump into definitions and examples, what do you need to know about dis‐ tributed systems to be effective with serverless? When you develop an application, you have to make a large number of assumptions. Some may be as simple as knowing that one step will occur after another. Others may be far more complex. Distributed systems will tear apart all your assumptions about the environment in which your code will run and how it will operate. When you develop for a single computer, many of the harsh realities of the physical world are abstracted away. As soon as you start building a system that resides on multiple computers, all of those realities suddenly surface—though they might not be obvious. This chapter will first offer a broad overview to better understand what you have signed up for. If you do not have experience developing backend systems, my goal is to explain what has changed about your world. But even if you have experience, you will find value here: distributed systems can bring out the pessimism and cynicism even in experi‐ enced software engineers and system administrators. We’ll talk about what can go wrong and what you can do about it. What Is a Distributed System? A distributed system is any system where the individual components are separated and communicate over a network. A distributed system can be part of a larger or smaller distributed system. The internet is one giant distributed system. Your cell phone provider operates a giant distributed system in order to connect you to an even bigger one. Their system contains wireless gear, network gear, and applications such as billing and customer information. 1
- 43. When working with apps, we usually expect determinism: given a specific input, the output, and the states and sequences to achieve that output, will always be the same. The reality of distributed systems, however, is nondeterminism. As the complexity of your application grows, it becomes difficult to predict the state of it at any given point. It is assumed that all parts of the system will be unreliable in either obvious or nonobvious ways, but it is your job to build a reliable system from these unreliable components. Your application does not live or process logic in a single place. If you have a browser- based application or a mobile application, the second you put a line of code in any other place, such as a function in the cloud, your system is distributed. Generally, components of a distributed system are asynchronous, meaning they pass off a task and do not wait directly for the result. But many important operations, such as the processing of credit card transactions, will be accessed synchronously, as they should block the progress of the calling task until completion of the vital transaction. A serverless system is inherently distributed. Given that a serverless function is by its very nature stateless, if your application involves any form of state, it is going to have to be distributed. But aren’t all modern applications distributed? While the answer is likely yes, it is definitely true for applications that have ambitions to grow and become more complex. If you are building a simple web app with a client frontend, a monolithic backend, and a database (also known as a three-tiered web application), then you are building a distributed system. However, many developers will neglect this fact when thinking about how the application stores its state in a database. And they will run into prob‐ lems as a result. At some point in scaling up their system, they will likely face an issue caused by an application server, regarded as being easily and horizontally scalable, connecting to their database (vertically scalable). This issue could range anywhere from needing more resources for the database to simply needing to update the data‐ base configuration to allow additional connections. But those developers who forget they are working on a distributed system will have all of the problems of one, without any of the common patterns to minimize issues. Serverless shifts many responsibilities to your cloud provider. However, as the soft‐ ware practitioner writing the business logic, there are still things you need to know and understand to make promises to your stakeholders in regard to the reliability, availability, and scalability of your software. 2 | Chapter 1: Distributed Systems
- 44. Why Do We Want a Distributed System? Do you need a solution that handles what happens when someone introduces a bug that causes your database to lock, preventing your main user-facing system from operating? Well, that’s actually a strength of distributed systems because you can kill the failing service, and all of the work expected to be done by it will be delayed but not lost as it queues up. If your user registration code ran the email sending code directly, you would be down completely. Designing so that one failure does not cas‐ cade and directly cause another is the subject of Chapters 4 and 11, but the resources listed in “Further Reading” on page 11 cover these concepts in much more depth. Any application intended to scale must be a distributed system. Otherwise, you will be limited to the compute and storage of one computer, and your users must visit this computer and use it in person, only one at a time. There are many advantages of dis‐ tributed systems, but there is no choice to be made. You are building a distributed system. You must learn the disadvantages of doing so to best limit their impact on your operations. The Harsh Realities of Distributed Systems Nothing about the network can be trusted. And in a distributed system, messages must be passed over the network. In Designing Data-Intensive Applications (O’Reilly), Martin Kleppmann expands on this interconnected relationship between your code and the source of so many problems, the network: A node in the network cannot know anything for sure—it can only make guesses based on the messages it receives (or doesn’t receive) via the network. A node can only find out what state another node is in (what data it has stored, whether it is correctly func‐ tioning, etc.) by exchanging messages with it. If a remote node doesn’t respond, there is no way of knowing what state it is in, because problems in the network cannot reliably be distinguished from problems at a node. Networks seem to be pretty reliable, but every now and then you have to hit the refresh button or wait. In a distributed system, your system has to deal with automat‐ ing that refresh. If one system goes down, and all of the other systems start attacking it with requests when it is already failing to keep up, what happens? There are far fewer things to consider when two pieces of code run in the same stack. Asynchronicity can create a lot of unintended effects, especially when unexpected by the programmer. Now add the reliability of a network to that. To illustrate these issues, let’s look at a common application. This application has a user registration process. New registrations go into a task queue to perform some operations, such as sending a welcome email. The developer made a smart choice to decouple the user’s registration and the back-of-the-house logic, such as sending an email. If the application was suffering from issues with sending email, it should not Why Do We Want a Distributed System? | 3
- 45. block the user from registering successfully. Other actions in the system may also cause a task to get queued up that will send some form of notification. Seems simple, right? Let’s get to it. The Physical World In the aftermath of Hurricane Sandy in 2012, a group of operational engineers found themselves in a precarious situation. The power was out in Lower Manhattan. The data center had generators and diesel fuel on hand, but the diesel pump had failed due to flooding; the pump was in the basement, and the generators were on the roof. Heroically, the engineers mounted a bucket brigade to bring diesel fuel, in 5-gallon buckets, up 17 flights of stairs, in the dark. The physical world itself is nowhere near perfect. Just because your organization does not own the servers, or can’t even touch or see them, does not mean you will not be affected by a fire, a power disruption, or another disaster, natural or otherwise. The companies relying on that particular data center were spared by the heroism of the bucket brigade, blissfully unaware of their servers’ potential to be cut off at any moment. When you host in the cloud, you may not be responsible for carrying a bucket, but you still have to deliver your application to end users despite the circum‐ stances. The cloud solves this problem as much as current technology allows with multiple availability zones, which generally come for free in serverless compute, but must be accounted for in your persistence of data, and other services as well. The physical world can be the root cause of many other failures we will encounter when working in the cloud: Network issues Someone may have tripped on a cable and pulled it out of its socket. Clock issues The physical hardware on the server responsible for keeping track of the time, a crystal, could be defective. Unresponsive node There could be a fire. Calling attention to this allows us to drastically simplify the rest of the issues we will face and focus more on the impact of these issues as you design your systems. Missing Messages Have you ever sent an email only to later find it stuck in the drafts or the outbox? There is no guarantee when you make a request over a network that it will be deliv‐ ered or processed. This is one of the many things we take for granted when working 4 | Chapter 1: Distributed Systems
- 46. on software that will run locally. These issues are the simple reality of computing that has been abstracted away from engineers enough that people have forgotten their existence. Networks can have congestion just like your local interstate during rush hour. The modern network in a cloud computing environment is a distributed system itself. The easiest way to observe this is when using a mobile network. We have all had experiences with apps that hang because they expect an instantaneous response from a remote computing system. How would this affect your code if it were some kind of live or real-time game? If your response gets too delayed, it could even be rejected by the remote system as anticheating logic. Messages go missing, show up late, or even show up at the wrong destination. The wires can’t be trusted. Unreliable Clocks How important is it for your system to know what time it is? What happens if you set your iPhone back in time before iPhones existed? Or what if you set it to a time 30 years into the future? Either way, there is a good chance it won’t boot. Apple has never confirmed the issue, but it has been attributed to timestamps on Unix systems having been started on January 1, 1970, creating a date of 0. Remember that the engineers working on the iPhone likely did not expect users to set back their date so far in the past, but they permitted users to do so. This has caused unexpected bugs, even for Apple. Servers have their system clock set automatically using the Network Time Protocol. While relying on your system clock seems like a sure thing, there are potential issues. Google published a paper on its internal Spanner database that details how they deal with time for this critical system. When their nodes were set to poll every 30 seconds, the system clock drifted by as much as 7 ms. That may not be an issue for you, even as both Google and Amazon offer enhanced synchronization based on GPS, and atomic clocks for hypersensitive systems such as trading stocks, though the common commodity system clock has some other quirks. When your clock drifts, it will even‐ tually be corrected in a way that can alter the effect of your time-sensitive code. Mul‐ tiple CPU cores have different references of the current time, and logic living inside a virtualized system on the cloud has an extra layer of separation from the reality of time passing in the outside world. Your code may experience jumps in time, forward or backward, at any time. It is important to utilize a monotonic clock when measur‐ ing the passage of time. A monotonic clock is one that is guaranteed to increase. In addition to the clock being susceptible to changing more than a second in any given second, there is no way to guarantee that all of your nodes’ clocks will be set to the same time. They are subject to the same issues of network reliability we have already discussed. As with all issues you will face, there will be a trade-off in the importance of an aspect of your system to the use case and amount of engineering resources available. Building a social network for pets? Those seconds may not be The Harsh Realities of Distributed Systems | 5
- 47. worth your trouble. Building a high-frequency trading system? You may have to uti‐ lize hardware atomic clocks set by GPS, as those microseconds can cost megabucks. The current time as it appears to your business logic can unexpectedly jump forward. Serverless functions, as with other forms of cloud compute, run your code in a vir‐ tualized or isolated way. The software that provides this virtualization or isolation can distort time in a number of ways. One distortion that can occur is when your code competes for shared resources, it may suffer from a pause due to multithreading. It will be put to sleep, then suddenly reactivated but with no understanding of the pas‐ sage of time that occurred in the outside world. This can similarly be caused by pro‐ cesses such as memory swaps, garbage collection, or even synchronously waiting on some resource that is accessed over the network. Keep this in mind when attempting to squeeze more performance by using threads or subprocesses to perform additional work in your system. These realities can manifest as issues where, for example, you can’t reliably know which change happened first in a series of events in a queue. In reality, when dealing with modern distributed systems, there is an expectation that your system may run in multiple different geographies. In this case, we have already learned that events can and will come out of order, and there is no real way to determine the order without some form of locking, which can be expensive and bring its own issues to bear. But if you need that kind of knowledge in your system, you won’t have any other choice. Even then you can and will be wrong about which task deserves to have the lock first. You have to handle that in software. No service will be offered in the short term that will solve this for you. Even if they start offering “consensus as a service” or some‐ thing similar, you will still have to understand the trade-offs and issues around their use when implementing your business logic. Cascading Failures Let’s say that you, the developer of the application in this example, did a great job loosely coupling the two components provided. If the user registration system goes down, the email system won’t really mind at all. In fact, if the email system is server‐ less, it won’t even run (how efficient!). If the email system goes down, the user regis‐ tration system stays up. Or so you might think. What happens if your task-queuing system becomes full and no longer accepts new tasks, and now your users can’t sign up? This is how issues compound, or cascade, to cause other issues. In this example, when one system (sending mail) failed long and hard enough, the outage caused another system to fail. When your system is composed of dominoes, space them to avoid a chain reaction when one falls. No matter how slow it is (it could have been an entire weekend before the queue filled up), a resilient system will be engineered to avoid this issue. You may not be able to afford such resilience in your current project, but you must be mindful of it. 6 | Chapter 1: Distributed Systems
- 48. Unexpected Ordering Have you ever shipped a new version of your code that included an additional time‐ stamp field, only to find that somehow inserts are still being committed without one? When operating in a distributed system, there is no guarantee for the order of execu‐ tion of logic split across multiple nodes. But how could your deployed changes not take effect? Simple: the old version of the code is running somewhere. It could be a task server that is faithfully chugging along while refusing to respond to requests for it to shut down so that it can be replaced with the new version of that code. Meanwhile, there is another change waiting to be pushed to production that includes some kind of mandatory field on registration, such as a first name, as well as includ‐ ing that name in the welcome email. You have a large day of new sign-ups, and this code makes it out to production. Instantly, people stop getting welcome emails, and you now have a big headache—what went wrong? Synchronicity was assumed. There were some number of welcome emails waiting to be sent out. When the new code hit production, the existing tasks were to send welcome emails to users that included their name, something those records don’t have. This particular issue can also occur due to network latency. Idempotency Idempotency is the idea that a certain task repeated more than once will have the same outcome. It is somewhat easy to build a system that will perform a given task at least once, but much more difficult, if not impossible to do in a guaranteed way, to build a system that performs a given task once and only once. However your system sends email, whether speaking SMTP directly to your users’ mail exchanger or using a third-party API, it’s not hard to imagine a situation where an email was successfully sent, but a failure is reported. This happens more than you would imagine when you start to scale, and chaos takes full hold. You try and send the email, and it gets sent, but right before the other side responds with success, the network connection is severed, and you never get that successful response. So as a responsible developer, you have designed the system to try again. This time it works. But your task that has been attempted twice, was completed twice, and as a result the user got two welcome emails. This is enough of an edge case that you may not try and over-optimize your system to always send exactly one welcome email, and you may not be able to without also hav‐ ing access to your user’s mailbox. But even then, what if they delete the message before you check? You will send it over and over again. A single node can never really know the truth of the outside world because it relies on the network to learn about the truth, and by the time it gets a response, that truth may be stale. Once you accept that, you can design for it. The Harsh Realities of Distributed Systems | 7
- 49. Another Random Scribd Document with Unrelated Content
- 50. underwood as on the northern bank of the Oxus. They, however, occurred in the same succession and formation as have been there described: they were quite soft, but the sand was not dusty, and the camels slid down them with their burthens. Here and there we came upon a sheet of indurated clay, as if the sand- hills here also rested on a base of that kind. In these hollows, and on the brow of the hills, we found a shrub like tamarisk, called “kasura,” also a kind of grass, or bent, called “salun.” There were likewise two thorny shrubs, called “kuzzak” and “karaghan”[1], neither of them the common camel thorn, but on which the camels delighted to browse. There was no water throughout the whole march, and no signs of inhabitants but a ruined fort, that had once served as a look-out from the Oxus. The Indian deserts of Jaysulmeer and Parkur sank into insignificance before this vast ocean of sand. No sight is more imposing than a desert; and the eye rests with a deep interest on the long line of camels, as it winds its crooked course through the frightful waste. The simile of a ship in the ocean and a camel in the desert may be hackneyed, but it is just. The objects animate impart a strong interest to inanimate nature. In the middle of our march through the desert, we met seven unfortunate Persians, who had been captured by the Toorkmuns, and were now on their road to Bokhara, where they would be sold. Five of them were chained together, and trod their way through the deep sand. There was a general shout of compassion, as the caravan passed these miserable beings; and the sympathy did not fail to affect the poor creatures themselves. They cried, and gave a longing look, as the last camel of the caravan passed to their dear native country. The camel on which I rode happened to be in the rear, and I stayed to hear their tale of woe. They had been seized by the Toorkmuns at Ghaeen, near Meshid, a few weeks before, when the culture of their fields had led them beyond the threshold of their homes. They were weary and thirsty, and I gave them all I could,—a single melon; a civility, little as it was, which was received with gratitude. What a frightful notion must not these unfortunate beings have entertained of the country they were entering, after their travels in such a desert. The Toorkmuns evince but little compassion for their Persian slaves; and what other treatment is to be expected from men who pass their lives in selling human beings. They give them but a scanty supply of food and water, that they may waste their strength, and prevent their escape; but beyond this the Toorkmun inflicts no other ills. The tales which have been circulated of their cutting the sinew of the heel, and of their passing a cord round the collar bone, are at variance with truth, since these blemishes would diminish the value of the slave. These unfortunate captives suffer a much heavier calamity,—they lose their liberty. As we reached our halting-ground in the morning, we had now an opportunity of observing the number and composition of the caravan. There were upwards of eighty camels, and about 150 persons, several of them men of the first respectability, who accompanied their merchandise to the markets of Persia. Some
- 51. travelled in panniers placed on camels; others rode, some on horses, many on donkeys; but every person, even the meanest, had some kind of conveyance. The horsemen preceded the camels; and, stretching themselves out on the sand with their bridles in their grasp, stole a few moments’ sleep, till the caravan overtook them. The scene was altogether curious and novel. Among the party there were eight or ten Persians, who had passed many years of slavery in Toorkistan, and, after purchasing their liberty, were now returning by stealth to their homes. These people were delighted with our enquiries; and, in the journey, many of them became attached to us. They would bring melons for us; kill a sheep; draw water; and were always at hand. Some of them had been no less than three times captured, and as often had they redeemed themselves; for the Uzbeks are readily imposed upon and cheated by their slaves, who make money in service. I conversed with several of them; and it was equally painful to hear their past sufferings and present anxiety. Their influential countrymen in the caravan had put several of them in charge of a portion of their merchandise, that they might be the less noticed, and considered rather as traders than emancipated slaves; for a Persian merchant in a caravan is generally safe. In spite of all this arrangement, some hard-hearted wretches had told tales on the banks of the Oxus: one individual had been forced to return to Bokhara; and some of the others had crossed with difficulty. One single hint to the people of Orgunje would, in all probability, yet arrest their further progress; but every one had been well tutored. What must be the feelings of some of these men as they approach Persia. One of them told me that he had had a wife and a numerous family when sold into captivity, twenty-two years before; of whom he had not heard any account since that period. If any of them are alive, the parent will show himself among them as an apparition from the tomb. Another of these unfortunate individuals had a tale which was not less touching. He had been seized along with his family, and, indeed, all the inhabitants of his village, near Toorshish; and delivered up, by one of the Khorasan chiefs, to the Toorkmuns, who drove, on this occasion, upwards of a hundred people to Bokhara. At Maimunu, which is on the road, they were disposed of to other Toorkmuns, and at Bokhara finally sold. There this unfortunate man saw his wife sold to one, his daughter and son to others, and himself to a different person. A humane man, hearing of his misfortunes, released him, since he believed it good in the sight of God; and the poor fellow lurked in Bokhara, like a bird near its nest that is robbed, in hopes of relieving the other members of his family. He had failed, and was now travelling into his own country, to excite the compassion and pity of those who had known him in his prosperity. It would harrow up a man’s heart, to listen to all the tales of the woe which is inflicted upon mankind by these plundering Toorkmuns. In marching from Karoul, we quitted the high road of the caravans, which leads to Merve, and proceeded westward into the desert, by a way that is altogether unfrequented. We had no option in the selection of such a route, since the officer
- 52. who commands the Orgunje army sent a messenger to direct our march upon his camp. We were thus thrown into the jaws of the lion, but were helpless; and the merchants appeared to regret it more than ourselves. After the usual halt, we reached the well of Balghooee twenty-four miles distant, on the morning of the 23d. It was a small and single well, about four feet in diameter, as deep as that at Karoul; and the Toorkmuns only discovered it after a zigzag search of some hours. We soon emptied it (for the water was good), and had to wait a night till it again filled. In this march the desert was overgrown with brushwood, but the tract was entirely destitute of water; and a few rats, lizards, and beetles, with here and there a solitary bird, were its only inhabitants. Some of the sand-hills now attained the height of sixty feet: but at that elevation they are invariably bare of all vegetation; which, I suppose, cannot thrive in such an exposed situation. The highest hills were about a distance of eight miles from the halting-place, and named “sheer i shootr,” or “the camel’s milk,” from some allusion to that useful animal. There was nothing peculiar in the colour of the sand, which was quartzose. There was no turf, grass, or creeping plants; every shrub grew separately; and the grass, which I before mentioned, was only to be found in clumps. The heat of the sand rose to 150°: and that of the atmosphere exceeded 100°, but the wind blew steadily; nor do I believe it would be possible to traverse this tract in summer, if it ceased to blow: the steady manner in which it comes from one direction is remarkable in this inland country. It is true, that in every direction except the north we have mountains, but they are too distant to impede the winds. Our caravan advanced at a firm and equal pace among the sand; nor can I discover that the progress of a camel is much impeded in the desert. They moved at the rate of two miles and one eighth in the hour (3740 yards); and I have since found, that the judicious Volney assigns the distance of 3600 yards as the hourly journey of a camel in the sands of Egypt and Syria. We had before heard of the deserts south of the Oxus; and had now the means of forming a judgment from personal observation. We saw the skeletons of camels and horses now bleaching in the sun, which had perished from thirst. The nature of the roads or pathways admits of their easy obliteration; and, if the beaten track be once forsaken, the traveller and his jaded animal generally perish. A circumstance of this very nature occurred but a few days previous to our leaving Charjooee. A party of three persons travelling from the Orgunje camp lost the road, and their supply of water failed them. Two of their horses sank under the parching thirst; and the unfortunate men opened the vein of their surviving camel, sucked its blood, and reached Charjooee from the nourishment which they thus derived. The camel died. These are facts of frequent occurrence. The Khan of Orgunje, in his late march into the desert, lost upwards of two thousand camels, that had been loaded with water and provisions for his men. He dug his wells as he advanced; but the supply of water was scanty. Camels are very patient under
- 53. thirst: it is a vulgar error, however, to believe that they can live any length of time without water. They generally pine, and die on the fourth day, and, under great heat, will even sink sooner. After a day’s detention to rest the camels, we marched at sunrise, and continued our progress, with a short halt, till the same time next day. We journied thirty-five miles, and alighted at a fetid well called Seerab; and from well to well we had no water. We appeared to have lost the great sand hills in our advance westward. The desert, though it had the same features as before, now presented an undulating and uneven country of sand, partially covered with shrubs. The soil was salt in some places; but the water of the well was good enough after it was some time drawn. Our Toorkmun Sirdar made his appearance shortly after our arrival, to claim his cup of tea; and never was a schoolboy more fond of sugar than this hoary-headed Toorkmun. I used to give it to him to have the pleasure of seeing him grind it, though some of the merchants wondered at our wasting it on such a person. I always felt the happier in the company of this man, for I looked upon him as the only bond between us and the barbarians we were to encounter. He used also to tell us the news of the caravan, and all the particulars of the country, which he even knew that we noted down. Ernuzzer did not deceive us, and the tea and the sugar which he consumed, were but a small tax for his service. In return for these favours, he promised to give me a bonne bouche, when we reached the first camp of the Toorkmuns; and when I expected nothing else than “kimmiz” or “boozu,” mare’s milk or fermented liquor, he brought me camel’s milk, which is the only drink of the Toorkmuns. The milk is mixed with water, and the cream is then drawn off. It is called “chal,” and has a salt, bitter taste. The thinner part of the milk is considered a grateful draught by the people, but to me it tasted sour and acrid. I believed that the Uzbeks and Toorkmuns drank mare’s milk and fermented liquors; but these are unknown in Bokhara, and only peculiar to the Kuzzaks and Kirgizzes, between that city and Russia. A caravan is a complete republic; but I do not believe that most republics are so orderly. Of our eighty camels every three or four belonged to different individuals; and there were four Cafila-bashees. Still there was no disputing about the arrangement or order of the march; and it is a point of honour, that the one shall at all times wait for the other. If a single camel throws its load, the whole line halts till it is replaced; and one feels pleased at such universal sympathy. These feelings make it agreeable to travel in a caravan, for the detentions are much fewer than would really be imagined. The more I mingled with Asiatics in their own sphere, and judged them by their own standard, I imbibed more favourable impressions regarding them. One does not see in civilised Europe that generous feeling, which induces the natives of Asia, great and small, to share with each other every mouthful that they possess. Among Mahommedans we have no distinction of gentleman and villain,—at least, so far as hospitality is concerned. The khan fares as simply as the peasant; and never offers to raise a morsel to his lips till he has
- 54. shared it with those near him. I myself frequently have been partaker of this bounty from rich and poor, for nothing is enjoyed without society. How different is the feeling that besets the sots of lower society in Britain! Nor is this good fellowship among the Asiatics confined to the travelled merchant: it is to be found in the towns as well as the country. It is a pity that civilisation, with all its advantages, does not retain for us these virtues. Barbarians are hospitable, civilised men are polite; but hospitality added to politeness makes it more acceptable. A caravan is an interesting scene at all times; and the shifts of the pious to prevent its detention in the Toorkmun desert were not unworthy of notice. The line was too extended to sound a general halt for prayers; and at the appointed time, each individual was to be seen on the back of his camel or in his pannier, performing his orisons before the Deity, in the best manner which he could accomplish them. The laws of the Prophet admit of a true believer being cleansed by sand, where there is no water; and the back of a horse or a camel is as legitimate a position for prayer, as the most splendid mosque of a city. The busy scene on our reaching the halting-ground in the evening, was both lively and entertaining. The Uzbeks, like ourselves, do not water their horses when they are warm: in this journey, we had no sooner arrived than we again took our departure, the horses were therefore permitted to quench their thirst; and to prevent any evil effects from the water, the animal was immediately mounted and galloped at speed over hill and dale, for miles. This brought the water, as the Uzbeks would tell you, to the heated temperature of the animal’s body. The easy carriage of some of the cavaliers, and the light saddles which they rode,—some of them little larger than racing saddles,—imparted an interest to these scampering freaks which was most exciting. Our next march brought us at midnight to Oochghooee, or the Three Wells, which we had great difficulty in finding. We wandered to the right and to the left, and the Toorkmuns dismounted in the dark, and felt for the pathway with their hands among the sand. We had almost despaired of recovering it, and were preparing for our bivouac, when the bark of a dog, and a distant answer to our repeated calls dispelled our anxiety, and we were soon encamped at the well. We here found a few wandering Toorkmuns, the first we had seen since leaving the Oxus. The well was bitter; but these shepherds seem indifferent to the quality of the water. The country continued to change still further as we advanced, becoming more flat and free from sand, but still running in alternate ridges and hollows. In these we discovered some small red sharp-edged pebbles, not unlike iron pyrites; nor did the wells which were dug in them, yet exceed the depth of thirty feet: in the Indian desert they are 300. The Toorkmuns rallied round us next morning, and we had the freest intercourse with them; for they were quite ignorant of our character, and the presence of one of their own tribe, our Toorkmun Ernuzzer, proved a sufficient attraction to these “children of the desert.” They spoke of the piercing cold of the winters in this country; and assured us that the snow
- 55. sometimes lay a foot deep. We ourselves had experienced a depression of ten degrees in the temperature since leaving the Oxus. We were now informed that we were approaching the camp of the Khan of Orgunje, which, it appears, was on the banks of the Moorghab, or Merve river, considerably below the place of that name, and about thirty miles distant from us. We set out at noon, and by the time the sun had set, found ourselves among the ruins of forts and villages, now deserted, which rose in castellated groups over an extensive plain. I have observed that we had been gradually emerging from the sand-hills; and these marks of human industry, which we had now approached, were the ancient remnants of civilisation of the famous kingdom of Merve, or, as our historians have erroneously called it, Meroo. Before we had approached them, we had not wanted signs of our being delivered from the ocean of sand, since several flocks of birds had passed over us. As the mariner is assured by such indications that he nears land, we had the satisfaction of knowing that we were approaching the water, after a journey of 150 miles through a sterile waste, where we had suffered considerable inconvenience from the want of it. We were not yet within the pale of habitations; but after a cool and pleasant march, over a perfectly flat and hard plain, every where interspersed with forts and ruins, we found ourselves, about nine in the following morning, at a large Toorkmun camp, (or, as it is called, an Oba,) near the banks of the Moorghab. The name of the place was Khwaju Abdoolla, and the whole colony sallied forth to meet the caravan. We took up a position on a hillock about two or three hundred yards distant; and the merchants instructed us to huddle together among themselves, and appear lowly and humble. We did so, and the Toorkmuns of the encampment soon crowded around us, begging for tobacco, for which they brought loads of the most luscious melons, that we cut up, and enjoyed in the company of camel drivers and slaves, braving the sun, though I cannot say to the detriment of our already sun-burned complexions. It now was discovered that the Orgunje camp lay on the other side of the river, which was not fordable but in certain places; and the merchants decided that they themselves, with all the Cafila-bashees, should forthwith proceed in person to the spot, and use their utmost to conciliate the officer in charge, for the Khan had returned within these few days to Khiva. Their great object seemed to be to effect a discharge of the duties in the spot where they were now encamped, since no one relished trusting their property within reach of an Orgunje detachment. If the party prayed for success, I can add that we were equally fervent, and the deputation set out accordingly with the good wishes of every one. We were left among the “oi polloi” of the caravan; and when night came, stretched our felts under a clear and cloudless sky, and slept without fear or anxiety from our man-selling neighbours. This state of security among such people and countries is very remarkable; but a Toorkmun, though he can engage in a foray, and execute it with unexampled address, cannot commit a theft in a quiet way, which is not congenial to his nature.
- 56. I have now a little leisure to speak of the desert which we had traversed on our route to the Moorghab. In a military point of view, the scarcity of water is a great obstacle. In some places the wells were thirty-six miles apart; and, generally, the water was both bitter and scanty. The water which we had transported with us from the Oxus was not less nauseous than that of the desert; for it must be carried in skins, and these must be oiled to preserve them from bursting. The grease mixes with the water, which latterly became so tainted that the horses even refused to drink it. There is nothing of which we feel the want so much as good water. In the march, several people of the caravan, particularly the camel-drivers, were attacked with inflammation of the eyes; I suppose, from the sand, glare, and dust. With such an enumeration of petty vexations and physical obstacles, it is dubious if an army could cross it at this point. The heavy sandy pathways, for there are no roads, might certainly be rendered passable to guns, by placing brushwood on the sand; but there is a great scarcity of grass for cattle, and the few horses which accompanied the caravan, were jaded and worn out before they reached the river. A horse which travels with a camel, has great injustice done to him; but an army could not outstrip the motions of a caravan, and fatigues would still fall heavily upon them. History tells us, that many armies have fought in and crossed this desert; but they consisted of hordes of light cavalry, that could move with rapidity. It is to be remembered, that we had not a foot-passenger in our party. Light horse might pass such a desert, by divisions, and separate routes; for besides the high road to Merve, there is a road both to the east and the west. It would, at all times, be a difficult task for a great body of men to pass from the Moorghab to the Oxus, since our caravan, of eighty camels, emptied the wells; and it would be easy to hide, or even fill up these scanty reservoirs. Where water lies within thirty feet of the surface, an energetic commander may remedy his wants, since we have an instance of it in the advance of the Orgunje Khan to the banks of the Moorghab. But after I have written, and, perhaps, diffusely, on the passage of such a desert, I may ask myself, who seeks to cross it, and in the line of what invader it lies? It is not in the route between India and Europe; and if the descendants of the Scythians and Parthians wish to invade and tyrannize over each other, they may do so without, perhaps, exciting even the notice of the “fierce Britons.” The Toorkmun camp, or “oba,” at which we halted, presented to us a scene of great novelty. It consisted of about 150 conical moveable huts, called “khirgahs,” which were perched on a rising ground. There was no order in the distribution, and they stood like so many gigantic beehives, which, if they had not had black roofs, might not be a bad comparison; and we might also take the children as the bees, for they were very numerous. I wondered at the collection of so many rising plunderers. Seeing the Toorkmuns in a body, it may be certainly distinguished, that they have something Tatar in their appearance; their eyes are small, and the eyelids appear swollen. They are a handsome race of people. All of them were
- 57. dressed in the “tilpak,” a square or conical black cap of sheep-skin, about a foot high, which is far more becoming than a turban, and gives to a party of Toorkmuns the appearance of a soldierlike and disciplined body. The Toorkmuns are remarkably fond of bright-coloured clothes, and choose the lightest shades of red, green, and yellow, as the patterns of their flowing “chupkuns,” or pelisses. They sauntered about their encampment in a great state of listlessness; and what have they to do but to live on the proceeds of their last foray? They have but few fields, and one or two individuals may tend their countless flocks at pasture. Their dogs, indeed, perform this office for them. These animals are very docile, but ferocious to a stranger: they are shaggy, appearing to be of the mastiff breed, and bear a high price even among these people. The martial habits of the Toorkmuns appeared in my eyes the more striking, as they had cleared the circle of their encampment of brushwood for about a mile round. It had, I believe, been cut for firewood; but the resemblance to an esplanade, or a parade ground, was none the less on that account. In my notice of the Toorkmuns, I must not now forget the ladies, whose head-dress would do honour to the galaxy of an English ball-room. It consists of a lofty white turban, shaped like a military chako, but higher, over which a red or white scarf is thrown, that falls down to the waist. Some of these Toorkmun females were fair and handsome, adorning themselves with a variety of ornaments, that were attached to their hair, which hangs in tresses over their shoulders. Their head-dress is, perhaps, a little large, but they themselves are generally on a large scale, and as they never veil it becomes them. The other part of their costume is a long gown that reaches to the ancle, and hides both it and the waist, the very standard points of beauty in our country; but so it is, that nations remote from each other differ not more in language and laws than in taste and manners. The party which had proceeded to the Orgunje camp, returned next morning with the deputy of the Yooz-bashee, or the Commander of an Hundred, and his very appearance made the hearts of the merchants thrill with fear. No taxes had been collected before this, and every thing was uncertain. The deputy was an elderly man, with a large “tilpak” stuck on his head, like a regimental cap. He was accompanied by a party of desert Toorkmuns, among whom was a chief, or “aksukal,” (literally, a white beard,) of the great tribe of Saruk. The merchants seated the deputation in the place of honour, addressed the deputy as he had been the Yooz-bashee himself, refreshed him with tea and tobacco (for they now smoked in public), and presented him with silks, cloths, raisins, and sugar, and then proceeded to display their merchandize. Every person made an offering, and we sent two handfuls of raisins and a bit of sugar as our homage. We sat at a short distance in our panniers, and witnessed the whole scene. The Yooz-bashee, as I also must call him, now spoke out to all the members of the caravan, and in the most candid language, said, that he had been directed to levy the lawful tax of one in forty, but that he would dispense with opening the bales. Truth, said he,
- 58. had better be told; for, if I have reason to doubt any of you, I will then examine them, and you will experience the wrath of the Khan of Orgunje, my lord and master. This speech was listened to with terror; some, I believe, actually said that they had more goods than they really possessed; and, as far as I could judge, no one deviated from the truth. Pen and ink were called for, and the congress proceeded to make a list of the merchandize, which was no easy matter. While the merchants were disputing about tillas, and flattering the Yooz-bashee, we had taken up a quiet position, and even pretended to be wrapped in sleep. I never was more awake in my life, and was near enough to hear and see every thing. There were several questions put regarding us, and the principal merchants spoke with earnestness and kindness. We had never instructed them, but they now chose to denominate us Hindoos from Cabool, who were proceeding on a pilgrimage to the flames of Bakoo, on the Caspian. We had been successively Englishmen, Afghans, Uzbeks, Armenians, and Jews, and they now denominated us Hindoos. These people are very simple; nor do they ever interrogate closely. Shortly after the subject of our character and objects had been discussed, the Toorkmun Aksukal rose from the party, and most unwelcomely seated himself by us. “Aksukal,” as I have said, means white beard, though this personage had a black enough plumage to his chin: he wore a splendid scarlet pelisse, and never did our national uniform appear to me more formidable than on his person; for he might have proved himself “a very Tartar” under his British colours. He spoke a little Persian, and said, “You are from Cabool?” to which I gave a nod of assent. The Doctor stretched himself back in his panier, and our visitor addressed himself to an Afghan, one of our people, of which I was glad, since it would keep up the illusion. It is said that the natives of Orgunje are, of all the tribes in Toorkistan, most hostile to Europeans, as well from their vicinity to Russia, as their knowledge that the Persians, who threaten their country, are assisted by them. They of course know nothing of the different nations of Europe, and look upon all Europeans as their enemies. I was not sorry when the Toorkmun chief selected another group, and that this patriarchal “white beard” had made no discovery, even after seeing us, and entering into conversation. The whole scene appeared to me a perfect riddle, for we ourselves had mixed with the Toorkmuns of our party as Europeans; and our real character was known to every individual of the caravan. Fear may have prevented some of them from making a full disclosure, but it was very creditable; since I have reason to believe that the people of the Orgunje Khan would not have willingly extended their favour towards us. We, however, had one instance of bad feeling, in a quarter where we least of all looked for it, at the hands of our Cafila-bashee. He required money to pay the just taxes on goods, which he had at the outset hoped to smuggle, and though all settlement had been made between us, and he had nearly received the full hire of his camels, he sent in the middle of the confusion to say, that the caravan would be detained on our account, if we did not lend him some tillas. What a moment,
- 59. and what a trial for the temper. It was useless to complain of ungenerous treatment, and it would have been worse to show that we felt it. I considered a couple of tillas enough to give the wretch, though we had provided ourselves with some three hundred of them, which I knew might befriend us, where men are sold and bought like sheep. Evening advanced, and our transactions with the Orgunje Yooz-bashee drew to a close. The commander of a hundred carried off two hundred golden tillas, and all the merchants accompanied him to his horse, and saw him beyond the limits of our camp. Such is the dread of authority, and the power of the meanest man who wears it. In the dusk the merchants came to visit us, and to relate the affairs of the day over a cup of tea. We had to thank an Uzbek, named Ullahdad, and Abdool, a Persian; but we had to make some acknowledgment to all, for we had now become intimate with every body. Whenever the horsemen of the caravan passed us on the road, they would shout out to us, “Ah, Meerza! how are you?” with all the consecutive compliments of their language. Little did many of them know, that the name of “Meerza Sikunder,” or the secretary Alexander, which they had given me, was so well merited; since I took every opportunity that I secretly could to use the pen and ink, and give a secretary’s account of all their proceedings. On this day I felt pleased with mankind, for we were now free to prosecute our journey. The Bokharees assured me that they were interested in our favour, from the commands of their minister the Koosh Begee; and the Persians, of whom there were many in the caravan, dreaded the friendship between Abbas Meerza and the English. Individually, I did not presume to believe that either of these great personages bestowed much care upon us, but it was pleasant to know that such were the opinions of our companions.
- 60. CHAP. XIII. CONTINUATION OF THE JOURNEY IN THE TOORKMUN DESERT. On the morning of the 29th of August we moved at dawn, with buoyant spirits, and followed the course of the Moorghab, or river of Merve, for twelve miles before we could cross it. We found it about eighty yards wide and five feet deep, running within steep clayey banks, at the rate of five miles an hour. We crossed by an indifferent ford, over a clay bottom with many holes. There was no village; but the place is called Uleesha. This river rises on the mountains of Huzara, and was long believed to fall into the Oxus or the Caspian. Both opinions are erroneous, since it forms a lake, or loses itself in one, about fifty miles N.W. of Merve. This river was formerly dammed above Merve, which turned the principal part of its waters to that neighbourhood, and raised that city to the state of richness and opulence which it once enjoyed. The dam was thrown down about forty-five years ago, by Shah Moorad, a king of Bokhara, and the river only now irrigates the country in its immediate vicinity, where it is covered with the tenements, or “obas,” of the Toorkmuns; for there are no fixed villages. These people cultivate by irrigation, and every thing grows in rich luxuriance. The Juwaree (holcus sorghum) has a stalk thicker than a walking-stick, and in the uncultivated parts there is the richest fodder for cattle and the finest thorny shrubs for the camel, an animal which is here found in vast herds. Above Merve the country is called Maroochak, and said to be unhealthy: there is a proverb, at least, which runs thus,—“Before God gets intelligence, the water of Maroochak has killed the man.”[2] This river is the Epardus of Arrian, a word which, I observe in one author, is said to mean irrigator,—nor is it here misapplied. The historian would even appear to have been acquainted with its course; for we are told that the Epardus “hides its streams in the sand, as did many other great rivers.”[3] The transition which we had experienced, from a sandy desert to the verge of a running stream, was most gratifying; every one seemed delighted, and even the animals appeared to feel the change. Throughout the day the banks presented a spectacle of merriment and joy; the Toorkmuns plunging into the water with their horses, and the greater part of the caravan sporting about in the stream. We hit upon a contrivance, which contributed not a little to our sport, and produced a “tunga,” or the third part of a rupee, which was to be the reward of the person who could first cross the river. The enormous sum was solemnly vested in a committee; I believe even the blessing was said; and sixteen competitors appeared on the occasion. It was won by a Toorkmun of Shurukhs, who had the art of running quickest in the deep water.
- 61. We were now in the vicinity of Merve, and several members of the caravan, on their approach to the river, declared that they had a view of the elevated mound of its ruined castle. I sought in vain, but the other spectators were looking for their native city, and wished, perhaps, to persuade themselves that they beheld it. I listened to the tales of valour which these people related to me of one Bairam Khan and a chosen body of seven hundred, that long resisted the arms of the Uzbeks of Bokhara, till Shah Moorad finally subdued them by a stratagem in war, and forcibly transferred the whole population to his capital. Nor was I less gratified to hear the patriotic tale of the heroines of Merve, the wives and daughters of the gallant band. It is recorded, and it is believed, that on one occasion, when the forces of Bokhara invaded the land of Merve, during the absence of Bairam Khan and his knights, these fair ones embodied and appeared in the field. The Uzbeks were intimidated at the sight of troops whom they believed they had surprised, and fled with precipitation, leaving the heroines of Merve their virtuous victors: nor is this a solitary instance of female triumph over man. The people of Merve, in their loss of country and liberty, retain the same reputation for valour which characterised their ancestors; and, to this day, when they quit the country, their valiant partners are held in Bokhara as a pledge of their fidelity, and may on no account cross the Oxus. Some circumstances here came to our knowledge that called for prudence and caution, and which appeared to excite the justest alarm. As our party had arrived at the Orgunje camp, they found the chief in the act of despatching a body of 350 Toorkmuns on a foray to the frontiers of Persia. Our friends had arrived even in time to give these barbarians the usual “fatha;” for, whatever they felt, it was impossible to appear otherwise than pleased at their intentions. The Yooz-bashee in their presence charged the robbers to be of good cheer, and remember the good work on which they were to be engaged, and the golden “tillas” to be reaped in the country of the Kuzzilbash. “Go,” exclaimed he, “and bring the Prince Royal of Persia, Abbas Meerza himself to the feet of the Khan Huzrut.” The Allamans mounted in a moment, and one of the merchants, who seemed to have had his senses about him, begged that the formidable band would spare our caravan. The Yooz-bashee gave instructions to that effect; but they now shook their heads, and seemed but little disposed to put the honesty of such men to a trial. They turned over all the bearings of the case in their minds, and looked very woful. As a member of the party, I could not help asking for information on the blessings which they had been called on gratuitously to bestow on such a horde. “Fatha,” said a Persian, “I did take the name of the holy Prophet, but it was that these man-selling scoundrels might never return.” Our conductor Ernuzzer himself said, that it was an abomination to have made such a use of the first sentence of the Koran; so easy is it to make the ritual of a faith correspond with the wishes. The doctor and myself, I believe, were the only members of the caravan who would have liked to have a peep at the ferocious Allamans; but I dare say it was
- 62. fortunate that our curiosity was not gratified. Since such a horde of plunderers was abroad, it was decided that we should march upon Shurukhs, a large Toorkmun settlement, and there await the result of their expedition, which the merchants of the caravan had more desire to hear of than witness. The party had been instructed to proceed by easy marches, as the Toorkmuns always do in their forays, and was expected to return on the tenth day. On the 30th of August we retraced the greater part of yesterday’s route, and travelled down the opposite bank of the river for about sixteen miles, when we again halted among the Toorkmuns in their native state, at an encampment called Kunjookoolan. We here mixed among them without hesitation, and gathered many particulars concerning them. The Toorkmuns are Toorks; but they differ from the Uzbeks, and are entirely devoted to a pastoral life. There are several great tribes of the race, all of whom claim a common origin; we had seen the Ersarees on the Oxus, and were now mingling with the tribe of Saruk, beyond which are the Salore. Towards the Caspian lie the Tuka, Goklan, and Yumood, all of them great tribes, and of which I shall speak as we advance. Among our Saruk acquaintances there was one individual who had passed his days in making incursions into Persia, and in his odious traffic had acquired a perfect knowledge of the language of that country, which enabled me to learn the genuine sentiments of a Toorkmun robber. His name was Noornyaz; and in his forays he had accompanied the largest and smallest parties: he had, indeed, only returned with three captives, that had been secured by the small number of six horsemen. He described the manner of approaching Persia by slow and short stages, and that, after reaching the frontiers, they frequently hovered for days in sight of a fort to watch for a favourable opportunity of capture. If none presents itself, they make a dash in upon the fields in the morning, as the shepherds and husbandmen pursue their occupations, and bear off with speed whoever they may seize. If hotly pursued, they relinquish a spare horse with which every two individuals is provided, and carry off the more valuable slave. In such a transient expedition every thing depends on the fleetness of their horses, and the Toorkmuns accordingly bestow the utmost care upon them. My Toorkmun acquaintance said, that he was now preparing his horse for another foray, which consists in exercising him most severely after a long abstinence from food and water, which brings the animal to a matchless state of hardihood. They do not permit them to taste green forage, but confine them to dry food, which they believe hardens the flesh. They sweat them till their fat entirely disappears, and of this they judge by the quantity of water which the horse drinks, since it is very small if his flesh has been properly reduced. The Toorkmun horse, with such a training, far surpasses in bottom those of Europe and Arabia; but he is a coarse-looking animal, and has neither the sleekness nor beauty of coat which we see in India or our own country. Since the life and fortune of the Toorkmun are identified with the goodness of his horse, we can account for the care and attention that he bestows upon him. The little food to which he is
- 63. inured enables his rider to provide with ease for his own wants: he carries the grain for the horse and himself, as well as bread and flour: in his advance he sometimes buries these in a well-known place, till he shall return from the foray; and when the Toorkmun retreats into his native desert, he is thus supplied with provisions, though he may have been weeks from his camp, which he shares with the victims of his capture, whom he drags into miserable servitude. In the catalogue of human miseries there are few more severely felt, and the consequences of which are more destructive to domestic happiness, than the cruel system of man-stealing. Great as are the miseries produced by this, the hordes who engage in it appear to derive none of the luxuries or enjoyment of human life from such an occupation, and live in rags and penury, seemingly without advantage from their devastations. The terror which the Toorkmuns inspire among the people of the neighbouring countries is fearful, nor is this surprising, since they evince such fortitude and persevering energy in their dangerous occupation. We cannot fail to admire their address, and acknowledge their valour, at the time that we deplore the lot of the unhappy country on which they display their prowess. The manners and customs of the Toorkmuns, in the odious practices which they pursue against their fellow man, sap the best principles of human nature, and we consequently find this people wanting in much of the honour which is often seen among half-civilised nations. “A Toorkmun,” the people will tell you, “is a dog, and will only be kept quiet with a bit of bread, like a dog: give it then, is the doctrine of the traveller, and pass on unmolested.” They have likewise the character of being perfidious and treacherous, nor is it altogether unmerited. The Persians have endeavoured, but without success, to put a stop to these reckless inroads of the Toorkmun, but he himself lives in a desert where he is safe, and is encouraged by the ready sale which he finds for his captives in the favoured countries that lie beyond his own desolate region. In their expeditions into Persia, some Toorkmuns are occasionally captured, and an exorbitant ransom has been placed upon their heads, but yet they have been redeemed by their kinsmen. A Toorkmun passes his life either in a foray, or in preparing for one; and it is a disgraceful fact, that the chiefs of Khorasan have long and unnaturally leagued with these enemies of their religion and their country, to barter a still greater portion of unfortunate Persians into their hands, and eternal slavery. Avarice is the most baneful of our vices. Now that we were beyond the power of the Orgunje troops, the merchants of the caravan assembled in conclave to bemoan the loss of their money in a new tax, and to devise ways and means to recover it. It appeared to the majority, that the Firingees, that is, ourselves, should bear a portion of the burthen, and the assemblage waited on us in the evening to express their wishes, and request that we would bear one fourth of all the duties. Since the payment of the regular customs had induced the officer to forego the usual fee on each pair of panniers, we had certainly escaped every kind of tax, and this was evidently owing to the
- 64. wealth and size of the caravan with which we were travelling. The Orgunje officer, too, it was now stated, had been bribed to the amount of ten tillas. It appeared both reasonable and just that we should bear our share of this outlay, and I therefore offered the usual tax of a tilla on each of our camels, since it would tend to diminish the general expenditure of the caravan. It was a point that called for the exercise of discretion and judgment, since a total denial might have converted a friendly into a hostile party; and, on the other hand, it at all times behoved us to be most sparing in our expenses. In the present instance, I had the good fortune to conciliate by my concession the principal merchants of the party. There were several who still called on us to pay a fourth share of the tax; but as I ascertained that no additional expenses had been incurred on our account, and the duties would have been levied whether we had been present or absent, I declined compliance, and stated to them that we were travellers, and their guests, in a foreign land, and hoped for their forbearance and justice. The Toorkmun chief, our friend Ernuzzer, appeared at this stage of our conversation, to enter his protest against such an outrage to hospitality as the demand which had been made upon us; but I had already made up my mind, and passed my word. The rights of the stranger are much respected among these people, and the cry of the many died away into the feeble vociferations of the poorer traders, whose scanty means made them feel more heavily the levy that had been made upon them. In one respect the character of an European in such countries is ill suited for a traveller; he is believed to possess boundless wealth, though he may be sunk in poverty; an Asiatic, in his expenses, has nothing in common with the opinions of an European. We now commenced our march in the desert westward of the Moorghab river, and made a progress of thirty-seven miles. The tract was entirely different from the opposite side, and about the middle of the journey the desert changed into a level, hard, flat surface, which it ever afterwards preserved. The camels moved up in four strings abreast of each other, and we continued to advance in that order. The tract put me much in mind of the Run of Cutch, though there were patches of bushes, which are not to be seen in that most singular region.[4] The country was destitute of water, but there were many remains of caravansarais and cisterns that had been built by the philanthropic Abdulla Khan of Bokhara. In this neighbourhood, and more particularly while on the banks of the river, we witnessed a constant succession of whirlwinds, that raised the dust to a great height, and moved over the plain like water-spouts at sea. In India these phenomena are familiarly known by the name of devils, where they sometimes unroof a house; but I had not seen them in that country either of such size or frequency as now prevailed in the Toorkmun desert. They appeared to rise from gusts of wind, for the air itself was not disturbed but by the usual north wind that blows steadily in this desert. As we halted in the morning of the 1st of September, at a ruin which bore the name of Kalournee, we descried the hills of Persian Khorasan. In the direction
- 65. where they rose I had observed the atmosphere to be clouded since we reached the banks of the Moorghab, and we might have perhaps seen them sooner, though they still appeared in the haze of distance. As we discovered these mountains at sunrise, a magnificent mirage shone in the same direction. One could trace a river, and its steep and opposite banks; but, as the sun ascended, the appearance vanished, and left the same flat and cheerless country in which we were now encamped. The high banks of the river had no existence, and the water was but vapour set in the rays of light. As we approached Shurukhs, we could distinguish a gradual, though almost imperceptible rise in the country. We exchanged the shrubs that I have before described, for the tamarisk and the camel’s thorn, which does not grow in the desert. The most singular of the plants which a new zone presented to us, was one called “gyk chenak” in the Toorkee language, which literally means, the deer’s cup. It grows like hemlock or assafœtida (and has as bad a smell), only that a leaf, shaped precisely like a cup, surrounds each knot or division of the plant’s stalk. In this natural bowl the rains of spring are collected, and supply the deer with water. Such is the popular belief, and such is the name. We afterwards saw a plant not unlike the deer’s cup among the hills eastward of Meshid. A gum, like tallow, exuded from it, and it shot up as an annual among the high lands. We had been treading in our last marches on the very ground which had been disturbed by the hoofs of the Toorkmuns who were advancing on Persia. It was with no small delight that we at last lost our traces of the formidable band, which we could discover had branched off the high road towards Meshid. Had we encountered them, a second negotiation would have been necessary, and the demands of robbers might not have been easily satisfied. “Allamans” seldom attack a caravan, but still there are authenticated instances of their having murdered a whole party in the very road we were travelling. Men with arms in their hands, and in power, are not to be restrained. After losing all traces of this band, we came suddenly upon a small party of Allamans, seven in number, who were returning from an unsuccessful expedition. They were young men, well mounted and caparisoned, in the Toorkmun manner; a lance and a sword formed their arms; they had no bows, and but one led horse. Their party had been discomfited, and four of them had fallen into the hands of the Persians. They told us of their disasters, and asked for bread, which some of our party gave them. I wish that all their expeditions would terminate like this. We reached Shurukhs at sun-rise on the 2d, after having performed a journey of seventy miles in forty-four hours, including every halt. During this period we had only marched for thirty-two hours, and the camels sometimes stepped out at the rate of two and a half miles an hour, which I had never before seen. All the camels were males, since they are believed to undergo fatigue better than females. Our caravan alighted round an old tomb, with a lofty dome, and it was unanimously
- 66. decided, that so long as the Allamans were abroad, it would not be prudent to prosecute our journey. It was therefore resolved to sleep in Shurukhs (to use a phrase of their own), the greatest haunt of the Toorkmun robbers; a paradox truly, since we were to settle among thieves to avoid the thieves abroad. We, however, possessed but humble influence in the party, and had only to meet the general wish. The merchandize was piled round the tomb, the people took up a position outside of it, and at night the camels and horses formed a triple barrier. Such were the arrangements for our protection, and, as will be seen, not more than were necessary. The Toorkmuns crowded among us during the day, and brought tunics of camel-cloth for sale, which were readily purchased; but there was not an individual of the caravan who trusted himself at a distance from it: and how could it be otherwise, when we hourly saw the “Allamans” passing and repassing in front of us, and knew that the chief subsistence of the people was derived from these “chupaos?” The Toorkmun settlement of Shurukhs consists of a small and weak fort, almost in ruins, situated on a hillock, under cover of which most of the inhabitants have pitched their tenements. There are a few mud houses, which have been built by the Jews of Meshid, who trade with these people; but the Toorkmuns themselves live in the conical houses or khirgahs, peculiar to their tribe. They are constructed of wood, surrounded by a mat of reeds, and covered in the roof with felts, that become black with soot. Shurukhs is the residence of the Salore Toorkmuns, the noblest of the race. Two thousand families are here domiciled, and an equal number of horses, of the finest blood, may be raised in case of need. If unable to cope with their enemies, these people flee to the deserts, which lie before them, and there await the termination of the storm. They pay a sparing and doubtful allegiance to Orgunje and Persia, but it is only an impending force that leads to their submission. When we were at Shurukhs they had a Persian ambassador in chains, and refused to grant a share of the transit duties to the Khan of Orgunje, which they had promised in the preceding month, when that chief was near them. These are commentaries on their allegiance. The Salore Toorkmuns are ruled by twelve aksukals, the heads of the different families; but they acknowledge no particular allegiance to any individual person. The country around Shurukhs is well watered by aqueducts from the rivulet of Tejend, which is a little brackish, but its waters are usefully employed in fertilising its fields. The soil is exceedingly rich, and possesses great aptness for agriculture; the seed is scattered, and vegetates almost without labour. The harvest is rich, and they reap it, like true republicans, without a tax. The inhabitants repeat a tradition, that the first of men tilled in Shurukhs, which was his garden, while Serendib or Ceylon was his house! There is not a tree or a bush to enliven the landscape, for the Toorkmuns despise gardening. The crops of wheat and juwaree are here most abundant, and the melons are only inferior to those of Bokhara.
- 67. Two days after our arrival at Shurukhs, and when I venture to say we had often congratulated ourselves at the near prospect of successfully terminating our journey, we experienced an alarm that at least showed our congratulations were premature. One of the Toorkmun chiefs of the place appeared in our part of the encampment, and summoned the Hajee, one of our people, to attend him, near enough for me to overhear their conversation. He commenced a long list of interrogatories regarding us, and stated that he had heard from persons in the caravan that we possessed great wealth, and had travelled into the remotest parts of Toorkistan. Such being the case, continued he, it was impossible for him to grant us permission to prosecute our journey, until the commands of Ullah Koli, Khan of Orgunje, were received concerning us. This formidable announcement would even have appeared more frightful, had not the Toorkmun added on his departure, that his fellow chiefs were ignorant of our presence in the caravan, and that we might perhaps consider his good wishes not unworthy of being purchased. The matter was however serious, since it discovered that there were persons in the caravan who were ill-disposed towards us, and it was certain that the Toorkmuns had the power of enforcing all which the person in question had threatened. Immediate measures were necessary, and I lost no time in adopting them. There were five or six merchants of respectability in the caravan, and I went to the two principal persons, whom I have before named, and related the affair to them with perfect candour. I should have gone to Ernuzzer the Toorkmun, but he had in former days lived at Shurukhs, and, in his change to the life of a citizen, had forfeited much of the influence he might be supposed to possess among his countrymen; nor did I even unfold to him the circumstances till we reached Meshid. I observed that the communication equally excited the uneasiness of the merchants, and once more discovered that these people were really concerned for our safety. They poured forth their wrath against the informer, and expressed in unequivocal language the fears which they entertained from the Vizier of Bokhara on one side, and the Prince Royal of Persia on the other. One of the merchants advised that I should immediately produce the firman of the King of Bokhara; but in this I differed, and the opinion of the other was more in consonance with my own judgment. Abdool undertook to negotiate the feeding of the dog of a Toorkmun; but it may be imagined that there was little to cheer us under such circumstances. A cheerful countenance was, however, indispensable, that we might the better meet the difficulties, and, if possible, frustrate the hopes of the villain who had betrayed us. The first piece of intelligence which assailed us on the following morning was the loss of a beautiful little black pony, which had been stolen from his pickets during night. It is customary in this country to chain the horse’s leg to the iron pin, and then padlock it; but we had not adopted this precaution. I regretted this loss more than I might have done a more serious misfortune. The sturdy little creature had followed me from Poona in the centre of India, had borne me in many a weary
- 68. journey, and I cannot tell how much it vexed me to leave him in such a country, and in such hands. The whole caravan assembled to express their regret at the theft, and assured me that I should either have the pony or his value; but they did not understand that in my estimation he stood above all price. I was obliged to turn to other matters, and it was a more solid source of consolation to find that we had satisfied the demands, and silenced the threats of the Toorkmun chief at a most moderate sacrifice. He became master of our stock of tea, and we should have added the sugar, had it been worth presenting; and this peace offering, crowned with two gold tillas (each valued at about six and a half rupees), satisfied a chief who had us in his power. Doonmus, for so he was named, was the “Aksukal” of 300 families, and one of those who share in the plunder and taxation of Shurukhs. We were much indebted in this difficulty to Abdool, who happened to be an acquaintance of the Toorkmun, and whom we had brought over to us by some acts of civility. We might not have escaped so readily from the talons of any of the others; and it was curious that the fellow who had wished to profit by us was the friend of the merchant with whom we were most intimate. This sunshine of our prosperity admitted of our entering with greater spirit into our enquiries regarding the Toorkmuns, and I gathered some characteristic incidents of the people. They are as romantic in their customs of marriage as in their habits of plunder. They do not enter into the conjugal state with the simple forms of Mahommedans; for the communication between the sexes is unrestrained, and attachments are formed that ripen into love. But the daughter of a Toorkmun has a high price, and the swain, in despair of making a legitimate purchase, seizes his sweetheart, seats her behind him on the same horse, and gallops off to the nearest camp, where the parties are united, and separation is impossible. The parents and relatives pursue the lovers, and the matter is adjusted by an intermarriage with some female relation of the bridegroom, while he himself becomes bound to pay so many camels and horses as the price of his bride. If the person be rich, these are generally paid on the spot; but if, as more often happens, he is without property, he binds himself to discharge his debt, which is viewed as one of honour; and he proceeds on forays to Persia, till he has gained enough to fulfil his engagement. His success in these generally converts him into a robber for the rest of his days; and the capture of the Kuzzilbash has now become indispensable to settle in life the family of a Toorkmun. The young lady, after her Gretna Green union, returns to the house of her parents, and passes a year in preparing the carpets and clothes, which are necessary for a Toorkmun tent; and, on the anniversary of her elopement, she is finally transferred to the arms and house of her gallant lover. A circumstance lately happened at Shurukhs, which was repeated to us by many of the people, and exhibits additional examples of the love of liberty, and the despair which is inspired by the loss of it. A Persian youth, who had been captured by the Toorkmuns, dragged out a miserable life of servitude in Shurukhs. He was
- 69. resolved to be free, and chose the opportunity of his master being at an entertainment, to effect his object. He saddled the best horse of his stable, and on the very eve of departure was discovered by the daughter of his lord, who attempted to give the alarm. He drew his sword, and put the girl to death. Her cries alarmed the mother, whom he also slew; and as he was bidding his final farewell to Shurukhs, the master himself arrived. The speed of the horse, which had so often been employed in the capture of his countrymen, now availed this fugitive, who was pursued, but not overtaken; and thus, by an exertion of desperate boldness, did he regain his liberty, leaving his master to deplore the loss of his wife and his daughter, his horse and his slave. I have mentioned that our camp at Shurukhs lay by the shrine of a Mahommedan saint. He flourished 824 years since, under the name of Aboolfuzzul Hoosn, as appears by an inscription on the tomb, and he is yet revered by all the Toorkmuns. If one of them fall sick, he invokes the manes of the saint; if his horse or his camel suffer from disease, he circumambulates his tomb, in the hope and conviction of relief. The Toorkmuns have no mosques; they say their prayers in the tent or in the desert, without ablution, and without a carpet. They have few Moollahs or priests, for the church has little honour among them, and they are but poor followers of the prophet. They have no education to assuage the fiercer passions, which renders the men unsusceptible of pity, and the women indifferent to chastity. The men perform all the out-door employments, and the women work at home. The Toorkmuns are a race of people who court alternate activity and idleness. Abroad they evince the greatest spirit, and at home saunter about in idleness and indolence. They are fond of their horses, and of singing songs in honour of them. At night I have listened to the panegyrics on the feats of the “Chupraslee” and “Karooghlee” horses, the never-ending theme of praise. “Karooghlee” means a warrior as well as a horse, but it describes a famous breed now said to be extinct. “Chupraslee,” though it means but swift, is applied to a particular horse of reputed speed. I longed to record some of these Toorkmun songs, but at Shurukhs we could gather only these few lines:— “I keep an Arab horse for the day of battle, I live on that day under his shade, In the conflict I slay a hero,— Keep an Arab horse, hold a shield of iron. Kurooghlee! “In the day of battle I bend my bow of iron, Erect on my horse, no one can dismount me. I am an only child, I have no brother or sister,— Keep an Arab horse, hold a shield of iron. Kurooghlee!
- 70. “If I breathe, the ice of the mountains melts, The water of my eyes would turn a mill, So said Jonas the Puree,— Keep an Arab horse, hold a shield of iron. Kurooghlee!” After the alarm which we had already experienced in Shurukhs, it was not desirable that we should mingle much with the people; but I had great curiosity to see them, and our Toorkmun Ernuzzer said I was invited to a friend’s house, and I accompanied him without further consideration. I was very agreeably surprised to find these wandering people living here, at least, in luxury. The tent or khirgah was spacious, and had a diameter of about twenty-five feet. The sides were of lattice-work, and the roof was formed of laths, which branched from a circular hoop, about three feet in diameter, through which the light is admitted. The floor was spread with felts and carpets, of the richest manufacture, which looked like velvet. Fringed carpets were also hung up round the tent, which gave it a great finish, and their beauty was no doubt enhanced by their being the work of wives and daughters. On one side of the tent was a small press, in which the females of the family kept their clothes, and above it were piled the quilts on which they slept. These are of variegated coloured cloth, both silk and cotton. From the circular aperture in the roof, three large tassels of silk were suspended, differing in colour, and neatly wrought by some fair young hand. Altogether, the apartment and its furniture bespoke any thing but an erratic people; yet the host explained to me that the whole house could be transported on one camel, and its furniture on another. On my return I expressed my surprise at such comfort, but my companions in the caravan bade me not wonder at such a display, since the Toorkmuns were man-eaters (adum khor), and got their food for nothing. Many a nation has been written down as cannibals on as slight grounds; but the people merely meant to tell me that they lived on the proceeds of man-selling. Before I quitted the tent, the host produced bread and melons, according to their custom, of which we partook, with about fifteen other Toorkmuns, who had dropped in. They cut up a melon with great dexterity and neatness, separate the pulp from the skin, which is not thicker than that of an orange, by a single sweep of the knife, then dividing it into a dozen pieces. I listened for about half an hour to their conversation, the subject of which I could comprehend to be slaves and horses. They took me for a native of Cabool, from the loongee which I wore as a turban, nor did I undeceive them. They all got up as I left, and bade me good-bye with all the respect of a good Mahommedan. They might not have injured me had they known the truth, but they would have detained me with endless questions; and, as it was, I saw their customs without inconvenience. I was never so much struck with the Tatar features as in this assemblage. The Toorkmun has a skull like a Chinese, his face is flat, his cheek bones project, and his countenance tapers to the chin, which has a most scanty crop of hair. He is by no means ugly, and his
- 71. body and features are alike manly. Their women are remarkably fair, and often handsome. I might have followed up my acquaintance, and dined with the Toorkmuns in the evening; but, since I did not do so, I shall describe their feast from Toorkmun authority. When they invite a stranger to dinner, they send to say they have killed a sheep. They are not very choice in their cookery. Their cakes are baked about two feet in diameter, and an inch thick, of the coarsest flour, and generally mixed up with slices of pumpkin. These are always eaten fresh. When the party assembles, the cloth is spread, and each person crumbles down the piece of cake which is laid before him. The meat is then brought, which consists of one entire sheep, boiled in a huge Russian pot. They separate the flesh from the bones, and tear it into as small pieces as the bread, with which it is mixed. They shred about a dozen of onions, and throw the whole mess into the pot where the meat has been boiled, and mix it up with the soup. It is then served out in wooden bowls, one of which is placed before every two persons. Their mode of eating is as singular as that of preparation; they fill their open hand, and commencing from the wrist, lick it up like dogs, holding the head over the bowl, which catches all that falls. Each of the two in his turn fills his hand, and holds his head over the bowl. Melons follow, and the repast concludes with a pipe of tobacco. The women do not eat with the men. On the seventh day after our arrival at Shurukhs, when every one was enquiring about the “Allamans,” or robbers, who had preceded us, they began to drop in upon us by twos and threes, with their horses lame and jaded, and by evening upwards of a hundred had arrived. They stopped by the caravan, and gave us a glowing account of their foray, congratulating themselves in boastful strains at their success. They had made their descent near Meshid four days previously, about ten in the morning, and rode up to the very walls of the city, driving men and animals before them. Not a soul appeared to arrest their progress; and when they numbered their spoil a few miles from the city, they found 115 human beings, 200 camels, and as many cattle. Since then they had returned without haste, and now skirted Shurukhs for refreshment. On the way they had already divided their booty. A fifth was given to the Khan of Orgunje, and the party had to congratulate themselves at the number of able-bodied men, and the few white-beards, old ones, among their prisoners. Returning through the hills, they encountered the videttes of a small party of horse, who are stationed to give information at Durbund, which lies between Shurukhs and Meshid. In the scuffle, one of the Toorkmuns was wounded, and they captured one of the videttes and fifteen horses. They put the unfortunate Persian to death, as an offering to God for the success which attended them; since they pretend to consider the murder of a heretic Kuzzilbash as grateful to the Almighty; and they generally kill most of the old persons who fall into their hands, as a propitiatory offering to the Creator. The Toorkmuns, indeed, defend their capture of these unfortunate human beings, on
- 72. the ground of their conversion to a true religion, and consequent salvation. Unhappily for mankind, the history of the world presents us with too many and similar instances of this mistaken and religious zeal. The Spaniards pursued their conquests in the New World under the specious pretence of disseminating Christianity; they sacked the empires of Mexico and Peru, and butchered their inoffending inhabitants; while their priests impiously blessed their inhuman outrages.[5] They, too, like the Toorkmuns, propitiated their king by a present of a fifth of their spoil. Human nature, under king or khan, is the same in all countries, whether we contemplate the frenzy and avarice of the Spaniards in America, or the roaming Toorkmun in the Scythian deserts. The opportunity which was afforded us of seeing these robbers, inspired a good opinion of their courage, for many of them were indifferently armed. They all had swords, most had light, long lances, quite different from those used by the Uzbeks, and a few had small matchlocks. Their horses looked quite done up, and walked as if on beds of gravel; but they had been thirteen days in motion, with scanty food and much work. While we admire the courage of these men, what shall we think of the Persians, who are encamped within two days’ journey of Meshid, under the heir-apparent of their throne, and numbering an army of twenty thousand men? The return of the Orgunje Allamans should have now settled our movements, but some timid being spread a rumour that half of the robbers yet lay in wait for our caravan on the Persian frontier. Our departure was therefore still put off, and I cannot say that I felt comfortable in such quarters. We had no tent or shelter for ten days but the rotten walls of an old tomb, which were infested with reptiles. Though our bed had always been the ground, and we had long ceased to feel the aches which one experiences from an occasional bivouac in civilised life, we could not now spread a carpet, lest we should appear too rich among the Toorkmuns, who stated in upon us at all times, and frequently asked us questions. Our bread, too, had been ten times coarser than “bannocks of barley meal,” not half so palatable. We could with great difficulty read or write for a single hour during the day, and the time passed as heavily as possible, exhausting our patience. During our detention, one of the camels was said to have gone mad, whether from ennui or some more cogent cause I knew not. The poor creature foamed at the mouth, groaned, and refused its food. The case was referred to us, as he was pronounced to be possessed of a devil; but of course without avail. At length they fell on the expedient of frightening the camel, by clashing a lighted torch before his eyes and body, and kindling reeds and furze under his nose. They also passed a red-hot iron over his head; and the animal assuredly improved under this rough treatment, of burning the devil who had lodged in so ugly a creature. At length, on the 11th of September, after a detention of ten long days, we joyfully quitted Shurukhs at sunrise. The Toorkmuns maintained their character to the last.
- 73. After giving us leave, and agreeing to tax us at the first stage, they waited till we had fairly started, and then sent orders to stop the caravan. They demanded a tilla and a half on every camel; which is the customary transit duty for an escort to the Persian frontier. The party came only a few miles, and then returned, tired of escorting; nor were we sorry to get so well rid of them. Our caravan had now been increased by the junction of two others, which had come up during our stay, and formed a numerous body: but I fear there were more timid than fighting hearts among us. There were men, women, and children; merchants, travellers, pilgrims, and emancipated slaves. There were Uzbeks, Arabs, Persians, Afghans, Hindoos, Jews, natives of Budukhshan and Cashmeer; Toorks and Toorkmuns; a Nogai Tatar, a wandering Kirghiz from Pameer, and ourselves, natives of Europe. Last, not least, was a young Persian girl, about fifteen years old, whom we had picked up at Shurukhs, and who was said to be of exquisite beauty. She had been captured by the Toorkmuns; and her loveliness overcoming their avarice, she had at first been detained by her captor. The arrival of our caravan and so many merchants, however, tempted his cupidity; and he offered his charge for sale. A merchant of Tehran purchased her for seventy-seven gold tillas; and the poor girl, who was walking about a few hours before, and saw and was seen by every one, was now literally packed up in a pannier. She had changed her character from slave to wife; for it signifies nought that she may have another husband, since she is surely born again who comes out of the hands of the Toorkmuns. This was a leap year; but a lady may be there allowed at all times to fall in love. The fair one of whom I speak made a set at the first merchant who visited her; and stated, as an inducement to her purchase, that she would join any creed they liked. This Persian girl is not the first of her sex who has changed her doctrines with her name. We halted in the afternoon at a cistern, eighteen miles distant from Shurukhs, the fort of which was yet visible; for we had travelled over level country, broken in some places by gravelly hillocks. At the third mile we crossed the dry and pebbly bed of the small river of Tejend, which rises in the neighbouring hills, and is lost in the sands. This is not the Herat river, nor is it the Ochus, for no such great river as appears in our maps has existence. Its pools were saline, and much of the soil was also salt. There were remnants of civilisation, but neither fields nor inhabitants. We again set out about eight at night with a full moon; and, after an advance of seven or eight miles, entered among defiles and hills, and found ourselves at Moozderan or Durbund, the frontier post in Persia, a little after sunrise, and forty- five miles from Shurukhs. The whole of the latter part of the route lay in a deep ravine, where there is imminent danger in travelling from the “Allamans” of the desert. We pushed on with great celerity and greater fear: every instrument of war was in requisition, every match was lit, and the slightest sound brought the horsemen to a halt; for we hourly expected to encounter the Toorkmuns. After a night of such anxiety, we beheld with pleasure the look-out towers of Durbund,
- 74. Welcome to Our Bookstore - The Ultimate Destination for Book Lovers Are you passionate about books and eager to explore new worlds of knowledge? At our website, we offer a vast collection of books that cater to every interest and age group. From classic literature to specialized publications, self-help books, and children’s stories, we have it all! Each book is a gateway to new adventures, helping you expand your knowledge and nourish your soul Experience Convenient and Enjoyable Book Shopping Our website is more than just an online bookstore—it’s a bridge connecting readers to the timeless values of culture and wisdom. With a sleek and user-friendly interface and a smart search system, you can find your favorite books quickly and easily. Enjoy special promotions, fast home delivery, and a seamless shopping experience that saves you time and enhances your love for reading. Let us accompany you on the journey of exploring knowledge and personal growth! ebookgate.com