Designing Data Intensive Applications The Big Ideas Behind Reliable Scalable and Maintainable Systems 6 [early release] Edition Martin Kleppmann

Designing Data Intensive Applications The Big
Ideas Behind Reliable Scalable and Maintainable
Systems 6 [early release] Edition Martin
Kleppmann download
https://textbookfull.com/product/designing-data-intensive-
applications-the-big-ideas-behind-reliable-scalable-and-
maintainable-systems-6-early-release-edition-martin-kleppmann/
Download more ebook from https://textbookfull.com

We believe these products will be a great fit for you. Click
the link to download now, or visit textbookfull.com
to discover even more!
Designing Distributed Systems Patterns and Paradigms
for Scalable Reliable Services Brendan Burns
https://textbookfull.com/product/designing-distributed-systems-
patterns-and-paradigms-for-scalable-reliable-services-brendan-
burns/
Distributed Services with Go Your Guide to Reliable
Scalable and Maintainable Systems 1st Edition Travis
Jeffery
https://textbookfull.com/product/distributed-services-with-go-
your-guide-to-reliable-scalable-and-maintainable-systems-1st-
edition-travis-jeffery/
Big Data Analytics Systems Algorithms Applications
C.S.R. Prabhu
https://textbookfull.com/product/big-data-analytics-systems-
algorithms-applications-c-s-r-prabhu/
React js Essentials A fast paced guide to designing and
building scalable and maintainable web apps with React
js 1st Edition Fedosejev
https://textbookfull.com/product/react-js-essentials-a-fast-
paced-guide-to-designing-and-building-scalable-and-maintainable-
web-apps-with-react-js-1st-edition-fedosejev/

MASTERING JAVASCRIPT DESIGN PATTERNS create scalable
and reliable applications with advanced Javascript
design patterns using reliable code 3rd Edition Tomas
Corral Cosas
https://textbookfull.com/product/mastering-javascript-design-
patterns-create-scalable-and-reliable-applications-with-advanced-
javascript-design-patterns-using-reliable-code-3rd-edition-tomas-
corral-cosas/
Big Data Applications and Services 2017 The 4th
International Conference on Big Data Applications and
Services Wookey Lee
https://textbookfull.com/product/big-data-applications-and-
services-2017-the-4th-international-conference-on-big-data-
applications-and-services-wookey-lee/
React 16 essentials a fast paced hands on guide to
designing and building scalable and maintainable web
apps with React 16 Second Edition. Edition Boduch
https://textbookfull.com/product/react-16-essentials-a-fast-
paced-hands-on-guide-to-designing-and-building-scalable-and-
maintainable-web-apps-with-react-16-second-edition-edition-
boduch/
The Movie Book: Big Ideas Simply Explained (DK Big
Ideas) Dk
https://textbookfull.com/product/the-movie-book-big-ideas-simply-
explained-dk-big-ideas-dk/
The Art Book: Big Ideas Simply Explained (DK Big Ideas)
Dk
https://textbookfull.com/product/the-art-book-big-ideas-simply-
explained-dk-big-ideas-dk/

978-1-491-90309-4
[FILL IN]
Designing Data-Intensive Applications
by Martin Kleppmann
Copyright © 2016 Martin Kleppmann. All rights reserved.
Printed in the United States of America.
Published by O’Reilly Media, Inc. , 1005 Gravenstein Highway North, Sebastopol, CA 95472.
O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are
also available for most titles ( http://safaribooksonline.com ). For more information, contact our corpo‐
rate/institutional sales department: 800-998-9938 or corporate@oreilly.com .
Editors: Mike Loukides and Ann Spencer
Production Editor: FILL IN PRODUCTION EDI‐
TOR
Copyeditor: FILL IN COPYEDITOR
Proofreader: FILL IN PROOFREADER
Indexer: FILL IN INDEXER
Interior Designer: David Futato
Cover Designer: Karen Montgomery
Illustrator: Rebecca Demarest
January -4712: First Edition
Revision History for the First Edition
2015-03-06: First Early Release
2015-04-01: Second Early Release
2015-06-18: Third Early Release
2016-01-26: Fourth Early Release
2016-05-05: Fifth Early Release
2016-07-11: Sixth Early Release
See http://oreilly.com/catalog/errata.csp?isbn=9781491903094 for release details.
The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Designing Data-Intensive Applica‐
tions, the cover image, and related trade dress are trademarks of O’Reilly Media, Inc.
While the publisher and the author(s) have used good faith efforts to ensure that the information and
instructions contained in this work are accurate, the publisher and the author(s) disclaim all responsibil‐
ity for errors or omissions, including without limitation responsibility for damages resulting from the use
of or reliance on this work. Use of the information and instructions contained in this work is at your own
risk. If any code samples or other technology this work contains or describes is subject to open source
licenses or the intellectual property rights of others, it is your responsibility to ensure that your use
thereof complies with such licenses and/or rights.

Martin Kleppmann
Boston
Designing Data-Intensive
Applications

Table of Contents
About this Book. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii
Part I. Foundations of Data Systems
1. Reliable, Scalable and Maintainable Applications. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Thinking About Data Systems 2
Reliability 4
Hardware faults 5
Software errors 6
Human errors 7
How important is reliability? 8
Scalability 8
Describing load 9
Describing performance 11
Approaches for coping with load 15
Maintainability 16
Operability: making life easy for operations 17
Simplicity: managing complexity 18
Evolvability: making change easy 19
Summary 20
2. Data Models and Query Languages. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
Relational Model vs. Document Model 26
The birth of NoSQL 27
The object-relational mismatch 28
Many-to-one and many-to-many relationships 31
Are document databases repeating history? 35
v

Relational vs. document databases today 38
Query Languages for Data 42
Declarative queries on the web 43
MapReduce querying 45
Graph-like Data Models 48
Property graphs 49
The Cypher query language 51
Graph queries in SQL 52
Triple-stores and SPARQL 55
The foundation: Datalog 59
Summary 62
3. Storage and Retrieval. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 67
Data Structures that Power Your Database 68
Hash indexes 70
SSTables and LSM-trees 74
B-trees 77
Other indexing structures 82
Keeping everything in memory 85
Transaction Processing or Analytics? 87
Data warehousing 88
Stars and snowflakes: schemas for analytics 90
Column-oriented storage 93
Column compression 94
Sort order in column storage 96
Writing to column-oriented storage 98
Aggregation: Data cubes and materialized views 98
Summary 100
4. Encoding and Evolution. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
Formats for Encoding Data 108
Language-specific formats 109
JSON, XML and binary variants 110
Thrift and Protocol Buffers 113
Avro 118
The merits of schemas 123
Modes of Data Flow 124
Data flow through databases 125
Data flow through services: REST and RPC 127
Message passing data flow 132
Summary 135
vi | Table of Contents

Part II. Distributed Data
5. Replication. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
Leaders and Followers 146
Synchronous vs. asynchronous replication 147
Setting up new followers 149
Handling node outages 150
Implementation of replication logs 152
Problems With Replication Lag 155
Reading your own writes 156
Monotonic reads 158
Consistent prefix reads 159
Solutions for replication lag 160
Multi-leader replication 161
Use cases for multi-leader replication 161
Handling write conflicts 164
Multi-leader replication topologies 168
Leaderless replication 171
Writing to the database when a node is down 171
Limitations of quorum consistency 175
Sloppy quorums and hinted handoff 177
Detecting concurrent writes 178
Summary 186
6. Partitioning. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191
Partitioning and replication 192
Partitioning of key-value data 193
Partitioning by key range 194
Partitioning by hash of key 195
Skewed workloads and relieving hot spots 196
Partitioning and secondary indexes 197
Partitioning secondary indexes by document 198
Partitioning secondary indexes by term 200
Rebalancing partitions 201
Strategies for rebalancing 201
Operations: automatic or manual rebalancing 204
Request routing 205
Parallel query execution 207
Summary 208
Table of Contents | vii

7. Transactions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213
The slippery concept of a transaction 214
The meaning of ACID 215
Single-object and multi-object operations 219
Weak isolation levels 224
Read committed 225
Snapshot isolation and repeatable read 228
Preventing lost updates 233
Preventing write skew and phantoms 237
Serializability 242
Actual serial execution 243
Two-phase locking (2PL) 248
Serializable snapshot isolation (SSI) 252
Summary 257
8. The Trouble with Distributed Systems. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265
Faults and Partial Failures 266
Cloud computing and supercomputing 267
Unreliable Networks 269
Network faults in practice 271
Detecting faults 272
Timeouts and unbounded delays 273
Synchronous vs. asynchronous networks 276
Unreliable Clocks 278
Monotonic vs. time-of-day clocks 279
Clock synchronization and accuracy 281
Relying on synchronized clocks 282
Process pauses 287
Knowledge, Truth and Lies 291
The truth is defined by the majority 292
Byzantine faults 295
System model and reality 298
Summary 302
9. Consistency and Consensus. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 311
Consistency Guarantees 312
Linearizability 314
What makes a system linearizable? 315
Relying on linearizability 320
Implementing linearizable systems 323
The cost of linearizability 326
Ordering Guarantees 329
viii | Table of Contents

Ordering and causality 330
Sequence number ordering 334
Total order broadcast 338
Distributed Transactions and Consensus 343
Atomic commit and two-phase commit (2PC) 344
Distributed transactions in practice 350
Fault-tolerant consensus 355
Membership and coordination services 360
Summary 363
Part III. Derived Data
10. Batch Processing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 377
Batch Processing with Unix Tools 379
Simple log analysis 379
The Unix philosophy 382
MapReduce and Distributed Filesystems 385
MapReduce job execution 387
Reduce-side joins and grouping 391
Map-side joins 396
The output of batch workflows 398
Comparing MapReduce to distributed databases 402
Beyond MapReduce 406
Materialization of intermediate state 407
Graphs and iterative processing 411
High-level APIs and languages 414
Summary 416
11. Stream Processing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 425
Transmitting Event Streams 426
Messaging systems 427
Partitioned logs 432
Databases and streams 436
Keeping systems in sync 437
Change data capture 439
Event sourcing 442
State, streams, and immutability 444
Processing Streams 448
Uses of stream processing 449
Reasoning about time 452
Stream joins 457
Table of Contents | ix

Fault tolerance 460
Summary 462
x | Table of Contents

Technology is a powerful force in our society. Data, software and communication can
be used for bad: to entrench unfair power structures, to undermine human rights, and
to protect vested interests. But they can also be used for good: to make underrepresented
people’s voices heard, to create opportunities for everyone, and to avert disasters. This
book is dedicated to everyone working towards the good.
Computing is pop culture. […] Pop culture holds a disdain for history. Pop culture is all
about identity and feeling like you’re participating. It has nothing to do with coopera‐
tion, the past or the future — it’s living in the present. I think the same is true of most
people who write code for money. They have no idea where [their culture came from]…
—Alan Kay, Dr Dobb’s Journal (2012)

About this Book
If you have worked in software engineering in recent years, especially in server-side
and backend systems, you have probably been bombarded with a plethora of buzz‐
words relating to storage and processing of data. NoSQL! Big Data! Web-scale!
Sharding! Eventual consistency! ACID! CAP theorem! Cloud services! MapReduce!
Real-time!
In the last decade we have seen many interesting developments in databases, dis‐
tributed systems and in the ways we build applications on top of them. There are var‐
ious driving forces for these developments, including:
• Internet companies such as Google, Yahoo!, Amazon, Facebook, LinkedIn and
Twitter are handling huge volumes of data and traffic, forcing them to create new
tools that enable them to efficiently handle such scale.
• Businesses need to be agile, test hypotheses cheaply, and respond quickly to new
market insights, by keeping development cycles short and data models flexible.
• Free and open source software has become very successful, and is now preferred
to commercial or bespoke in-house software in many environments.
• CPU clock speeds are barely increasing, but multi-core processors are standard,
and networks are getting faster. This means parallelism is only going to increase.
• Even if you work on a small team, you can now build systems that are distributed
across many machines and even multiple geographic regions, thanks to infra‐
structure as a service (IaaS) such as Amazon Web Services.
• Many services are now expected to be highly available; extended downtime due
to outages or maintenance is becoming increasingly unacceptable.
Data-intensive applications are pushing the boundaries of what is possible by making
use of these technological developments. We call an application data-intensive if data
is its primary challenge: the quantity of data, the complexity of data, or the speed at
xiii

which it is changing (as opposed to compute-intensive, where CPU cycles are the
bottleneck).
The tools and technologies that help data-intensive applications store and process
data have been rapidly adapting to these changes. New types of database systems
(“NoSQL”) have been getting lots of attention, but message queues, caches, search
indexes, frameworks for batch and stream processing, and related technologies are
very important too. Many applications use some combination of these.
The buzzwords that fill this space are a sign of enthusiasm for the new possibilities,
which is a great thing. However, as software engineers and architects, we also need to
have a technically accurate and precise understanding of the various technologies and
their trade-offs if we want to build good applications. For that understanding, we
have to dig deeper than buzzwords.
Fortunately, behind the rapid changes in technology, there are enduring principles
that remain true, no matter which version of a particular tool you are using. If you
understand those principles, you’re in a position to see where each tool fits in, how to
make good use of it, and how to avoid its pitfalls. That’s where this book comes in.
The goal of this book is to help you navigate the diverse and fast-changing landscape
of technologies for processing and storing data. This book is not a tutorial for one
particular tool, nor is it a textbook full of dry theory. Instead, we will look at examples
of successful data systems: technologies that form the foundation of many popular
applications, and that have to meet scalability, performance and reliability require‐
ments in production every day.
We will dig into the internals of those systems, tease apart their key algorithms, dis‐
cuss their principles and the trade-offs they have to make. On this journey, we will try
to find useful ways of thinking about data systems — not just how they work, but also
why they work that way, and what questions we need to ask.
After reading this book, you will be in a great position to decide which kind of tech‐
nology is appropriate for which purpose, and understand how tools can be combined
to form the foundation of a good application architecture. You won’t be ready to
build your own database storage engine from scratch, but fortunately that is rarely
necessary. You will, however, develop a good intuition for what your systems are
doing under the hood, so that you can reason about their behavior, make good design
decisions, and track down any problems that may arise.
Who Should Read this Book?
If you develop applications that have some kind of server/backend for storing or pro‐
cessing data, and your applications use the internet (e.g. web applications, mobile
apps, or internet-connected sensors), then this book is for you.
xiv | About this Book

This book is for software engineers, software architects and technical managers who
love to code. It is especially relevant if you need to make decisions about the architec‐
ture of the systems you work on — for example, if you need to choose tools for solv‐
ing a given problem, and figure out how best to apply them. But even if you have no
choice over your tools, this book will help you better understand their strengths and
weaknesses.
You should have some experience building web-based applications or network serv‐
ices, and you should be familiar with relational databases and SQL. Any non-
relational databases and other data-related tools you know are a bonus, but not
required. A general understanding of common network protocols like TCP and
HTTP is helpful. Your choice of programming language or framework makes no dif‐
ference for this book.
If any of the following are true for you, you’ll find this book valuable:
• You want to learn how to make data systems scalable, for example to support
web or mobile apps with millions of users.
• You need to make applications highly available (minimizing downtime) and
operationally robust.
• You are looking for ways of making systems easier to maintain in the long run,
even as they grow, and as requirements and technologies change.
• You have a natural curiosity for the way things work, and want to know what
goes on inside major websites and online services. This book breaks down the
internals of various databases and data processing systems, and it’s great fun to
explore the bright thinking that went into their design.
Sometimes, when discussing scalable data systems, people make comments along the
lines of “you’re not Google or Amazon, stop worrying about scale and just use a rela‐
tional database”. There is truth in that statement: building for scale that you don’t
need is wasted effort, and may lock you into an inflexible design. In effect, it is a form
of premature optimization. However, it’s also important to choose the right tool for
the job, and different technologies each have their own strengths and weaknesses. As
we shall see, relational databases are important, but not the final word on dealing
with data.
Scope of this Book
This book does not attempt to give detailed instructions on how to install or use spe‐
cific software packages or APIs, since there is already plenty of documentation for
those things. Instead we discuss the various principles and trade-offs that are funda‐
mental to data systems, and we explore the different design decisions taken by differ‐
ent products.
About this Book | xv

Most of what we discuss in this book has already been said elsewhere in some form or
another — in conference presentations, research papers, blog posts, code, bug track‐
ers, and engineering folklore. This book summarizes the most important ideas from
many different sources, and it includes pointers to the original literature throughout
the text. The references at the end of each chapter are a great resource if you want to
explore an area in more depth.
We look primarily at the architecture of data systems and the ways how they are inte‐
grated into data-intensive applications. This book doesn’t have space to cover deploy‐
ment, operations, security, ethics, management and other areas — those are complex
and important topics, and we wouldn’t do them justice by making them superficial
side-notes in this book. They deserve books of their own.
Many of the technologies described in this book fall within the realm of the Big Data
buzzword. However, the term Big Data is so over-used and under-defined that it is
not useful in a serious engineering discussion. This book uses less ambiguous terms,
such as single-node vs. distributed systems, or online/interactive vs. offline/batch
processing systems.
This book has a bias towards free and open source software (FOSS), because reading,
modifying and executing source code is a great way to understand how something
works in detail. Open platforms also reduce the risk of vendor lock-in. However,
where appropriate, we also discuss proprietary software (closed-source software, soft‐
ware as a service, or companies’ in-house software that is only described in literature
but not released publicly).
Outline of this Book
This book is arranged into three parts:
1. In Part I, we will discuss the fundamental ideas that we need in order to design
data-intensive applications. We’ll start in Chapter 1 by discussing what we’re
actually trying to achieve: reliability, scalability and maintainability — how we
need to think about them, and how we can achieve them. In Chapter 2 we will
compare several different data models and query languages, and see how they are
appropriate to different situations. In Chapter 3 we will talk about storage
engines: how databases arrange data on disk so that you can find it again effi‐
ciently. Chapter 4 turns to formats for data encoding (serialization) and evolu‐
tion of schemas over time.
2. In Part II, we will move from data stored on one machine to data that is dis‐
tributed across multiple machines. This is often necessary for scalability, but
brings with it a variety of unique challenges. We’ll first discuss replication (Chap‐
ter 5), partitioning/sharding (Chapter 6), and transactions (Chapter 7). We will
then go into more detail on the problems with distributed systems (Chapter 8)
xvi | About this Book

and what it means to achieve consistency and consensus in a distributed system
(Chapter 9).
3. In Part III, we move up another step and discuss building heterogeneous systems
that consist of several different components. As there is no one database for all
use cases, applications often need to integrate several different databases, caches,
indexes and so on. In Chapter 10 we will start with a batch processing approach,
and build upon this in Chapter 11 to describe stream processing. Finally, in ???
we will put everything together and discuss how we can integrate different data
systems into reliable, scalable and maintainable applications.
Early Release Status and Feedback
This is an early release copy of Designing Data-Intensive Applications. The text, fig‐
ures and examples are a work in progress, and several chapters are yet to be written.
We are releasing the book before it is finished because we hope that it is already use‐
ful in its current form, and because we would love your feedback in order to create
the best possible finished product.
If you find any errors or glaring omissions, if you find anything confusing, or if you
have any ideas for improving the book, please email the author and editors at feed‐
back@dataintensive.net.
About this Book | xvii

PART I
Foundations of Data Systems
The first four chapters go through the fundamental ideas that apply to all data sys‐
tems, whether running on a single machine or distributed across a cluster of
machines:
1. Chapter 1 introduces the terminology and approach that we’re going to use
throughout this book. It examines what we actually mean with words like relia‐
bility, scalability and maintainability, and how we can try to achieve them.
2. Chapter 2 compares several different data models and query languages — the
most visible difference between different databases from a developer’s point of
view. We will see how different models are appropriate to different situations.
3. Chapter 3 turns to the internals of storage engines, and looks at how databases
lay out data on disk. Different storage engines are optimized for different work‐
loads, and choosing the right one can have a huge effect on performance.
4. Chapter 4 compares various formats for data encoding (serialization), and espe‐
cially examines how they fare in an environment where application requirements
change and schemas need to adapt over time.
Later, Part II will turn to the particular issues of distributed data systems.

CHAPTER 1
Reliable, Scalable and Maintainable
Applications
The Internet was done so well that most people think of it as a natural resource like the
Pacific Ocean, rather than something that was man-made. When was the last time a
technology with a scale like that was so error-free?
—Alan Kay, Dr Dobb’s Journal (2012)
Many applications today are data-intensive, as opposed to compute-intensive. Raw
CPU power is rarely a limiting factor for these applications — bigger problems are
usually the amount of data, the complexity of data, and the speed at which it is
changing.
A data-intensive application is typically built from standard building blocks which
provide commonly needed functionality. For example, many applications need to:
• Store data so that they, or another application, can find it again later (databases),
• Remember the result of an expensive operation, to speed up reads (caches),
• Allow users to search data by keyword or filter it in various ways (search
indexes),
• Send a message to another process, to be handled asynchronously (stream pro‐
cessing),
• Periodically crunch a large amount of accumulated data (batch processing).
If that sounds painfully obvious, that’s just because these data systems are such a suc‐
cessful abstraction: we use them all the time without thinking too much. When build‐
1

ing an application, most engineers wouldn’t dream of writing a new data storage
engine from scratch, because databases are a perfectly good tool for the job.
But reality is not that simple. There are many database systems with different charac‐
teristics, because different applications have different requirements. There are vari‐
ous approaches to caching, several ways of building search indexes, and so on. When
building an application, we still need to figure out which tools and which approaches
are the most appropriate for the task at hand. Sometimes it can be hard to combine
several tools when you need to do something that a single tool cannot do alone.
This book is a journey through both the principles and the practicalities of data sys‐
tems, and how you can use them to build data-intensive applications. We will explore
what different tools have in common, what distinguishes them, and how they achieve
their characteristics.
In this Chapter 1, we will start by exploring the fundamentals of what we are trying to
achieve: reliable, scalable and maintainable data systems. We’ll clarify what those
things mean, outline some ways of thinking about them, and go over the basics that
we will need for later chapters. In the following chapters we will continue layer by
layer, looking at different design decisions which need to be considered when work‐
ing on a data-intensive application.
Thinking About Data Systems
We typically think of databases, queues, caches etc. as being very different categories
of tools. Although a database and a message queue have some superficial similarity —
both store data for some time — they have very different access patterns, which
means different performance characteristics, and thus very different implementa‐
tions.
So why should we lump them all together under an umbrella term like data systems?
Many new tools for data storage and processing have emerged in recent years. They
are optimized for a variety of different use cases, and they no longer neatly fit into
traditional categories [1]. For example, there are data stores that are also used as mes‐
sage queues (Redis), and there are message queues with database-like durability guar‐
antees (Kafka). The boundaries between the categories are becoming blurred.
Secondly, increasingly many applications now have such demanding or wide-ranging
requirements that a single tool can no longer meet all of its data processing and stor‐
age needs. Instead, the work is broken down into tasks that can be performed effi‐
ciently on a single tool, and those different tools are stitched together using
application code.
For example, if you have an application-managed caching layer (using memcached or
similar), or a full-text search server separate from your main database (such as Elas‐
2 | Chapter 1: Reliable, Scalable and Maintainable Applications

ticsearch or Solr), it is normally the application code’s responsibility to keep those
caches and indexes in sync with the main database. Figure 1-1 gives a glimpse of what
this may look like (we will go into detail in later chapters).
When you combine several tools in order to provide a service, the service’s interface
or API usually hides those implementation details from clients. Now you have essen‐
tially created a new, special-purpose data system from smaller, general-purpose com‐
ponents. Your composite data system may provide certain guarantees, e.g. that the
cache will be correctly invalidated or updated on writes, so that outside clients see
consistent results. You are now not only an application developer, but also a data sys‐
tem designer.
If you are designing a data system or service, a lot of tricky questions arise. How do
you ensure that the data remains correct and complete, even when things go wrong
internally? How do you provide consistently good performance to clients, even when
parts of your system are degraded? How do you scale to handle an increase in load?
What does a good API for the service look like?
Application code
Application code
Application code
first check if
data is cached
Read requests Asynchronous tasks
In-memory
cache
Primary
database
Full-text
index
Message
queue
Cache misses
and writes
Capture
changes
to data
Search
requests
Client
requests
Invalidate
or update
cache
Apply updates
to search index
e.g. send email
“Outside world”
API
Figure 1-1. One possible architecture for a data system that combines several compo‐
nents.
There are many factors that may influence the design of a data system, including the
skills and experience of the people involved, legacy system dependencies, the time‐
Thinking About Data Systems | 3

scale for delivery, your organization’s tolerance of different kinds of risk, regulatory
constraints, etc. Those factors depend very much on the situation.
In this book, we focus on three concerns that are important in most software systems:
Reliability
The system should continue to work correctly (performing the correct function at
the desired performance) even in the face of adversity (hardware or software
faults, and even human error). See “Reliability” on page 4.
Scalability
As the system grows (in data volume, traffic volume or complexity), there should
be reasonable ways of dealing with that growth. See “Scalability” on page 8.
Maintainability
Over time, many different people will work on the system (engineering and oper‐
ations, both maintaining current behavior and adapting the system to new use
cases), and they should all be able to work on it productively. See “Maintainabil‐
ity” on page 16.
These words are often cast around without a clear understanding of what they mean.
In the interest of thoughtful engineering, we will spend the rest of this chapter
exploring ways of thinking about reliability, scalability and maintainability. Then, in
the following chapters, we will look at various techniques, architectures and algo‐
rithms that are used in order to achieve those goals.
Reliability
Everybody has an intuitive idea of what it means for software to be reliable or unreli‐
able. For software, typical expectations include:
• The application performs the function that the user expected.
• It can tolerate the user making mistakes, or using the software in unexpected
ways.
• Its performance is good enough for the required use case, under expected load
and data volume.
• The system prevents any unauthorized access and abuse.
If all those things together mean “working correctly”, then we can understand relia‐
bility as meaning, roughly, “continuing to work correctly, even when things go
wrong”.
The things that can go wrong are called faults, and systems that anticipate faults and
can cope with them are called fault-tolerant or resilient. The term is slightly mislead‐
ing: it suggests that we could make a system tolerant of every possible kind of fault,

which in reality is not feasible. If the entire planet Earth (and all servers on it) were
swallowed by a black hole, tolerance of that fault would require web hosting in space
— good luck getting that budget item approved. So it only makes sense to talk about
tolerating certain types of fault.
Note that a fault is not the same as a failure [2]. A fault is usually defined as one com‐
ponent of the system deviating from its spec, whereas a failure is when the system as a
whole stops providing the required service to the user. It is impossible to reduce the
probability of a fault to zero; therefore it is usually best to design fault tolerance
mechanisms that prevent faults from causing failures. In this book we cover several
techniques for building reliable systems from unreliable parts.
Counter-intuitively, in such fault-tolerant systems, it can make sense to increase the
rate of faults by triggering them deliberately — for example, by randomly killing indi‐
vidual processes without warning. Many critical bugs are actually due to poor error
handling [3]; by deliberately inducing faults, you ensure that the fault-tolerance
machinery is continually exercised and tested, which can increase your confidence
that faults will be handled correctly when they occur naturally. The Netflix chaos
monkey [4] is an example of this approach.
Although we generally prefer tolerating faults over preventing faults, there are cases
where prevention is better than cure (e.g. because no cure exists). This is the case
with security matters, for example: if an attacker has compromised a system and
gained access to sensitive data, that event cannot be undone. However, this book
mostly deals with the kinds of fault that can be cured, as described in the following
sections.
Hardware faults
When we think of causes of system failure, hardware faults quickly come to mind.
Hard disks crash, RAM becomes faulty, the power grid has a blackout, someone
unplugs the wrong network cable. Anyone who has worked with large data centers
can tell you that these things happen all the time when you have a lot of machines.
Hard disks are reported as having a mean time to failure (MTTF) of about 10 to 50
years [5, 6]. Thus, on a storage cluster with 10,000 disks, we should expect on average
one disk to die per day.
Our first response is usually to add redundancy to the individual hardware compo‐
nents in order to reduce the failure rate of the system. Disks may be set up in a RAID
configuration, servers may have dual power supplies and hot-swappable CPUs, and
data centers may have batteries and diesel generators for backup power. When one
component dies, the redundant component can take its place while the broken com‐
ponent is replaced. This approach cannot completely prevent hardware problems
Reliability | 5

i. Defined in “Approaches for coping with load” on page 15 below.
from causing failures, but it is well understood, and can often keep a machine run‐
ning uninterrupted for years.
Until recently, redundancy of hardware components was sufficient for most applica‐
tions, since it makes total failure of a single machine fairly rare. As long as you can
restore a backup onto a new machine fairly quickly, the downtime in case of failure is
not catastrophic in most applications. Thus, multi-machine redundancy was only
required by a small number of applications for which high availability was absolutely
essential.
However, as data volumes and applications’ computing demands increase, more
applications are using larger numbers of machines, which proportionally increases
the rate of hardware faults. Moreover, in some “cloud” platforms such as Amazon
Web Services it is fairly common for virtual machine instances to become unavailable
without warning [7], as the platform is designed to prioritize flexibility and elasticityi
over single-machine reliability.
Hence there is a move towards systems that can tolerate the loss of entire machines,
by using software fault-tolerance techniques in preference to hardware redundancy.
Such systems also have operational advantages: a single-server system requires plan‐
ned downtime if you need to reboot the machine (to apply operating system security
patches, for example), whereas a system that can tolerate machine failure can be
patched one node at a time, without downtime of the entire system.
Software errors
We usually think of hardware faults as being random and independent from each
other: one machine’s disk failing does not imply that another machine’s disk is going
to fail. There may be weak correlations (for example due to a common cause, such as
the temperature in the server rack), but otherwise it is unlikely that a large number of
hardware components will fail at the same time.
Another class of fault is a systematic error within the system [8]. Such faults are
harder to anticipate, and because they are correlated across nodes, they tend to cause
many more system failures than uncorrelated hardware faults [5]. Examples include:
• A software bug that causes every instance of an application server to crash when
given a particular bad input. For example, consider the leap second on June 30,
2012 that caused many applications to hang simultaneously, due to a bug in the
Linux kernel [9].
• A runaway process uses up some shared resource — CPU time, memory, disk
space or network bandwidth.

• A service that the system depends on slows down, becomes unresponsive or
starts returning corrupted responses.
• Cascading failures, where a small fault in one component triggers a fault in
another component, which in turn triggers further faults [10].
The bugs that cause these kinds of software fault often lie dormant for a long time
until they are triggered by an unusual set of circumstances. In those circumstances, it
is revealed that the software is making some kind of assumption about its environ‐
ment — and while that assumption is usually true, it eventually stops being true for
some reason [11].
There is no quick solution to the problem of systematic faults in software. Lots of
small things can help: carefully thinking about assumptions and interactions in the
system, thorough testing, process isolation, allowing processes to crash and restart,
measuring, monitoring and analyzing system behavior in production. If a system is
expected to provide some guarantee (for example, in a message queue, that the num‐
ber of incoming messages equals the number of outgoing messages), it can constantly
check itself while it is running, and raise an alert if a discrepancy is found [12].
Human errors
Humans design and build software systems, and the operators who keep the system
running are also human. Even when they have the best intentions, humans are
known to be unreliable. For example, one study of large internet services found that
configuration errors by operators were the leading cause of outages, whereas hard‐
ware faults (servers or network) played a role in only 10–25% of outages [13].
How do we make our system reliable, in spite of unreliable humans? The best systems
combine several approaches:
• Design systems in a way that minimizes opportunities for error. For example,
well-designed abstractions, APIs and admin interfaces make it easy to do “the
right thing”, and discourage “the wrong thing”. However, if the interfaces are too
restrictive, people will work around them, negating their benefit, so this is a
tricky balance to get right.
• Decouple the places where people make the most mistakes from the places where
they can cause failures. In particular, provide fully-featured non-production
sandbox environments where people can explore and experiment safely, using
real data, without affecting real users.
• Test thoroughly at all levels, from unit tests to whole-system integration tests and
manual tests [3]. Automated testing is widely used, well understood, and espe‐
cially valuable for covering corner cases that rarely arise in normal operation.
Reliability | 7

Random documents with unrelated
content Scribd suggests to you:

"Alice's father was named Marshall. He and her mother both are
dead. She has no near relatives."
"I remember Marshall--he was a refiner."
"Precisely; he met with reverses a few years ago."
Uncle John looked after Alice with his feeble, questioning grin.
"Fine looking," he muttered, still looking after her much as the
toothless giant looked after Christian as he passed his cave. "Fine
looking."
Dolly was annoyed: "Oh, you're always thinking about fine
looks! She is nice."
Uncle John smiled undismayed. "Is she?"
CHAPTER VI
Alice had been married five years--it seemed a long time. The first
five years of married life are likely to be long enough to chart pretty
accurately the currents of the future, however insufficient to predict
just where those currents will carry one.
Much disillusioning comes in the first five years; when they have
passed we know less of ourselves and more of our consort.
Undoubtedly the complement of this is true, and our consort knows
more of us; but this thought, not always reassuring, comes only
when we reflect concerning ourselves, which fortunately, perhaps, is
not often. Married people, if we may judge from what they say, tend
to reflect more concerning their mates.

Alice, it is certain, knew less of herself. Much of the confidence
of five years earlier she had parted with, some of it cruelly. Yet
coming at twenty-five into the Kimberly circle, and with the
probability of remaining in it, of its being to her a new picture of life,
Alice gradually renewed her youth. Some current flowing from this
joy of living seemed to revive in her the illusions of girlhood. All that
she now questioned was whether it really was for her.
Her husband enjoyed her promise of success in their new
surroundings without realizing in the least how clearly those about
them discriminated between his wife and himself. She brought one
quality that was priceless among those with whom she now mingled-
-freshness.
Among such people her wares of mental aptness, intelligence,
amiability, not to discuss a charm of person that gave her a place
among women, were rated higher than they could have been
elsewhere. She breathed in her new atmosphere with a renewed
confidence, for nothing is more gratifying than to be judged by what
we believe to be the best in us; and nothing more reassuring after
being neglected by stupid people than to find ourselves approved by
the best.
Walter MacBirney, her husband, representing himself and his
Western associates, and now looked on by them as a man who had
forced recognition from the Kimberly interests, made on his side,
too, a favorable impression among the men with whom his affairs
brought him for the first time in contact.
If there was an exception to such an impression it was with
Robert Kimberly, but even with him MacBirney maintained easily the

reputation accorded to Western men for general capacity and a
certain driving ability for putting things through.
He was described as self-made; and examined with the quiet
curiosity of those less fortunate Eastern men who were unwilling or
unable to ascribe their authorship to themselves, he made a
satisfactory showing.
In the Kimberly coterie of men, which consisted in truth more of
the staff associates in the Kimberly activities than of the Kimberlys
themselves, the appearance of MacBirney on the scene at Second
Lake was a matter of interest to every one of the fledgling
magnates, who, under the larger wing of the Kimberlys, directed the
commercial end of their interests.
McCrea, known as Robert Kimberly's right-hand man; Cready
Hamilton, one of the Kimberly bankers, and brother of Doctor
Hamilton, Robert's closest friend; Nelson, the Kimberly counsel--all
took a hand in going over MacBirney, so to say, and grading him up.
They found for one thing that he could talk without saying anything;
which in conducting negotiations was an excellent trait. And if not
always a successful story-teller, he was a shrewd listener. In
everything his native energy gave him a show of interest which,
even when factitious, told in his favor.
Soon after the call on Uncle John, Dolly arranged a dinner for
the MacBirneys, at which Charles Kimberly and his wife and Robert
Kimberly were to be the guests. It followed a second evening spent
at the Nelsons', whence Robert Kimberly had come home with the
De Castros and MacBirneys. Alice had sung for them. After accepting
for the De Castro dinner, Robert at the last moment sent excuses.
Dolly masked her feelings. Imogene and Charles complained a little,

but Arthur De Castro was so good a host that he alone would have
made a dinner go.
MacBirney, after he and Alice had gone to their rooms for the
night, spoke of Robert's absence. "I don't quite understand that
man," he mused. "What do you make of him, Alice?"
Alice was braiding her hair. She turned from her table. "I've met
him very little, you know--when we called at his house, and twice at
the Nelsons'. And I saw very little of him last night. He was with that
drinking set most of the evening."
MacBirney started. "Don't say 'that drinking set.'"
"Really, that describes them, Walter. I don't see that they excel
in anything else. I hate drinking women."
"When you're in Rome, do as the Romans do," suggested
MacBirney, curtly.
Alice's tone hardened a trifle. "Or at least let the Romans do as
they please, without comment."
"Exactly," snapped her husband. "I don't know just what to
make of Kimberly," he went on.
"Charles, or the brother?"
"Robert, Robert. He's the one they all play to here." MacBirney,
sitting in a lounging-chair, emphasized the last words, as he could do
when impatient, and shut his teeth and lips as he did when
perplexed. "I wonder why he didn't come to-night?"
Alice had no explanation to offer. "Charles," she suggested,
tying her hair-ribbon, "is very nice."
"Why, yes--you and Charles are chummy already. I wish we
could get better acquainted with Robert," he continued, knitting his
brows. "I thought you were a little short with him last night, Alice."

"Short? Oh, Walter! We didn't exchange a dozen words."
"That's just the way it struck me."
"But we had no chance to. I am sure I didn't mean to be short.
I sang, didn't I? And more on his account, from what Dolly had said
to me, than anybody else's. He didn't like my singing, but I couldn't
help that. He didn't say a single word."
"Why, he did say something!"
"Just some stiff remark when he thanked me."
Alice, rising, left her table. MacBirney laughed.
"Oh, I see. That's what's the matter. Well, you're quite mistaken,
my dear." Catching Alice in his arms as she passed, in a way he did
when he wished to seem affectionate, MacBirney drew his wife to
him. "He did like it. He remarked to me just as he said good-night,
that you had a fine voice."
"That does not sound like him--possibly he was ironical."
"And when I thanked him," continued MacBirney, "he took the
trouble to repeat: 'That song was beautifully sung.' Those were his
exact words."
In spite of painful experiences it rarely occurred to Alice that her
husband might be deceiving her, nor did she learn till long afterward
that he had lied to her that night. With her feelings in some degree
appeased she only made an incredulous little exclamation: "He didn't
ask me to sing again," she added quietly.
MacBirney shrugged his shoulders. "He is peculiar."
"I try, Walter," she went on, lifting her eyes to his with an effort,
"to be as pleasant as I can to all of these people, for your sake."
"I know it, Alice." He kissed her. "I know it. Let us see now what
we can do to cultivate Robert Kimberly. He is the third rail in this

combination, and he is the only one on the board of directors who
voted finally against taking us in."
"Is that true?"
"So Doane told Lambert, in confidence, and Lambert told me."
"Oh, Lambert! That detestable fellow. I wouldn't believe
anything he said anyway."
MacBirney bared his teeth pleasantly. "Pshaw! You hate him
because he makes fun of your Church."
"No. I despise him, because he is a Catholic and ridicules his
own."
Her husband knew controversy was not the way to get a favor.
"I guess you're right about that, Allie. Anyway, try being pleasant to
Kimberly. The way you know how to be, Allie--the way you caught
me, eh?" He drew her to him with breezy enthusiasm. Alice showed
some distress.
"Don't say such things, please."
"That was only a joke."
"I hate such jokes."
"Very well, I mean, just be natural," persisted MacBirney
amiably, "you are fascinating enough any old way."
Alice manifested little spirit. "Does it make so much difference to
you, Walter, whether we pay attention to him?"
MacBirney raised his eyebrows with a laughing start. "What an
innocent you are," he cried in a subdued tone. And his ways of
speech, if ever attractive, were now too familiar. "Difference!" he
exclaimed cheerily. "When they buy he will name the figure."
"But I thought they had decided to buy."

"The executive committee has authorized the purchase. But he,
as president, has been given the power to fix the price. Don't you
see? We can afford to smile a little, eh?"
"It would kill me to smile if I had to do it for money."
"Oh, you are a baby in arms, Allie," exclaimed her husband
impatiently, "just like your father! You'd starve to death if it weren't
for me."
"No doubt."
MacBirney was still laughing at the idea when he left his wife's
room, and entering his own, closed the door.
Alice, in her room, lay in the darkness for a long time with open
eyes.
CHAPTER VII
The test of Alice's willingness to smile came within a brief fortnight,
when with the De Castros, she was the guest of Imogene Kimberly
at The Cliffs, Imogene's home.
"This is all most informal," said Imogene, as she went
downstairs arm-in-arm with Alice; "as you see, only one-half the
house is open."
"The open half is so lovely," returned Alice, "that I'm glad to
take the other half on faith."
"It was my only chance--this week, and as Dolly says, I 'jumped
at it'! I am sorry your husband has disappointed us."
"He was called to town quite unexpectedly."

"But Providence has provided a substitute. Robert Kimberly is
coming." Alice almost caught her breath. "He is another of those
men," continued Imogene, "whom you never can get when you want
them. Fortunately he telephoned a moment ago saying he must see
Charles. I answered that the only possible way to see him was to
come over now, for he is going fishing and leaves at midnight. The
guides wired this morning that the ice is out. And when the ice goes
out," Imogene raised her hands, "neither fire nor earthquake can
stop Charles. Here is Robert now. Oh, and he has Doctor Hamilton
with him. All the better. If we can get both we shall have no lack of
men."
Robert Kimberly and Doctor Hamilton were coming down the
hall. "How delightful!" cried Imogene, advancing, "and I am so glad
you've come, doctor."
Kimberly paused. He saw Alice lingering behind her hostess and
the De Castros with Fritzie Venable coming downstairs.
"You have a dinner on," he said to Imogene.
"Only a small one."
"But you didn't tell me----"
"Just to give you a chance to show your indifference to
surprises, Robert."
She introduced Doctor Hamilton to Alice. "These two are always
together," she explained to Alice, lifting her fan toward the doctor
and her brother-in-law. "But any hostess is fortunate to capture
them like this, just the right moment."
Hamilton, greeting Alice, turned to Imogene: "What is this
about your husband's going to Labrador to-morrow?"

"He is going to-night. The salmon are doing something or
other."
"Deserted Gaspé, has he?"
"Temporarily," said Imogene, pausing to give an order to a
butler. Robert waited a moment for her attention. "I brought the
doctor," he explained, "because I couldn't leave him to dine alone.
And now----"
"And now," echoed Imogene, "you see how beautifully it turns
out. The Nelsons declined, Mr. MacBirney disappoints me, Charles
goes fishing, and can't get home to-night in time to dine. But there
are still seven of us--what could be better? Mrs. De Castro will claim
the doctor. Arthur won't desert me, and, Robert, you may give an
arm to Fritzie and one to Mrs. MacBirney."
There was now no escape from a smile, and Alice resolved to be
loyal to her hostess. The party moved into the drawing-room.
Fritzie Venable tried to engage Kimberly in answering her
questions about a saddle-horse that one of his grooms had
recommended. Kimberly professed to know nothing about it. When it
became apparent that he really did know nothing of the horse,
Fritzie insisted on explaining.
Her spirited talk, whether concerning her own troubles or those
of other people, was not uninteresting. Soon she talked more
especially to Alice. Kimberly listened not inattentively but somewhat
perfunctorily, and the manner, noticeable at their second meeting,
again impressed Alice.
Whether it was a constraint or an unpleasing reserve was not
clear; and it might have been the abstraction of a busied man, one

of that type familiar in American life who are inherently interesting,
but whose business affairs never wholly release their thought.
Whatever the cause, Fritzie was sufficiently interested in her
own stories to ignore it and in a degree to overcome the effect of it.
She was sure of her ground because she knew her distinguished
connection had a considerate spot in his heart for her. She finally
attacked him directly, and at first he did not go to the trouble of a
defence. When she at length accused him, rather sharply, of letting
business swallow him up, Kimberly, with Alice listening, showed a
trace of impatience.
"The old sugar business!" Fritzie exclaimed reproachfully, "it is
taking the spirituality completely out of the Kimberly family."
Robert looked at her in genuine surprise and burst into a laugh.
"What's that?" he demanded, bending incredulously forward.
Fritzie tossed her head. "I don't care!"
"Spirituality?" echoed Kimberly, with a quiet malice. His laugh
annoyed Fritzie, but she stuck to her guns: "Spirits, then; or gayety,
or life!" she cried. "I don't care what you call it. Anything besides
everlastingly piling up money. Oh, these almighty dollars!"
"You tire of them so quickly, is it, Fritzie? Or is it that they don't
feel on familiar terms enough to stay long with you?" he asked,
while Alice was smiling at the encounter.
Fritzie summoned her dignity and pointed every word with a
nod. "I simply don't want to see all of my friends--ossify! Should
you?" she demanded, turning to Alice for approval.
"Certainly not," responded Alice.
"Bone black is very useful in our business," observed Kimberly.

Fritzie's eyes snapped. "Then buy it! Don't attempt to supply the
demand out of your own bones!"
It would have been churlish to refuse her her laugh. Kimberly
and Alice for the first time laughed together and found it pleasant.
Fritzie, following up her advantage, asked Doctor Hamilton
whether he had heard Dora Morgan's latest joke. "She had a
dispute," continued Fritzie, "with George Doane last night about
Unitarians and Universalists----"
"Heavens, have those two got to talking religion?" demanded
Kimberly, wearily.
"George happened to say to Cready Hamilton that Unitarians
and Universalists believed just about the same doctrine. When Dora
insisted it was not so, George told her she couldn't name a
difference. 'Why, nonsense, George,' said Dora, 'Unitarians deny the
divinity of Christ, but Universalists don't believe in a damned thing.'
And the funny part of it was, George got furious at her," concluded
Fritzie with merriment.
"I suppose you, too, fish," ventured Alice to Kimberly as the
party started for the dining-room.
"My fishing is something of a bluff," he confessed. "That is, I
fish, but I don't get anything. My brother really does get the fish,"
he said as he seated her. "He campaigns for them--one has to
nowadays, even for fish. I can't scrape up interest enough in it for
that. I whip one pool after another and drag myself wearily over
portages and chase about in boats, and my guides fable wisely but I
get next to nothing."
Alice laughed. Even though he assumed incompetence it
seemed assumed. And in saying that he got no fish one felt that he

did get them.
Arthur was talking of Uncle John's nurse--whom the circle had
nicknamed "Lazarus." He referred to the sacrifices made sometimes
by men.
"It won't do to say," De Castro maintained, "that these men are
mere clods, that they have no nerves, no sensitiveness. The first one
you meet may be such a one; the next, educated or of gentle
blood."
"'Lazarus,'" he continued, "is by no means a common man. He
is a gentleman, the product of centuries of culture--this is evident
from five minutes' talk with him. Yet he has abandoned everything--
family, surroundings, luxuries--for a work that none of us would
dream of undertaking."
"And what about women, my dear?" demanded Dolly. "I don't
say, take a class of women--take any woman. A woman's life is
nothing but sacrifice. The trouble is that women bear their burdens
uncomplainingly. That is where all women make a mistake. My life
has been a whole series of sacrifices, and I propose people shall
know it."
"No matter, Dolly," suggested Imogene, "your wrongs shall be
righted in the next world."
"I should just like the chance to tell my story up there,"
continued Dolly, fervently.
Kimberly turned to Alice: "All that Dolly fears," said he, in an
aside, "is that heaven will prove a disappointment. But to change the
subject from heaven abruptly--you are from the West, Mrs.
MacBirney."

"Do you find the change so abrupt? and must I confess again to
the West?"
"Not if you feel it incriminates you."
"But I don't," protested Alice with spirit.
"Has your home always been there?"
"Yes, in St. Louis; and it is a very dear old place. Some of my
early married life was spent much farther West."
"How much farther?"
"So much that I can hardly make anybody comprehend it--
Colorado."
"How so?"
"They ask me such wild questions about buffalos and Indians. I
have found one woman since coming here who has been as far West
as Chicago, once."
"In what part of Colorado were you?"
"South of Denver."
"You had beautiful surroundings."
"Oh, do you know that country?"
"Not nearly as well as I should like to. It is beautiful."
Alice laughed repentantly as she answered: "More beautiful to
me now, I'm afraid, than it was then."
"Any town is quiet for a city girl, of course. Was it a small
town?"
"Quite small. And odd in many ways."
"I see; where the people have 'best clothes'----"
"Don't make fun."
"And wear them on Sunday. And there is usually one three-story
building in the town--I was marooned over Sunday once in a little

Western town, with an uncle. I saw a sign on a big building: 'Odd
Fellows' Hall.' Who are the Odd Fellows, uncle?' I asked. He was a
crusty old fellow: 'Optimists, my son, optimists,' he growled, 'They
build three-story buildings in two-story towns.' What was your town,
by the way?"
"Piedmont."
"Piedmont?" Kimberly paused a moment. "I ought to know
something of that town."
Alice looked surprised. "You?"
"The uncle I spoke of built a railroad through there to the Gulf.
Isn't there a town below Piedmont named Kimberly?"
"To be sure there is. How stupid! I never thought it was named
after your uncle."
"No, that uncle was a Morgan,", interposed Imogene, listening,
"the town was named after your next neighbor."
"How interesting! And how could you make such fun of me--
having me tell you of a country you knew all about! And a whole
town named after you!"
"That is a modest distinction," remarked Kimberly. "As a boy I
was out there with an engineering party and hunted a little. My
uncle gave me the town as a Christmas present."
"A town for a Christmas present!"
"I suspected after I began paying taxes on my present that my
uncle had got tired of it. They used to sit up nights out there to
figure out new taxes. In the matter of devising taxes it is the most
industrious, progressive, tireless community I have ever known. And
their pleas were so ingenious; they made you feel that if you
opposed them you were an enemy to mankind."

"Then they beguiled Robert every once in a while," interposed
Fritzie, "into a town hall or public library or a park or electric lighting
plant. Once they asked him for a drinking fountain." Fritzie laughed
immoderately at the recollection. "He put in the fountain and
afterward learned there was no water within fifteen miles; they then
urged him to put in a water-works system to get water to it."
"I suggested a brewery to supply the fountain," said Arthur,
looking over, "and that he might work out even by selling the surplus
beer. There were difficulties, of course; if he supplied the fountain
with beer, nobody would buy it in bottles. Then it was proposed to
sell the surplus beer to the neighboring towns. But with the fountain
playing in Kimberly, these would pretty certainly be depopulated. Per
contra, it was figured that this might operate to raise the price of his
Kimberly lots. But while we were working the thing out for him, what
do you think happened?"
"I haven't an idea," laughed Alice.
"The town voted for prohibition."
"Fancy," murmured Imogene, "and named Kimberly!"
"And what became of the fountain?"
"Oh, it is running; he put in the water-works."
"Generous man!"
"Generous!" echoed Hamilton. "Don't be deceived, Mrs.
MacBirney. You should see what he charges them for water. I should
think it would be on his conscience, if he has one. He is Jupiter with
the frogs. Whatever they ask, he gives them. But when they get it--
how they do get it!"
"Don't believe Doctor Hamilton, Mrs. MacBirney," said Robert
Kimberly. "I stand better with my Western friends than I do with

these cynical Easterners. And if my town will only drink up the
maintenance charges, I am satisfied."
"The percentage of lime in the water he supplies is something
fierce," persisted the doctor. "It is enough to kill off the population
every ten years. I suggested a hospital."
"But didn't Mr. MacBirney tell me they have a sugar factory
there?" asked Alice.
"They have," said De Castro. "One of Robert's chemists was out
there once trying to analyze the taxes. Incidentally, he brought back
some of the soil, thinking there might be something in it to account
for the tax mania. And behold, he found it to be fine for sugar beets!
Irrigation ditches and a factory were put in. You should see how
swell they are out there now."
"Robert has had all kinds of resolutions from the town," said
Fritzie.
Kimberly turned to Alice to supplement the remark. "Quite true,
I have had all kinds--they are strong on resolutions. But lately these
have been less sulphurous."
"Well, isn't it odd? My father's ranch once extended nearly all
the way from Piedmont to the very town you are speaking of!"
exclaimed Alice.
Kimberly looked at her with interest. "Was that really yours--the
big ranch north of Kimberly?"
"I spent almost every summer there until I was fifteen."
"That must have been until very lately."
Alice returned his look with the utmost simplicity. "No, indeed, it
is ten years ago."

Kimberly threw back his head and it fell forward a little on his
chest. "How curious," he said reflectively; "I knew the ranch very
well."
When they were saying good-night, Imogene whispered to
Alice: "I congratulate you."
Alice, flushed with the pleasure of the evening, stood in her
wraps. She raised her brows in pleased surprise. "Pray what for?"
"Your success. The evening, you know, was in your honor; and
you were decidedly the feature of it."
"I really didn't suspect it."
"And you made a perfect success with your unexpected
neighbor."
"But I didn't do anything at all!"
"It isn't every woman that succeeds without trying. We have
been working for a long time to pull Robert out of the dumps."
Imogene laughed softly. "I noticed to-night while you were talking to
him that he tossed back his head once or twice. When he does that,
he is waking up! Here is your car, Dolly," she added, as the De
Castros came into the vestibule.
"Arthur is going to take Doctor Hamilton and Fritzie in our car,
Imogene," explained Dolly. "Robert has asked Mrs. MacBirney and
me to drive home around the south shore with him."
CHAPTER VIII

Charles Kimberly was at The Towers the morning after the return
from his fishing trip, to confer with Uncle John and his brother upon
the negotiations for the MacBirney properties. In the consideration
of any question each of the three Kimberlys began with a view-point
quite distinct from those of the others.
John Kimberly, even in old age and stricken physically to an
appalling degree, swerved not a hair's-breadth from his constant
philosophy of life. He believed first and last in force, and that feeble
remnant of vitality which disease, or what Dolly would have termed,
"God's vengeance," had left him, was set on the use of force.
To the extent that fraud is an element of force, he employed
fraud; but it was only because fraud is a part of force, and whoever
sets store by the one will not always shrink from the other. Any
disposition of a question that lacked something of this complexion
seemed to Uncle John a dangerous one.
Charles had so long seen bludgeoning succeed that it had
become an accepted part of his business philosophy. But in the day
he now faced, new forces had arisen. Public sentiment had become
a factor in industrial problems; John was blind to its dangerous
power; Charles was quite alive to it.
New views of the problem of competition had been advanced,
and in advocating them, one of the Kimberlys, Robert, was known to
be a leader. This school sought to draw the sting of competitive loss
through understandings, coöperation, and peace, instead of
suspicion, random effort, and war.
Charles saw this tendency with satisfaction; Uncle John saw it
sceptically. But Charles, influenced by the mastery of his uncle,
became unsettled in his conclusions and stood liable to veer in his

judgment to one side or the other of the question, as he might be
swayed by apprehensions concerning the new conditions or rested in
confidence in the policies of the old.
Between these two Kimberly make-ups, the one great in attack,
the other in compromise, stood Robert. "Say what you please,"
Nelson often repeated to McCrea, "John may be all right, but his day
is past. Charlie forgets every day more than the opposition know, all
told. But I call Robert the devil of the family. How does he know
when to be bold? Can you tell? How does he know when to be
prudent? I know men, if I do anything, McCrea--but I never can
measure that fellow."
Whatever Robert liked at least enlisted all of his activities and
his temperament turned these into steam cylinders. John Kimberly
influenced Robert in no way at all and after some years of profanity
and rage perceived that he never should. This discovery was so
astounding that after a certain great family crisis he silently and
secretly handed the sceptre of family infallibility over to his nephew.
Left thus to himself, Robert continued to think for himself. The
same faculties that had served John a generation earlier now served
Robert. John had forgotten that when a young man he had never let
anybody think for him, and the energy that had once made John,
also made his younger nephew.
The shrewdness that had once overcome competition by war
now united with competitors to overcome the public by peace. The
real object of industrial endeavor being to make money, a white-
winged and benevolent peace, as Nelson termed it, should be the
policy of all interests concerned. And after many hard words, peace

with eighty per cent. of the business was usually achieved by the
united Kimberlys.
It had cost something to reach this situation; and now that the
West had come into the sugar world it became a Kimberly problem
to determine how the new interests should be taken care of.
On the morning that Charles called he found Uncle John in his
chair. They sent for Robert, and pending his appearance opened the
conference. At the end of a quarter of an hour Robert had not
appeared. Charles looked impatiently at his watch and despatched a
second servant to summon his brother. After twenty-five minutes a
third call was sent.
During this time, in the sunniest corner of the south garden,
sheltered by a high stone wall crested with English ivy and
overgrown with climbing roses, sat Robert Kimberly indolently
watching Brother Francis and a diminutive Skye terrier named Sugar.
Sugar was one of Kimberly's dogs, but Francis had nursed Sugar
through an attack after the kennel keepers had given him up. And
the little dog although very sick and frowsy had finally pulled
through. The intimacy thus established between Sugar and Francis
was never afterward broken but by death.
In this sunny corner, Kimberly, in a loose, brown suit of tweed,
his eyes shaded by a straw hat, sat in a hickory chair near a table. It
was the corner of the garden in which Francis when off duty could
oftenest be found. A sheltered walk led to the pergola along which
he paced for exercise. Near the corner of the wall stood an oak. And
a bench, some chairs and a table made the spot attractive. Sugar
loved the bench, and, curled up on it, usually kept watch while
Francis walked. On cold days the dog lay with one hair-curtained eye

on the coming and going black habit. On warm days, cocking one
ear for the measured step, he dozed.
Francis, when Sugar had got quite well, expressed himself as
scandalized that the poor dog had never been taught anything. He
possessed, his new master declared, neither manners nor
accomplishments, and Francis amid other duties had undertaken, in
his own words, to make a man of the little fellow.
Robert, sitting lazily by, instead of attending the conference call,
and apparently thinking of nothing--though no one could divine just
what might be going on under his black-banded hat--was watching
Francis put Sugar through some of the hard paces he had laid out
for him.
"That dog is naturally stupid, Francis--all my dogs are. They
continually cheat me on dogs," said Kimberly presently. "You don't
think so? Very well, I will bet you this bank-note," he took one from
his waistcoat as he spoke, "that you cannot stop him this time on
'two'."
"I have no money to bet you, Robert."
"I will give you odds."
"You well know I do not bet--is it not so?"
"You are always wanting money; now I will bet you the bank-
note against one dollar, Francis, that you cannot stop him on 'two'."
Francis threw an eye at the money in Kimberly's hand. "How
much is the bank-note, Robert?"
"One hundred dollars."
Francis put the temptation behind him. "You would lose your
money. Sugar knows how to stop. In any case, I have no dollar."
"I will bet the money against ten cents."

"I have not even ten cents."
"I am sorry, Francis, to see a man receiving as large a salary as
you do, waste it in dissipation and luxury. However, if you have no
money, I will bet against your habit."
"If I should lose my habit, what would I do?"
"You could wear a shawl," argued Kimberly.
"All would laugh at me. In any case, to bet the clothes off my
back would be a sin."
"I am so sure I am right, I will bet the money against your
snuff-box, Francis," persisted Kimberly.
"My snuff-box I cannot bet, since Cardinal Santopaolo gave it to
me."
"Francis, think of what you could do for your good-for-nothing
boys with one hundred dollars."
Francis lifted his dark eyes and shook his head.
"I will bet this," continued the tempter, "against the snuff in
your box, that you can't stop him this time on 'two'."
"Sugar will stop on 'two'," declared Francis, now wrought up.
"Dare you bet?"
"Enough! I bet! It is the snuff against the money. May my poor
boys win!"
The sunny corner became active. Kimberly straightened up, and
Francis began to talk to Sugar.
"Now tell me again," said Kimberly, "what this verse is."
"I say to him," explained Francis, "that the good soldier goes to
war----"
"I understand; then you say, 'One, two, three!'"
"Exactly."

"When you say 'three,' he gets the lump?"
"Yes."
"But the first time you say the verse you stop at 'two.' Then you
repeat the verse. If the dog takes the lump before you reach the end
the second time and say 'three'----"
"You get the snuff!" Francis laid the box on the table beside
Kimberly's bank-note.
"Sugar! Guarda!" The Skye terrier sat upright on his haunches
and lifted his paws. Francis gave him a preliminary admonition, took
from a mysterious pocket a lump of sugar, laid it on the tip of the
dog's nose, and holding up his finger, began in a slow and clearly
measured tone:
"Buon soldato
Va alia guerra,
Mangia male,
Dorme in terra.
Uno, due--
Buon soldato
Va--"
But here Sugar, to Francis's horror, snapped the lump into his mouth
and swallowed it.
"You lose," announced Kimberly.
Francis threw up his hands. "My poor boys!"
"This is the time, Francis, your poor boys don't get my money. I
get your snuff."

"Ah, Sugar, Sugar! You ruin us." The little Skye sitting fast,
looked innocently and affectionately up at his distressed master.
"Why," demanded the crestfallen Francis, "could you not wait for the
lump one little instant?"
"Sugar is like me," suggested Kimberly lazily, "he wants what he
wants when he wants it."
Alice, this morning, had been deeply in his thoughts. From the
moment he woke he had been toying indolently with her image--
setting it up before his imagination as a picture, then putting it away,
then tempting his lethargy again with the pleasure of recalling it.
He drew a cigar-case from his pocket and carefully emptied the
snuff out of the box into it. "When do you get more snuff, Francis?"
"On Saturday."
"This is Tuesday. The box is nearly full. It looks like good stuff."
He paused between each sentence. "But you would bet."
Francis without looking busied himself with his little pupil.
"I have emptied the box," announced Kimberly. There was no
answer. "Do you want any of it back?"
Francis waved the offer aside.
"A few pinches, Francis?"
"Nothing."
"That dog," continued Kimberly, rapping the box to get every
grain out and perceiving the impossibility of harrying Francis in any
other way, "is good for nothing anyway. He wasn't worth saving."
"That dog," returned Francis earnestly, "is a marvel of
intelligence and patience. He has so sweet a temper, and he is so
quick, Robert, to comprehend."
"I fail to see it."

"You will see it. The fault is in me."
"I don't see that either."
Francis looked at Kimberly appealingly and pointed benevolently
at Sugar. "I ask too much of that little dog. He will learn. 'Patience,
Francis,' he says to me, 'patience; I will learn.'"
Summoning his philosophy to bridge over the disappointment,
Francis, as he stood up, absent-mindedly felt in his deep pocket for
his snuff-box. It was in difficulties such as this that recourse to a
frugal pinch steadied him. He recollected instantly that the snuff was
gone, and with some haste and stepping about, he drew out his
handkerchief instead--glancing toward Kimberly as he rubbed his
nose vigorously to see if his slip had been detected.
Needless to say it had been--less than that would not have
escaped Kimberly, and he was already enjoying the momentary
discomfiture. Sugar at that moment saw a squirrel running down the
walk and tore after him.
Francis with simple dignity took the empty snuff-box from the
table and put it back in his pocket. His composure was restored and
the incident to him was closed.
Kimberly understood him so well that it was not hard to turn the
talk to a congenial subject. "I drove past the college the other day. I
see your people are doing some building."
Francis shrugged his shoulders. "A laundry, Robert."
"Not a big building, is it?"
"We must go slow."
"It is over toward where you said the academy ought to go."
"My poor academy! They do not think it will ever come."

"You have more buildings now than you have students. What do
you want with more buildings?"
"No, no. We have three hundred students--three hundred now."
Francis looked at his questioner with eyes fiercely eager. "That is the
college, Robert. The academy is something else--for what I told
you."
"What did you tell me?" Kimberly lighted a cigar and Francis
began again to explain.
"This is it: Our Sisters in the city take now sixteen hundred boys
from seven to eight years old. These boys they pick up from the
orphan courts, from the streets, from the poor parents. When these
boys are twelve the Sisters cannot keep them longer, they must let
them go and take in others.
"Here we have our college and these boys are ready for it when
they are sixteen. But, between are four fatal years--from twelve to
sixteen. If we had a school for such boys, think what we could do.
They would be always in hand; now, they drift away. They must go
to work in the city filth and wickedness. Ah, they need the protection
we could give them in those terrible four years, Robert. They need
the training in those years to make of them mechanics and artisans--
to give them a chance, to help them to do more than drift without
compass or rudder--do you not see?
"Those boys that are bright, that we find ready to go further,
they are ready at sixteen for our college; we keep and educate
them. But the others--the greater part--at sixteen would leave us,
but trained to earn. And strengthened during those four critical years
against evil. Ah!"

Francis paused. He spoke fast and with an intensity that
absorbed him.
Kimberly, leaning comfortably back, sat with one foot resting on
his knee. He knocked the ash of his cigar upon the heel of his shoe
as he listened--sometimes hearing Francis's words, sometimes not.
He had heard all of them before at one time or another; the plea
was not new to him, but he liked the fervor of it.
"Ah! It is not for myself that I beg." Brother Francis's hands fell
resignedly on his knees. "It is for those poor boys, to keep them,
Robert, from going to hell--from hell in this world and in the next. To
think of it makes me always sorrowful--it makes a beggar of me--a
willing beggar."
Kimberly moved his cigar between his lips.
"But where shall I get so much money?" exclaimed Francis,
helplessly. "It will take a million dollars to do what we ought to do.
You are a great man, Robert; tell me, how shall I find it?"
"I can't tell you how to find it; I can tell you how to make it."
"How?"
"Go into the sugar business."
"Then I must leave God's business."
"Francis, if you will pardon me, I think for a clever man you are
in some respects a great fool. I am not joking. What I have often
said about your going into the sugar business, I repeat. You would
be worth ten thousand dollars a year to me, and I will pay you that
much any day."
Francis looked at Kimberly as if he were a madman, but
contented himself with moving his head slowly from side to side in

protest. "I cannot leave God's business, Robert. I must work for him
and pray to him for the money. Sometime it will come."
"Then tell Uncle John to raise your wages," suggested Kimberly,
relapsing into indifference.
"Robert, will you not sometime give me a letter to introduce me
to the great banker who comes here, Hamilton?"
"He will not give you anything."
"He has so much money; how can he possibly need it all?"
"You forget, Francis, that nobody needs money so much as
those that have it."
"Ah!"
"Hamilton may have no more money than I have, and you don't
ask me for a million dollars."
"It is not necessary to ask you. You know I need it. If you could
give it to me, you would."
"If I gave you a million dollars how should I ever get it back?"
Francis spoke with all seriousness. "God will pay you back."
"Yes, but when? That is a good deal of money to lend to God."
"It is a good deal."
"When do I get it back, and how?"
"He will surely pay you, Robert; God pays over there."
"That won't do--over there. It isn't honest."
Francis started. "Not honest?"
"You are offering deferred dividends, Francis. What would my
stockholders say if I tried that kind of business? Gad, they would
drag me into court."
"Ah, yes! But, Robert; you pay for to-day: he pays for eternity."

Kimberly smoked a moment. "In a proposition of that kind,
Francis, it seems to me the question of guarantees is exceedingly
important. You good men are safe enough; but where would the bad
men come in on your eternal dividends?"
"You are not with the bad men, Robert. Your heart is not bad.
You are, perhaps, cruel----"
"What?"
"But generous. Sometime God will give you a chance."
"You mean, sometime I will give God a chance."
"No, Robert, what I say I mean--sometime, God will give you a
chance."
Charles Kimberly's impatient voice was heard from the pergola.
"Robert! We've been waiting thirty minutes," he stormed.
"I am just coming."

CHAPTER IX
That afternoon MacBirney played golf with Charles Kimberly. Toward
five o'clock, Alice in one of the De Castro cars drove around to The
Hickories after him. When he came in, she was sitting on the porch
with a group of women, among them Fritzie Venable and Lottie
Nelson.
"I must be very displeasing to Mrs. Nelson," Alice said to her
husband as they drove away. "It upsets me completely to meet that
woman."
"Why, what's the matter with her?" asked MacBirney, in a tone
which professing friendly surprise really implied that the grievance
might after all be one of imagination.
"I haven't an idea," declared Alice a little resentfully. "I am not
conscious of having done a thing to offend her."
"You are oversensitive."
"But, Walter, I can tell when people mean to be rude."
"What did Mrs. Nelson do that was rude?" asked her husband in
his customary vein of scepticism.
"She never does anything beyond ignoring me," returned Alice.
"It must be, I think, that she and I instinctively detest each other.
They were talking about a dinner and musicale Thursday night that
Mr. Robert Kimberly is giving at The Towers. Miss Venable said she
supposed we were going, and I had to say I really didn't know. We
haven't been asked, have we?"

"Not that I know of."
"Mrs. Nelson looked at me when Fritzie spoke; I think it is the
first time that she ever has looked at me, except when she had to
say 'good-morning' or 'good-evening.' I was confused a little when I
answered, I suppose; at any rate, she enjoyed it. Mr. Kimberly would
not leave us out, would he?"
"I don't think so. He was playing golf this afternoon with Cready
Hamilton, and he stopped to offer me his yacht for the week of the
cup races."
"Why, how delightful! How came he ever to do that?"
"And I think he has made up his mind what he is going to do
about placing me on the board," continued MacBirney, resuming his
hard, thin manner and his eager tone of business. "I wish I knew
just what is coming."
Alice had scarcely reached her room when she found the dinner
invitation. She felt a little thrill of triumph as she read it. Her maid
explained that the note had been laid in the morning with Mrs. De
Castro's letters.
Late in the evening Kimberly came over with his sister-in-law,
Imogene. The De Castros were at the seashore overnight and the
visitors' cards were sent up to the MacBirneys. It was warm and the
party sat on the south veranda. Kimberly talked with Alice and she
told him they hoped to be present at his dinner.
"You are sure to be, aren't you?" he asked. "The evening is
given for you."
"For us?"
"No, not for 'us,' but for you," he said distinctly. "Mr. MacBirney
has said he is fond of the water--you like music; and I am trying

something for each of you. I should have asked you about your
engagements before the cards went out. If there is any conflict the
date can easily be recalled."
"Oh, no. That would be a pity."
"Not at all. I change my arrangements when necessary every
ten minutes."
"But there isn't any conflict, and I shall be delighted to come.
Pray, how do you know I like music?"
"I heard you say so once to Arthur De Castro. Tell me what you
are amused about?"
"Have I betrayed any amusement?"
"For just about the hundredth part of a second, in your eyes."
They were looking at each other and his gaze though within
restraint was undeniably alive. Alice knew not whether she could
quite ignore it or whether her eyes would drop in an annoying
admission of self-consciousness. She avoided the latter by
confessing. "I am sure I don't know at all what you are talking
about----"
"I am sure you do, but you are privileged not to tell if you don't
want to."
"Then--our dinner card was mislaid and until to-night we didn't
know whether----"
"There was going to be any dinner."
"Oh, I knew that. I was at the Casino this afternoon----"
"I saw you."
"And when I was asked whether I was going to the dinner at
The Towers I couldn't, of course, say."
"Who asked you, Mrs. Nelson?"

Welcome to our website – the ideal destination for book lovers and
knowledge seekers. With a mission to inspire endlessly, we offer a
vast collection of books, ranging from classic literary works to
specialized publications, self-development books, and children's
literature. Each book is a new journey of discovery, expanding
knowledge and enriching the soul of the reade
Our website is not just a platform for buying books, but a bridge
connecting readers to the timeless values of culture and wisdom. With
an elegant, user-friendly interface and an intelligent search system,
we are committed to providing a quick and convenient shopping
experience. Additionally, our special promotions and home delivery
services ensure that you save time and fully enjoy the joy of reading.
Let us accompany you on the journey of exploring knowledge and
personal growth!
textbookfull.com

Designing Data Intensive Applications The Big Ideas Behind Reliable Scalable and Maintainable Systems 6 [early release] Edition Martin Kleppmann

More Related Content

Similar to Designing Data Intensive Applications The Big Ideas Behind Reliable Scalable and Maintainable Systems 6 [early release] Edition Martin Kleppmann (20)

Recently uploaded (20)

Designing Data Intensive Applications The Big Ideas Behind Reliable Scalable and Maintainable Systems 6 [early release] Edition Martin Kleppmann