Immediate download Kubernetes Best Practices 1st Edition Brendan Burns ebooks 2024

Full download ebook at ebookname.com
Kubernetes Best Practices 1st Edition Brendan
Burns
https://ebookname.com/product/kubernetes-best-practices-1st-
edition-brendan-burns/
OR CLICK BUTTON
DOWLOAD NOW
Download more ebook from https://ebookname.com

More products digital (pdf, epub, mobi) instant
download maybe you interests ...
Kubernetes Up and Running Dive Into the Future of
Infrastructure 2nd Edition Brendan Burns
https://ebookname.com/product/kubernetes-up-and-running-dive-
into-the-future-of-infrastructure-2nd-edition-brendan-burns/
Manufacturing Best Practices 1st Edition Bobby Hull
https://ebookname.com/product/manufacturing-best-practices-1st-
edition-bobby-hull/
Technology Best Practices 1st Edition Robert H. Spencer
https://ebookname.com/product/technology-best-practices-1st-
edition-robert-h-spencer/
Java Database Best Practices 1st Edition George Reese
https://ebookname.com/product/java-database-best-practices-1st-
edition-george-reese/

ACCOUNTING Accounting Best Practices Steven M. Bragg
https://ebookname.com/product/accounting-accounting-best-
practices-steven-m-bragg/
Digital forensics threatscape and best practices 1st
Edition Sammons
https://ebookname.com/product/digital-forensics-threatscape-and-
best-practices-1st-edition-sammons/
Visual Studio 2010 Best Practices 1st Edition Peter
Ritchie
https://ebookname.com/product/visual-studio-2010-best-
practices-1st-edition-peter-ritchie/
Information Operations Matters Best Practices 1st
Edition Leigh Armistead
https://ebookname.com/product/information-operations-matters-
best-practices-1st-edition-leigh-armistead/
Accounting Best Practices Sixth edition Steven M. Bragg
https://ebookname.com/product/accounting-best-practices-sixth-
edition-steven-m-bragg/

Brendan Burns, Eddie Villalba,
Dave Strebel & Lachlan Evenson
Kubernetes
Best Practices
Blueprints for Building Successful Applications
on Kubernetes

Brendan Burns, Eddie Villalba,
Dave Strebel, and Lachlan Evenson
Kubernetes Best Practices
Blueprints for Building Successful
Applications on Kubernetes
Boston Farnham Sebastopol Tokyo
Beijing Boston Farnham Sebastopol Tokyo
Beijing

978-1-492-05647-8
[LSI]
Kubernetes Best Practices
by Brendan Burns, Eddie Villalba, Dave Strebel, and Lachlan Evenson
Copyright © 2020 Brendan Burns, Eddie Villalba, Dave Strebel, and Lachlan Evenson. All rights reserved.
Printed in the United States of America.
Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472.
O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are
also available for most titles (http://oreilly.com). For more information, contact our corporate/institutional
sales department: 800-998-9938 or corporate@oreilly.com.
Acquisitions Editor: John Devins
Development Editor: Virginia Wilson
Production Editor: Elizabeth Kelly
Copyeditor: Charles Roumeliotis
Proofreader: Sonia Saruba
Indexer: WordCo Indexing Services, Inc.
Interior Designer: David Futato
Cover Designer: Karen Montgomery
Illustrator: Rebecca Demarest
November 2019: First Edition
Revision History for the First Release
2019-11-12: First Release
2020-07-10: Second Release
See https://www.oreilly.com/catalog/errata.csp?isbn=0636920273219 for release details.
The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Kubernetes Best Practices, the cover
image, and related trade dress are trademarks of O’Reilly Media, Inc.
The views expressed in this work are those of the authors, and do not represent the publisher’s views.
While the publisher and the authors have used good faith efforts to ensure that the information and
instructions contained in this work are accurate, the publisher and the authors disclaim all responsibility
for errors or omissions, including without limitation responsibility for damages resulting from the use of
or reliance on this work. Use of the information and instructions contained in this work is at your own
risk. If any code samples or other technology this work contains or describes is subject to open source
licenses or the intellectual property rights of others, it is your responsibility to ensure that your use
thereof complies with such licenses and/or rights.

Table of Contents
Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi
1. Setting Up a Basic Service. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
Application Overview 1
Managing Configuration Files 2
Creating a Replicated Service Using Deployments 3
Best Practices for Image Management 4
Creating a Replicated Application 4
Setting Up an External Ingress for HTTP Traffic 6
Configuring an Application with ConfigMaps 7
Managing Authentication with Secrets 9
Deploying a Simple Stateful Database 11
Creating a TCP Load Balancer by Using Services 15
Using Ingress to Route Traffic to a Static File Server 16
Parameterizing Your Application by Using Helm 17
Deploying Services Best Practices 19
Summary 19
2. Developer Workflows. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
Goals 21
Building a Development Cluster 22
Setting Up a Shared Cluster for Multiple Developers 23
Onboarding Users 24
Creating and Securing a Namespace 27
Managing Namespaces 28
Cluster-Level Services 29
Enabling Developer Workflows 29
Initial Setup 30
iii

Enabling Active Development 31
Enabling Testing and Debugging 31
Setting Up a Development Environment Best Practices 32
Summary 33
3. Monitoring and Logging in Kubernetes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
Metrics Versus Logs 35
Monitoring Techniques 35
Monitoring Patterns 36
Kubernetes Metrics Overview 37
cAdvisor 37
Metrics Server 38
kube-state-metrics 38
What Metrics Do I Monitor? 39
Monitoring Tools 40
Monitoring Kubernetes Using Prometheus 42
Logging Overview 46
Tools for Logging 47
Logging by Using an EFK Stack 48
Alerting 50
Best Practices for Monitoring, Logging, and Alerting 52
Monitoring 52
Logging 52
Alerting 52
Summary 53
4. Configuration, Secrets, and RBAC. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
Configuration Through ConfigMaps and Secrets 55
ConfigMaps 55
Secrets 56
Common Best Practices for the ConfigMap and Secrets APIs 57
RBAC 63
RBAC Primer 64
RBAC Best Practices 65
Summary 67
5. Continuous Integration, Testing, and Deployment. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
Version Control 70
Continuous Integration 70
Testing 71
Container Builds 71
Container Image Tagging 72
iv | Table of Contents

Continuous Deployment 73
Deployment Strategies 73
Testing in Production 77
Setting Up a Pipeline and Performing a Chaos Experiment 79
Setting Up CI 79
Setting Up CD 82
Performing a Rolling Upgrade 82
A Simple Chaos Experiment 82
Best Practices for CI/CD 83
Summary 84
6. Versioning, Releases, and Rollouts. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
Versioning 85
Releases 86
Rollouts 87
Putting It All Together 88
Best Practices for Versioning, Releases, and Rollouts 91
Summary 92
7. Worldwide Application Distribution and Staging. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
Distributing Your Image 94
Parameterizing Your Deployment 95
Load-Balancing Traffic Around the World 95
Reliably Rolling Out Software Around the World 96
Pre-Rollout Validation 96
Canary Region 99
Identifying Region Types 99
Constructing a Global Rollout 100
When Something Goes Wrong 101
Worldwide Rollout Best Practices 102
Summary 103
8. Resource Management. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
Kubernetes Scheduler 105
Predicates 105
Priorities 106
Advanced Scheduling Techniques 107
Pod Affinity and Anti-Affinity 107
nodeSelector 108
Taints and Tolerations 108
Pod Resource Management 110
Resource Request 110
Table of Contents | v

Resource Limits and Pod Quality of Service 111
PodDisruptionBudgets 113
Managing Resources by Using Namespaces 114
ResourceQuota 115
LimitRange 117
Cluster Scaling 118
Application Scaling 119
Scaling with HPA 120
HPA with Custom Metrics 121
Vertical Pod Autoscaler 121
Resource Management Best Practices 122
Summary 122
9. Networking, Network Security, and Service Mesh. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
Kubernetes Network Principles 123
Network Plug-ins 126
Kubenet 126
Kubenet Best Practices 127
The CNI Plug-in 127
CNI Best Practices 127
Services in Kubernetes 128
Service Type ClusterIP 129
Service Type NodePort 130
Service Type ExternalName 131
Service Type LoadBalancer 132
Ingress and Ingress Controllers 133
Services and Ingress Controllers Best Practices 135
Network Security Policy 136
Network Policy Best Practices 138
Service Meshes 139
Service Mesh Best Practices 141
Summary 141
10. Pod and Container Security. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
PodSecurityPolicy API 143
Enabling PodSecurityPolicy 143
Anatomy of a PodSecurityPolicy 145
PodSecurityPolicy Challenges 153
PodSecurityPolicy Best Practices 154
PodSecurityPolicy Next Steps 155
Workload Isolation and RuntimeClass 155
Using RuntimeClass 156
vi | Table of Contents

Runtime Implementations 156
Workload Isolation and RuntimeClass Best Practices 157
Other Pod and Container Security Considerations 158
Admission Controllers 158
Intrusion and Anomaly Detection Tooling 158
Summary 158
11. Policy and Governance for Your Cluster. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
Why Policy and Governance Are Important 159
How Is This Policy Different? 159
Cloud-Native Policy Engine 160
Introducing Gatekeeper 160
Example Policies 161
Gatekeeper Terminology 161
Defining Constraint Templates 162
Defining Constraints 163
Data Replication 164
UX 164
Audit 165
Becoming Familiar with Gatekeeper 166
Gatekeeper Next Steps 166
Policy and Governance Best Practices 167
Summary 167
12. Managing Multiple Clusters. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 169
Why Multiple Clusters? 169
Multicluster Design Concerns 171
Managing Multiple Cluster Deployments 173
Deployment and Management Patterns 173
The GitOps Approach to Managing Clusters 175
Multicluster Management Tools 177
Kubernetes Federation 178
Managing Multiple Clusters Best Practices 180
Summary 181
13. Integrating External Services and Kubernetes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183
Importing Services into Kubernetes 183
Selector-Less Services for Stable IP Addresses 184
CNAME-Based Services for Stable DNS Names 185
Active Controller-Based Approaches 186
Exporting Services from Kubernetes 187
Exporting Services by Using Internal Load Balancers 188
Table of Contents | vii

Exporting Services on NodePorts 188
Integrating External Machines and Kubernetes 189
Sharing Services Between Kubernetes 190
Third-Party Tools 191
Connecting Cluster and External Services Best Practices 191
Summary 192
14. Running Machine Learning in Kubernetes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193
Why Is Kubernetes Great for Machine Learning? 193
Machine Learning Workflow 194
Machine Learning for Kubernetes Cluster Admins 195
Model Training on Kubernetes 195
Distributed Training on Kubernetes 198
Resource Constraints 198
Specialized Hardware 199
Libraries, Drivers, and Kernel Modules 200
Storage 200
Networking 201
Specialized Protocols 201
Data Scientist Concerns 202
Machine Leaning on Kubernetes Best Practices 202
Summary 203
15. Building Higher-Level Application Patterns on Top of Kubernetes. . . . . . . . . . . . . . . . 205
Approaches to Developing Higher-Level Abstractions 205
Extending Kubernetes 206
Extending Kubernetes Clusters 206
Extending the Kubernetes User Experience 208
Design Considerations When Building Platforms 208
Support Exporting to a Container Image 209
Support Existing Mechanisms for Service and Service Discovery 209
Building Application Platforms Best Practices 210
Summary 210
16. Managing State and Stateful Applications. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213
Volumes and Volume Mounts 214
Volume Best Practices 215
Kubernetes Storage 215
PersistentVolume 215
PersistentVolumeClaims 216
Storage Classes 217
Kubernetes Storage Best Practices 218
viii | Table of Contents

Stateful Applications 219
StatefulSets 220
Operators 221
StatefulSet and Operator Best Practices 222
Summary 223
17. Admission Control and Authorization. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225
Admission Control 225
What Are They? 226
Why Are They Important? 226
Admission Controller Types 227
Configuring Admission Webhooks 227
Admission Control Best Practices 229
Authorization 231
Authorization Modules 232
Authorization Best Practices 234
Summary 235
18. Conclusion. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237
Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239
Table of Contents | ix

Preface
Who Should Read This Book
Kubernetes is the de facto standard for cloud native development. It is a powerful tool
that can make your next application easier to develop, faster to deploy, and more reli‐
able to operate. However, unlocking the power of Kubernetes requires using it cor‐
rectly. This book is intended for anyone who is deploying real-world applications to
Kubernetes and is interested in learning patterns and practices they can apply to the
applications that they build on top of Kubernetes.
Importantly, this book is not an introduction to Kubernetes. We assume that you have
a basic familiarity with the Kubernetes API and tools, and that you know how to cre‐
ate and interact with a Kubernetes cluster. If you are looking to learn Kubernetes,
there are numerous great resources out there, such as Kubernetes: Up and Running
(O’Reilly) that can give you an introduction.
Instead, this book is a resource for anyone who wants to dive deep on how to deploy
specific applications and workloads on Kubernetes. It should be useful to you
whether you are about to deploy your first application onto Kubernetes or you’ve
been using Kubernetes for years.
Why We Wrote This Book
Between the four of us, we have significant experience helping a wide variety of users
deploy their applications onto Kubernetes. Through this experience, we have seen
where people struggle, and we have helped them find their way to success. When sit‐
ting down to write this book, we attempted to capture these experiences so that many
more people could learn by reading the lessons that we learned from these real-world
experiences. It’s our hope that by committing our experiences to writing, we can scale
our knowledge and allow you to be successful deploying and managing your applica‐
tion on Kubernetes on your own.
xi

Navigating This Book
Although you might read this book from cover to cover in a single sitting, that is not
really how we intended you to use it. Instead, we designed this book to be a collection
of standalone chapters. Each chapter gives a complete overview of a particular task
that you might need to accomplish with Kubernetes. We expect people to dive into
the book to learn about a specific topic or interest, and then leave the book alone,
only to return when a new topic comes up.
Despite this standalone approach, there are some themes that span the book. There
are several chapters on developing applications on Kubernetes. Chapter 2 covers
developer workflows. Chapter 5 discusses Continuous Integration and testing. Chap‐
ter 15 covers building higher-level platforms on top of Kubernetes, and Chapter 16
discusses managing state and stateful applications. In addition to developing applica‐
tions, there are several chapters on operating services in Kubernetes. Chapter 1 covers
the setup of a basic service, and Chapter 3 covers monitoring and metrics. Chapter 4
covers configuration management, while Chapter 6 covers versioning and releases.
Chapter 7 covers deploying your application around the world.
There are also several chapters on cluster management, including Chapter 8 on
resource management, Chapter 9 on networking, Chapter 10 on pod security, Chap‐
ter 11 on policy and governance, Chapter 12 on managing multiple clusters, and
Chapter 17 on admission control and authorization. Finally there are several chapters
that are truly independent; these cover machine learning (Chapter 14) and integrat‐
ing with external services (Chapter 13).
Though it can be useful to read all of the chapters before you actually attempt the
topic in the real world, our primary hope is that you will treat this book as a refer‐
ence. It is intended as a guide as you put these topics to practice in the real world.
Conventions Used in This Book
The following typographical conventions are used in this book:
Italic
Indicates new terms, URLs, email addresses, filenames, and file extensions.
Constant width
Used for program listings, as well as within paragraphs to refer to program ele‐
ments such as variable or function names, databases, data types, environment
variables, statements, and keywords.
Constant width bold
Shows commands or other text that should be typed literally by the user.
xii | Preface

Constant width italic
Shows text that should be replaced with user-supplied values or by values deter‐
mined by context.
This element signifies a tip or suggestion.
This element signifies a general note.
This element indicates a warning or caution.
Using Code Examples
Supplemental material (code examples, exercises, etc.) is available for download at
https://oreil.ly/KBPsample.
If you have a technical question or a problem using the code examples, please send
email to bookquestions@oreilly.com.
This book is here to help you get your job done. In general, if example code is offered
with this book, you may use it in your programs and documentation. You do not
need to contact us for permission unless you’re reproducing a significant portion of
the code. For example, writing a program that uses several chunks of code from this
book does not require permission. Selling or distributing examples from O’Reilly
books does require permission. Answering a question by citing this book and quoting
example code does not require permission. Incorporating a significant amount of
example code from this book into your product’s documentation does require
permission.
We appreciate, but generally do not require, attribution. An attribution usually
includes the title, author, publisher, and ISBN. For example: “Kubernetes Best Practi‐
ces by Brendan Burns, Eddie Villalba, Dave Strebel, and Lachlan Evenson (O’Reilly).
Copyright 2020 Brendan Burns, Eddie Villalba, Dave Strebel, and Lachlan Evenson,
978-1-492-05647-8.”
Preface | xiii

If you feel your use of code examples falls outside fair use or the permission given
above, feel free to contact us at permissions@oreilly.com.
O’Reilly Online Learning
For more than 40 years, O’Reilly Media has provided technol‐
ogy and business training, knowledge, and insight to help
companies succeed.
Our unique network of experts and innovators share their knowledge and expertise
through books, articles, conferences, and our online learning platform. O’Reilly’s
online learning platform gives you on-demand access to live training courses, in-
depth learning paths, interactive coding environments, and a vast collection of text
and video from O’Reilly and 200+ other publishers. For more information, please
visit http://oreilly.com.
How to Contact Us
Please address comments and questions concerning this book to the publisher:
O’Reilly Media, Inc.
1005 Gravenstein Highway North
Sebastopol, CA 95472
800-998-9938 (in the United States or Canada)
707-829-0515 (international or local)
707-829-0104 (fax)
We have a web page for this book, where we list errata, examples, and any additional
information. You can access this page at https://oreil.ly/KubBP.
Email bookquestions@oreilly.com to comment or ask technical questions about this
book.
For more information about our books, courses, conferences, and news, see our web‐
site at http://www.oreilly.com.
Find us on Facebook: http://facebook.com/oreilly
Follow us on Twitter: http://twitter.com/oreillymedia
Watch us on YouTube: http://www.youtube.com/oreillymedia
xiv | Preface

Acknowledgments
Brendan would like to thank his wonderful family, Robin, Julia, and Ethan, for the
love and support of everything he does; the Kubernetes community, without whom
none of this would be possible; and his fabulous coauthors, without whom this book
would not exist.
Dave would like to thank his beautiful wife, Jen, and their three children, Max, Mad‐
die, and Mason, for all of their support. He would also like to thank the Kubernetes
community for all the advice and help they have provided over the years. Finally, he
would like to thank his coauthors in making this adventure a reality.
Lachlan would like to thank his wife and three children for their love and support. He
would also like to thank everyone in the Kubernetes community, including the won‐
derful individuals who have taken the time to teach him over the years. He also would
like to send a special thanks to Joseph Sandoval for his mentorship. And, finally, he
would like to thank his fantastic coauthors for making this book possible.
Eddie would like to thank his wife, Sandra, for her moral support and for letting him
disappear for hours on end to write while she was in the final trimester of their first
pregnancy. He would also like to thank his new daughter, Giavanna, for giving him
the drive to push forward. Finally, he would like to thank the Kubernetes community
and his coauthors who have always been guideposts in his journey to be cloud native.
We would all like to thank Virginia Wilson for her work in developing the manu‐
script and helping us bring all of our ideas together, and Bridget Kromhout, Bilgin
Ibryam, Roland Huß, and Justin Domingus for their attention to the finishing
touches.
Preface | xv

CHAPTER 1
Setting Up a Basic Service
This chapter describes the practices for setting up a simple multitier application in
Kubernetes. The application consists of a simple web application and a database.
Though this might not be the most complicated application, it is a good place to start
to orient to managing an application in Kubernetes.
Application Overview
The application that we will use for our sample isn’t particularly complex. It’s a simple
journal service that stores its data in a Redis backend. It has a separate static file
server using NGINX. It presents two web paths on a single URL. The paths are one
for the journal’s RESTful application programming interface (API), https://my-host.io/
api, and a file server on the main URL, https://my-host.io. It uses the Let’s Encrypt ser‐
vice for managing Secure Sockets Layer (SSL) certificates. Figure 1-1 presents a dia‐
gram of the application. Throughout this chapter, we build up this application, first
using YAML configuration files and then Helm charts.
1

Figure 1-1. An application diagram
Managing Configuration Files
Before we get into the details of how to construct this application in Kubernetes, it is
worth discussing how we manage the configurations themselves. With Kubernetes,
everything is represented declaratively. This means that you write down the desired
state of the application in the cluster (generally in YAML or JSON files), and these
declared desired states define all of the pieces of your application. This declarative
approach is far preferable to an imperative approach in which the state of your cluster
is the sum of a series of changes to the cluster. If a cluster is configured imperatively,
it is very difficult to understand and replicate how the cluster came to be in that state.
This makes it very challenging to understand or recover from problems with your
application.
When declaring the state of your application, people typically prefer YAML to JSON,
though Kubernetes supports them both. This is because YAML is somewhat less ver‐
bose and more human editable than JSON. However, it’s worth noting that YAML is
indentation sensitive; often errors in Kubernetes configurations can be traced to
incorrect indentation in YAML. If things aren’t behaving as expected, indentation is a
good thing to check.
Because the declarative state contained in these YAML files serves as the source of
truth for your application, correct management of this state is critical to the success of
your application. When modifying your application’s desired state, you will want to
be able to manage changes, validate that they are correct, audit who made changes,
2 | Chapter 1: Setting Up a Basic Service

and possibly roll things back if they fail. Fortunately, in the context of software engi‐
neering, we have already developed the tools necessary to manage both changes to
the declarative state as well as audit and rollback. Namely, the best practices around
both version control and code review directly apply to the task of managing the
declarative state of your application.
These days most people store their Kubernetes configurations in Git. Though the spe‐
cific details of the version control system are unimportant, many tools in the Kuber‐
netes ecosystem expect files in a Git repository. For code review there is much more
heterogeneity, though clearly GitHub is quite popular, others use on-premises code
review tools or services. Regardless of how you implement code review for your
application configuration, you should treat it with the same diligence and focus that
you apply to source control.
When it comes to laying out the filesystem for your application, it’s generally worth‐
while to use the directory organization that comes with the filesystem to organize
your components. Typically, a single directory is used to encompass an Application
Service for whatever definition of Application Service is useful for your team. Within
that directory, subdirectories are used for subcomponents of the application.
For our application, we lay out the files as follows:
journal/
frontend/
redis/
fileserver/
Within each directory are the concrete YAML files needed to define the service. As
you’ll see later on, as we begin to deploy our application to multiple different regions
or clusters, this file layout will become more complicated.
Creating a Replicated Service Using Deployments
To describe our application, we’ll begin at the frontend and work downward. The
frontend application for the journal is a Node.js application implemented in Type‐
Script. The complete application is slightly too large to include in the book. The
application exposes an HTTP service on port 8080 that serves requests to the /api/*
path and uses the Redis backend to add, delete, or return the current journal entries.
This application can be built into a container image using the included Dockerfile
and pushed to your own image repository. Then, substitute this image name in the
YAML examples that follow.
Creating a Replicated Service Using Deployments | 3

Best Practices for Image Management
Though in general, building and maintaining container images is beyond the scope of
this book, it’s worthwhile to identify some general best practices for building and
naming images. In general, the image build process can be vulnerable to “supply-
chain attacks.” In such attacks, a malicious user injects code or binaries into some
dependency from a trusted source that is then built into your application. Because of
the risk of such attacks, it is critical that when you build your images you base them
on only well-known and trusted image providers. Alternately, you can build all your
images from scratch. Building from scratch is easy for some languages (e.g., Go) that
can build static binaries, but it is significantly more complicated for interpreted lan‐
guages like Python, JavaScript, or Ruby.
The other best practices for images relate to naming. Though the version of a con‐
tainer image in an image registry is theoretically mutable, you should treat the ver‐
sion tag as immutable. In particular, some combination of the semantic version and
the SHA hash of the commit where the image was built is a good practice for naming
images (e.g., v1.0.1-bfeda01f). If you don’t specify an image version, latest is used by
default. Although this can be convenient in development, it is a bad idea for produc‐
tion usage because latest is clearly being mutated every time a new image is built.
Creating a Replicated Application
Our frontend application is stateless; it relies entirely on the Redis backend for its
state. As a result, we can replicate it arbitrarily without affecting traffic. Though our
application is unlikely to sustain large-scale usage, it’s still a good idea to run with at
least two replicas so that you can handle an unexpected crash or roll out a new ver‐
sion of the application without downtime.
Though in Kubernetes, a ReplicaSet is the resource that manages replicating a con‐
tainerized application, so it is not a best practice to use it directly. Instead, you use the
Deployment resource. A Deployment combines the replication capabilities of Repli‐
caSet with versioning and the ability to perform a staged rollout. By using a Deploy‐
ment you can use Kubernetes’ built-in tooling to move from one version of the
application to the next.
The Kubernetes Deployment resource for our application looks as follows:
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
labels:
app: frontend
name: frontend
namespace: default
spec:

replicas: 2
selector:
matchLabels:
app: frontend
template:
metadata:
labels:
app: frontend
spec:
containers:
- image: my-repo/journal-server:v1-abcde
imagePullPolicy: IfNotPresent
name: frontend
resources:
request:
cpu: "1.0"
memory: "1G"
limits:
cpu: "1.0"
memory: "1G"
There are several things to note in this Deployment. First is that we are using Labels
to identify the Deployment as well as the ReplicaSets and the pods that the Deploy‐
ment creates. We’ve added the app: frontend label to all of these resources so that we
can examine all resources for a particular layer in a single request. You’ll see that as
we add other resources, we’ll follow the same practice.
Additionally, we’ve added comments in a number of places in the YAML. Although
these comments don’t make it into the Kubernetes resource stored on the server, just
like comments in code, they serve to help guide people who are looking at this con‐
figuration for the first time.
You should also note that for the containers in the Deployment we have specified
both Request and Limit resource requests, and we’ve set Request equal to Limit.
When running an application, the Request is the reservation that is guaranteed on the
host machine where it runs. The Limit is the maximum resource usage that the con‐
tainer will be allowed. When you are starting out, setting Request equal to Limit will
lead to the most predictable behavior of your application. This predictability comes at
the expense of resource utilization. Because setting Request equal to Limit prevents
your applications from overscheduling or consuming excess idle resources, you will
not be able to drive maximal utilization unless you tune Request and Limit very, very
carefully. As you become more advanced in your understanding of the Kubernetes
resource model, you might consider modifying Request and Limit for your applica‐
tion independently, but in general most users find that the stability from predictabil‐
ity is worth the reduced utilization.
Now that we have the Deployment resource defined, we’ll check it into version con‐
trol, and deploy it to Kubernetes:
Creating a Replicated Service Using Deployments | 5

git add frontend/deployment.yaml
git commit -m "Added deployment" frontend/deployment.yaml
kubectl apply -f frontend/deployment.yaml
It is also a best practice to ensure that the contents of your cluster exactly match the
contents of your source control. The best pattern to ensure this is to adopt a GitOps
approach and deploy to production only from a specific branch of your source con‐
trol, using Continuous Integration (CI)/Continuous Delivery (CD) automation. In
this way you’re guaranteed that source control and production match. Though a full
CI/CD pipeline might seem excessive for a simple application, the automation by
itself, independent of the reliability it provides, is usually worth the time taken to set
it up. And CI/CD is extremely difficult to retrofit into an existing, imperatively
deployed application.
There are also some pieces of this application description YAML (e.g., the ConfigMap
and secret volumes) as well as pod Quality of Service that we examine in later
sections.
Setting Up an External Ingress for HTTP Traffic
The containers for our application are now deployed, but it’s not currently possible
for anyone to access the application. By default, cluster resources are available only
within the cluster itself. To expose our application to the world, we need to create a
Service and load balancer to provide an external IP address and to bring traffic to our
containers. For the external exposure we are actually going to use two Kubernetes
resources. The first is a Service that load-balances Transmission Control Protocol
(TCP) or User Datagram Protocol (UDP) traffic. In our case, we’re using the TCP
protocol. And the second is an Ingress resource, which provides HTTP(S) load bal‐
ancing with intelligent routing of requests based on HTTP paths and hosts. With a
simple application like this, you might wonder why we choose to use the more com‐
plex Ingress, but as you’ll see in later sections, even this simple application will be
serving HTTP requests from two different services. Furthermore, having an Ingress
at the edge enables flexibility for future expansion of our service.
Before the Ingress resource can be defined, there needs to be a Kubernetes Service for
the Ingress to point to. We’ll use Labels to direct the Service to the pods that we cre‐
ated in the previous section. The Service is significantly simpler to define than the
Deployment and looks as follows:
apiVersion: v1
kind: Service
metadata:
labels:
app: frontend
name: frontend
namespace: default

spec:
ports:
- port: 8080
protocol: TCP
targetPort: 8080
selector:
app: frontend
type: ClusterIP
After you’ve defined the Service, you can define an Ingress resource. Unlike Service
resources, Ingress requires an Ingress controller container to be running in the clus‐
ter. There are a number of different implementations you can choose from, either
provided by your cloud provider, or implemented using open source servers. If you
choose to install an open source ingress provider, it’s a good idea to use the Helm
package manager to install and maintain it. The nginx or haproxy Ingress providers
are popular choices:
kind: Ingress
metadata:
name: frontend-ingress
spec:
rules:
- http:
paths:
- path: /api
backend:
serviceName: frontend
servicePort: 8080
Configuring an Application with ConfigMaps
Every application needs a degree of configuration. This could be the number of jour‐
nal entries to display per page, the color of a particular background, a special holiday
display, or many other types of configuration. Typically, separating such configura‐
tion information from the application itself is a best practice to follow.
There are a couple of different reasons for this separation. The first is that you might
want to configure the same application binary with different configurations depend‐
ing on the setting. In Europe you might want to light up an Easter special, whereas in
China you might want to display a special for Chinese New Year. In addition to this
environmental specialization, there are agility reasons for the separation. Usually a
binary release contains multiple different new features; if you turn on these features
via code, the only way to modify the active features is to build and release a new
binary, which can be an expensive and slow process.
The use of configuration to activate a set of features means that you can quickly (and
even dynamically) activate and deactivate features in response to user needs or
Configuring an Application with ConfigMaps | 7

application code failures. Features can be rolled out and rolled back on a per-feature
basis. This flexibility ensures that you are continually making forward progress with
most features even if some need to be rolled back to address performance or correct‐
ness problems.
In Kubernetes this sort of configuration is represented by a resource called a Config‐
Map. A ConfigMap contains multiple key/value pairs representing configuration
information or a file. This configuration information can be presented to a container
in a pod via either files or environment variables. Imagine that you want to configure
your online journal application to display a configurable number of journal entries
per page. To achieve this, you can define a ConfigMap as follows:
kubectl create configmap frontend-config --from-literal=journalEntries=10
To configure your application, you expose the configuration information as an envi‐
ronment variable in the application itself. To do that, you can add the following to the
container resource in the Deployment that you defined earlier:
...
# The containers array in the PodTemplate inside the Deployment
containers:
- name: frontend
...
env:
- name: JOURNAL_ENTRIES
valueFrom:
configMapKeyRef:
name: frontend-config
key: journalEntries
...
Although this demonstrates how you can use a ConfigMap to configure your applica‐
tion, in the real world of Deployments, you’ll want to roll out regular changes to this
configuration with weekly rollouts or even more frequently. It might be tempting to
roll this out by simply changing the ConfigMap itself, but this isn’t really a best prac‐
tice. There are several reasons for this: the first is that changing the configuration
doesn’t actually trigger an update to existing pods. Only when the pod is restarted is
the configuration applied. Because of this, the rollout isn’t health based and can be ad
hoc or random.
A better approach is to put a version number in the name of the ConfigMap itself.
Instead of calling it frontend-config, call it frontend-config-v1. When you want
to make a change, instead of updating the ConfigMap in place, you create a new v2
ConfigMap, and then update the Deployment resource to use that configuration.
When you do this, a Deployment rollout is automatically triggered, using the appro‐
priate health checking and pauses between changes. Furthermore, if you ever need to
rollback, the v1 configuration is sitting in the cluster and rollback is as simple as
updating the Deployment again.

Managing Authentication with Secrets
So far, we haven’t really discussed the Redis service to which our frontend is connect‐
ing. But in any real application we need to secure connections between our services.
In part this is to ensure the security of users and their data, and in addition, it is
essential to prevent mistakes like connecting a development frontend with a produc‐
tion database.
The Redis database is authenticated using a simple password. It might be convenient
to think that you would store this password in the source code of your application, or
in a file in your image, but these are both bad ideas for a variety of reasons. The first
is that you have leaked your secret (the password) into an environment where you
aren’t necessarily thinking about access control. If you put a password into your
source control, you are aligning access to your source with access to all secrets. This is
probably not correct. You probably will have a broader set of users who can access
your source code than should really have access to your Redis instance. Likewise,
someone who has access to your container image shouldn’t necessarily have access to
your production database.
In addition to concerns about access control, another reason to avoid binding secrets
to source control and/or images is parameterization. You want to be able to use the
same source code and images in a variety of environments (e.g., development, canary,
and production). If the secrets are tightly bound in source code or image, you need a
different image (or different code) for each environment.
Having seen ConfigMaps in the previous section, you might immediately think that
the password could be stored as a configuration and then populated into the applica‐
tion as an application-specific configuration. You’re absolutely correct to believe that
the separation of configuration from application is the same as the separation of
secrets from application. But the truth is that a secret is an important concept by
itself. You likely want to handle access control, handling, and updates of secrets in a
different way than a configuration. More important, you want your developers think‐
ing differently when they are accessing secrets than when they are accessing configu‐
ration. For these reasons, Kubernetes has a built-in Secret resource for managing
secret data.
You can create a secret password for your Redis database as follows:
kubectl create secret generic redis-passwd --from-literal=passwd=${RANDOM}
Obviously, you might want to use something other than a random number for your
password. Additionally, you likely want to use a secret/key management service,
either via your cloud provider, like Microsoft Azure Key Vault, or an open source
project, like HashiCorp’s Vault. When you are using a key management service, they
generally have tighter integration with Kubernetes secrets.
Managing Authentication with Secrets | 9

Secrets in Kubernetes are stored unecrypted by default. If you want
to store secrets encrypted, you can integrate with a key provider to
give you a key that Kubernetes will use to encrypt all of the secrets
in the cluster. Note that although this secures the keys against
direct attacks to the etcd database, you still need to ensure that
access via the Kubernetes API server is properly secured.
After you have stored the Redis password as a secret in Kubernetes, you then need to
bind that secret to the running application when deployed to Kubernetes. To do this,
you can use a Kubernetes Volume. A Volume is effectively a file or directory that can
be mounted into a running container at a user-specified location. In the case of
secrets, the Volume is created as a tmpfs RAM-backed filesystem and then mounted
into the container. This ensures that even if the machine is physically compromised
(quite unlikely in the cloud, but possible in the datacenter), the secrets are much
more difficult to obtain by the attacker.
To add a secret volume to a Deployment, you need to specify two new entries in the
YAML for the Deployment. The first is a volume entry for the pod that adds the vol‐
ume to the pod:
...
volumes:
- name: passwd-volume
secret:
secretName: redis-passwd
With the volume in the pod, you need to mount it into a specific container. You do
this via the volumeMounts field in the container description:
...
volumeMounts:
readOnly: true
mountPath: "/etc/redis-passwd"
...
This mounts the secret volume into the redis-passwd directory for access from the
client code. Putting this all together, you have the complete Deployment as follows:
kind: Deployment
metadata:
labels:
app: frontend
name: frontend
namespace: default
spec:
replicas: 2
selector:
matchLabels:

app: frontend
template:
metadata:
labels:
app: frontend
spec:
containers:
- image: my-repo/journal-server:v1-abcde
imagePullPolicy: IfNotPresent
name: frontend
volumeMounts:
readOnly: true
mountPath: "/etc/redis-passwd"
resources:
requests:
cpu: "1.0"
memory: "1G"
limits:
cpu: "1.0"
memory: "1G"
volumes:
secret:
At this point we have configured the client application to have a secret available to
authenticate to the Redis service. Configuring Redis to use this password is similar;
we mount it into the Redis pod and load the password from the file.
Deploying a Simple Stateful Database
Although conceptually deploying a stateful application is similar to deploying a client
like our frontend, state brings with it more complications. The first is that in Kuber‐
netes a pod can be rescheduled for a number of reasons, such as node health, an
upgrade, or rebalancing. When this happens, the pod might move to a different
machine. If the data associated with the Redis instance is located on any particular
machine or within the container itself, that data will be lost when the container
migrates or restarts. To prevent this, when running stateful workloads in Kubernetes
its important to use remote PersistentVolumes to manage the state associated with the
application.
There is a wide variety of different implementations of PersistentVolumes in Kuber‐
netes, but they all share common characteristics. Like secret volumes described ear‐
lier, they are associated with a pod and mounted into a container at a particular
location. Unlike secrets, PersistentVolumes are generally remote storage mounted
through some sort of network protocol, either file based, such as Network File System
(NFS) or Server Message Block (SMB), or block based (iSCSI, cloud-based disks,
Deploying a Simple Stateful Database | 11

etc.). Generally, for applications such as databases, block-based disks are preferable
because they generally offer better performance, but if performance is less of a con‐
sideration, file-based disks can sometimes offer greater flexibility.
Managing state in general is complicated, and Kubernetes is no
exception. If you are running in an environment that supports
stateful services (e.g., MySQL as a service, Redis as a service), it is
generally a good idea to use those stateful services. Initially, the cost
premium of a stateful Software as a Service (SaaS) might seem
expensive, but when you factor in all the operational requirements
of state (backup, data locality, redundancy, etc.), and the fact that
the presence of state in a Kubernetes cluster makes it difficult to
move applications between clusters, it becomes clear that, in most
cases, storage SaaS is worth the price premium. In on-premises
environments where storage SaaS isn’t available, having a dedicated
team provide storage as a service to the entire organization is defi‐
nitely a better practice than allowing each team to roll its own.
To deploy our Redis service, we use a StatefulSet resource. Added after the initial
Kubernetes release as a complement to ReplicaSet resources, a StatefulSet gives
slightly stronger guarantees such as consistent names (no random hashes!) and a
defined order for scale-up and scale-down. When you are deploying a singleton, this
is somewhat less important, but when you want to deploy replicated state, these
attributes are very convenient.
To obtain a PersistentVolume for our Redis, we use a PersistentVolumeClaim. You
can think of a claim as a “request for resources.” Our Redis declares abstractly that it
wants 50 GB of storage, and the Kubernetes cluster determines how to provision an
appropriate PersistentVolume. There are two reasons for this. The first is so that we
can write a StatefulSet that is portable between different clouds and on-premises,
where the details of disks might be different. The other reason is that although many
PersistentVolume types can be mounted to only a single pod, we can use volume
claims to write a template that can be replicated and yet have each pod assigned its
own specific PersistentVolume.
The following example shows a Redis StatefulSet with PersistentVolumes:
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: redis
spec:
serviceName: "redis"
replicas: 1
selector:
matchLabels:

app: redis
template:
metadata:
labels:
app: redis
spec:
containers:
- name: redis
image: redis:5-alpine
ports:
- containerPort: 6379
name: redis
volumeMounts:
- name: data
mountPath: /data
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 10Gi
This deploys a single instance of your Redis service, but suppose you want to replicate
the Redis cluster for scale-out of reads and resiliency to failures. To do this you need
to obviously increase the number of replicas to three, but you also need to ensure that
the two new replicas connect to the write master for Redis.
When you create the headless Service for the Redis StatefulSet, it creates a DNS entry
redis-0.redis; this is the IP address of the first replica. You can use this to create a
simple script that can launch in all of the containters:
#!/bin/sh
PASSWORD=$(cat /etc/redis-passwd/passwd)
if [[ "${HOSTNAME}" == "redis-0" ]]; then
redis-server --requirepass ${PASSWORD}
else
redis-server --slaveof redis-0.redis 6379 --masterauth ${PASSWORD} --
requirepass ${PASSWORD}
fi
You can create this script as a ConfigMap:
kubectl create configmap redis-config --from-file=./launch.sh
You then add this ConfigMap to your StatefulSet and use it as the command for the
container. Let’s also add in the password for authentication that we created earlier in
the chapter.
Deploying a Simple Stateful Database | 13

The complete three-replica Redis looks as follows:
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: redis
spec:
serviceName: "redis"
replicas: 3
selector:
matchLabels:
app: redis
template:
metadata:
labels:
app: redis
spec:
containers:
- name: redis
image: redis:5-alpine
ports:
- containerPort: 6379
name: redis
volumeMounts:
- name: data
mountPath: /data
- name: script
mountPath: /script/launch.sh
subPath: launch.sh
mountPath: /etc/redis-passwd
command:
- sh
- -c
- /script/launch.sh
volumes:
- name: script
configMap:
name: redis-config
defaultMode: 0777
secret:
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 10Gi

Creating a TCP Load Balancer by Using Services
Now that we’ve deployed the stateful Redis service, we need to make it available to
our frontend. To do this, we create two different Kubernetes Services. The first is the
Service for reading data from Redis. Because Redis is replicating the data to all three
members of the StatefulSet, we don’t care which read our request goes to. Conse‐
quently, we use a basic Service for the reads:
apiVersion: v1
kind: Service
metadata:
labels:
app: redis
name: redis
namespace: default
spec:
ports:
- port: 6379
protocol: TCP
targetPort: 6379
selector:
app: redis
sessionAffinity: None
type: ClusterIP
To enable writes, you need to target the Redis master (replica #0). To do this, create a
headless Service. A headless Service doesn’t have a cluster IP address; instead, it pro‐
grams a DNS entry for every pod in the StatefulSet. This means that we can access
our master via the redis-0.redis DNS name:
apiVersion: v1
kind: Service
metadata:
labels:
app: redis-write
name: redis-write
spec:
clusterIP: None
ports:
- port: 6379
selector:
app: redis
Thus, when we want to connect to Redis for writes or transactional read/write pairs,
we can build a separate write client connected to the redis-0.redis-write server.
Creating a TCP Load Balancer by Using Services | 15

Using Ingress to Route Traffic to a Static File Server
The final component in our application is a static file server. The static file server is
responsible for serving HTML, CSS, JavaScript, and image files. It’s both more
efficient and more focused for us to separate static file serving from our API serving
frontend described earlier. We can easily use a high-performance static off-the-shelf
file server like NGINX to serve files while we allow our development teams to focus
on the code needed to implement our API.
Fortunately, the Ingress resource makes this source of mini-microservice architecture
very easy. Just like the frontend, we can use a Deployment resource to describe a
replicated NGINX server. Let’s build the static images into the NGINX container and
deploy them to each replica. The Deployment resource looks as follows:
kind: Deployment
metadata:
labels:
app: fileserver
name: fileserver
namespace: default
spec:
replicas: 2
selector:
matchLabels:
app: fileserver
template:
metadata:
labels:
app: fileserver
spec:
containers:
# This image is intended as an example, replace it with your own
# static files image.
- image: my-repo/static-files:v1-abcde
imagePullPolicy: Always
name: fileserver
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
resources:
request:
cpu: "1.0"
memory: "1G"
limits:
cpu: "1.0"
memory: "1G"
dnsPolicy: ClusterFirst
restartPolicy: Always

Now that there is a replicated static web server up and running, you will likewise cre‐
ate a Service resource to act as a load balancer:
apiVersion: v1
kind: Service
metadata:
labels:
app: fileserver
name: fileserver
namespace: default
spec:
ports:
- port: 80
protocol: TCP
targetPort: 80
selector:
app: fileserver
sessionAffinity: None
type: ClusterIP
Now that you have a Service for your static file server, extend the Ingress resource to
contain the new path. It’s important to note that you must place the / path after
the /api path, or else it would subsume /api and direct API requests to the static file
server. The new Ingress looks like this:
kind: Ingress
metadata:
name: frontend-ingress
spec:
rules:
- http:
paths:
- path: /api
backend:
serviceName: fileserver
servicePort: 8080
# NOTE: this should come after /api or else it will hijack requests
- path: /
backend:
serviceName: fileserver
servicePort: 80
Parameterizing Your Application by Using Helm
Everything that we have discussed so far focuses on deploying a single instance of our
service to a single cluster. However, in reality, nearly every service and every service
team is going to need to deploy to multiple different environments (even if they share
a cluster). Even if you are a single developer working on a single application, you
likely want to have at least a development version and a production version of your
Parameterizing Your Application by Using Helm | 17

application so that you can iterate and develop without breaking production users.
After you factor in integration testing and CI/CD, it’s likely that even with a single
service and a handful of developers, you’ll want to deploy to at least three different
environments, and possibly more if you consider handling datacenter-level failures.
An initial failure mode for many teams is to simply copy the files from one cluster to
another. Instead of having a single frontend/ directory, have a frontend-production/
and frontend-development/ pair of directories. The reason this is so dangerous is
because you are now in charge of ensuring that these files remain synchronized with
one another. If they were intended to be entirely identical, this might be easy, but
some skew between development and production is expected because you will be
developing new features; it’s critical that the skew is both intentional, and easily
managed.
Another option to achieve this would be to use branches and version control, with
the production and development branches leading off from a central repository, and
the differences between the branches clearly visible. This can be a viable option for
some teams, but the mechanics of moving between branches are challenging when
you want to simultaneously deploy software to different environments (e.g., a CI/CD
system that deploys to a number of different cloud regions).
Consequently, most people end up with a templating system. A templating system
combines templates, which form the centralized backbone of the application configu‐
ration, with parameters that specialize the template to a specific environment configu‐
ration. In this way, you can have a generally shared configuration, with intentional
(and easily understood) customization as needed. There are a variety of different
template systems for Kubernetes, but the most popular by far is a system called Helm.
In Helm, an application is packaged in a collection of files called a chart (nautical
jokes abound in the world of containers and Kubernetes).
A chart begins with a chart.yaml file, which defines the metadata for the chart itself:
apiVersion: v1
appVersion: "1.0"
description: A Helm chart for our frontend journal server.
name: frontend
version: 0.1.0
This file is placed in the root of the chart directory (e.g., frontend/). Within this direc‐
tory, there is a templates directory, which is where the templates are placed. A tem‐
plate is basically a YAML file from the previous examples, with some of the values in
the file replaced with parameter references. For example, imagine that you want to
parameterize the number of replicas in your frontend. Previously, here’s what the
Deployment had:

...
spec:
replicas: 2
...
In the template file (frontend-deployment.tmpl), it instead looks like the following:
...
spec:
replicas: {{ .replicaCount }}
...
This means that when you deploy the chart, you’ll substitute the value for replicas
with the appropriate parameter. The parameters themselves are defined in a val‐
ues.yaml file. There will be one values file per environment where the application
should be deployed. The values file for this simple chart would look like this:
replicaCount: 2
Putting this all together, you can deploy this chart using the helm tool, as follows:
helm install path/to/chart --values path/to/environment/values.yaml
This parameterizes your application and deploys it to Kubernetes. Over time these
parameterizations will grow to encompass the variety of different environments for
your application.
Deploying Services Best Practices
Kubernetes is a powerful system that can seem complex. But setting up a basic appli‐
cation for success can be straightforward if you use the following best practices:
• Most services should be deployed as Deployment resources. Deployments create
identical replicas for redundancy and scale.
• Deployments can be exposed using a Service, which is effectively a load balancer.
A Service can be exposed either within a cluster (the default) or externally. If you
want to expose an HTTP application, you can use an Ingress controller to add
things like request routing and SSL.
• Eventually you will want to parameterize your application to make its configura‐
tion more reusable in different environments. Packaging tools like Helm are the
best choice for this kind of parameterization.
Summary
The application built in this chapter is a simple one, but it contains nearly all of the
concepts you’ll need to build larger, more complicated applications. Understanding
Deploying Services Best Practices | 19

how the pieces fit together and how to use foundational Kubernetes components is
key to successfully working with Kubernetes.
Laying the correct foundation via version control, code review, and continuous deliv‐
ery of your service ensures that no matter what you build, it is built in a solid manner.
As we go through the more advanced topics in subsequent chapters, keep this foun‐
dational information in mind.

CHAPTER 2
Developer Workflows
Kubernetes was built for reliably operating software. It simplifies deploying and man‐
aging applications with an application-oriented API, self-healing properties, and use‐
ful tools like Deployments for zero downtime rollout of software. Although all of
these tools are useful, they don’t do much to make it easier to develop applications for
Kubernetes. Furthermore, even though many clusters are designed to run production
applications and thus are rarely accessed by developer workflows, it is also critical to
enable development workflows to target Kubernetes, and this typically means having
a cluster or at least part of a cluster that is intended for development. Setting up such
a cluster to facilitate easy development of applications for Kubernetes is a critical part
of ensuring success with Kubernetes. Clearly if there is no code being built for your
cluster, the cluster itself isn’t accomplishing much.
Goals
Before we describe the best practices for building out development clusters, it is
worth stating our goals for such clusters. Obviously, the ultimate goal is to enable
developers to rapidly and easily build applications on Kubernetes, but what does that
really mean in practice and how is that reflected in practical features of the develop‐
ment cluster?
It is useful to identify phases of developer interaction with the cluster.
The first phase is onboarding. This is when a new developer joins the team. This
phase includes giving the user a login to the cluster as well as getting them oriented to
their first deployment. The goal for this phase is to get a developer’s feet wet in a min‐
imal amount of time. You should set a key performance indicator (KPI) goal for this
process. A reasonable goal would be that a user could go from nothing to the current
21

application at HEAD running in less than half an hour. Every time someone is new to
the team, test how you are doing against this goal.
The second phase is developing. This is the day-to-day activity of the developer. The
goal for this phase is to ensure rapid iteration and debugging. Developers need to
quickly and repeatedly push code to the cluster. They also need to be able to easily
test their code and debug it when it isn’t operating properly. The KPI for this phase is
more challenging to measure, but you can estimate it by measuring the time to get a
pull request (PR) or change up and running in the cluster, or with surveys of the
user’s perceived productivity, or both. You will also be able to measure this in the
overall productivity of your teams.
The third phase is testing. This phase is interleaved with developing and is used to
validate the code before submission and merging. The goals for this phase are two-
fold. First, the developer should be able to run all tests for their environment before a
PR is submitted. Second, all tests should automatically run before code is merged into
the repository. In addition to these goals you should also set a KPI for the length of
time the tests take to run. As your project becomes more complex, it’s natural for
more and more tests to take a longer time. As this happens, it might become valuable
to identify a smaller set of smoke tests that a developer can use for initial validation
before submitting a PR. You should also have a very strict KPI around test flakiness. A
flaky test is one that occasionally (or not so occasionally) fails. In any reasonably
active project, a flakiness rate of more than one failure per one thousand runs will
lead to developer friction. You need to ensure that your cluster environment does not
lead to flaky tests. Whereas sometimes flaky tests occur due to problems in the code,
they can also occur because of interference in the development environment (e.g.,
running out of resources and noisy neighbors). You should ensure that your develop‐
ment environment is free of such issues by measuring test flakiness and acting
quickly to fix it.
Building a Development Cluster
When people begin to think about developing on Kubernetes, one of the first choices
that occurs is whether to build a single large development cluster or to have one clus‐
ter per developer. Note that this choice only makes sense in an environment in which
dynamic cluster creation is easy, such as the public cloud. In physical environments,
its possible that one large cluster is the only choice.
If you do have a choice you should consider the pros and cons of each option. If you
choose to have a development cluster per user, the significant downside of this
approach is that it will be more expensive and less efficient, and you will have a large
number of different development clusters to manage. The extra costs come from the
fact that each cluster is likely to be heavily underutilized. Also, with developers creat‐
ing different clusters, it becomes more difficult to track and garbage-collect resources
22 | Chapter 2: Developer Workflows

that are no longer in use. The advantage of the cluster-per-user approach is simplic‐
ity: each developer can self-service manage their own cluster, and from isolation, it’s
much more difficult for different developers to step on one another’s toes.
On the other hand, a single development cluster will be significantly more efficient;
you can likely sustain the same number of developers on a shared cluster for one-
third the price (or less). Plus, it’s much easier for you to install shared cluster services,
for example, monitoring and logging, which makes it significantly easier to produce a
developer-friendly cluster. The downside of a shared development cluster is the pro‐
cess of user management and potential interference between developers. Because the
process of adding new users and namespaces to the Kubernetes cluster isn’t currently
streamlined, you will need to activate a process to onboard new developers. Although
Kubernetes resource management and Role-Based Access Control (RBAC) can
reduce the probability that two developers conflict, it is always possible that a user
will brick the development cluster by consuming too many resources so that other
applications and developers won’t schedule. Additionally, you will still need to ensure
that developers don’t leak and forget about resources they’ve created. This is some‐
what easier, though, than the approach in which developers each create their own
clusters.
Even though both approaches are feasible, generally, our recommendation is to have a
single large cluster for all developers. Although there are challenges in interference
between developers, they can be managed and ultimately the cost efficiency and abil‐
ity to easily add organization-wide capabilities to the cluster outweigh the risks of
interference. But you will need to invest in a process for onboarding developers,
resource management, and garbage collection. Our recommendation would be to try
a single large cluster as a first option. As your organization grows (or if it is already
large), you might consider having a cluster per team or group (10 to 20 people) rather
than a giant cluster for hundreds of users. This can make both billing and manage‐
ment easier.
Setting Up a Shared Cluster for Multiple Developers
When setting up a large cluster, the primary goal is to ensure that multiple users can
simultaneously use the cluster without stepping on one another’s toes. The obvious
way to separate your different developers is with Kubernetes namespaces. Namespa‐
ces can serve as scopes for the deployment of services so that one user’s frontend ser‐
vice doesn’t interfere with another user’s frontend service. Namespaces are also scopes
for RBAC, ensuring that one developer cannot accidentally delete another developer’s
work. Thus, in a shared cluster it makes sense to use a namespace as a developer’s
workspace. The processes for onboarding users and creating and securing a name‐
space are described in the following sections.
Setting Up a Shared Cluster for Multiple Developers | 23

Onboarding Users
Before you can assign a user to a namespace, you have to onboard that user to the
Kubernetes cluster itself. To achieve this, there are two options. You can use
certificate-based authentication to create a new certificate for the user and give them
a kubeconfig file that they can use to log in, or you can configure your cluster to use
an external identity system (for example, Microsoft Azure Active Directory or AWS
Identity and Access Management [IAM]) for cluster access.
In general, using an external identity system is a best practice because it doesn’t
require that you maintain two different sources of identity, but in some cases this isn’t
possible and you need to use certificates. Fortunately, you can use the Kubernetes cer‐
tificate API for creating and managing such certificates. Here’s the process for adding
a new user to an existing cluster.
First, you need to generate a certificate signing request to generate a new certificate.
Here is a simple Go program to do this:
package main
import (
"crypto/rand"
"crypto/rsa"
"crypto/x509"
"crypto/x509/pkix"
"encoding/asn1"
"encoding/pem"
"os"
)
func main() {
name := os.Args[1]
user := os.Args[2]
key, err := rsa.GenerateKey(rand.Reader, 1024)
if err != nil {
panic(err)
}
keyDer := x509.MarshalPKCS1PrivateKey(key)
keyBlock := pem.Block{
Type: "RSA PRIVATE KEY",
Bytes: keyDer,
}
keyFile, err := os.Create(name + "-key.pem")
if err != nil {
panic(err)
}
pem.Encode(keyFile, &keyBlock)
keyFile.Close()

commonName := user
// You may want to update these too
emailAddress := "someone@myco.com"
org := "My Co, Inc."
orgUnit := "Widget Farmers"
city := "Seattle"
state := "WA"
country := "US"
subject := pkix.Name{
CommonName: commonName,
Country: []string{country},
Locality: []string{city},
Organization: []string{org},
OrganizationalUnit: []string{orgUnit},
Province: []string{state},
}
asn1, err := asn1.Marshal(subject.ToRDNSequence())
if err != nil {
panic(err)
}
csr := x509.CertificateRequest{
RawSubject: asn1,
EmailAddresses: []string{emailAddress},
SignatureAlgorithm: x509.SHA256WithRSA,
}
bytes, err := x509.CreateCertificateRequest(rand.Reader, &csr, key)
if err != nil {
panic(err)
}
csrFile, err := os.Create(name + ".csr")
if err != nil {
panic(err)
}
pem.Encode(csrFile, &pem.Block{Type: "CERTIFICATE REQUEST", Bytes:
bytes})
csrFile.Close()
}
You can run this as follows:
go run csr-gen.go client <user-name>;
This creates files called client-key.pem and client.csr. You then can run the following
script to create and download a new certificate:
#!/bin/bash
csr_name="my-client-csr"

name="${1:-my-user}"
csr="${2}"
cat <<EOF | kubectl create -f -
apiVersion: certificates.k8s.io/v1beta1
kind: CertificateSigningRequest
metadata:
name: ${csr_name}
spec:
groups:
- system:authenticated
request: $(cat ${csr} | base64 | tr -d 'n')
usages:
- digital signature
- key encipherment
- client auth
EOF
echo
echo "Approving signing request."
kubectl certificate approve ${csr_name}
echo
echo "Downloading certificate."
kubectl get csr ${csr_name} -o jsonpath='{.status.certificate}'
| base64 --decode > $(basename ${csr} .csr).crt
echo
echo "Cleaning up"
kubectl delete csr ${csr_name}
echo
echo "Add the following to the 'users' list in your kubeconfig file:"
echo "- name: ${name}"
echo " user:"
echo " client-certificate: ${PWD}/$(basename ${csr} .csr).crt"
echo " client-key: ${PWD}/$(basename ${csr} .csr)-key.pem"
echo
echo "Next you may want to add a role-binding for this user."
This script prints out the final information that you can add to a kubeconfig file to
enable that user. Of course, the user has no access privileges, so you will need to apply
Kubernetes RBAC for the user in order to grant them privileges to a namespace.

Creating and Securing a Namespace
The first step in provisioning a namespace is actually just creating it. You can do this
using kubectl create namespace my-namespace.
But the truth is that when you create a namespace, you want to attach a bunch of
metadata to that namespace, for example, the contact information for the team that
builds the component deployed into the namespace. Generally, this is in the form of
annotations; you can either generate the YAML file using some templating, such as
Jinja or others, or you can create and then annotate the namespace. A simple script to
do this looks like:
ns='my-namespace'
kubectl create namespace ${ns}
kubectl annotate namespace ${ns} annotation_key=annotation_value
When the namespace is created, you want to secure it by ensuring that you can grant
access to the namespace to a specific user. To do this, you can bind a role to a user in
the context of that namespace. You do this by creating a RoleBinding object within
the namespace itself. The RoleBinding might look like this:
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: example
namespace: my-namespace
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: edit
subjects:
- apiGroup: rbac.authorization.k8s.io
kind: User
name: myuser
To create it, you simply run kubectl create -f role-binding.yaml. Note that you
can reuse this binding as much as you want so long as you update the namespace in
the binding to point to the correct namespace. If you ensure that the user doesn’t have
any other role bindings, you can be assured that this namespace is the only part of the
cluster to which the user has access. A reasonable practice is to also grant reader
access to the entire cluster; in this way developers can see what others are doing in
case it is interfering with their work. Be careful in granting such read access, however,
because it will include access to secret resources in the cluster. Generally, in a devel‐
opment cluster this is OK because everyone is in the same organization and the
secrets are used only for development; however, if this is a concern, then you can cre‐
ate a more fine-grained role that eliminates the ability to read secrets.

If you want to limit the amount of resources consumed by a particular namespace,
you can use the ResourceQuota resource to set a limit to the total number of resour‐
ces that any particular namespace consumes. For example, the following quota limits
the namespace to 10 cores and 100 GB of memory for both Request and Limit for the
pods in the namespace:
apiVersion: v1
kind: ResourceQuota
metadata:
name: limit-compute
namespace: my-namespace
spec:
hard:
requests.cpu: "10"
requests.memory: 100Gi
limits.cpu: "10"
limits.memory: 100Gi
Managing Namespaces
Now that you have seen how to onboard a new user and how to create a namespace to
use as a workspace, the question remains how to assign a developer to the namespace.
As with many things, there is no single perfect answer; rather, there are two
approaches. The first is to give each user their own namespace as part of the onboard‐
ing process. This is useful because after a user is onboarded, they always have a dedi‐
cated workspace in which they can develop and manage their applications. However,
making the developer’s namespace too persistent encourages the developer to leave
things lying around in the namespace after they are done with them, and garbage-
collecting and accounting individual resources is more complicated. An alternate
approach is to temporarily create and assign a namespace with a bounded time to live
(TTL). This ensures that the developer thinks of the resources in the cluster as transi‐
ent and that it is easy to build automation around the deletion of entire namespaces
when their TTL has expired.
In this model, when the developer wants to begin a new project, they use a tool to
allocate a new namespace for the project. When they create the namespace, it has a
selection of metadata associated with the namespace for management and account‐
ing. Obviously, this metadata includes the TTL for the namespace, but it also includes
the developer to which it is assigned, the resources that should be allocated to the
namespace (e.g., CPU and memory), and the team and purpose of the namespace.
This metadata ensures that you can both track resource usage and delete the name‐
space at the right time.
Developing the tooling to allocate namespaces on demand can seem like a challenge,
but simple tooling is relatively simple to develop. For example, you can achieve the

allocation of a new namespace with a simple script that creates the namespace and
prompts for the relevant metadata to attach to the namespace.
If you want to get more integrated with Kubernetes, you can use custom resource def‐
initions (CRDs) to enable users to dynamically create and allocate new namespaces
using the kubectl tool. If you have the time and inclination, this is definitely a good
practice because it makes namespace management declarative and also enables the
use of Kubernetes RBAC.
After you have tooling to enable the allocation of namespaces, you also need to add
tooling to reap namespaces when their TTL has expired. Again, you can accomplish
this with a simple script that examines the namespaces and deletes those that have an
expired TTL.
You can build this script into a container and use a ScheduledJob to run it at an inter‐
val like once per hour. Combined together, these tools can ensure that developers can
easily allocate independent resources for their project as needed, but those resources
will also be reaped at the proper interval to ensure that you don’t have wasted resour‐
ces and that old resources don’t get in the way of new development.
Cluster-Level Services
In addition to tooling to allocate and manage namespaces, there are also useful
cluster-level services, and it’s a good idea to enable them in your development cluster.
The first is log aggregation to a central Logging as a Service (LaaS) system. One of the
easiest things for a developer to do to understand the operation of their application is
to write something to STDOUT. Although you can access these logs via kubectl
logs, that log is limited in length and is not particularly searchable. If you instead
automatically ship those logs to a LaaS system such as a cloud service or an Elastic‐
search cluster, developers can easily search through logs for relevant information as
well as aggregate logging information across multiple containers in their service.
Enabling Developer Workflows
Now that we succesfully have a shared cluster setup and we can onboard new applica‐
tion developers to the cluster itself, we need to actually get them developing their
application. Remember that one of the key KPIs that we are measuring is the time
from onboarding to an initial application running in the cluster. It’s clear that via the
just-described onboarding scripts we can quickly authenticate a user to a cluster and
allocate a namespace, but what about getting started with the application? Unfortu‐
nately, even though there are a few techniques that help with this process, it generally
requires more convention than automation to get the initial application up and run‐
ning. In the following sections, we describe one approach to achieving this; it is by no
Enabling Developer Workflows | 29

means the only approach or the only solution. You can optionally apply the approach
as is or be inspired by the ideas to arrive at your own solution.
Initial Setup
One of the main challenges to deploying an application is the installation of all of the
dependencies. In many cases, especially in modern microservice architectures, to
even get started developing on one of the microservices requires the deployment of
multiple dependencies, either databases or other microservices. Although the deploy‐
ment of the application itself is relatively straightforward, the task of identifying and
deploying all of the dependencies to build the complete application is often a frustrat‐
ing case of trial and error married with incomplete or out-of-date instructions.
To address this issue, it is often valuable to introduce a convention for describing and
installing dependencies. This can be seen as the equivalent of something like npm
install, which installs all of the required JavaScript dependencies. Eventually, there
is likely to be a tool similar to npm that provides this service for Kubernetes-based
applications, but until then, the best practice is to rely on convention within your
team.
One such option for a convention is the creation of a setup.sh script within the root
directory of all project repositories. The responsibility of this script is to create all
dependencies within a particular namespace to ensure that all of the application’s
dependencies are correctly created. For example, a setup script might look like the
following:
kubectl create my-service/database-stateful-set-yaml
kubectl create my-service/middle-tier.yaml
kubectl create my-service/configs.yaml
You then could integrate this script with npm by adding the following to your
package.json:
{
...
"scripts": {
"setup": "./setup.sh",
...
}
}
With this setup, a new developer can simply run npm run setup and the cluster
dependencies will be installed. Obviously, this particular integration is Node.js/npm
specific. In other programming languages, it will make more sense to integrate with
the language-specific tooling. For example, in Java you might integrate with a Maven
pom.xml file instead.

Enabling Active Development
Having set up the developer workspace with required dependencies, the next task is
to enable them to iterate on their application quickly. The first prerequisite for this is
the ability to build and push a container image. Let’s assume that you have this
already set up; if not, you can read how to do this in a number of other online resour‐
ces and books.
After you have built and pushed a container image, the task is to roll it out to the
cluster. Unlike traditional rollouts, in the case of developer iteration, maintaining
availability is really not a concern. Thus, the easiest way to deploy new code is to sim‐
ply delete the Deployment object associated with the previous Deployment and then
create a new Deployment pointing to the newly built image. It is also possible to
update an existing Deployment in place, but this will trigger the rollout logic in the
Deployment resource. Although it is possible to configure a Deployment to roll out
code quickly, doing so introduces a difference between the development environment
and the production environment that can be dangerous or destabilizing. Imagine, for
example, that you accidentally push the development configuration of the Deploy‐
ment into production; you will suddenly and accidentally deploy new versions to pro‐
duction without appropriate testing and delays between phases of the rollout. Because
of this risk and because there is an alternative, the best practice is to delete and re-
create the Deployment.
Just like installing dependencies, it is also a good practice to make a script for per‐
forming this deployment. An example deploy.sh script might look like the following:
kubectl delete -f ./my-service/deployment.yaml
perl -pi -e 's/${old_version}/${new_version}/' ./my-service/deployment.yaml
kubectl create -f ./my-service/deployment.yaml
As before, you can integrate this with existing programming language tooling so that
(for example) a developer can simply run npm run deploy to deploy their new code
into the cluster.
Enabling Testing and Debugging
After a user has successfully deployed their development version of their application,
they need to test it and, if there are problems, debug any issues with the application.
This can also be a hurdle when developing in Kubernetes because it is not always
clear how to interact with your cluster. The kubectl command line is a veritable
Swiss army knife of tools to achieve this, from kubectl logs to kubectl exec and
kubectl port-forward, but learning how to use all of the different options and ach‐
ieving familiarity with the tool can take a considerable amount of experience. Fur‐
thermore, because the tool runs in the terminal, it often requires the composition of
Enabling Active Development | 31

multiple windows to simultaneously examine both the source code for the application
and the running application itself.
To streamline the testing and debugging experience, Kubernetes tooling is increas‐
ingly being integrated into development environments, for example, the open source
extension for Visual Studio (VS) Code for Kubernetes. The extension is easily
installed for free from the VS Code marketplace. When installed, it automatically dis‐
covers any clusters that you already have in your kubeconfig file, and it provides a
tree-view navigation pane for you to see the contents of your cluster at a glance.
In addition to being able to see your cluster state at a glance, the integration allows a
developer to use the tools available via kubectl in an intuitive, discoverable way.
From the tree view, if you right-click a Kubernetes pod, you can immediately use port
forwarding to bring a network connection to the pod directly to the local machine.
Likewise, you can access the logs for the pod or even get a terminal within the run‐
ning container.
The integration of these commands with prototypical user interface expectations
(e.g., right-click shows a context menu), as well as the integration of these experiences
alongside the code for the application itself, enable developers with minimal Kuber‐
netes experience to rapidly become productive in the development cluster.
Of course this VS Code extension isn’t the only integration between Kubernetes and a
devlopment environment; there are several others that you can install depending on
your choice of programming environment and style (vi, emacs, etc.).
Setting Up a Development Environment Best Practices
Setting up successful workflows on Kubernetes is key to productivity and happiness.
Following these best practices will help to ensure that developers are up and running
quickly:
• Think about developer experience in three phases: onboarding, developing, and
testing. Make sure that the development environment you build supports all
three of these phases.
• When building a development cluster, you can choose between one large cluster
and a cluster per developer. There are pros and cons to each, but generally a sin‐
gle large cluster is a better approach.
• When you add users to a cluster, add them with their own identity and access to
their own namespace. Use resource limits to restrict how much of the cluster they
can use.
• When managing namespaces, think about how you can reap old, unused resour‐
ces. Developers will have bad hygiene about deleting unused things. Use automa‐
tion to clean it up for them.

Another random document with
no related content on Scribd:

Mame welcomed with enthusiasm the prospect of a small room on
the top floor. The landlady repeated the once-over without
enthusiasm. Should she? Or should she not? An outlandish girl,
American to the bone, but this attic would be none the worse for a
tenant, provided, of course, that she was really a paying one.
Elmer P. Dobree had told Mame more than once that “she was cute
as a bag of monkeys.” The zoölogical resources of five continents
could not have exceeded the flair with which Miss Durrance opened
her vanity bag and produced an impressive roll of Bradburys.
“I’ll be happy to pay a fortnight in advance.” It was Mame’s best
Broadway manner. “Here is the money. I am a very respectable girl.”
Reassured by the sight of the Bradburys rather than by the
Broadway manner, which to the insular taste had a decidedly
cosmopolitan flavour, the landlady went so far as to ask for a name
and references.
“I’m a special European correspondent.” Mame gave a slow and
careful value to each word.
A faint beam pierced the landlady’s gloom. She had feared “an
actress”; although to be just to the girl she didn’t look that sort.
“Here is my card,” Broadway cold drawn and pure, with a dash of
Elmer P. talking over the phone.
The châtelaine of Fotheringay House adjusted a pair of gold-rimmed
eyeglasses and read:

Miss Amethyst Du Rance
New York City, U. S. A.
European Correspondent
Cowbarn Independent
The card was returned to its owner with polite thanks. A subtle
gesture indicated that a sudden rise had occurred in the stock of
Miss Amethyst Du Rance.
“I’m not quite sure, Miss Du Rance, but I may be able to find you a
bedroom on the second floor.”
Victory! The roll had begun the good work, but the card had
consummated it. One up for the Cowbarn Independent.
Iowa’s shrewd daughter had realized already that it would not do to
make a poor mouth in Europe. All the same Aunt Lou’s legacy was
melting like snow. Money must appear to be no object as far as Miss
Amethyst Du Rance was concerned; yet she must watch out or a
whole dollar would not pull more than fifty cents.
“Top floor’ll fix me.” Mame it was who spoke, yet with the lofty voice
of Miss Amethyst Du Rance. “Tariff’ll be less, I reckon, but”—
haughtily detaching a second Bradbury from the wad—“I’ll be most
happy to pay spot cash in advance for a fortnight’s board and
residence.”
Money is quite as eloquent in London, England, as it is in New York
or Seattle or Milpitas, Cal. Mame’s air of affluence combined with a
solid backing of notes did the trick, although the well-bred fashion in
which a dyed-in-the-wool British landlady glossed over the fact
seemed to render it non-existent.
“You have luggage, I presume?”
There was a trunk outside on the taxi.
“The porter will take it up to your room. I will ring for him now.”
Mrs. Toogood suited the word to the action, the action to the word.
She was crisp and decisive, final and definite. Mame felt this lady

was wasted in private life. She ought to have been in Congress.

IV
FIVE minutes later Miss Amethyst Du Rance and all her worldly
goods were assembled in a small musty bedroom at the top of
Fotheringay House. It smelt of damp. There was no grate or stove or
any means of heating. The floor was shod with a very cold-looking
brand of lino. Only a thin layer of cement divided the ceiling from
the tiles of the roof—so thin, indeed, that the all-pervading yellow
fog could almost be seen in the act of percolating through them.
Mrs. Toogood, who had personally conducted her new guest up
three pairs of stairs, lit the gas and drew the curtains across the
narrow window. She then informed Miss Du Rance that dinner was
at half-past seven, but there would be afternoon tea in the drawing
room on the first floor in about half an hour.
Mame took off her coat and hat, removed the stains of travel from a
frank and good-humoured countenance, re-did her hair and applied
a dab of powder to a nose which had a tendency to freckle; and
then she went downstairs. Stirred by a feeling of adventure she
forgot how cold she was; also she forgot the chill that had gathered
about her heart. London, England, was a long, long way from home.
Its climate was thoroughly depressing and the same could be said of
its landladies. Whether the climate produced the landladies or the
landladies produced the climate she had not been long enough in
the island to say.
The light in the drawing room was dim. It half concealed a glory of
aspidistra, lace curtains, anti-macassars and wax fruits. There was a
solemnity about it which by some means had been communicated to
a unique collection of old women who upon sofas and chairs were
collected in a semi-circle round an apology for a fire. Mame could
not repress a shiver as one sibyl after another looked up from her

wool-work or her book and gave the firm-footed and rather
impulsive intruder the benefit of a frozen stare from a glacial eye.
By the time Mame had subsided on the only unoccupied seat within
the fire’s orbit, she felt that a jury of her sex having duly marked and
digested her had come to the unanimous conclusion that she was
guilty of presumption in being upon the earth at all. The detachment
of this bunch of sibyls gave density and weight to the feeling. Their
silence was uncanny. It was only disturbed by the click click of
knitting pins and the occasional creak of the fire.
Mame had been five minutes in a situation which every second made
more irksome, since for the first time in her life she was at a total
loss for speech, when, as Mrs. Toogood had predicted, tea appeared.
That lady, in a démodé black silk dress, which looked like an
heirloom, preceded a metal urn, a jug of hot water, an array of
cracked saucers and cups, some doubtful-looking bread and butter,
and still more doubtful-looking cake. All these things were borne
upon a tray by the prim maid who had first admitted Mame to
Fotheringay House.
The sight of “the eats” cheered Mame up a bit. And in conjunction
with Mrs. Toogood’s arrival, they certainly went some way towards
unsealing the frozen atmosphere of the witches’ parlour. A place was
found for the hostess near the fire; the small maid set up a tea-
table; the cups and saucers began to circulate.
Mame was served last. By then the brew, not strong to begin with,
had grown very thin. “Rather weak, Miss Du Rance, I fear,” loftily
said its dispenser. “I hope you don’t mind.”
Mame, for whom that peculiarly British function which the French
speak of as “le five-o’clock” was a new experience, promptly said
she didn’t mind at all. Her voice was so loud that it fairly shattered
the hush; it was almost as if a bomb had fallen into a prayer-
meeting. Every ear was startled by the power of those broad nasal
tones.

“This is Miss Du Rance of America.” The hostess spoke for the
benefit of the company. It was not so much an introduction as an
explanation; a defence and a plea rather than an attempt at mixing.
Some of the sibyls glared at Mame, some of them scowled. No other
attention was paid her. Yet they were able to make clear that her
invasion of the ancient peace of Fotheringay House was resented.
Little cared the visitor. Set of old tabbies. Bunch of fossilised
mossbacks. She was as good as they. And better. In drawing
comparisons, Miss Du Rance was not in the habit of underrating
herself or of overrating others. And she was a born fighter.
Was it not sheer love of a fight that had brought her to Europe? She
already saw that London was going to be New York over again: a
city of four-flushers, with all sorts of dud refinements and false
delicacies, unknown to Cowbarn, Iowa. Of course she was a little
hick. But cosmopolitan experience was going to improve her. And
cuteness being her long suit, these dames had no need to rub her
rusticity into her quite so good and hearty.
As Mame toyed with a cup of tea that was mere coloured water and
took a chance with the last surviving piece of “spotted dog,” which
had just one raisin beneath a meagre scrape of butter, her quick
mind brought her right up against the facts of the case. Somehow
she wasn’t in the picture. She must study how to fit herself into her
surroundings. That was what she was there for; to see the world
and to put herself right with it.
When in Rome you must do like the Romans, or you’ll bite granite.
Paula Wyse Ling had sprung that. And Paula knew, for she had
travelled. What she really meant was that Mame Durrance must
unlearn most of what she had learned at Cowbarn, Iowa, if she was
going to fire the East.
Cowbarn was the home of the roughneck. But in New York City and
London, England, highbrows swarmed like bees. These places were
the native haunts of that Culture in which Mame had omitted to take
a course.

She was soon convinced this was the dullest party she had ever
been at. But it didn’t prevent her mind from working. Never in her
life had she been more cast down. These people made her feel like
thirty cents. They spoke in hushed and solemn voices. If this was
Europe it would have been wiser to stay on her native continent.
Presently the small maid bore away the teapot and the crockery.
With massive dignity the mistress followed her out. The landlady’s
withdrawal seemed, if it were possible, to add new chills to the
gloom, and Mame having reached the point where she could stand it
no longer, had just decided to make a diversion by going up to her
bedroom and unpacking her trunk when a new interest was lent to
the scene. A man entered the room. Mame’s first thought was that
he must have a big streak of natural folly to venture alone and
unprotected into this nest of sleeping cats.
Strange to say, the temerarious male was made welcome. The
density of the atmosphere lessened as soon as he came in. One old
tabby after another began to sit up and take notice; and Mame,
while busily engaged in watching the newcomer, had a feeling of
gratitude towards him for having cast by his mere presence a ray of
light upon that inspissated gloom.
Certainly he was no common man. As well as Mame could tell he
was as old as the tabbies to whom he made himself so agreeable.
Yet he was old with a difference. His abundant hair, which was snow
white, was brushed in a dandified way, and the note of gallantry was
repeated in every detail of his personality. His clothes, though not
strikingly smart, were worn with an air. There was style in the set of
his necktie, and if his trousers might bag a little at the knees they
somehow retained the cut of a good tailor. Also he wore a monocle
in a way that seemed to add grace and charm to his manner and to
cancel his curious pallor and his look of eld.
Mame was at once deeply interested in this new arrival. No doubt he
was one of the blood-peers, of whom she had read. Down on his
luck perhaps, and for all his pleasant touch of swank, he somehow
suggested it. Besides it stood to reason that a real blood-peer, used

to the best that was going, as this old beau plainly was, would not
be spending his time in a dead-alive hole playing purry-purry puss-
puss if he were not up against it.
All the same there was absolutely nothing in his manner to suggest
a shortage of dough. It was so grand that it seemed to banish any
vulgar question of ways and means. To judge by the way he dandled
his eyeglass while he entertained the tabbies with his measured yet
copious and genial talk, he might have been the King of England
with his beard off.
When the clock on the chimneypiece struck seven the ladies rose in
a body and withdrew to prepare for the evening meal. Their example
was followed by Mame. She would have liked to stay and get into
conversation with the intriguing stranger, but dinner was in half an
hour and it would take some little time to unpack her trunk in which
was a new one-piece dress she was going to wear. Besides there
would be a chance later in the evening, no doubt, of making his
acquaintance.
As it happened this pleasure had to be postponed. To Mame’s
disappointment the old boy did not appear at dinner. But she did not
blame him. The food was meagre and those who ate it were quite
the dullest set she had ever seen. Some of the guests were
accommodated with small separate tables. One of these had been
provided for Miss Du Rance. It was in a draughty corner, exposed to
a strong current of air between two open doors which led to a large
and antiquated lift whereby the meal ascended from the basement.
Mame felt small-town, but she had come to Europe to learn. Even if
the seclusion of a private table had its conveniences she would have
much preferred to mingle with her fellow p.g.’s. She was social by
nature. Besides, she was determined to be a mixer. All these folks
had it in their power to teach her something, duds though they
were. Britain must give up all its secrets to Miss Amethyst Du Rance.
Judging by the dead-beats who swarmed in this fog-bound isle they
might amount to nothing; at the same time one cannot know too

much of one’s subject. For some little time to come the subject for
Miss Du Rance was going to be London, England.

V
THE next morning the fog had lifted and Mame set out for Fleet
Street. By Mrs. Toogood’s advice she boarded Bus 26 which passed
the end of Montacute Square; and having made a friend of the
conductor, a kindly and cheerful young man, he promised to let her
know when they came to Tun Court.
He was as good as his word. In about ten minutes he pulled the
cord and popped his head into the bus. “Y’are, miss. Tun Court’s just
opper-site.” And then as a concession to Mame’s accent, which was a
long way from home: “Watch out, missy, when you cross the street.”
Mame with her recent experience of Broadway and Fifth Avenue felt
she could have crossed this street on her head. It was so narrow.
And although there was no lack of traffic it was moving slow with a
remarkable sense of order and alignment. But Mame liked the young
conductor for his briskness and his courtesy; and as she stepped off
the knife-board and with the fleetness of a slender-ankled nymph
she dodged between the delivery vans of the Westminster Gazette
and the Morning Post she winged him a bright smile.
London, so far, was a city of disappointments. Tun Court added to
their number. It was mean, insignificant, tumble-down, grimy. But
Mame had read that Doctor Johnson or some other famous guy had
either lived or died in it. The latter probably. No man would have
chosen Tun Court to live in, unless he had gone off the handle, as
the famous, she had also read, were more apt to do than ordinary
folks.
The salient fact about Tun Court, however, had nothing to do with
Doctor Johnson. It was the home of the well-known Society journal
High Life. Mame would not have looked to find a paper of repute
housed in this nest of frowsty, mildewed offices in which there was

not space to swing a cat. But she did look and with such poor
success that she had to open her bag and produce the address
which Paula Ling had given her in order to verify it. Yes, it was O.K.:
Number Nine, Tun Court, Fleet Street. Yonder, through that decaying
arch, which by some means had evaded the great fire of B.C. 1666—
or it may have been A.D.?—was the footpath the ancient Romans
had laid along the Fleet Ditch; and the cobblestones upon which
Mame stood, which no doubt had been laid by the Romans also,
indubitably rejoiced in the name Tun Court, since straight before her
eyes a sign was up to say so.
The puzzle was to find Number Nine. Tun Court dealt in names, not
numbers. Among the names High Life was not to be found. There
was the registered London office of the Quick Thinkers’ Chronicle;
also of the Broadcasters’ Review; also of the official organ of the
Amalgamated Society of Pew Openers. These were the portents
which leaped to Mame’s eye, but the one she sought did not seem to
be there. At the far end of the alley, however, where the light was so
bad that it was difficult to see anything, she was just able to
decipher the legend, High Life. Top Floor. It was painted on a wall,
inside a doorway.
Mame boldly attacked some dark stairs, very hollow sounding and
decrepit and full of sharp turns, passing en route the outer portals of
the Eatanswill Gazette and other influential journals. The higher rose
the stairs the darker they grew. But at last patience was rewarded.
High Life—Inquiries, met the pilgrim’s gaze at the top of the second
pair of stairs; yet had that gaze not been young and keen a match
would have been needed to read the inscription on the wall.
She knocked on the door and went in. A pig-tailed flapper lifted her
eyes slowly from Volume 224 of the Duchess Library.
In her best Broadway manner Mame asked if the editor was in. Miss
Pigtail did not appear to be impressed by the Broadway manner. She
made a bluff at concealing an out-size in yawns, laid aside her
novelette with an air of condescension for which Mame longed to
smack her face, and said, “I’ll take in your name.”

Mame felt discouraged, but she was determined not to let the minx
know it. With an air she took a card from her bag; and Miss Pigtail
after one supercilious glance at it went forth to an inner room whose
door was marked Private.
In about thirty seconds Miss Pigtail reappeared. “This way, please,”
she said haughtily. Mame still had a desire to put one over on the
young madam; but evidently she was coming to business all right.
Seated before a roll-top desk, in a stuffy room twelve feet by twelve,
whose only other furniture were an almanac and a vacant chair, was
the editor of High Life. At least Mame surmised that the gentleman
who received her occupied that proud position, even if he did not
quite fulfil her idea of the part. It was difficult to say just where he
fell short, but somehow he did fall short. He was one of those large
flabby men who are only seen without a pipe in their mouths when
they are putting liquids into it. His eyes were tired, his front teeth
didn’t seem to fit, and he had that air of having been born three
highballs below par which some men inherit and others acquire.
The editor of High Life was not a prepossessing man, although the
most striking thing about him, his large moustache, was so
wonderfully pointed and waxed, that Mame felt quite hypnotised by
it. However, she took a pull on herself, made her best bow and
elegantly presented Paula Wyse Ling’s introduction letter.
The visitor was invited to a chair. Then after brief examination of the
envelope the editor made clear that he was not the person to whom
it was addressed. “My name is Judson,” he said, “Digby Judson. I
took over from Walter Waterson about nine months ago.”
“So long as you’re the main guy,” Mame assured him, “it’ll be all
right. I want to connect up with this paper.”
With a slight frown of perplexity Mr. Digby Judson opened Miss Paula
Ling’s letter. “It says nothing about experience,” he remarked mildly.
“And to be quite candid I don’t know Paul M. Wing from Adam.”
“It’s a her,” said Mame matter-of-factly. “Paula Ling’s the name.”

“I beg her pardon, but I don’t know her from Eve.”
Mame had a feeling that she had struck a concealed rock. “Old Man
Waterson would have, anyway,” she said; and with a royal gesture
she indicated her own card, now lying on the editorial blotting pad.
Digby Judson took up the card and laughed. Mame was determined
not to be sensitive, she simply could not afford to be, but that laugh
somehow jarred her nerves. “Cowbarn Independent.” He gave her a
comic look from the extreme corner of a bleared eye. “Holy Jones!”
Mame’s heart sank. It was New York over again. This guy was not
quite so brusque, but he had the same sneer in his manner. A sick
feeling came upon her that she was up against it.
“Cowbarn Independent! I don’t think you’ll be able to get away with
that.”
It was almost like casting an aspersion upon Mame’s parents.
Natural pugnacity leaped to her eyes. In fact it was as much as she
could do to prevent it from jumping off the end of her tongue. “A lot
you know about it,” she yearned to say, but prudently didn’t.
The editor of High Life toyed with the card and drew a mock serious
sigh for which Mame could have slain him. “When did you arrive in
this country, Miss Du Rance?”
“I landed Liverpool yesterday morning.”
“And may I ask what you propose to do now you’ve landed?”
For all the grim depth of her conviction that she could not afford to
be thin-skinned, she resented the subtle impertinence of this
catechism. Yes, it was New York over again. New York had advised
her to cut out the Cowbarn and already she rather wished she had.
But she had figured it out that London being a foreign city would not
guess the sort of burg her home town was.
All the same her faith in herself was not shaken. It was weak to
have these qualms. Mame Durrance was Mame Durrance if she

hailed from Cowbarn, Iowa, and Abe Lincoln was Abe Lincoln even if
he was raised in the wilds of Kentucky.
She crimsoned with mortification, but took herself vigorously in
hand. “What’ll I do now I’ve landed? What do you suppose I’ll do?”
“Knock us endways, I expect.”
“That’d be too easy, I guess—with some of you.”
Mr. Digby Judson was by way of being a human washout but he
liked this power of repartee. Few were the things he admired, but
foremost among them was what he called “vim.” This amusing
spitfire certainly had her share of that.
“You think I’m a little hick.” It is difficult to be wise when your
temper breaks a string. “But I ain’t. Leastways I ain’t goin’ to be
always. With European experience I’ll improve some.”
“Ye-es, I daresay.”
The dry composure of the editor’s voice caused Mame to see red.
“I’m over here to pull the big stuff. An’ don’t forget it.”
Mr. Digby Judson found it hard to conceal his amusement. He gave
his moustache a twirl and said patronisingly: “Well, Miss Du Rance,
what can we do for you in the meantime?”
“Help me to a few dollars.”
Mr. Judson threw up his hands with an air of weary scorn. “My good
girl, to seek dollars in Fleet Street is like looking for a flea in a five-
acre plot. Never have they been so scarce or so many people after
’em. And pretty spry too, you know. They’ve studied the newspaper
public and can give it just what it wants.”
Mame was undaunted. “A chance to see what I can do—that’s all I
ask.”
“What can you do?”
“Suppose I write a bunch of articles on British social life as it strikes
an on-time American.”

“Let us suppose it.” The editor had no enthusiasm.
“Will you print the guff and pay for it?”
This was in the nature of a leading question. Time was needed for
Mr. Judson’s reply. “Rather depends, you know, on the sort of thing it
is.” Out of deference for the feelings of his visitor he did his best to
hide the laugh in his eyes. “You see what we chiefly go for is first-
hand information about the aristocracy.”
Miss Du Rance was aware of that.
“Are you in a position to supply it?”
“I expect I’ll be able to supply it as well as most if I get the chance.”
“Well,” said Digby Judson, fixing Mame with a fishlike eye, “when
you find yourself included in a party at a smart country house you
can send along an account of the sayings and doings of your fellow
guests, a description of their clothes, where they are going to spend
the summer, who is in love with who and all that kind of bilge, and
I’ll be very glad to consider it.”
Mame thanked Mr. Judson for his sporting offer. “I’m sure you’ll fall
for my junk when you see it. There’ll be pep in it. But of course I’ll
want intros to start in.”
“You have introductions, I presume?” The editor still hid his smile.
Unfortunately Miss Du Rance was rather short of introductions. But
she hoped High Life would be able to make good the deficiency.
High Life, it seemed, was not in a position to do so. But it had a
suggestion to offer. Mr. Digby Judson looked through a litter of
papers on his desk. Detaching one from the pile he refreshed his
memory by a careful perusal. Then he said: “There is a vacancy for a
housemaid, I believe, at Clanborough House, Mayfair.”
The news left Mame cold.
“We have influence with the housekeeper at Clanborough House.
She is not exactly a member of our staff, but she receives a fee to

keep our interests at heart. Clanborough House is still a power in the
political and social world. The position of housemaid offers
considerable scope for a person of intelligence such as you appear to
be, Miss Du Rance.”
“I? Housemaid! Me?” The voice of Miss Du Rance went up a whole
octave.
“Of course,” said the editor, “to be quite candid, you would have
rather to put a crimp in your style. These great houses are decidedly
conservative. But you would find opportunities, large opportunities,
believe me, in such a position for obtaining the information we
require.”
Mame was staggered. The rôle of hired girl, even in a mansion, had
not entered her calculations. “What do you take me for?” She
rebuttoned her gloves, snorting blood and fire. “Don’t you see I’m a
lady?”
Mr. Digby Judson gazed fixedly at Mame, stroking his exotic
moustache in the process. “There are ladies and ladies. Frankly, Miss
Du Rance, I can’t promise much success over that course. You see,
in this country at the present time we are overstocked, even with
the genuine article. We are as prolific of ladies in England as they
are of rabbits in Australia. But what we want here is pep and that’s
where you Americans have got the pull. It’s pep, Miss Du Rance, we
are out for, and that, I take it, you are able to supply.”
Mame looked death at the editor. But she said nothing.
“If you’re wise you’ll give ladyism the go by. Better let me see if I
can wangle this billet for you at Clanborough House. A rare chance,
believe me, for a girl like yourself, to study our upper class from the
inside. You’ll be lucky if you get another such opportunity. If you
really give your mind to the job I feel sure you’ll do well.”
“No hired girling for me, I thank you,” Mame spoke in a level voice.
Mr. Digby Judson looked a trifle disappointed. “Well, think it over. But
I am fully convinced of one thing.”

A down-and-out feeling upon her, Mame asked dully what the thing
was.
“It’s the only terms on which you are ever likely to find yourself at
Clanborough House or any other place of equal standing.”
Mame bit her lip to conceal her fury. The insult went deeper than
any she had received in New York. As she bowed stiffly and turned
to go she had a sudden thirst for Mr. Digby Judson’s blood.
She had reached the door, its clumsy knob was in her hand when
she turned again, and said with a slow smile over-spreading a
crimson face, “You’ll excuse me asking, won’t you, but do you mind
telling me if it’s very difficult to train canaries to roost on your
moustache?”

VI
BITTERLY disappointed, Mame promptly found her way back into
Fleet Street. The beginning was bad and there was no disguising it.
Her New York experience had prepared her for difficulties further
east; but she had not reckoned to bite granite so soon or quite so
hard. If she put a crimp in her style she might take a situation as a
housemaid!
Moving towards the Strand she had now a feeling of hostility
towards the people around her. These mossbacks who crowded the
sidewalks had some conceit of themselves. But who were they?
Mame asked herself that. Who were they, anyway?
Luncheon at a cheap restaurant hardly improved Mame’s temper.
The “eats” seemed queer. But at any rate they appeared to stimulate
the mind. In the course of the meal, with the help of a newspaper
propped against the cruet, she did a lot of thinking.
To begin with, she must not look for too much success in London. As
these Cockneys had it, Mame Durrance was not going to set the
Thames on fire. The same applied with equal force to Miss Amethyst
Du Rance. She must watch her step. Aunt Lou’s legacy had now
dwindled to something under five hundred dollars. That plain fact
was the writing on the wall.
Mame foresaw that her trip to Europe was not likely to prove a long
one, unless by some happy chance she struck oil. But of this there
was no sign. Since coming east she had met nothing but bad luck.
And her reception in Tun Court that morning told her to expect no
immediate change. Such being the case, she must put by half the
money she still had, for a passage home across the Atlantic.
By counting every dime she might carry on in London six weeks.
Within that time she hoped, of course, to find a means of adding to

her slender store. But at the moment she could not say that things
looked rosy. The folks here did not cotton to her, still less did she
cotton to them.
After a slice of ham and a cup of coffee she ordered a piece of
pumpkin pie. It was as if she had ordered the moon. The waitress
had not even heard of the national delicacy.
Mame sighed. “Not heard of pumpkin pie!” This was a backward
land. There was a lot of spadework to be done in it. A suspicion was
growing that she would have done better to stay in New York. She
had to be content with custard and stewed figs, upon which poor
substitute she walked slowly along the Strand to Trafalgar Square.
Here she turned into the National Gallery. As she ascended the many
steps of the building she tried to raise a feeling of awe. Even if she
was a little hick she knew that a true American citizeness mentally
takes off her shoes when she enters a dome of Culture.
The feeling of awe was not very powerful. But she was a sane and
cool observer of things and people; and if she was too honest to
pretend to emotions she didn’t possess there was no reason why
those walls should not be a mental stimulus.
She took a seat on a comfortable divan, before a large and lurid
Turner, all raging sea and angry sky. This picture impressed Mame
Durrance considerably less than it had impressed Ruskin. But a hush
of culture all around enabled her to sit two good hours putting her
ideas in order. A plan of some sort was necessary if she was going to
make good. When she set out from Cowbarn, six months back, she
felt that her natural abilities would carry her through anything. Now,
she was not so sure. Things were no longer rose colour. There was a
terrible lot of leeway to make up. She just had not guessed that she
was so far behind the game. Perhaps it had been wiser to stay at
home and put herself through college.
Chastened by the buffets of the day she returned to Montacute
Square about five. When she entered the gloomy drawing room the
tabbies received her icily. Not a sign of success on the horizon so far.

The tea drinking was as depressing as ever. Nobody took any notice
of her.
The aloofness of these frumps was as hard to bear as the insolence
of the editor of High Life. Mame’s resentment grew. Presently she
rose and went up to her cold bedroom. She got out her writing case.
Perched with knees crossed, on the end of her bed, she spent a
prosperous hour jotting down her first day’s impressions of London,
England.
Vigorous mental exercise seemed to take a load off her spirit. What
she had written ought to raise a smile in Elmer P. To-morrow she
would go at it again; then it should be typed and mailed. If the boob
fell for it, and born optimist that Mame Durrance was, she felt he
sure would, where that stuff came from there was good and plenty
more to pull.

VII
WHEN Mame returned to the drawing room she was dressed for the
evening meal. It was only seven o’clock and she counted on having
the place to herself, since at that hour the tabbies would be
occupied with their own preparations. But as it happened the room
was not quite empty. There was just one person in it.
The old elegant, who had already excited Mame’s curiosity, stood
before the meagre fire warming his thin hands. As soon as she came
in he turned towards her with a little bow of rare politeness.
“I am told,” he said in the deepest, most measured tone Mame had
ever heard, “you are an American.”
Mame owned to that in the half-humorous manner she had already
adopted for the benefit of these islanders. Some folks might have
been abashed by this obvious grandee. Not so Miss Amethyst Du
Rance. She was as good as the best and she was in business to
prove it. These bums were not to be taken at their own valuation.
Back of everything her faith in her own shrewd wits was unshakable.
“I have a very warm corner in my heart for all Americans.”
“Have you so?” said Mame.
There was not a hint of patronage in the old buck’s manner, yet in
spite of his air of simple kindness, Mame somehow felt the King-of-
England-with-his-beard-off feeling creeping upon her. He was the
goods all right, this old john, but she was determined to take him in
her stride as she would have taken President Harding or any other
regular fellow.
“Won’t you tell me your name?”
Mame opened the small bag which she never parted with, even at
meal times, and took out her card. The old man fixed his eyeglass

and scanned it with prodigious solemnity. “Cowbarn.” A bland pause.
“Now tell me, what state is that in? It’s very ignorant not to know,”
he apologetically added.
“No, it ain’t.” Mame was captivated by the air of humility, although
not sure that it was real. “Cowbarn’s in the state of Iowa. On’y a
one-horse burg.”
“Ah, yes, to be sure, Iowa.” The grandee made play with his
eyeglass. “I remember touring the Middle West with Henry Irving in
’89.”
So long was ’89 before Mame was born that she was a trifle vague
upon the subject of Henry Irving. But she knew all about Lloyd
George, Arthur, Earl of Balfour, and even Old Man Gladstone of an
earlier day. She surmised that Henry was one of these.
“A senator, I guess?”
“My dear young lady, no.” The tone of surprise was comically tragic.
“Henry Irving was the greatest ac-torr Eng-laand ever produced.”
“You don’t say!” The awe in Mame’s voice was an automatic
concession to the awe in the voice of the speaker.
“Yes, Eng-laand’s greatest ac-torr.” There was a note of religious
exaltation in the old grandee. “I toured the United States three times
with Henry Irving.”
“Did you visit Cowbarn?” Mame asked for politeness’ sake. To the
best of her information, Cowbarn, like herself, had not been invented
in ’89.
“I seem to remember playing the Duke there to Henry Irving’s
Shylock at a one-night stand,” said the grandee also for politeness’
sake. He had never heard of Cowbarn, he had never been to Iowa,
and in ’89 Henry the August had given up the practice of playing
one-night stands. But the higher amenities of the drawing room are
not always served by a conservative handling of raw fact.

Mame, with that sharp instinct of hers, knew the old chap was lying.
But it didn’t lessen her respect. He had an overwhelming manner
and when he gave the miserable fire a simple poke he used the
large gesture of one who feels that the eyes of the universe are
upon him. Still, with every deduction made, and a homely daughter
of a republic felt bound to make many, he was the most human
thing she had met so far in her travels.
His name was Falkland Vavasour. And in confiding to Mame this
bright jewel of the English theatre, which his pronunciation of it led
her to think it must be, he yet modestly said that its lustre was
nought compared with the blinding effulgence of the divine Henry.
“Some ac-torr, old man Henry Irv,” Mame was careful to pronounce
the sacred word “actor” in the manner of this old-timer.
“My de-ah young lady, Henry Irving was a swell. Never again shall
we look upon his like.”
“I’ll say not.” And then with an instinct to hold the conversation at
the level to which it had now risen Mame opened the door of fancy.
“Come to think of it, I’ve heard my great-uncle Nel speak of Henry
Irving. You’ve heard, I guess, of Nelson E. Grice, the Federal
general, one of the signatures to the peace of Appomattox. He was
the brother of my mother’s mother. Many’s the time I’ve set on
Great-uncle’s knee and played with the gold watch and chain that his
old friend General Sherman give him the day after the battle of
Gettysburg.”
Great-uncle Nel had really nothing to do with the case, but Mame
felt he was a sure card to play in this high-class conversation. The
old Horse had not been a general, he had not signed the peace of
Appomattox, and there was a doubt whether General Sherman,
whose friend he certainly was, had ever given him a gold watch and
chain; but he was a real asset in the Durrance family. Apart from this
hero there was nothing to lift it out of the rut of mediocrity. Quite
early in life Mame had realized the worth of great-uncle Nel.

Mr. Falkland Vavasour had the historical sense. He rose to General
Nelson E. Grice like a trout to a may fly. The old buck refixed his
eyeglass and recalled the Sixties. He was playing junior lead at the
Liverpool Rotunda when the news came of President Lincoln’s
assassination. He remembered— But what did he not remember?
Yes, great-uncle Nel was going to be a sure card in London, England.
Mame was getting on with the old beau like a house on fire when
the clock on the chimneypiece struck half-past seven. Mr. Falkland
Vavasour gave a little sigh and said he must go. Mame, quickened
already by a regard for this charming old man, who lit the gloom of
Fotheringay House, expressed sorrow that he was not dining in.
But it seemed that Mr. Falkland Vavasour never did dine in. This
applied, in fact, to all his meals. All his meals were taken out.
“But why?” asked Mame disappointedly.
“My dear young lady”—the old-timer’s shrug was so whimsical yet so
elegant it sure would have made Henry Irving jealous—“one has
nothing against the cook of our hostess— But!”
Until that moment Mame had not realized what a world of meaning
a simple word can hold.
She was keenly disappointed. As the first tabby invaded the drawing
room Mr. Falkland Vavasour passed out. A glamour, a warmth,
passed out with him. Everything grew different. It was a change
from light to darkness; it was like a swift cloud across the sun.

VIII
THE days to follow wrought havoc with Aunt Lou’s legacy. Mame’s
idea had been to support herself with her pen during her stay in
England. She had a gift, or thought she had, for expressing herself
on paper; she had a sharp eye for things, she had energy and she
had ideas; yet soon was she to learn that as far as London went the
market for casual writing was no better than New York.
Times there were when she began to regret her safe anchorage at
Cowbarn. But she did not spend much time looking back. She was
determined to be a go-getter. Briskly she went about the town,
seeing and hearing and jotting down what she saw and heard. And
every Friday morning she mailed a packet of her observations to the
Cowbarn Independent.
Weeks went quickly by. No word came from the only friend she had
in the small and tight world of editors. Even Elmer P. Dobree, on
whom she had optimistically counted, had turned her down. And
things were not going well with her. She had not been able to earn a
dollar in Europe, yet daily the wad was growing less. The time was
sure coming when she would have to go home with her money
spent and only a few chunks of raw experience to show for it.
Perhaps she had tried to prise off a bit too much. Wiser, perhaps, to
have stuck to her job. She had been a fairly efficient stenographer
and typist. But her active mind was bored. Hovering around that
hard stool in the Independent office it wanted all there was in the
world. Yet first New York and now London, in their sharp reality, had
taught her that the world was a bigger place than she had allowed
for. And there were more folks in it. There were simply millions of
Mame Durrances around: every sort of go-getter, all wanting the
earth and with just her chance of connecting up. But if the worst
came, so she had figured it out, and she failed to click in what these

Britishers called “journalism,” she would always be able to return to
an office stool.
Now, however, she was not so sure. As far as London went, it had
six million people and half of them seemed to be looking for jobs.
Just to keep in touch she had applied for one or two vacancies
advertised in the papers. She had no real wish to get them. It would
have been an admission of defeat and it was early days for that. But
it would be well, in case the time really came, to know how to come
in out of the rain.
She had applied in person but results were not encouraging.
Secretarial work was badly paid in London and the struggle for it
terrible. There was nothing like enough to go round. Besides, when
all was said, Miss Amethyst Du Rance had not come to England to
adorn an office stool.
It was not yet time to say good-bye to ambition. She still firmly
believed it was in her to make good as a newspaper girl. But it was
not easy to conquer the East. Compared with most of these high-
flyers and four-flushers she was up against, with their finesse and
their culture and their slick talk, Mame Durrance was a little hick. No
use disguising it; she was a little hick. “A sense of ignorance is the
beginning of knowledge” was one of the mottoes for 1921 in the
office calendar which had adorned the fly-walk at the back of her
typewriter and which, with an eye to the future, she had committed
to memory in her spare moments.
The beginning of knowledge for Mame Durrance meant covering up
your tracks. She must see that none of her fellow go-getters put one
over on her. But even that simple precaution was not easy. They
smiled every time she opened her face, these college boobs. And
being as sharp as a hawk she had soon decided that her first duty
was to get rid of her accent.
To this end she went freely about, she dressed to the limit of her
slim purse, she laid herself out to meet interesting folks. Her fellow
p.g.’s of Fotheringay House could not be considered interesting. But

there was one exception. Mr. Falkland Vavasour continued to show
himself much her friend. And he was quite the most interesting
creature she had met. He was, also, very useful. It was a joy to hear
him speak what he called the King’s English. She felt she could not
do better than model herself on this living fount of euphony.
First she must cast out the nasal drawl that raised a smile wherever
it was heard. Then she must mobilise the vowels and consonants of
the mother tongue in the style which gave Mr. Falkland Vavasour an
assured position in the drawing room of Fotheringay House. The
tabbies, for the most part man-haters, simply hung on the words of
Mr. Falkland Vavasour. This was mainly due, in Mame’s opinion, to his
faultless voice production; and she soon set to work to study it.
Paula Wyse Ling, the most accomplished go-getter of her
acquaintance, had said it was worth any girl’s while to visit London
in order to acquire an English accent. Mame had doubted this. What
was good enough for Cowbarn, Iowa, should have been good
enough for the whole world. But the world, it seemed, was some
place. She now began to see what Paula meant.
No sooner had Miss Amethyst Du Rance won the friendship of Mr.
Falkland Vavasour than she decided that something must be done in
the matter. She discreetly asked if he could tell her how to improve
her voice. The old man tactfully said it didn’t need improvement. In
his opinion it was a fine and powerful organ. Mame felt this was
politeness. Her voice didn’t lack force but force was the trouble. “It
needs a soft pedal,” said Mame. “Refinement, you know, and charm
and all the frills of the West End theatres, restaurants and shops.”
The old actor had a sense of humour and a kind heart. He was
amused by Mame, and he liked her. It would have been hard for an
artist in life not to like such naïveté, such enthusiasm, such
concentration, such fire. But this voice of hers was a problem. It had
a natural punch that treated brick walls as if they were brown paper.
Toying with his monocle, in all things the perfect john, he drawled:
“My de-ah young lady, if I may say so, your voice is mag-nif-i-cent,

simply mag-nif-i-cent.”
Mame’s smile crumpled under the sheer action-pressure of her mind.
“Some ways it ain’t. Some ways it’s wanting.”
“A shade more of—er, distinction perhaps?”
“Distinction.” Mame darted on the word like a bird on an insect. “You
said it. Distinction’s mine. And I’ll get it, too—if it kills me.”
“My de-ah young lady, perfectly simple for a girl of your talent.”
“Honest? You think that?” The good grey eye glowed hopefully. “I
wish I could say mag-nif-i-cent as poyfect as you can.”
“You will, my de-ah young lady, believe me, you will.”
“Well, I’ll start to learn right now.”
Mr. Falkland Vavasour smiled approval. He advised her to draw three
deep breaths from the lower chest and to pronounce the word
syllable by syllable.
Mame stood to her full height. She inflated. “Mag-nif-i-cent! Mag-nif-
i-cent! Mag-nif-i-cent!”
“My de-ah young lady, what could be better?”
This was vastly encouraging. But it was only a beginning and her
nature was to leave nothing to chance. A little later that day she
acquired a second-hand copy of Bell’s Standard Elocutionist from a
bookshop in the Charing Cross Road. And then she arranged for Mr.
Falkland Vavasour to hear her say a little piece every morning, when
the drawing room was empty.

IX
MAME’S friendship with “the mystery man” continued to grow. That
was a name his fellow guests had given him. His comings and goings
were indeed mysterious. Nobody knew where he took his meals.
Nobody knew what his circumstances were. All the time he had been
at Fotheringay House, which was quite a number of years, his name
had never been seen in a playbill. But there was a legend that he
had once had an engagement with Bancroft at the old Prince of
Wales.
He was always dressed immaculately, he was the soul of courtesy,
his talk was urbane, and to Mame at any rate, it seemed highly
informed. But there was no concealing from her keen eyes that the
old boy was thin as a rail. In fact she would hardly have been
surprised if some bright morning a wind from the east had blown
him away altogether. As for his clothes, in spite of the wonderful air
with which he wore them, and good as they had been, they were
almost threadbare and literally shone with age.
Mame gathered from one of the tabbies, who in the process of time
began to thaw a little, that Mr. Falkland Vavasour was a distant
connection of the landlady’s. This fact was held to explain why he
was allowed to live at Fotheringay House while invariably taking his
meals at his club. At least it was generally understood that it was at
his club that he took his meals. But wherever he may have taken
them, even if the food was more delicate than at Fotheringay House,
it could hardly have been more abundant. Week by week the old
man grew thinner and thinner. His step on the drawing room carpet
grew lighter and more feeble. Even his wonderful voice lost
something of its timbre. Yet amid all these signs of decay, he
retained that alert, sprightly man-of-the-worldliness which Mame
found so curiously fascinating.

One morning, soon after breakfast, when she had been nearly five
weeks at Fotheringay House, she sat in a corner of the dismal
drawing room adding up her accounts and gloomily wondering
whether the time had not come to look for “board and residence”
that would cost less. Suddenly there came a rude shock. Mrs.
Toogood entered in a state of agitation. Mr. Falkland Vavasour had
just been found dead in his bed.
A doctor had already been sent for. But until he arrived the cause of
Mr. Falkland Vavasour’s death must remain, like the old man himself,
a mystery. The landlady as well as her p.g.’s were quite at a loss to
account for the tragic occurrence. Miss Glendower, the most
conversational of the tabbies, opined that it must be sheer old age.
Dear Mr. Falkland Vavasour must certainly be very old.
Miss Du Rance agreed that he must be. For was he not playing the
junior lead at the Liverpool Rotunda when the news came of
President Lincoln’s assassination?
“What year was that?” asked Miss Glendower.
“’Sixty-five.” Mame gave that outstanding date in history with pride
and with promptitude. Before starting east she had fortified a
memory naturally good by a correspondence course; therefore she
could trust it.
Miss Glendower had no doubt at all that old age was the cause of
death. But Mame was visited suddenly by a grim suspicion. It might
be old age. Or it might not— Before giving an opinion she would
await the doctor’s verdict.
In a few minutes came the doctor. He was received by Mrs.
Toogood, who led him slowly up two flights of stairs to the room of
Mr. Falkland Vavasour. Overmastered by curiosity, and with an ever-
deepening agitation fixing itself upon her—Mame had really liked this
kindly and charming old man—she followed a small procession up
the stairs.
She stood on the threshold of the room while the doctor bent over
the bed. First he took one frail and shrunken hand and then he took

Immediate download Kubernetes Best Practices 1st Edition Brendan Burns ebooks 2024

More Related Content

Similar to Immediate download Kubernetes Best Practices 1st Edition Brendan Burns ebooks 2024 (20)

Recently uploaded (20)

Immediate download Kubernetes Best Practices 1st Edition Brendan Burns ebooks 2024