Data Pipelines with Apache Airflow 1st Edition Bas P Harenslak Julian Rutger De Ruiter
- 1. Data Pipelines with Apache Airflow 1st Edition Bas P Harenslak Julian Rutger De Ruiter download https://textbookfull.com/product/data-pipelines-with-apache- airflow-1st-edition-bas-p-harenslak-julian-rutger-de-ruiter/ Download more ebook from https://textbookfull.com
- 2. We believe these products will be a great fit for you. Click the link to download now, or visit textbookfull.com to discover even more! Data Pipelines with Apache Airflow 1st Edition Bas P Harenslak Julian Rutger De Ruiter https://textbookfull.com/product/data-pipelines-with-apache- airflow-1st-edition-bas-p-harenslak-julian-rutger-de-ruiter-2/ Learning Apache Drill Query and Analyze Distributed Data Sources with SQL 1st Edition Charles Givre https://textbookfull.com/product/learning-apache-drill-query-and- analyze-distributed-data-sources-with-sql-1st-edition-charles- givre/ Data Pipelines Pocket Reference: Moving and Processing Data for Analytics 1st Edition James Densmore https://textbookfull.com/product/data-pipelines-pocket-reference- moving-and-processing-data-for-analytics-1st-edition-james- densmore/ Azure Data Factory Cookbook: Data engineers guide to build and manage ETL and ELT pipelines with data integration , 2nd Edition Dmitry Foshin https://textbookfull.com/product/azure-data-factory-cookbook- data-engineers-guide-to-build-and-manage-etl-and-elt-pipelines- with-data-integration-2nd-edition-dmitry-foshin/
- 3. Data Engineering with AWS: Acquire the skills to design and build AWS-based data transformation pipelines like a pro 2nd Edition Eagar https://textbookfull.com/product/data-engineering-with-aws- acquire-the-skills-to-design-and-build-aws-based-data- transformation-pipelines-like-a-pro-2nd-edition-eagar/ Watermarking Security 1st Edition Patrick Bas https://textbookfull.com/product/watermarking-security-1st- edition-patrick-bas/ Beginning Jenkins Blue Ocean: Create Elegant Pipelines With Ease 1st Edition Nikhil Pathania https://textbookfull.com/product/beginning-jenkins-blue-ocean- create-elegant-pipelines-with-ease-1st-edition-nikhil-pathania/ Data Protection: Ensuring Data Availability 1st Edition Preston De Guise https://textbookfull.com/product/data-protection-ensuring-data- availability-1st-edition-preston-de-guise/ Data Protection: Ensuring Data Availability 2nd Edition Preston De Guise https://textbookfull.com/product/data-protection-ensuring-data- availability-2nd-edition-preston-de-guise/
- 4. M A N N I N G Bas Harenslak Julian de Ruiter
- 5. Pipeline as DAG Task 1 Task 2 Task 3 Task 4 Schedule interval = @daily DAG file (Python) Dependency between tasks, indicating task 3 must run before task 4 Which schedule to use for running the DAG Represents a task/operation we want to run
- 6. Data Pipelines with Apache Airflow
- 8. Data Pipelines with Apache Airflow BAS HARENSLAK AND JULIAN DE RUITER M A N N I N G SHELTER ISLAND
- 9. For online information and ordering of this and other Manning books, please visit www.manning.com. The publisher offers discounts on this book when ordered in quantity. For more information, please contact Special Sales Department Manning Publications Co. 20 Baldwin Road PO Box 761 Shelter Island, NY 11964 Email: orders@manning.com ©2021 by Manning Publications Co. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by means electronic, mechanical, photocopying, or otherwise, without prior written permission of the publisher. Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in the book, and Manning Publications was aware of a trademark claim, the designations have been printed in initial caps or all caps. Recognizing the importance of preserving what has been written, it is Manning’s policy to have the books we publish printed on acid-free paper, and we exert our best efforts to that end. Recognizing also our responsibility to conserve the resources of our planet, Manning books are printed on paper that is at least 15 percent recycled and processed without the use of elemental chlorine. Development editor: Tricia Louvar Technical development editor: Arthur Zubarev Manning Publications Co. Review editor: Aleks Dragosavljević 20 Baldwin Road Production editor: Deirdre S. Hiam PO Box 761 Copy editor: Michele Mitchell Shelter Island, NY 11964 Proofreader: Keri Hales Technical proofreader: Al Krinker Typesetter: Dennis Dalinnik Cover designer: Marija Tudor ISBN: 9781617296901 Printed in the United States of America
- 10. v brief contents PART 1 GETTING STARTED ........................................................1 1 ■ Meet Apache Airflow 3 2 ■ Anatomy of an Airflow DAG 20 3 ■ Scheduling in Airflow 40 4 ■ Templating tasks using the Airflow context 60 5 ■ Defining dependencies between tasks 85 PART 2 BEYOND THE BASICS..................................................113 6 ■ Triggering workflows 115 7 ■ Communicating with external systems 135 8 ■ Building custom components 157 9 ■ Testing 186 10 ■ Running tasks in containers 220 PART 3 AIRFLOW IN PRACTICE ...............................................253 11 ■ Best practices 255 12 ■ Operating Airflow in production 281
- 11. BRIEF CONTENTS vi 13 ■ Securing Airflow 322 14 ■ Project: Finding the fastest way to get around NYC 344 PART 4 IN THE CLOUDS .........................................................365 15 ■ Airflow in the clouds 367 16 ■ Airflow on AWS 375 17 ■ Airflow on Azure 394 18 ■ Airflow in GCP 412
- 12. vii contents preface xv acknowledgments xvii about this book xix about the authors xxiii about the cover illustration xxiv PART 1 GETTING STARTED ..............................................1 1 Meet Apache Airflow 3 1.1 Introducing data pipelines 4 Data pipelines as graphs 4 ■ Executing a pipeline graph 6 Pipeline graphs vs. sequential scripts 6 ■ Running pipeline using workflow managers 9 1.2 Introducing Airflow 10 Defining pipelines flexibly in (Python) code 10 ■ Scheduling and executing pipelines 11 ■ Monitoring and handling failures 13 Incremental loading and backfilling 15 1.3 When to use Airflow 17 Reasons to choose Airflow 17 ■ Reasons not to choose Airflow 17 1.4 The rest of this book 18
- 13. CONTENTS viii 2 Anatomy of an Airflow DAG 20 2.1 Collecting data from numerous sources 21 Exploring the data 21 2.2 Writing your first Airflow DAG 22 Tasks vs. operators 26 ■ Running arbitrary Python code 27 2.3 Running a DAG in Airflow 29 Running Airflow in a Python environment 29 ■ Running Airflow in Docker containers 30 ■ Inspecting the Airflow UI 31 2.4 Running at regular intervals 33 2.5 Handling failing tasks 36 3 Scheduling in Airflow 40 3.1 An example: Processing user events 41 3.2 Running at regular intervals 42 Defining scheduling intervals 42 ■ Cron-based intervals 44 Frequency-based intervals 46 3.3 Processing data incrementally 46 Fetching events incrementally 46 ■ Dynamic time references using execution dates 48 ■ Partitioning your data 50 3.4 Understanding Airflow’s execution dates 52 Executing work in fixed-length intervals 52 3.5 Using backfilling to fill in past gaps 54 Executing work back in time 54 3.6 Best practices for designing tasks 55 Atomicity 55 ■ Idempotency 57 4 Templating tasks using the Airflow context 60 4.1 Inspecting data for processing with Airflow 61 Determining how to load incremental data 61 4.2 Task context and Jinja templating 63 Templating operator arguments 64 ■ What is available for templating? 66 ■ Templating the PythonOperator 68 Providing variables to the PythonOperator 73 ■ Inspecting templated arguments 75 4.3 Hooking up other systems 77
- 14. CONTENTS ix 5 Defining dependencies between tasks 85 5.1 Basic dependencies 86 Linear dependencies 86 ■ Fan-in/-out dependencies 87 5.2 Branching 90 Branching within tasks 90 ■ Branching within the DAG 92 5.3 Conditional tasks 97 Conditions within tasks 97 ■ Making tasks conditional 98 Using built-in operators 100 5.4 More about trigger rules 100 What is a trigger rule? 101 ■ The effect of failures 102 Other trigger rules 103 5.5 Sharing data between tasks 104 Sharing data using XComs 104 ■ When (not) to use XComs 107 ■ Using custom XCom backends 108 5.6 Chaining Python tasks with the Taskflow API 108 Simplifying Python tasks with the Taskflow API 109 When (not) to use the Taskflow API 111 PART 2 BEYOND THE BASICS ........................................113 6 Triggering workflows 115 6.1 Polling conditions with sensors 116 Polling custom conditions 119 ■ Sensors outside the happy flow 120 6.2 Triggering other DAGs 122 Backfilling with the TriggerDagRunOperator 126 Polling the state of other DAGs 127 6.3 Starting workflows with REST/CLI 131 7 Communicating with external systems 135 7.1 Connecting to cloud services 136 Installing extra dependencies 137 ■ Developing a machine learning model 137 ■ Developing locally with external systems 143 7.2 Moving data from between systems 150 Implementing a PostgresToS3Operator 151 ■ Outsourcing the heavy work 155
- 15. CONTENTS x 8 Building custom components 157 8.1 Starting with a PythonOperator 158 Simulating a movie rating API 158 ■ Fetching ratings from the API 161 ■ Building the actual DAG 164 8.2 Building a custom hook 166 Designing a custom hook 166 ■ Building our DAG with the MovielensHook 172 8.3 Building a custom operator 173 Defining a custom operator 174 ■ Building an operator for fetching ratings 175 8.4 Building custom sensors 178 8.5 Packaging your components 181 Bootstrapping a Python package 182 ■ Installing your package 184 9 Testing 186 9.1 Getting started with testing 187 Integrity testing all DAGs 187 ■ Setting up a CI/CD pipeline 193 ■ Writing unit tests 195 ■ Pytest project structure 196 ■ Testing with files on disk 201 9.2 Working with DAGs and task context in tests 203 Working with external systems 208 9.3 Using tests for development 215 Testing complete DAGs 217 9.4 Emulate production environments with Whirl 218 9.5 Create DTAP environments 219 10 Running tasks in containers 220 10.1 Challenges of many different operators 221 Operator interfaces and implementations 221 ■ Complex and conflicting dependencies 222 ■ Moving toward a generic operator 223 10.2 Introducing containers 223 What are containers? 223 ■ Running our first Docker container 224 ■ Creating a Docker image 225 Persisting data using volumes 227 10.3 Containers and Airflow 230 Tasks in containers 230 ■ Why use containers? 231
- 16. CONTENTS xi 10.4 Running tasks in Docker 232 Introducing the DockerOperator 232 ■ Creating container images for tasks 233 ■ Building a DAG with Docker tasks 236 Docker-based workflow 239 10.5 Running tasks in Kubernetes 240 Introducing Kubernetes 240 ■ Setting up Kubernetes 242 Using the KubernetesPodOperator 245 ■ Diagnosing Kubernetes- related issues 248 ■ Differences with Docker-based workflows 250 PART 3 AIRFLOW IN PRACTICE .....................................253 11 Best practices 255 11.1 Writing clean DAGs 256 Use style conventions 256 ■ Manage credentials centrally 260 Specify configuration details consistently 261 ■ Avoid doing any computation in your DAG definition 263 ■ Use factories to generate common patterns 265 ■ Group related tasks using task groups 269 ■ Create new DAGs for big changes 270 11.2 Designing reproducible tasks 270 Always require tasks to be idempotent 271 ■ Task results should be deterministic 271 ■ Design tasks using functional paradigms 272 11.3 Handling data efficiently 272 Limit the amount of data being processed 272 ■ Incremental loading/processing 274 ■ Cache intermediate data 275 Don’t store data on local file systems 275 ■ Offload work to external/source systems 276 11.4 Managing your resources 276 Managing concurrency using pools 276 ■ Detecting long-running tasks using SLAs and alerts 278 12 Operating Airflow in production 281 12.1 Airflow architectures 282 Which executor is right for me? 284 ■ Configuring a metastore for Airflow 284 ■ A closer look at the scheduler 286 12.2 Installing each executor 290 Setting up the SequentialExecutor 291 ■ Setting up the LocalExecutor 292 ■ Setting up the CeleryExecutor 293 Setting up the KubernetesExecutor 296
- 17. CONTENTS xii 12.3 Capturing logs of all Airflow processes 302 Capturing the webserver output 303 ■ Capturing the scheduler output 303 ■ Capturing task logs 304 ■ Sending logs to remote storage 305 12.4 Visualizing and monitoring Airflow metrics 305 Collecting metrics from Airflow 306 ■ Configuring Airflow to send metrics 307 ■ Configuring Prometheus to collect metrics 308 Creating dashboards with Grafana 310 ■ What should you monitor? 312 12.5 How to get notified of a failing task 314 Alerting within DAGs and operators 314 ■ Defining service-level agreements 316 ■ Scalability and performance 318 ■ Controlling the maximum number of running tasks 318 ■ System performance configurations 319 ■ Running multiple schedulers 320 13 Securing Airflow 322 13.1 Securing the Airflow web interface 323 Adding users to the RBAC interface 324 ■ Configuring the RBAC interface 327 13.2 Encrypting data at rest 327 Creating a Fernet key 328 13.3 Connecting with an LDAP service 330 Understanding LDAP 330 ■ Fetching users from an LDAP service 333 13.4 Encrypting traffic to the webserver 333 Understanding HTTPS 334 ■ Configuring a certificate for HTTPS 336 13.5 Fetching credentials from secret management systems 339 14 Project: Finding the fastest way to get around NYC 344 14.1 Understanding the data 347 Yellow Cab file share 348 ■ Citi Bike REST API 348 Deciding on a plan of approach 350 14.2 Extracting the data 350 Downloading Citi Bike data 351 ■ Downloading Yellow Cab data 353 14.3 Applying similar transformations to data 356
- 18. CONTENTS xiii 14.4 Structuring a data pipeline 360 14.5 Developing idempotent data pipelines 361 PART 4 IN THE CLOUDS...............................................365 15 Airflow in the clouds 367 15.1 Designing (cloud) deployment strategies 368 15.2 Cloud-specific operators and hooks 369 15.3 Managed services 370 Astronomer.io 371 ■ Google Cloud Composer 371 Amazon Managed Workflows for Apache Airflow 372 15.4 Choosing a deployment strategy 372 16 Airflow on AWS 375 16.1 Deploying Airflow in AWS 375 Picking cloud services 376 ■ Designing the network 377 Adding DAG syncing 378 ■ Scaling with the CeleryExecutor 378 Further steps 380 16.2 AWS-specific hooks and operators 381 16.3 Use case: Serverless movie ranking with AWS Athena 383 Overview 383 ■ Setting up resources 384 ■ Building the DAG 387 ■ Cleaning up 393 17 Airflow on Azure 394 17.1 Deploying Airflow in Azure 394 Picking services 395 ■ Designing the network 395 Scaling with the CeleryExecutor 397 ■ Further steps 398 17.2 Azure-specific hooks/operators 398 17.3 Example: Serverless movie ranking with Azure Synapse 400 Overview 400 ■ Setting up resources 401 ■ Building the DAG 404 ■ Cleaning up 410 18 Airflow in GCP 412 18.1 Deploying Airflow in GCP 413 Picking services 413 ■ Deploying on GKE with Helm 415 Integrating with Google services 417 ■ Designing the network 419 ■ Scaling with the CeleryExecutor 419
- 19. CONTENTS xiv 18.2 GCP-specific hooks and operators 422 18.3 Use case: Serverless movie ranking on GCP 427 Uploading to GCS 428 ■ Getting data into BigQuery 429 Extracting top ratings 432 appendix A Running code samples 436 appendix B Package structures Airflow 1 and 2 439 appendix C Prometheus metric mapping 443 index 445
- 20. xv preface We’ve both been fortunate to be data engineers in interesting and challenging times. For better or worse, many companies and organizations are realizing that data plays a key role in managing and improving their operations. Recent developments in machine learning and AI have opened a slew of new opportunities to capitalize on. However, adopting data-centric processes is often difficult, as it generally requires coordinating jobs across many different heterogeneous systems and tying everything together in a nice, timely fashion for the next analysis or product deployment. In 2014, engineers at Airbnb recognized the challenges of managing complex data workflows within the company. To address those challenges, they started developing Airflow: an open source solution that allowed them to write and schedule workflows and monitor workflow runs using the built-in web interface. The success of the Airflow project quickly led to its adoption under the Apache Software Foundation, first as an incubator project in 2016 and later as a top-level proj- ect in 2019. As a result, many large companies now rely on Airflow for orchestrating numerous critical data processes. Working as consultants at GoDataDriven, we’ve helped various clients adopt Air- flow as a key component in projects involving the building of data lakes/platforms, machine learning models, and so on. In doing so, we realized that handing over these solutions can be challenging, as complex tools like Airflow can be difficult to learn overnight. For this reason, we also developed an Airflow training program at GoData- Driven, and have frequently organized and participated in meetings to share our knowledge, views, and even some open source packages. Combined, these efforts have
- 21. PREFACE xvi helped us explore the intricacies of working with Airflow, which were not always easy to understand using the documentation available to us. In this book, we aim to provide a comprehensive introduction to Airflow that cov- ers everything from building simple workflows to developing custom components and designing/managing Airflow deployments. We intend to complement many of the excellent blogs and other online documentation by bringing several topics together in one place, using a concise and easy-to-follow format. In doing so, we hope to kickstart your adventures with Airflow by building on top of the experience we’ve gained through diverse challenges over the past years.
- 22. xvii acknowledgments This book would not have been possible without the support of many amazing people. Colleagues from GoDataDriven and personal friends supported us and provided valu- able suggestions and critical insights. In addition, Manning Early Access Program (MEAP) readers posted useful comments in the online forum. Reviewers from the development process also contributed helpful feedback: Al Krinker, Clifford Thurber, Daniel Lamblin, David Krief, Eric Platon, Felipe Ortega, Jason Rendel, Jeremy Chen, Jiri Pik, Jonathan Wood, Karthik Sirasanagandla, Kent R. Spillner, Lin Chen, Philip Best, Philip Patterson, Rambabu Posa, Richard Meinsen, Robert G. Gimbel, Roman Pavlov, Salvatore Campagna, Sebastián Palma Mardones, Thorsten Weber, Ursin Stauss, and Vlad Navitski. At Manning, we owe special thanks to Brian Sawyer, our acquisitions editor, who helped us shape the initial book proposal and believed in us being able to see it through; Tricia Louvar, our development editor, who was very patient in answering all our questions and concerns, provided critical feedback on each of our draft chapters, and was an essential guide for us throughout this entire journey; and to the rest of the staff as well: Deirdre Hiam, our project editor; Michele Mitchell, our copyeditor; Keri Hales, our proofreader; and Al Krinker, our technical proofreader. Bas Harenslak I would like to thank my friends and family for their patience and support during this year-and-a-half adventure that developed from a side project into countless days, nights, and weekends. Stephanie, thank you for always putting up with me working at
- 23. ACKNOWLEDGMENTS xviii the computer. Miriam, Gerd, and Lotte, thank you for your patience and belief in me while writing this book. I would also like to thank the team at GoDataDriven for their support and dedication to always learn and improve, I could not have imagined being the author of a book when I started working five years ago. Julian de Ruiter First and foremost, I’d like to thank my wife, Anne Paulien, and my son, Dexter, for their endless patience during the many hours that I spent doing “just a little more work” on the book. This book would not have been possible without their unwavering support. In the same vein, I’d also like to thank our family and friends for their sup- port and trust. Finally, I’d like to thank our colleagues at GoDataDriven for their advice and encouragement, from whom I’ve also learned an incredible amount in the past years.
- 24. xix about this book Data Pipelines with Apache Airflow was written to help you implement data-oriented work- flows (or pipelines) using Airflow. The book begins with the concepts and mechanics involved in programmatically building workflows for Apache Airflow using the Python programming language. Then the book switches to more in-depth topics such as extending Airflow by building your own custom components and comprehensively testing your workflows. The final part of the book focuses on designing and managing Airflow deployments, touching on topics such as security and designing architectures for several cloud platforms. Who should read this book Data Pipelines with Apache Airflow is written both for scientists and engineers who are looking to develop basic workflows in Airflow, as well as engineers interested in more advanced topics such as building custom components for Airflow or managing Air- flow deployments. As Airflow workflows and components are built in Python, we do expect readers to have intermediate experience with programming in Python (i.e., have a good working knowledge of building Python functions and classes, understand- ing concepts such as *args and **kwargs, etc.). Some experience with Docker is also beneficial, as most of our code examples are run using Docker (though they can also be run locally if you wish).
- 25. ABOUT THIS BOOK xx How this book is organized: A road map The book consists of four sections that cover a total of 18 chapters. Part 1 focuses on the basics of Airflow, explaining what Airflow is and outlining its basic concepts. ■ Chapter 1 discusses the concept of data workflows/pipelines and how these can be built using Apache Airflow. It also discusses the advantages and disadvantages of Airflow compared to other solutions, including in which situations you might not want to use Apache Airflow. ■ Chapter 2 goes into the basic structure of pipelines in Apache Airflow (also known as DAGs), explaining the different components involved and how these fit together. ■ Chapter 3 shows how you can use Airflow to schedule your pipelines to run at recurring time intervals so that you can (for example) build pipelines that incrementally load new data over time. The chapter also dives into some intrica- cies in Airflow’s scheduling mechanism, which is often a source of confusion. ■ Chapter 4 demonstrates how you can use templating mechanisms in Airflow to dynamically include variables in your pipeline definitions. This allows you to reference things such as schedule execution dates within your pipelines. ■ Chapter 5 demonstrates different approaches for defining relationships between tasks in your pipelines, allowing you to build more complex pipeline structures with branches, conditional tasks, and shared variables. Part 2 dives deeper into using more complex Airflow topics, including interfacing with external systems, building your own custom components, and designing tests for your pipelines. ■ Chapter 6 shows how you can trigger workflows in other ways that don’t involve fixed schedules, such as files being loaded or via an HTTP call. ■ Chapter 7 demonstrates workflows using operators that orchestrate various tasks outside Airflow, allowing you to develop a flow of events through systems that are not connected. ■ Chapter 8 explains how you can build custom components for Airflow that allow you to reuse functionality across pipelines or integrate with systems that are not supported by Airflow’s built-in functionality. ■ Chapter 9 discusses various options for testing Airflow workflows, touching on several properties of operators and how to approach these during testing. ■ Chapter 10 demonstrates how you can use container-based workflows to run pipeline tasks within Docker or Kubernetes and discusses the advantages and disadvantages of these container-based approaches.
- 26. ABOUT THIS BOOK xxi Part 3 focuses on applying Airflow in practice and touches on subjects such as best practices, running/securing Airflow, and a final demonstrative use case. ■ Chapter 11 highlights several best practices to use when building pipelines, which will help you to design and implement efficient and maintainable solutions. ■ Chapter 12 details several topics to account for when running Airflow in a pro- duction setting, such as architectures for scaling out, monitoring, logging, and alerting. ■ Chapter 13 discusses how to secure your Airflow installation to avoid unwanted access and to minimize the impact in the case a breach occurs. ■ Chapter 14 demonstrates an example Airflow project in which we periodically process rides from New York City’s Yellow Cab and Citi Bikes to determine the fastest means of transportation between neighborhoods. Part 4 explores how to run Airflow in several cloud platforms and includes topics such as designing Airflow deployments for the different clouds and how to use built-in operators to interface with different cloud services. ■ Chapter 15 provides a general introduction by outlining which Airflow compo- nents are involved in (cloud) deployments, introducing the idea behind cloud- specific components built into Airflow, and weighing the options of rolling out your own cloud deployment versus using a managed solution. ■ Chapter 16 focuses on Amazon’s AWS cloud platform, expanding on the previ- ous chapter by designing deployment solutions for Airflow on AWS and demon- strating how specific components can be used to leverage AWS services. ■ Chapter 17 designs deployments and demonstrates cloud-specific components for Microsoft’s Azure platform. ■ Chapter 18 addresses deployments and cloud-specific components for Google’s GCP platform. People new to Airflow should read chapters 1 and 2 to get a good idea of what Airflow is and what it can do. Chapters 3–5 provide important information about Airflow’s key functionality. The rest of the book discusses topics such as building custom compo- nents, testing, best practices, and deployments and can be read out of order, based on the reader’s particular needs. About the code All source code in listings or text is in a fixed-width font like this to separate it from ordinary text. Sometimes code is also in bold to highlight code that has changed from previous steps in the chapter, such as when a new feature adds to an existing line of code. In many cases, the original source code has been reformatted; we’ve added line breaks and reworked indentation to accommodate the available page space in the book. In rare cases, even this was not enough, and listings include line-continuation
- 27. ABOUT THIS BOOK xxii markers (➥). Additionally, comments in the source code have often been removed from the listings when the code is described in the text. Code annotations accompany many of the listings, highlighting important concepts. References to elements in the code, scripts, or specific Airflow classes/variables/ values are often in italics to help distinguish them from the surrounding text. Source code for all examples and instructions to run them using Docker and Docker Compose are available in our GitHub repository (https://github.com/BasPH/ data-pipelines-with-apache-airflow) and can be downloaded via the book’s website (www.manning.com/books/data-pipelines-with-apache-airflow). NOTE Appendix A provides more detailed instructions on running the code examples. All code samples have been tested with Airflow 2.0. Most examples should also run on older versions of Airflow (1.10), with small modifications. Where possible, we have included inline pointers on how to do so. To help you account for differences in import paths between Airflow 2.0 and 1.10, appendix B provides an overview of changed import paths between the two versions. LiveBook discussion forum Purchase of Data Pipelines with Apache Airflow includes free access to a private web forum run by Manning Publications where you can make comments about the book, ask technical questions, and receive help from the author and other users. To access the forum and subscribe to it, go to https:/ /livebook.manning.com/#!/book/data- pipelines-with-apache-airflow/discussion. This page provides information on how to get on the forum once you’re registered, what kind of help is available, and its rules of conduct. Manning’s commitment to our readers is to provide a venue where a meaningful dialogue between individual readers and between readers and the authors can take place. It is not a commitment to any specific amount of participation on the part of the authors, whose contribution to the forum remains voluntary (and unpaid). We suggest you try asking the authors some challenging questions lest their interest stray! The forum and the archives of previous discussions will be accessible from the pub- lisher’s website as long as the book is in print.
- 28. xxiii about the authors BAS HARENSLAK is a data engineer at GoDataDriven, a company developing data- driven solutions located in Amsterdam, Netherlands. With a background in software engineering and computer science, he enjoys working on software and data as if they are challenging puzzles. He favors working on open source software, is a committer on the Apache Airflow project, and is co-organizer of the Amsterdam Airflow meetup. JULIAN DE RUITER is a machine learning engineer with a background in computer and life sciences and has a PhD in computational cancer biology. As an experienced soft- ware developer, he enjoys bridging the worlds of data science and engineering by using cloud and open source software to develop production-ready machine learning solutions. In his spare time, he enjoys developing his own Python packages, contribut- ing to open source projects, and tinkering with electronics.
- 29. xxiv about the cover illustration The figure on the cover of Data Pipelines with Apache Airflow is captioned “Femme de l’Isle de Siphanto,” or Woman from Island Siphanto. The illustration is taken from a collection of dress costumes from various countries by Jacques Grasset de Saint- Sauveur (1757–1810), titled Costumes de Différents Pays, published in France in 1797. Each illustration is finely drawn and colored by hand. The rich variety of Grasset de Saint-Sauveur’s collection reminds us vividly of how culturally apart the world’s towns and regions were just 200 years ago. Isolated from each other, people spoke different dialects and languages. In the streets or in the countryside, it was easy to identify where they lived and what their trade or station in life was just by their dress. The way we dress has changed since then and the diversity by region, so rich at the time, has faded away. It is now hard to tell apart the inhabitants of different conti- nents, let alone different towns, regions, or countries. Perhaps we have traded cultural diversity for a more varied personal life—certainly for a more varied and fast-paced technological life. At a time when it is hard to tell one computer book from another, Manning cele- brates the inventiveness and initiative of the computer business with book covers based on the rich diversity of regional life of two centuries ago, brought back to life by Grasset de Saint-Sauveur’s pictures.
- 30. Part 1 Getting started This part of the book will set the stage for your journey into building pipe- lines for all kinds of wonderful data processes using Apache Airflow. The first two chapters are aimed at giving you an overview of what Airflow is and what it can do for you. First, in chapter 1, we’ll explore the concepts of data pipelines and sketch the role Apache Airflow plays in helping you implement these pipelines. To set expectations, we’ll also compare Airflow to several other technologies, and dis- cuss when it might or might not be a good fit for your specific use case. Next, chapter 2 will teach you how to implement your first pipeline in Airflow. After building the pipeline, we’ll also examine how to run this pipeline and monitor its progress using Airflow’s web interface. Chapters 3–5 dive deeper into key concepts of Airflow to give you a solid understanding of Airflow’s underpinnings. Chapter 3 focuses on scheduling semantics, which allow you to configure Air- flow to run your pipelines at regular intervals. This lets you (for example) write pipelines that load and process data efficiently on a daily, weekly, or monthly basis. Next, in chapter 4, we’ll discuss templating mechanisms in Airflow, which allow you to dynamically reference variables such as execution dates in your pipelines. Finally, in chapter 5, we’ll dive into different approaches for defining task dependencies in your pipelines, which allow you to define complex task hierarchies, including conditional tasks, branches, and so on. If you’re new to Airflow, we recommend making sure you understand the main concepts described in chapters 3–5, as these are key to using it effectively.
- 31. 2 PART 1 Getting started Airflow’s scheduling semantics (described in chapter 3) can be especially confusing for new users, as they can be somewhat counterintuitive when first encountered. After finishing part 1, you should be well-equipped to write your own basic pipe- lines in Apache Airflow and be ready to dive into some more advanced topics in parts 2–4.
- 32. 3 Meet Apache Airflow People and companies are continuously becoming more data-driven and are devel- oping data pipelines as part of their daily business. Data volumes involved in these business processes have increased substantially over the years, from megabytes per day to gigabytes per minute. Though handling this data deluge may seem like a considerable challenge, these increasing data volumes can be managed with the appropriate tooling. This book focuses on Apache Airflow, a batch-oriented framework for building data pipelines. Airflow’s key feature is that it enables you to easily build scheduled data pipelines using a flexible Python framework, while also providing many building blocks that allow you to stitch together the many different technologies encountered in modern technological landscapes. This chapter covers Showing how data pipelines can be represented in workflows as graphs of tasks Understanding how Airflow fits into the ecosystem of workflow managers Determining if Airflow is a good fit for you
- 33. Another Random Document on Scribd Without Any Related Topics
- 34. there chanced to fall under my eye in the editorial column of a Sunday school paper the statement that Col. Alexander Hogeland of Louisville, Ky., had visited Nashville, Tenn., in the interest of the "Curfew Law." Other items in the column caused a momentary disturbance of my brain cells, then passed away to be recalled no more. But this one lingered in my memory and would not down, for thereby hangs the following tale: The expedition against the Federal force at Huntsville was commanded by Col. Gracie, of Alabama, and consisted of the 12th Ga. Battalion, a portion of an Alabama regiment, and a few cavalry. Leaving Clinton at 4 p. m., Aug. 12, we camped near Jacksonboro on the night of the 13th and on the morning of the 14th started for Huntsville by a rough mountain path that crossed a spur of the Cumberland range. After a toilsome tramp we halted at 9 p. m. and after an hour's rest were again on the march. The path is narrow and the overarching trees shut out every ray of starlight. Groping along in the dark we follow the tramp of the feet in front, reaching out occasionally to touch the file just ahead, lest our ears have deceived us. Our pathway passes on the edge of a precipitous bluff and my brother in Crump's company loses his footing and topples over it. The fall fails to disable him, but he loses his hat and in the darkness is unable to recover it. Hatless he rejoins the command and the procession moves on. Just before daylight we halt for another rest. At 5 a. m. we resume the march and in the early morning reach the vicinity of the Federal camp. Deploying into line of battle we advance through a belt of woodland and entering a cornfield beyond, our right is fired upon by the Federal pickets. As we drive them in a scattering fire is kept up until we come in sight of their camp and near it a rude log fort built upon the crest of a tall hill, over whose precipitous slope the forest trees have been felled, making an almost impassable abattis. While arrangements are being made for an attack upon the fort, Tom Tutt and the writer, who are both on the color guard, see a thin line four or five hundred yards to our right, near a church, and whom we take to be the pickets, who had been resisting our advance. Tom, whose rule is to shoot at
- 35. everything in sight, selects his man and fires and the writer follows suit. We load and fire again. After a few rounds I become convinced that it is a portion of Capt. Crump's company, which had been detached and sent to the right and in which I have two brothers. As Tom raises his gun again I said, "Hold on, Tom, you are shooting at your own company." He made no reply and continued firing until the order to advance was given. A deep gully lay partially in our front and as its passage caused some confusion in the ranks, we halted to reform the line. Crump's company was hurrying forward to join us and before they had reached their position in line Col. Gracie gave the command, "Charge." From underneath the head logs of the fort the Belgian rifles were barking at us and the heavy balls they carried whistled by us like young shells. We were waiting for Crump, and Gracie, ignorant of the cause of the delay, shouted: "What is the matter with the 12th Ga. Battalion?" Just then a lone cavalryman passed the line on foot and with drawn sabre made his way towards the fort with the evident intention of capturing the whole business himself. Crump's company came up at a "double quick" and the whole line moved forward with a yell. Sergeant Harwell, our color- bearer, had never been under fire and the boys, uncertain as to his grit, had asked Tom Tutt, who did not know what fear meant, to take the colors when the charge began. Tom made the effort to seize them, but Harwell, a tall, gaunt man, and brother of two honored Methodist preachers, declined to give them up and bore them forward bravely. As we advanced the fire from the fort suddenly ceased and we thought they were waiting to see the whites of our eyes. Reaching the steep ascent we climbed up over logs and brush until the fort was gained. Lieut. Joe Taliaferro, of Augusta, was the first to enter, and with his sword cut down the floating flag. The fort was empty—not a Yankee to be seen. Under cover of the thick forest growth in their rear they had hid to other haunts, under the idea, perhaps, that "He who fights and runs away, Will live to fight another day."
- 36. Their camp, located just below the fort gave ample evidence of their hasty exit. Our attack was something of a "surprise party" and their unfinished morning meal was boiling, baking and frying on the camp fires. We were unexpected and uninvited guests and yet our reception was warm, although unfriendly. Our all-night tramp enabled us to do full justice to the breakfast they had prepared, as well as the sugar cured hams and other supplies their commissary had kindly left for our use. We appropriated an ample outfit of blankets, canteens, haversacks, etc., and burned what we could not carry away. The skirmish on our side, and probably on theirs was almost bloodless. W. W. Bussey, of the Oglethorpes, and Garyhan, of Crump's company, were slightly wounded. I recall no other casualty except the killing of a nice horse ridden by Col. Gracie. And now what has all this to do with the item I read in a Sunday school paper? Simply this: Among the assets and effects secured that day by the writer from the officer's tent and administered upon without "Letter's Testamentary" was a pocket diary belonging to Capt. Alexander Hogeland, of the 10th Indiana Regt. On reading the paragraph referred to, the coincidence in names suggested the possibility that Col. Alexander Hogeland, of Louisville, Ky., "Father of the Curfew," might have been Capt. Alexander Hogeland, of the 10th Ind. Regt., whose property had been in my possession for thirty- seven years. To test the matter, I wrote Col. Hogeland and from his reply the following extract is taken: "Your deeply interesting favor of the 4th inst received and for the information it contains accept my hearty thanks. I am the identical person referred to in your letter. Was first lieutenant Co. D, 10th Indiana Regiment in the West Virginia campaign and afterwards Captain of Co. G. In May, '62, was made lieutenant-colonel of 7th East Tennessee Regiment, commanded by Col. Wm. Cliff, and stationed at Huntsville, Tenn., in August, '62. We lost everything on the occasion you refer to and this is the first information I have received as to the whereabouts of my effects. I am very glad to avail myself of your proffer to return my
- 37. diary and enclose herewith necessary postage." Col. Hogeland's diary was duly returned to him and in acknowledging its receipt he took occasion to thank me for looking him up after all these years and assured me that he would endeavor to return that kindness by visiting Augusta in the early future and giving the citizens of this goodly city the benefit of the "Curfew Law." It will furnish additional evidence of the truthfulness of the opening statement in this sketch if the capture of a war diary nearly forty years ago, should result in the adoption of a "Curfew" ordinance in Augusta. In illustration of the adage that "Every dog has his day," it may not be amiss to say that Col. Hogeland's escapade from Fort Cliff at the instance of four companies of the old First Georgia Regiment, was only partial compensation for the 100-mile run made by those self-same companies from Laurel Hill, Va., in '61, with Capt. Hogeland's regiment as one of the exciting causes. JACKSBORO. On our return from Huntsville, Joe Derry and J. W. Lindsay, of the Oglethorpes, unable to keep pace with the command, straggled and were captured by "bush-whackers." Joe was exchanged a few days, later, Lindsay preferring to remain a prisoner. After a short stay at Clinton we moved up to Jacksboro and remained there until Oct. 9th, guarding Bragg's line of communications. Our service at this place was uneventful. Buell's army had retreated into Kentucky and there was nothing to disturb our "otium cum dignitate" save a moderate amount of picket duty and the one subject ever uppermost in the soldier's mind—"rations." The following incidents of our stay at this camp furnish some illustrations of this fact: THE PARSON AND THE GRAVY. A continuous diet of salt bacon had made the boys ravenous for fresh meat and as war has no tendency to strengthen respect for
- 38. property rights where a soldier's appetite is involved, they were not, as a rule, very scrupulous as to the methods adopted to procure a supply. The means most in use at the date referred to were known in camp parlance as "flip ups." As no encyclopedia of my acquaintance describes this mechanical contrivance and its specifications have never encumbered the records of the patent office, it may not be amiss to say that it consisted of a bent sapling, a slip noose with a trigger attachment and a bait of corn. The unsuspecting porker, tempted by the bait, sprang the trigger and the sapling freed from its confinement, sought to resume its normal position, while the shote caught in the noose and partially suspended in the air gave noisy notice that the game was up. On one occasion the catch, by right of discovery or otherwise, fell to a mess, of which Parson H——, a minister of the Presbyterian persuasion, was a member. When dinner was served that day a dish of smoking pork chops was passed to the Parson, but he declined with the remark that his conscience did not allow him to eat stolen meat. As the meal progressed the fragrant odor from the dish struck his olfactories with increasingly tempting force and he finally passed up his tin plate and said: "I'll take a little of the gravy if you please." He had made a brave fight for principle and his final compromise was probably due to the fact that Paul's vow, "If meat make my brother to offend I will eat no flesh while the world standth," failed to include gravy in its inhibition. He may have been further influenced by the reflection that his refusal to indulge could not possibly restore the porker to life again. As Jim Wilson said, "'Twas Greece (grease), but living Greece no more." This incident recalls the fact that Jim and the writer had on this subject the same scruples as the Parson, and in order to place ourselves on the line of strongest resistance we entered into an agreement with each other binding ourselves to total abstinence from all meat of questionable origin until mutually released from the obligation. The compact was religiously observed until Hood's
- 39. campaign in Tennessee in the winter of '64. Transportation was scarce and rations were scarcer. On one occasion two ears of corn were issued to each soldier. Some wag in the company, probably Elmore Dunbar, seeing that horse rations were being furnished sang out, "come and get your fodder." On another occasion beef was issued but no bread. We had neither lard to fry nor salt to season, but our digestive apparatus was not then fastidious as to condiments. It was unimportant whether it was taken "cum grano salis" or without, so the void was filled. A fire was built of dried limbs from a brush pile and the beef placed in a shallow frying pan to stew, Frank Stone being the chef de cuisine. The mess sat around with anxious faces and whetted appetites. Finally one of them, in shifting his position, struck the end of a limb on which the pan was resting and dumped the whole business into the dirt and ashes. The catastrophe placed us rather than the beef in a stew and we went to bed supperless. Under such conditions it is, perhaps, but natural that the case should be re-opened, a new trial granted and a verdict rendered to follow Paul's other injunction, "Whatsoever is set before you, eat, asking no questions for conscience sake." I can not recall positively that either of us ever indulged even as to gravy, but I think I can say that neither of us was particepts criminis in the act of impressment. If guilty, we were only accessories after the fact. "THEM MOLASSES." During our stay at Jacksboro the farmers in that section were making sorghum syrup, which most of them called "them molasses." Near one of our picket posts lived a Baptist minister named Lindsay, from whose better half we purchased vegetables and other edibles. On one occasion I was unable to make exact change and left owing her 12 1-2 cents in Confederate money. Two weeks later I was on
- 40. picket again and paid her the balance due. She was so much surprised that a soldier should have the moral sense to recognize and meet such an obligation that she formed a very exalted estimate of my honesty and when I afterwards went to buy some of "them molasses" she requested her husband to take it from a barrel she had reserved for her own use "for," he said "she likes 'em powerful thick." I had occasion to regret her kindness, for it was so thick that it was with difficulty that I could get it either into or out of my canteen, and in view of her partiality I did not have the heart to suggest that a thinner grade would be preferred. She was a kind and motherly soul, and yet some of the soldiers would steal from her. To prevent or minimize their depredations she cooped a noisy rooster underneath her bedroom as a sort of watch dog to notify her of any midnight foragers. A few mornings afterwards she awoke to find, aside from other losses, that her feathered sentinel had been caught asleep upon his post by some soldier, who was chicken-mouthed, if he was not chicken-hearted. RATIONS. Rations as one of the sinews of war, deserve something more than incidental mention in these memories and as no more favorable opportunity may occur, it may be as well to give them more extended notice in connection with the incident just related. Confederate rations during the early years of the war were as I recollect them, not only fair in quality but ample in quantity. As evidence of this fact I remember that the boys were sometimes so indifferent when rations hour arrived that it was difficult to induce them to draw their allowance promptly. Charles Catlin was our company commissary and I can hear now his clear, sharp tones as they rang out on the frosty evening air among the Virginia mountains in '61, "Come up and get your beef. Are you going to keep a man standing out here in the cold all night?"
- 41. As the war progressed the resources of the Confederacy, limited to its own production by the cordon of hostile gunboats that girded its ports, became more and more heavily taxed and its larder grew leaner and leaner. But little wheat was raised in the Gulf States and few beeves except in Texas. We were reduced largely to meal and bacon rations, and the supply of these sometimes recalled the instructions in regard to loading a squirrel rifle given by its owner to a friend to whom he had loaned it: "Put in very little powder, if any." Cooking squads were detailed from each company and once a day the wagons would drive up and issue three small corn pones to each man. Some of the boys, whose hunger was chronic, would begin on theirs and never stop until the last pone had been eaten. Bob Winter belonged to this class and eight or ten hours after his daily rations had disappeared Dick Morris would draw a pone or half a pone from his haversack and say, "Bob, here's some bread if you want it," and Bob would reply, "Dick, I don't want to take it if you need it," and Dick would answer, "Bob, I've told you a thousand times that I wouldn't give you anything that I wanted," and Bob would succumb and so would the bread. When our changes of base were rapid the squads would cook up two or three days' rations and in hot weather the bread would mould and when broken open the fungus growth looked very much like cobweb. Some of the pones had also the appearance of slow convalescence from chill and fever. Under such conditions it could hardly be considered very palatable except upon the idea of a rustic friend of mine, who, in commending the virtues of India Cholagogue, was asked as to its palatability. "O," said he, "it's very palatable, but the meanest stuff to take you ever saw." Most of the boys had left well-to-do homes to enter the service and while they bore privation and hunger without a murmur, there would sometimes come into their hard lives a craving for the good things they had left behind. Gathered about the camp-fire, cold and tired and hungry, they would discuss the dish that each liked best and their lips would grow tremulous as they thought of the day
- 42. when hope would become realization. Joe Derry, I remember, could never be weaned away from the memory of his mother's nice mince pies and black-berry jam. I can see his eyes dance now as he magnified their merits. Bob Winter's ultimate thule in the gastronomic line was sliced potato pie, while Jim Thomas would never tire of singing the praises of 'possum baked with potatoes. Louis Picquet said to him one day, "Jim, if I ever get home again I am going to have one dinner of 'possum and 'taters if it kills me." But it was left to the epicurean taste of John Henry Casey to reach the acme of these unsatisfied longings when, recognizing the value of quantity as well as quality he declared that nothing less would satisfy him than "a chicken pie big enough to trot a horse and buggy around on." But for extending this ration sketch to an irrational length I might have said something of the May Pop leaves that we cooked for "greens" in North Georgia, of the half hardened corn transformed into meal by means of an improvised grater prepared by driving nails through the side of a tin canteen, of the pork issued to us in Tennessee with the hair still on it, of the hog skins that we ate at Inka, Miss., and of many other such things, but they would probably fail to interest the reader as they did the actors in those far off days.
- 43. CHAPTER IV. TRANSFERRED TO THE COAST. Our enlistment as artillery had so far proven a delusion and a snare. The Confederacy had no guns with which to equip us and we had found no opportunity to capture any. During our stay at Jacksboro Capt. Allen succeeded in securing from the War Department the transfer of the Oglethorpes to the 2nd South Carolina Artillery, then in service at Charleston. Oct. 9, '62, at 6 p. m. we fell into line, gave three cheers for our late companions in arms and as the setting sun crimsoned with its last rays the lofty summit of the Cumberland, we filed out of the village to the tune of "We are sons of old Aunt Dinah, And we go where we've amind to And we stay where we're inclined to, And we don't care a——cent." and our sojourn in Jacksonboro was a thing of the past. Reaching Augusta Oct. 13, we were dismissed until the 23rd, when we went into camp at the Bush Ground, near the city. Why we did not proceed at once to our command in Charleston has always been to the writer an unsolved problem. We remained in Augusta until Dec. 9, when orders were received to report to Gen. H. W. Mercer, at Savannah. Col. Geo. A. Gordon, in command of the 13th Ga. Battalion was endeavoring to raise it to a regiment. As he lacked two companies and as the Oglethorpes had 120 men on its roll an effort was made to divide the company. On Dec. 11 a vote was taken, the result showing a majority against division. Dec. 15 we were formally attached to the 63rd Ga. Regiment, ranking as Co. A.
- 44. Our quarters were located just in the rear of Thunderbolt Battery and here we remained for more than twelve months in the discharge of semi-garrison duty. A STUDY IN INSECT LIFE. The period covered by our service on the coast formed a sort of oasis in our military life. The Federal gunboats were kind enough to extend social courtesies to us only at long range and longer intervals. We fought and bled, it is true, but not on the firing line. The foes that troubled us most, were the fleas and sand fleas and mosquitoes that infested that sections. They never failed to open the spring campaign promptly and from their attacks by night and day no vigilance on the picket line could furnish even slight immunity. If the old time practice of venesection as a therapeutic agent was correct in theory our hygienic condition ought to have been comparatively perfect. During the "flea season" it was not an unusual occurrence for the boys after fruitless efforts to reach the land of dreams, to rise from their couches, divest themselves of their hickory shirts and break the silence of the midnight air by vigorously threshing them against a convenient tree in the hope of finding temporary "surcease of sorrow" from this ever-present affliction. It was said that if a handful of sand were picked up half of it would jump away. I can not vouch for the absolute correctness of this statement, but I do know that I killed, by actual count, one hundred and twenty fleas in a single blanket on which I had slept the preceding night and I can not recall that the morning was specially favorable for that species of game either. I remember further that as we had in camp no "Society for the Prevention of Cruelty to Animals," I corked up an average specimen of these insects to see how long he would live without his daily rations. At the end of two weeks he had grown a trifle thin, but was still a very lively corpse. But these were not the only "ills, that made calamity of so long a life," for as Moore might have said, if his environment had been different,
- 45. "Oft in the stilly night, Ere slumber's chain had bound me, I felt the awful bite Of 'skeeters buzzing 'round me." Their bills were presented on the first day of the day of the month and, unfortunately, on every other day. At our picket stations on Wilmington and White marsh Islands and at the "Spindles" on the river where the young alligators amused themselves by crawling up on the bank and stealing our rations, there was a larger variety known as gallinippers, from whose attacks the folds of a blanket thrown over our faces was not full protection. But there were still others. On dress parade in the afternoons, while the regiment was standing at "parade rest" and no soldier was allowed to move hand or foot until Richter's band, playing Capt. Sheppards Quick step, had completed its daily tramp to the left of the line and back to its position on the right, the sandflies seemed to be aware of our helplessness and "in prejudice of good order and military discipline" were especially vicious in their attack upon every exposed part of our anatomy. Capt. C. W. Howard, I remember, was accustomed to fill his ears with cotton as a partial protection. I have seen Charlie Goetchius, while on the officers' line in front of the regiment, squirm and shiver in such apparent agony that the veins in his neck seemed ready to burst. Neither whistling minies, nor shrieking shells, nor forced marches with no meal in the barrel nor oil in the cruse ever seemed to disturb his equanimity in the slightest degree. Quietly and modestly and bravely he met them all. But the sandfly brigade was a little too much for him. In addition to these discomforts, the salt water marsh, near which we were camped, never failed to produce a full crop of chills and fever as well as of that peculiar species of crabs known as "fiddlers." Gen. Early was once advised by one of his couriers that the Yankees were in his rear. "Rear the d—l," said old Jubal, "I've got no rear. I'm front all round." These fiddlers seemed to be in the same happy
- 46. condition. Their physical conformation was such that no matter from what side they were approached, they retired in an exactly opposite direction without the necessity of changing front. But of the chills. Of the one hundred and fifteen men in our ranks only three escaped an attack of this disease. The writer was fortunately one of the three. One man had fifty-three chills before a furlough was allowed him. Quinine was scarce and boneset tea and flannel bandages saturated with turpentine were used as substitutes. Whiskey was sometimes issued as a preventative. In pursuance of a resolution formed on entering the service I never tasted the whiskey and as soon as my habit on this line became known, I was not subjected to the trouble of looking up applicants for the extra ration. The dearth in medical supplies recalls other facts showing the straits to which the Confederacy was reduced on other lines by the blockade of its ports. Letters written in '63, and now in my possession, show that my brother, then Assistant Surgeon at Tallahassee, Fla., could not purchase in that place a pair of suspenders nor a shirt collar—that my mess could not buy an oven in Savannah, though willing to pay $30 for it and that I ordered shoes for Capt. Picquet, and other members of the company from a Mr. Campbell at Richmond Factory, as no suitable ones could be had in Savannah. Our service at Thunderbolt was entirely devoid of any exciting incident or episode in a martial way. If the company fired a single shot at a Yankee during our stay I can not recall it. On one occasion 8 or 10 volunteers from each regiment stationed there were wanted for "a secret and dangerous expedition," as it was termed in the order. There was a ready response from the Oglethorpes for the entire number wanted from the regiment. Among those volunteers I recall the names of W. J. Steed, J. E. Wilson, R. B. Morris, J. C. Kirkpatrick and F. I. Stone. We never knew whether it was a contemplated attack on Fort Pulaski or the capture of a Federal gunboat, as the expedition failed to materialize. April 18, '63, Henry Wombke of the Oglethorpes, was drowned while bathing in Warsaw Sound, and on July 12, '63, John Quincy
- 47. Adams, while returning from picket at the Spindles was accidentally shot by George Mosher, who had gone up on the boat to kill alligators. Some official changes took place in the company during our stay at this camp. To fill the vacancy occasioned by the resignation of Lieut. W. G. Johnson, Charles T. Goetchius was elected, but I have no record of the date. On July 5, '63, the death of Major John R. Giles resulted in the promotion on July 12, of Capt. J. V. H. Allen to that field office in the regiment. Louis Picquet became captain of the company, and on July 14, Geo. W. McLaughlin was elected Jr. 2nd. Lieut. As a part of the "res gestae" of our soldier life at Thunderbolt, the following incident may be of some interest: SOAP AND WATER. My earliest recollections of Thunderbolt is associated with a fruitless effort to mix turpentine soap and salt water. We had reached the place tired and dusty and dirty. As soon as the ranks were broken, the boys divested themselves of their clothing and soaping their bodies thoroughly plunged into the salt water for a bath. The result may be imagined. The dirt and dust accumulated in streaks, which no amount of scrubbing could dislodge for it stuck closer than a postage stamp. A SUGARED TONGUE. Col. Geo. A. Gordon was a pleasant, persuasive speaker and in his address to the company urging its division so as to complete the quota necessary for a regimental organization he held out to us a tempting array of promises as to our treatment if his wishes were complied with. An Irish member of his old company heard the
- 48. speech and in commenting on it said, "Faith, the sugar on his tongue is an inch thick." The Oglethorpes, though serving as infantry, had retained their artillery organization and Gordon in his plea for a division, said that the incorporation of such an organization into an infantry regiment would be an anomaly—that we would be "nyther fish, flesh nor fowl," giving the English pronunciation to the word "neither." Some time afterward the Colonel was making his Sunday morning inspection of quarters and had reached Elmore Dunbar's tent. As some of Dunbar's mess were sick, he had hoisted a yellow handkerchief over the tent and with a piece of charcoal had placed on its front the sign, "Wayside Home." Gordon saluted as he came up, and then noticing the sign said, "Sergeant, what is your bill of fare today," "Nyther fish, flesh nor fowl," said Dunbar, and the Colonel smiled and went his way. FIRE AND FALL BACK The monotony of garrison duty and our comparative exemption from danger during our stay at Thunderbolt, developed the spirit of mischief in the boys to an inordinate degree and no opportunity for its exercise was allowed to go unimproved. Bob Lassiter, while off duty one day, was taking a nap on a "bunk" in his cabin. His unhosed feet protruded from the window, probably with a view to fumigation by the salt sea breeze. Jim McLaughlin passed by and taking in the situation called Jim Thomas. Twisting and greasing a strip of paper they placed it gently between Bob's unsuspecting toes, fired the ends and then made themselves scarce in that locality. As the lambent flame "lipped the Southern strand" of Bob's pedal extremities, he, doubtless, felt in the language of Henry Timrod, "Strange tropic warmth and hints of summer seas" and probably dreamed of "A Hot Time in the Old Town" that day. But if so his dreams were short-lived. With a yell of pain he fell back on the floor of his cabin, and then,
- 49. He hotly hurried to and fro, To find the author of his woe; The search was vain for chance was slim To fasten guilt on either Jim. SKIRMISHING FOR PIE Dessert was not a standing item on our army bill of fare, and when, by chance or otherwise, our menu culminated in such a course, moderation in our indulgence was one of the lost arts. One day in '63, W. J. Steed and I, with several other comrades chanced to be in Savannah at the dinner hour. Our rations for a long time had known no change from the daily round of corn bread and fat bacon, and we decided to vary this monotony by a meal at the Screven House. The first course was disposed of and dessert was laid before us. Steed finished his but his appetite for pie was still unsatisfied. Calling a waiter he said, "Bring me some more pie." "We furnish only one piece," said the waiter. The first course plates had not been removed from the table, but simply shoved aside. The waiter passed on and Steed pushed the dessert plate from him and gently drawing the other back in his front, awaited results. Another waiter passed and thinking Steed had not been served, brought him another piece of pie. This being disposed of the program was again repeated and still another waiter supplied dessert. The shifting process was continued until his commissary department could hold no more and he was forced to retire upon the laurels he had won in the field of gastronomic diplomacy. STEED AND THE SUGAR My friend's penchant for pie may have had its influence in the origin of a problem in the company, which like the squaring of the circle has never received a satisfactory solution. He held during his
- 50. term of service the office of commissary sergeant for the company, a position in which it was difficult at any time and impossible when rations were scarce, to give entire satisfaction. These difficulties in his case were, perhaps, enhanced by the peculiarities of his poetic temperament, which caused him to live among the stars and gave him a distaste for the bread and meat side of life, except possibly as to pie. Try as faithfully as he would to show strict impartiality in the distribution, there was sometimes a dim suspicion that the bone in the beef fell oftener to other messes than his own and that the scanty rations of sugar issued weekly were heaped a little higher when his mess had in contemplation a pie or pudding on the following day. These suspicions finally culminated in an inquiry, which became a proverb of daily use; an inquiry, which formed the concluding argument in every camp discussion, whether on a disputed point in military tactics or on the reconciliation of geological revelation with the Mosaic cosmogony; an inquiry with which Jim McLaughlin and Jim Fleming still salute their former commissary: "What has that to do with Steed and the sugar?" Of course there was never any foundation for such a feeling and probably never any real suspicion of favoritism in the matter. These things formed the minor key of our soldier life and served as they were intended, to enliven its sometimes dull monotony. My friend, and I am glad to have been honored so long by his friendship, will pardon, I know, in the gentleness of his heart a revival of these memories. Aside from the faithful discharge of the difficult duties of his position, it gives me pleasure to add my willing testimony to the silent witness of his armless sleeve, that on the firing line and in all the sphere of duty, to which the service called him, he was every inch a soldier. "BUTTER ON MY GREENS." For the convenience and comfort of the soldiers going to and returning from their commands, "Wayside Homes" were established
- 51. at different points in the Confederacy where free lunches were served by the fair and willing hands of patriotic young ladies living in the vicinity. A uniform of grey was the only passport needed. One of these "Homes" was located at Millen, Ga. Detained there on one occasion, en route to my command at Thunderbolt I was glad to accept their hospitality. Seated at the table enjoying the spread they had prepared one of these fair waiting maids approached me and asked if I would take some butter on my "greens." My gastronomic record as a soldier had been like Joseph's coat, "of many colors." I had eaten almost everything from "cush" and "slapjacks" to raw corn and uncooked bacon. I had made up dough on the top of a stump for a tray and cooked it on a piece of split hickory for an oven. I had eaten salt meat to which the government had good title, and fresh meat to which neither I nor the government had any title, good or bad. But butter on "greens" was a combination new to my experience and as my digestive outfit had, during my school days, been troubled with a dyspeptic trend, I felt compelled to decline such an addition to a dish that had been boiled with fat bacon. Notwithstanding the absence of my friend Steed the supply of pie that day was short, and with a degree of self-denial, for which I can not now account, I asked for none. A soldier next me at the table, however, filed his application and when our winsome waitress returned, she handed the desert to me and left my neighbor pieless. I could not recall her fair young face as one I had ever seen before, and I had always been noted for my lack of personal comeliness. I was at a loss therefore to understand why the unsolicited discrimination in my favor had been made. A few minutes later the problem was solved. Standing on the porch after the meal had ended, this self-same maiden approached me a little timidly and asked, "When did you hear from your brother Sammie?" She and my younger brother, it seemed, had been schoolmates, and, as I learned afterwards, "sweethearts" as well, and the pie business was no longer a mystery.
- 52. If she still lives as maid or matron and this sketch should meet her eye, it gives me pleasure to assure her that the fragrance of her kindly deed though based upon no merit of my own, still lingers lovingly in my memory, like the echo of "faint, fairy footfalls down blossoming ways." OUR CAMP POET. "Dropping into poetry" has not been a peculiarity confined to that singular creation of Dickens' fancy, "Silag Wegg." While not a contagious disease, it is said that a majority of men suffer from it at some period in life. Like measles and whooping cough it usually comes early, is rarely fatal and complete recovery, as a rule, furnishes exemption from further attacks, without vaccination. Under these conditions it is but natural that the Oglethorpes should have had a poet in their ranks. In fact we had two, James E. Wilson and W. J. Steed, who has already figured somewhat in these memories, and who was called Phunie, for short. The latter was, however, only an ex-poet, not ex-officio, nor ex-cathedra, but ex-post facto. His attack had been light, very light, a sort of poetical varioloid. He had recovered and so far as the record shows, there had been no relapse. On the first appearance of the symptoms he had mounted his "Pegasus," which consisted of a stack of barrels in rear of his father's barn, and after an hour's mental labor, he rose and reported progress, but did not ask leave to sit again. The results are summed up in the following poetic gem: "Here sits Phunie on a barrel, With his feet on another barrel." He has always claimed that while the superficial reader might find in these lines an apparent lack of artistic finish, with some possible defects as to metre and an unfortunate blending of anapestic and iambic verse, the rhyme was absolutely perfect. I have been unable to discover in them the rhythmic and liquid cadence that marks
- 53. Buchannan Reade's "Drifting," or the perfection in measure attributed by Poe to Byron's "Ode" to his sister, yet my tender regard for my old comrade disinclines me to take issue with him as to the merits of this, the sole offspring of his poetic genius. My inability to find it in any collection of poetical quotations has induced me to insert it here with the hope of rescuing it from a fate of possibly undeserved oblivion. Jim Wilson's case was different. His was a chronic attack. "He lisped in numbers for the numbers came." As a poet he was not only a daisy, but, as Tom Pilcher would say, he was a regular geranium. I regret that my memory has retained, with a single exception, only fragments of his many wooings of the muse. A young lady friend, Miss Eve, of Nashville, asked from Jim a christening contribution to an album she had just purchased. He was equal to the occasion. The man and the hour had met. He was in it from start to finish. He filled every page in the book with original verse. I recall now only the following stanza:
- 54. "Newton, the man of meditation, The searcher after hidden cause, Who first discovered gravitation And ciphered out attractions laws, Could not, with all his cogitation, Find rules to govern woman's jaws." But his special forte was parody. A competitive examination was ordered at Thunderbolt in '63 to fill the position of second sergeant in the company. After studying Hardee's Tactics for a week Jim relieved his feelings in the following impromptu effort: Tell me not the mournful numbers From a "shoulder" to a "prime," For I murmur in my slumbers Make two "motions in one time." The Oglethorpes, though serving as infantry had clung tenaciously to their artillery organization and to the red stripes and chevrons which marked the heavier arm of the service. On our assignment to Gordon's regiment, the Colonel had made a very strong appeal to us to divide the company and to discard our artillery trimmings. At the next Sunday morning inspection Jim's tent bore a placard with this inscription, intended for the Colonel's eye: "You may cheat or bamboozle us as much as you will, But the sign of artillery will hang round us still." Probably his masterpiece was a parody on "Maryland," written at Jacksonboro, Tenn., on the eve of our transfer from the 12th Ga. Battalion. That the reader may understand the personal allusion in the verses it is necessary to say that Edgar Derry, Jim Russell, Ed Clayton and Alph Rogers had been detailed by Col. Capers to fill certain staff positions with the battalion; that Miles Turpin was company drummer and Stowe—whose camp sobriquet was "Calline," was fifer; that in the skirmish at Huntsville, Tenn., W. W. Bussey,
- 55. who was known in camp as "Busky," had been shot in the temple; that before the final charge on the fort, Col. Capers in crossing a ditch had mired in its bottom and had found some difficulty in extricating himself; that the war horse of the male persuasion ridden by Col. Gracie had been killed in the skirmish and that Randolph was Secretary of War. When the transfer had been effected it was uncertain whether the detailed men would retain their position or would return to the company, and the following verses were written by Jim as an appeal to them to go with us: Come 'tis the red dawn of the day, Here's your mule, Come, details, join our proud array, Here's your mule. With Clayton panting for the fray, With Rogers urging on that bay, With Derry bold and Russell gay, Here's your mule. Oh! Here's your mule. Come for your limbs are stout and strong, Here's your mule, Come for your loafing does you wrong, Here's your mule, Come with your muskets light and long, Rejoin the crowd where you belong, And help us sing this merry song, Here's your mule, Oh! Here's your mule. Dear fellows break your office chains, Here's your mule, The "Web-feet" should not call in vain, Here's your mule, But if it goes against the grain, "Sick furlough" is the proud refrain, By which you may get off again, Here's your mule. Oh! Here's your mule.
- 56. We trust you will not from us scud, Here's your mule, And nip your glory in the bud, Here's your mule, Remember "Busky" bathed in blood, Remember Capers stuck in mud, And gallant Gracie's dying stud, Here's your mule, Oh! Here's your mule. Ah, though you may awhile stay mum, Here's your mule, To "Calline's" fife and Turpin's drum, Here's your mule, When orders come from Randolph grum, You will not then be deaf nor dumb, Ah, then we know you'll come, you'll come, Here's your mule, Oh! Here's your mule. And now in conclusion, I am unwilling that my friend, Jim Wilson should be judged solely by these rhymes. If any allusion in them sounds harshly to ears polite, it must be remembered that they were intended, only for soldiers eyes and ears. The son of a Presbyterian missionary to India, he was an educated Christian gentleman, one of the brightest and wittiest men I have ever known, as brave as Julius Caesar and as true to the flag for which he fought as any man who wore the grey.
- 57. CHAPTER V. THE DALTON AND ATLANTA CAMPAIGN. Our service on the coast ended April 28, 1864. On April 23 orders were received transferring our regiment to Gen. A. R. Wright's Brigade, Army of Northern Virginia. Gen. H. W. Mercer in command, had been ordered to report for duty to Gen. Johnston at Dalton, Ga. As Gordon and Mercer were both Savannah men and their war service to that date had thrown them together, they succeeded in inducing the War Department to change our orders and assign us to Johnston's Army. April 28 we left Savannah, reaching Dalton at 3 a. m. April 30, and on May 4 were attached to Gen. W. H. T. Walker's division, three miles east of Dalton. On May 7 Sherman opened his Atlanta campaign and for one hundred days the rattle of musketry, the roar of cannon, the shrieking of shells and the zip of minies, grew very familiar to us, if not very amusing. Our first sight of the enemy was at Rocky Face Ridge, May 9. Our pickets were driven in and our trenches shelled, causing some casualties in the regiment, but none in the Oglethorpes. Lieut. Reddick of Co. B, while reading a newspaper in rear of the trenches was killed by a Federal sharpshooter. No assault was made on our position, but at three other points in Johnston's line efforts were made to carry the trenches, though the attacks were all repulsed. On the same day Sherman, probably anticipating such a result, began his flanking plan of campaign by sending McPherson through Snake Creek Gap to threaten Johnston's line of communications at Resaca. The Federal superiority in numbers at a ratio of nearly two to one, enabled Sherman to cover Johnston's entire front and gave him besides a large force with which to conduct his flanking operations, a policy he pursued persistently and successfully to the end of the campaign. As it is not my purpose to give the general features of this campaign,
- 58. but simply to record the share borne in it by the 63rd Ga. regiment, I can, perhaps best subserve that purpose by furnishing the following condensed extracts from my "War Diary" for that period, elaborating afterward any special features or incidents that may seem to merit more extended notice. May 10. Left trenches 1 a. m., marched to a point 3 miles from Resaca. (11). Marched to Resaca and returned. (12). Marched to a position one mile above Calhoun. (13). Quiet. Being unwell, on invitation of Lieut. Daniel spent the night with Rev. I. S. Hopkins and himself at the house of his mother in Calhoun. 14. Battle of Resaca. Rejoined command on its way to the front. Walker's division held in reserve until 12 p. m. Then ordered up to reinforce Stewart's division. Exposed to heavy artillery fire while crossing pontoon bridge at Resaca. Heavy fighting in our front. Enemy repulsed. 10 p. m., marched back through Calhoun to Tanner's Ferry. 15. In line of battle. Jackson's brigade charged enemy's line at the Ferry but were repulsed. 10 p. m., returned to Calhoun. 16. Marched to Tanner's Ferry. Heavy skirmishing between Steven's brigade and the enemy. Junius T. Steed of the Oglethorpes, wounded. Slept on our arms. 17. At 1 a. m. aroused and ordered to fall back to Adairsville. Remained in line of battle until 12 p. m. 18. Fell back four miles below Kingston. 19. Advanced and took position 2 miles from Kingston. Under fire from sharpshooters and skirmishers H. L. Hill killed and T. F. Burbanks wounded. 12 or 15 casualties in regiment. Retired to Cass station and formed line of battle. Johnston's battle order issued. 20. At 1 a. m. crossed the Etowah and fell back to within two miles of Altoona.
- 59. 21-22. Quiet. (23). Marched five miles in the direction of Dallas. 24. Aroused at daylight and marched 15 miles, camping near Powder Springs. 25. At 1 a. m. marched four miles back. At 2 p. m. moved forward a mile and formed line of battle. After night moved three miles and bivouacked. 26. At 3 a. m. went forward and took position in rear of Stewart's division. Skirmishing in front all day. 27. Moved to the left near Dallas and then a mile or two to the right. H. B. Jackson wounded. Oglethorpes and Co. I thrown out as skirmishers. At 11 p. m. brigade ordered away, leaving us on skirmish line without support. 28. Skirmishing all day. Capt. Picquet wounded in leg, A. W. McCurdy in head. 29. At 4 p. m. relieved from duty on skirmish line and rejoined regiment on Ellsbury Ridge. 30-June 1. Quiet. (2). Heavy rain. Division moved four miles to the right in rear of Stevenson, slippery march. 3. Quiet day. At 11 p. m. moved off to the right. Jackson's brigade and a portion of ours detached in the darkness, lost their way and forced to lie over till morning. 4. Rejoined division and built breastworks. Oglethorpes and Co. G on picket. Skirmishing with the enemy. At 12 p. m. relieved by Wheeler's cavalry and told to "git," as our army had fallen back. Overtook regiment after five mile tramp over muddiest road I ever saw. Moved 3 miles further and took position in rear of Gist's brigade. (6-7). Quiet. 8. Brigade on picket. 63d Ga. in reserve. 9-11. Quiet, and rain, rain, rain.
- 60. Welcome to our website – the ideal destination for book lovers and knowledge seekers. With a mission to inspire endlessly, we offer a vast collection of books, ranging from classic literary works to specialized publications, self-development books, and children's literature. Each book is a new journey of discovery, expanding knowledge and enriching the soul of the reade Our website is not just a platform for buying books, but a bridge connecting readers to the timeless values of culture and wisdom. With an elegant, user-friendly interface and an intelligent search system, we are committed to providing a quick and convenient shopping experience. Additionally, our special promotions and home delivery services ensure that you save time and fully enjoy the joy of reading. Let us accompany you on the journey of exploring knowledge and personal growth! textbookfull.com