Red Hat Enterprise Linux Troubleshooting Guide Cane B
- 1. Red Hat Enterprise Linux Troubleshooting Guide Cane B download https://ebookbell.com/product/red-hat-enterprise-linux- troubleshooting-guide-cane-b-6724310 Explore and download more ebooks at ebookbell.com
- 2. Here are some recommended products that we believe you will be interested in. You can click the link to download. Red Hat Enterprise Linux Troubleshooting Guide Cane B https://ebookbell.com/product/red-hat-enterprise-linux- troubleshooting-guide-cane-b-6723446 Red Hat Enterprise Linux 9 Administration 2nd Edition Pablo Iranzo Gmez https://ebookbell.com/product/red-hat-enterprise- linux-9-administration-2nd-edition-pablo-iranzo-gmez-47290428 Red Hat Enterprise Linux 5 Administration Unleashed 1st Edition Tammy Fox https://ebookbell.com/product/red-hat-enterprise- linux-5-administration-unleashed-1st-edition-tammy-fox-2094116 Red Hat Enterprise Linux 8 Administration Master Linux Administration Skills And Prepare For The Rhcsa Certification Exam Miguel Prez Colino https://ebookbell.com/product/red-hat-enterprise- linux-8-administration-master-linux-administration-skills-and-prepare- for-the-rhcsa-certification-exam-miguel-prez-colino-36306514
- 3. Red Hat Enterprise Linux 6 Administration Real World Skills For Red Hat Administrators 1st Edition Sander Van Vugt https://ebookbell.com/product/red-hat-enterprise- linux-6-administration-real-world-skills-for-red-hat- administrators-1st-edition-sander-van-vugt-4315804 Red Hat Enterprise Linux 4 For Dummies Terry Collings https://ebookbell.com/product/red-hat-enterprise-linux-4-for-dummies- terry-collings-919766 Red Hat Enterprise Linux 8 Essentials Neil Smyth https://ebookbell.com/product/red-hat-enterprise-linux-8-essentials- neil-smyth-10969354 Red Hat Enterprise Linux Server Cookbook Leemans William https://ebookbell.com/product/red-hat-enterprise-linux-server- cookbook-leemans-william-11860776 Red Hat Enterprise Linux 7 Desktops And Administration Richard Petersen https://ebookbell.com/product/red-hat-enterprise-linux-7-desktops-and- administration-richard-petersen-60571652
- 6. Red Hat Enterprise Linux Troubleshooting Guide
- 7. Table of Contents Red Hat Enterprise Linux Troubleshooting Guide Credits About the Author About the Reviewers www.PacktPub.com Support files, eBooks, discount offers, and more Why subscribe? Free access for Packt account holders Preface What this book covers What you need for this book Who this book is for Conventions Reader feedback Customer support Downloading the example code Errata Piracy Questions 1. Troubleshooting Best Practices Styles of troubleshooting The Data Collector The Educated Guesser The Adaptor Choosing the appropriate style Troubleshooting steps Understanding the problem statement Asking questions Tickets Humans Attempting to duplicate the issue Running investigatory commands Establishing a hypothesis
- 8. Putting together patterns Is this something that I've encountered before? Trial and error Start by creating a backup Getting help Books Team Wikis or Runbooks Google Man pages Reading a man page Name Synopsis Description Examples Additional sections Info documentation Referencing more than commands Installing man pages Red Hat kernel docs People Following up Documentation Root cause analysis The anatomy of a good RCA The problem as it was reported The actual root cause of the problem A timeline of events and actions taken Any key data points to validate the root cause A plan of action to prevent the incident from reoccurring Establishing a root cause Sometimes you must sacrifice a root cause analysis Understanding your environment Summary 2. Troubleshooting Commands and Sources of Useful Information Finding useful information Log files The default location
- 9. Common log files Finding logs that are not in the default location Checking syslog configuration Checking the application's configuration Other examples Using the find command Configuration files Default system configuration directory Finding configuration files Using the rpm command Using the find command The proc filesystem Troubleshooting commands Command-line basics Command flags The piping command output Gathering general information w – show who is logged on and what they are doing rpm – RPM package manager Listing all packages installed Listing all files deployed by a package Using package verification df – report file system space usage Showing available inodes free – display memory utilization What is free, is not always free The /proc/meminfo file ps – report a snapshot of current running processes Printing every process in long format Printing a specific user's processes Printing a process by process ID Printing processes with performance information Networking ip – show and manipulate network settings Show IP address configuration for a specific device Show routing configuration Show network statistics for a specified device
- 10. netstat – network statistics Printing network connections Printing all ports listening for tcp connections Delay Performance iotop – a simple top-like I/O monitor iostat – report I/O and CPU statistics Manipulating the output vmstat – report virtual memory statistics sar – collect, report, or save system activity information Using the sar command Summary 3. Troubleshooting a Web Application A small back story The reported issue Data gathering Asking questions Duplicating the issue Understanding the environment Where is this blog hosted? Lookup IPs with nslookup What about ping, dig, or other tools? Ok, it's within our environment; now what? What services are installed and running? Validate the web server Validating the database service Validating PHP A summary of installed and running services Looking for error messages Apache logs Finding the location of Apache's logs Reviewing the logs Using curl to call our web application Requesting a non-PHP page Reviewing generated log entries What we learned from httpd logs Verifying the database
- 11. Verifying the WordPress database Finding the installation path for WordPress Checking the default configuration Finding the database credentials Connecting as the WordPress user Validating the database structure What we learned from the database validation Establishing a hypothesis Resolving the issue Understanding database data files Finding the MariaDB data folder Resolving data file issues Validating Final validation Summary 4. Troubleshooting Performance Issues Performance issues It's slow Performance Application CPU Top – a single command to look at everything What does this output tell us about our issue? Individual processes from top Determining the number of CPUs available Threads and Cores lscpu – Another way to look at CPU info ps – Drill down deeper on individual processes with ps Using ps to determine process CPU utilization Putting it all together A quick look with top Digging deeper with ps Memory free – Looking at free and used memory Linux memory buffers and caches Swapped memory What free tells us about our system
- 12. Checking for oomkill ps - Checking individual processes memory utilization vmstat – Monitoring memory allocation and swapping Putting it all together Taking a look at the system's memory utilization with free Watch what is happening with vmstat Finding the processes that utilize the most memory with ps Disk iostat – CPU and device input/output statistics CPU details Reviewing I/O statistics Identifying devices Who is writing to these devices? ps – Using ps to identify processes utilizing I/O iotop – A top top-like command for disk i/o Putting it all together Using iostat to determine whether there is a I/O bandwidth problem Using iotop to determine which processes are consuming disk bandwidth Using ps to understand more about processes Network ifstat – Review interface statistics Quick review of what we have identified Comparing historical metrics sar – System activity report CPU Memory Disk Network Review what we learned by comparing historical statistics Summary 5. Network Troubleshooting Database connectivity issues Data collection Duplicating the issue Finding the database server Testing connectivity
- 13. Telnet from blog.example.com Telnet from our laptop Ping Troubleshooting DNS Checking DNS with dig Looking up DNS with nslookup What did dig and nslookup tell us? A bit about /etc/hosts DNS summary Pinging from another location Testing port connectivity with cURL Showing current network connections with netstat Using netstat to watch for new connections Breakdown of netstat states Capturing network traffic with tcpdump Taking a look at the server's network interfaces What is a network interface? Viewing device configuration Specifying the interface with tcpdump Reading the captured data A quick primer on TCP Types of TCP packet Reviewing collected data Taking a look on the other side Identifying the network configuration Testing connectivity from db.example.com Looking for connections with netstat Tracing network connections with tcpdump Routing Viewing the routing table The default route Utilizing IP to show the routing table Looking for routing misconfigurations More specific routes win Hypothesis Trial and error Removing the invalid route
- 14. Configuration files Summary 6. Diagnosing and Correcting Firewall Issues Diagnosing firewalls Déjà vu Troubleshooting from historic issues Basic troubleshooting Validating the MariaDB service Troubleshooting with tcpdump Understanding ICMP Understanding connection rejections A quick summary of what you have learned so far Managing the Linux firewall with iptables Verify that iptables is running Show iptables rules being enforced Understanding iptables rules Ordering matters Default policies Breaking down the iptables rules Putting the rules together Viewing iptables counters Correcting the iptables rule ordering How iptables rules are applied Modifying iptables rules Testing our changes Summary 7. Filesystem Errors and Recovery Diagnosing filesystem errors Read-only filesystems Using the mount command to list mounted filesystems A mounted filesystem Using fdisk to list available partitions Back to troubleshooting NFS – Network Filesystem NFS and network connectivity Using the showmount command NFS server configuration
- 15. Exploring /etc/exports Identifying the current exports Testing NFS from another client Making mounts permanent Unmounting the /mnt filesystem Troubleshooting the NFS server, again Finding the NFS log messages Reading /var/log/messages Read-only filesystems Identifying disk issues Recovering the filesystem Unmounting the filesystem Filesystem checks with fsck The fsck and xfs filesystems How do these tools repair a filesystem? Mounting the filesystem Repairing the other filesystems Recovering the / (root) filesystem Validation Summary 8. Hardware Troubleshooting Starting with a log entry What is a RAID? RAID 0 – striping RAID 1 – mirroring RAID 5 – striping with distributed parity RAID 6 – striping with double distributed parity RAID 10 – mirrored and striped Back to troubleshooting our RAID How RAID recovery works Checking the current RAID status Summarizing the key information Looking at md status with /proc/mdstat Using both /proc/mdstat and mdadm Identifying a bigger issue Understanding /dev More than just disk drives
- 16. Device messages with dmesg Summarizing what dmesg has provided Using mdadm to examine the superblock Checking /dev/sdb2 What we have learned so far Re-adding the drives to the arrays Adding a new disk device When disks are not added cleanly Another way to watch the rebuild status Summary 9. Using System Tools to Troubleshoot Applications Open source versus home-grown applications When the application won't start Exit codes Is the script failing, or the application? A wealth of information in the configuration file Watching log files during startup Checking whether the application is already running Checking open files Understanding file descriptors Getting back to the lsof output Using lsof to check whether we have a previously running process Finding out more about the application Tracing an application with strace What is a system call? Using strace to identify why the application will not start Resolving the conflict Summary 10. Understanding Linux User and Kernel Limits A reported issue Why is the job failing? Background questions Is the cron job even running? User crontabs Understanding user limits The file size limit The max user processes limit
- 17. The open files limit Changing user limits The limits.conf file Future proofing the scheduled job Running the job again Kernel tunables Finding the kernel parameter for open files Changing kernel tunables Permanently changing a tunable Temporarily changing a tunable Running the job one last time A look back Too many open files A bit of clean up Summary 11. Recovering from Common Failures The reported problem Is Apache really down? Why is it down? What else was happening at that time? Searching the messages log Breaking down this useful one-liner The cut command The sort command The uniq command Tying it all together What happens when a Linux system runs out of memory? Minimum free memory A quick recap How oom-kill works Adjusting the oom score Determining whether our process was killed by oom-kill Why did the system run out of memory? Resolving the issue in the long-term and short-term Long-term resolution Short-term resolution Summary
- 18. 12. Root Cause Analysis of an Unexpected Reboot A late night alert Identifying the issue Did someone reboot this server? What do the logs tell us? Learning about new processes and services What caused the high load average? What are the run queue and load average? Load average Investigating the filesystem being full The du command Why wasn't the queue directory processed? A checkpoint on what you learned Sometimes you cannot prove everything Preventing reoccurrence Immediate action Long-term actions A sample Root Cause Analysis Problem summary Problem details Root cause Action plan Further actions to be taken Summary Index
- 19. Red Hat Enterprise Linux Troubleshooting Guide
- 20. Red Hat Enterprise Linux Troubleshooting Guide Copyright © 2015 Packt Publishing All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews. Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the author, nor Packt Publishing, and its dealers and distributors will be held liable for any damages caused or alleged to be caused directly or indirectly by this book. Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information. First published: October 2015 Production reference: 1131015 Published by Packt Publishing Ltd. Livery Place 35 Livery Street Birmingham B3 2PB, UK. ISBN 978-1-78528-355-0 www.packtpub.com
- 21. Credits Author Benjamin Cane Reviewers Brian C Galura Deepak G Kulkarni Warren Myers Siddhesh Poyarekar Commissioning Editor Nadeem Baghban Acquisition Editor Reshma Raman Content Development Editor Mamata Walkar Technical Editor Ryan Kochery Copy Editors Tani Kothari Merilyn Pereira Project Coordinator
- 22. Shipra Chawhan Proofreader Safis Editing Indexer Tejal Daruwale Soni Production Coordinator Aparna Bhagat Cover Work Aparna Bhagat
- 23. About the Author Benjamin Cane has nearly 10 years of experience in Linux systems administration. His first systems administration role was in 2006. At that time, he worked for a web hosting company supporting thousands of FreeBSD and Linux systems. Afterwards, he joined a managed services company that specialized in managing mission-critical systems. There, he worked his way to the position of a lead systems engineer, providing 24x7 support for highly critical enterprise systems that ran Red Hat Enterprise Linux. Now, Benjamin is a systems architect. He focuses on building High and Continuous Availability environments within the financial services industry. He is also currently a Red Hat Certified Engineer and Certified Ethical Hacker. With his experience in mission-critical environments, he has learned to identify and troubleshoot very complex issues quickly, because often these environments have a low tolerance for downtime. Being able to identify the root causes of very complex systems issues quickly is a skill that requires extensive knowledge of Linux and troubleshooting best practices. In addition to this book, Benjamin writes about Linux systems administration and DevOps topics on his blog at http://bencane.com. He is also the project founder for Runbook (https://github.com/Runbook/runbook), an open source application designed to monitor and automatically resolve infrastructure and application issues.
- 24. About the Reviewers Brian C Galura spent his childhood tinkering with subjects such as Java programming and Linux; his professional experience started with VoIP testing at 3Com in suburban Chicago. He then spent two years studying computer engineering at Purdue University before leaving to pursue freelance consulting in Los Angeles. Following several years of freelancing, he developed his expertise in enterprise infrastructure and cloud computing by working for a variety of start-ups and large corporations. Later, he completed a bachelor's in IT while working at Citrix. Brian is currently working on Citrix's cloud engineering and systems architecture team in Santa Barbara, California. Deepak G Kulkarni has around 10 years of experience in software engineering including nine years of strong experience in product development using C/C++/Java. He has over four years of experience in JUnit, Core JAVA, Web Services, XML, Spring, Spring MVC, and Hibernate. He also has extensive software development experience using C/C++ on UNIX (HP-UX) and Sun-Solaris, shell scripting, and STL (Standard Template Library) and cross-platform development. Deepak also has over two years of experience in the Mac OS using Objective C/C++, Cocoa framework, Xcode, and MetroWorks IDE. Deepak has a working knowledge of Perl/SNMP/TCP/IP, OS Internals, web services, XML, XSD, database (Oracle)/IPC (inter process communication)/system calls/Pro*C. He has been exposed to UML / use case diagrams / class diagrams / sequence diagrams / activity diagrams / state diagrams, IBM Rational Rose, and Design Patterns (GoF Patterns) / Design Principles. He has hands-on experience with Ant, Maven, JUnit, JProfiler, JProbe, JSON, Servlets, Python, and Ruby on Rails, as well as experience in remote service calls and client server programming. Warren Myers has been a professional data center and cloud automation architect for over eight years, with exposure to dozens of platforms, scores of customers, and hundreds of interesting problems.
- 25. He was a reviewer on Raspberry Pi Server Essentials and wrote the freely- available e-book, Debugging and Supporting Software Systems (http://cnx.org/contents/aa87c3d5-e350-458e-a948- 8cd1a9bf8e36@2.4:1/Debugging_and_Supporting_Softw). Siddhesh Poyarekar has been a free and open source software programmer for over seven years and has worked on a variety of projects beginning with writing his own dialer program for an ISP that did not have a UI-based dialer for Linux. He spent a number of years troubleshooting problems in various domains from the Linux desktop, shell, and the kernel to the core system runtime on Linux systems, that is, the GNU C Library for which he is now a maintainer.
- 26. www.PacktPub.com
- 27. Support files, eBooks, discount offers, and more For support files and downloads related to your book, please visit www.PacktPub.com. Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.PacktPub.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at <service@packtpub.com> for more details. At www.PacktPub.com, you can also read a collection of free technical articles, sign up for a range of free newsletters and receive exclusive discounts and offers on Packt books and eBooks. https://www2.packtpub.com/books/subscription/packtlib Do you need instant solutions to your IT questions? PacktLib is Packt's online digital book library. Here, you can search, access, and read Packt's entire library of books.
- 28. Why subscribe? Fully searchable across every book published by Packt Copy and paste, print, and bookmark content On demand and accessible via a web browser
- 29. Free access for Packt account holders If you have an account with Packt at www.PacktPub.com, you can use this to access PacktLib today and view 9 entirely free books. Simply use your login credentials for immediate access. This book is dedicated to my wife Ruby and our two sons Ethan and Jacob. Ruby, without your love and support I would have never been able to write this book. Thank you for putting up with the long hours and sleepless nights while trying to complete this.
- 30. Preface Red Hat Enterprise Linux is a widely popular Linux distribution that is used in everything from Cloud to enterprise mainframe computers. If you include downstream distributions such as CentOS, the adoption of the Red Hat Enterprise Linux distribution is even greater. As with most things, there is always someone responsible for resolving issues with all of these various systems running Red Hat Enterprise Linux. Red Hat Enterprise Linux Troubleshooting Guide is written to provide basic to advanced troubleshooting practices and commands for Linux systems, with these troubleshooting techniques specifically focused on systems running Red Hat Enterprise Linux. This book is designed to provide you with steps and the knowledge required to remedy a wide variety of scenarios. The examples in this book use real-world issues with real-world resolutions. While the examples in this book are situational, this book can also be used as a reference for Linux-related topics and commands. They provide the reader with the ability to reference both troubleshooting steps and specific commands to resolve complex issues.
- 31. What this book covers Chapter 1, Troubleshooting Best Practices, covers the troubleshooting process at a high level. By equating the troubleshooting process with the scientific method, this book will explain how to break down a problem to identify the root cause, no matter how complicated the problem. Chapter 2, Troubleshooting Commands and Sources of Useful Information, provides the reader with a simple introduction to common locations of useful information. It will also provide a reference for fundamental Linux commands that can be used for troubleshooting many types of issues. Chapter 3, Troubleshooting a Web Application, takes the process learned in Chapter 1 and the commands learned in Chapter 2 to work through a complicated problem. The problem outlined in this chapter is "By example" meaning that the flow of this chapter is designed to walk you through the entire troubleshooting process, from end to end. Chapter 4, Troubleshooting Performance Issues, deals with performance issues and some of the most complicated problems to troubleshoot. Often, the complications are compounded by the perception of users versus expected levels of performance. In this chapter, the tools and information discussed in Chapter 2 will, once again, be used to resolve a real-world performance problem. Chapter 5, Network Troubleshooting, talks about networking being a critical component of any modern day system. This chapter will cover the core commands necessary for the configuration and diagnostics of Linux networking. Chapter 6, Diagnosing and Correcting Firewall Issues, covers the complex nature of Linux firewalls, in a continuation of Chapter 5. This chapter will introduce and highlight commands and techniques necessary to troubleshoot Linux software firewalls. Chapter 7, Filesystem Errors and Recovery, teaches you that being able to recover a filesystem could mean the difference between losing and retaining data. This chapter will introduce some core Linux filesystem concepts and will demonstrate how to recover a read-only filesystem.
- 32. Chapter 8, Hardware Troubleshooting, starts to touch on the process of troubleshooting hardware issues. This chapter will walk you through the restoration of a failed hard drive. Chapter 9, Using System Tools to Troubleshoot Applications, explores how often a system administrator's role is not only to troubleshoot OS issues but also application issues. This chapter will show you how to utilize common system tools to identify the root cause of an application issue. Chapter 10, Understanding Linux User and Kernel Limits, demonstrates that Red Hat Enterprise Linux has many components in place to prevent users from overloading the system. This chapter will explore these components and explain how to modify them to allow legitimate resource utilization. Chapter 11, Recovering from Common Failures, walks you through troubleshooting out-of-memory conditions. This scenario is very common in heavily utilized environments and can be difficult to troubleshoot. This chapter will cover not only how to troubleshoot this issue, but also why the issue occurred. Chapter 12, Root Cause Analysis of an Unexpected Reboot, puts the troubleshooting process and commands learned in the previous chapters to the test. This chapter walks you through performing a Root Cause Analysis on a server that has unexpectedly rebooted.
- 33. What you need for this book Although this book can be standalone, readers will benefit greatly from having a system with Red Hat Enterprise Linux release 7, with the operating system available. You will more effectively learn the commands and resources discussed in this book when you have the ability to execute them on a test system. While it is possible to use many of the commands, processes, and resources covered in this book with other Linux distributions, it is highly recommended to utilize a Red Hat downstream distribution such as CentOS 7 if Red Hat Enterprise Linux 7 is not available to the reader.
- 34. Who this book is for If you are a competent RHEL administrator or consultant with a desire to increase your troubleshooting skills and your knowledge of Red Hat Enterprise Linux, then this book is perfect for you. A good knowledge level and understanding of basic Linux commands are expected.
- 35. Conventions In this book, you will find a number of text styles that distinguish between different kinds of information. Here are some examples of these styles and an explanation of their meaning. Code words in text, database table names, folder names, filenames, file extensions, pathnames, dummy URLs, user input, and Twitter handles are shown as follows: "Within reason, it is not required to include every cd or ls command executed." When we wish to draw your attention to a particular part of a code block, the relevant lines or items are set in bold: 192.168.33.12 > 192.168.33.11: ICMP host 192.168.33.12 unreachable - admin prohibited, length 68 Any command-line input or output is written as follows: # yum install man-pages New terms and important words are shown in bold. Words that you see on the screen, for example, in menus or dialog boxes, appear in the text like this: "we will see a message on our screen that says still here?." Note Warnings or important notes appear in a box like this. Tip Tips and tricks appear like this.
- 36. Reader feedback Feedback from our readers is always welcome. Let us know what you think about this book—what you liked or disliked. Reader feedback is important for us as it helps us develop titles that you will really get the most out of. To send us general feedback, simply e-mail <feedback@packtpub.com>, and mention the book's title in the subject of your message. If there is a topic that you have expertise in and you are interested in either writing or contributing to a book, see our author guide at www.packtpub.com/authors.
- 37. Customer support Now that you are the proud owner of a Packt book, we have a number of things to help you to get the most from your purchase.
- 38. Downloading the example code You can download the example code files from your account at http://www.packtpub.com for all the Packt Publishing books you have purchased. If you purchased this book elsewhere, you can visit http://www.packtpub.com/support and register to have the files e-mailed directly to you.
- 39. Errata Although we have taken every care to ensure the accuracy of our content, mistakes do happen. If you find a mistake in one of our books—maybe a mistake in the text or the code—we would be grateful if you could report this to us. By doing so, you can save other readers from frustration and help us improve subsequent versions of this book. If you find any errata, please report them by visiting http://www.packtpub.com/submit-errata, selecting your book, clicking on the Errata Submission Form link, and entering the details of your errata. Once your errata are verified, your submission will be accepted and the errata will be uploaded to our website or added to any list of existing errata under the Errata section of that title. To view the previously submitted errata, go to https://www.packtpub.com/books/content/support and enter the name of the book in the search field. The required information will appear under the Errata section.
- 40. Piracy Piracy of copyrighted material on the Internet is an ongoing problem across all media. At Packt, we take the protection of our copyright and licenses very seriously. If you come across any illegal copies of our works in any form on the Internet, please provide us with the location address or website name immediately so that we can pursue a remedy. Please contact us at <copyright@packtpub.com> with a link to the suspected pirated material. We appreciate your help in protecting our authors and our ability to bring you valuable content.
- 41. Questions If you have a problem with any aspect of this book, you can contact us at <questions@packtpub.com>, and we will do our best to address the problem.
- 42. Chapter 1. Troubleshooting Best Practices This chapter, which happens to be the first chapter, is probably the most important and least technical. Most chapters in this book cover specific issues and the commands necessary to troubleshoot those issues. This chapter, however, will cover some troubleshooting best practices that can be applied to any issue. You can think of this chapter as the principles behind the practices being applied.
- 43. Styles of troubleshooting Before covering the best practices of troubleshooting, it is important to understand the different styles of troubleshooting. In my experience, I have found that people tend to use one of three styles of troubleshooting, which are as follows: The Data Collector The Educated Guesser The Adaptor Each of these styles have their own strengths and weaknesses. Let's have a look at the characteristics of these styles.
- 44. The Data Collector I like to call the first style of troubleshooting, the Data Collector. The Data Collector is someone who generally utilizes a systematic approach to solve issues. The systematic troubleshooting approach is generally characterized as follows: Asking specific questions to parties reporting issues, expecting detailed answers Running commands to identify system performance for most issues Running through a predefined set of troubleshooting steps before stepping into action The strength of this style is that it is effective, no matter what level of engineer or administrator is using it. By going through issues systematically, collecting each data point, and understanding the results before executing any resolution, the Data Collector is able to resolve issues that they might not necessarily be familiar with. The weakness of this style is that the data collection is not usually the fastest method to resolve issues. Depending on the issue, collecting data can take a long time and some of that data might not be necessary to find the resolution.
- 45. The Educated Guesser I like to call the second style of troubleshooting, the Educated Guesser. The Educated Guesser is someone who generally utilizes an intuitive approach to solve issues. The intuitive approach is generally characterized by the following: Identifying the cause of issues with minimal information Running a few commands before resolving the issue Utilizing previous experience to identify root cause The strength of this style of troubleshooting is that it allows you to come up with resolutions sooner. When confronted with an issue, this type of troubleshooter tends to pull from experience and requires minimal information to find a resolution. The weakness of this style is that it relies heavily on experience, and thus requires time before being effective. When focusing on resolution, this troubleshooter might also attempt multiple actions to resolve the issue, which can make it seem like the Educated Guesser does not fully understand the issue at hand.
- 46. The Adaptor There is a third and often-overlooked style of troubleshooting; this style utilizes both the systematic and intuitive styles. I like to call this style the Adaptor. The Adaptor has a personality which enables it to switch between systematic and intuitive troubleshooting styles. This combined style is often faster than the Data Collector style and is more detail oriented than the Educated Guesser style. This is because they are able to apply the troubleshooting style appropriate for the task at hand.
- 47. Choosing the appropriate style While it is easy to say that one method is better than the other, the fact of the matter is that picking the appropriate troubleshooting style depends greatly on the person. It is important to understand which troubleshooting style best fits your own personality. By understanding which style fits you better, you can learn and use techniques that fit that style. You can also learn and adopt techniques from other styles to apply troubleshooting steps that you would normally overlook. This book will show both the Data Collector and Educated Guesser styles of troubleshooting, and periodically highlighting which personality style the steps best fit.
- 48. Troubleshooting steps Troubleshooting is a process that is both rigid and flexible. The rigidity of the troubleshooting process is based on the fact that there are basic steps to be followed. In this way, I like to equate the troubleshooting process to the scientific method, where the scientific method has a specific list of steps that must be followed. The flexibility of the troubleshooting process is that these steps can be followed in any order that makes sense. Unlike the scientific method, the troubleshooting process often has the goal of resolving the issue quickly. Sometimes, in order to resolve an issue quickly, you might need to skip a step or execute them out of order. For example, with the troubleshooting process, you might need to resolve the immediate issue, and then identify the root cause of that issue. The following list has five steps that make up the troubleshooting process. Each of these steps could also include several sub-tasks, which may or may not be relevant to the issue. It is important to follow these steps with a grain of salt, as not every issue can be placed into the same bucket. The following steps are meant to be used as a best practice but, as with all things, it should be adapted to the issue at hand: 1. Understanding the problem statement. 2. Establishing a hypothesis. 3. Trial and error. 4. Getting help. 5. Documentation.
- 49. Understanding the problem statement With the scientific method, the first step is to establish a problem statement, which is another way of saying: to identify and understand the goal of the experiment. With the troubleshooting process, the first step is to understand the problem being reported. The better we understand an issue, the easier it is to resolve the issue. There are a number of tasks we can perform that will help us understand issues better. This first step is where a Data Collector's personality stands out. Data Collectors, by nature, will gather as much data as they can before moving on to the next step, whereas, the Educated Guessers generally tend to run through this step quickly and then move on to the next step, which can sometimes cause critical pieces of information to be missed. Adaptors tend to understand which data collecting steps are necessary and which ones are not. This allows them to collect data as a Data Collector would, but without spending time gathering data that does not add value to the issue at hand. The sub-task in this troubleshooting step is asking the right questions. Asking questions Whether via human or automated processes such as a ticket system, the reporter of the issue is often a great source of information. Tickets When they receive a ticket, the Educated Guesser personality will often read the heading of the ticket, make an assumption of the issue and move to the next stage of understanding the issue. The Data Collector personality will generally open the ticket and read the full details of the ticket. While it depends on the ticketing and monitoring system, in general, there can be useful information within a ticket. Unless the issue is a common issue and you are able to understand all that you know from the header, it is generally a good idea to read the ticket description. Even small amounts of information might help with particularly tricky issues.
- 50. Humans Gathering additional information from humans, however, can be inconsistent. This varies greatly depending on the environment being supported. In some environments, the person reporting an issue can provide all of the details required to resolve the issue. In other environments, they might not understand the issue and simply explain the symptoms. No matter what troubleshooting style fits your personality best, being able to get important information from the person reporting the issue is an important skill. Intuitive problem solvers such as the Educated Guesser or Adaptor tend to find this process easier as compared to Data Collector personalities, not because these personalities are necessarily better at obtaining details from people but rather because they are able to identify patterns with less information. Data Collectors, however, can get the information they need from those reporting the issue if they are prepared to ask troubleshooting questions. Note Don't be afraid to ask obvious questions My first technical job was in a webhosting technical support call center. There I often received calls from users who did not want to perform the basic troubleshooting steps and simply wanted the issue escalated. These users simply felt that they had performed all of the troubleshooting steps themselves and had found an issue beyond first level support. While sometimes this was true, more often, the issue was something basic that they had overlooked. In that role, I quickly learned that even if the user is reluctant to answer basic or obvious questions, at the end of the day, they simply want their issue resolved. If that meant going through repetitive steps, that was ok, as long as the issue is resolved. Even today, as I am now the escalation point for senior engineers, I find that many times engineers (even with years of troubleshooting experience under their belt) overlook simple basic steps. Asking simple questions that might seem basic are sometimes a great time saver; so don't be afraid to ask them.
- 51. Attempting to duplicate the issue One of the best ways to gather information and understand an issue is to experience it. When an issue is reported, it is best to duplicate the issue. While users can be a source of a lot of information, they are not always the most reliable; oftentimes a user might experience an error and overlook it or simply forget to relay the error when reporting the issue. Often, one of the first questions I will ask a user is how to recreate the issue. If the user is able to provide this information, I will be able to see any errors and often identify the resolution of the issue faster. Note Sometimes duplicating the issue is not possible While it is always best to duplicate the issue, it is not always possible. Every day, I work with many teams; sometimes, those teams are within the company but many times they are external vendors. Every so often during a critical issue, I will see someone make a blanket statement such as "If we can't duplicate it, we cannot troubleshoot it." While it is true that duplicating an issue is sometimes the only way to find the root cause, I often hear this statement abused. Duplicating an issue should be viewed like a tool; it is simply one of many tools in your troubleshooting tool belt. If it is not available, then you simply have to make do with another tool. There is a significant difference between not being able to find a resolution and not attempting to find a resolution due to the inability to duplicate an issue. The latter is not only unhelpful, but also unprofessional. Running investigatory commands Most likely, you are reading this book to learn techniques and commands to troubleshoot Red Hat Enterprise Linux systems. The third sub-task in understanding the problem statement is just that—running investigative commands to identify the cause of the issue. Before executing investigatory commands, however, it is important to know that the previous steps are in a
- 52. logical order. It is a best practice to first ask the user reporting an issue some basic details of the issue, then after obtaining enough information, duplicate the issue. Once the issue has been duplicated, the next logical step is to run the necessary commands to troubleshoot and investigate the cause of the issue. It is very common to find yourself returning to previous steps during the troubleshooting process. After you have identified some key errors, you might find that you must ask the original reporter for additional information. When troubleshooting, do not be afraid to take a few steps backwards in order to gain clarity of the issue at hand.
- 53. Establishing a hypothesis With the scientific method, once a problem statement has been formulated it is then time to establish a hypothesis. With the troubleshooting process, after you have identified the issue, gathered the information about the issue such as errors, system current state, and so on, it is also time to establish what you believe caused or is causing the issue. Some issues, however, might not require much of a hypothesis. It is common that errors in log files or the systems current state might answer why the issue occurred. In such scenarios, you can simply resolve the issue and move on to the Documentation step. For issues that are not cut and dry, you will need to put together a hypothesis of the root cause. This is necessary as the next step after forming a hypothesis is attempting to resolve the issue. It is difficult to resolve an issue if you do not have at least, a theory of the root cause. Here are a few techniques that can be used to help form a hypothesis. Putting together patterns While performing data collection during the previous steps, you might start to see patterns. Patterns can be something as simple as similar log entries across multiple services, the type of failure that occurred (such as, multiple services going offline), or even a reoccurring spike in system resource utilization. These patterns can be used to formulate a theory of the issue. To drive the point home, let's go through a real-world scenario. You are managing a server that both runs a web application and receives e-mails. You have a monitoring system that detected an error with the web service and created a ticket. While investigating the ticket, you also receive a call from an e- mail user stating they are getting e-mail bounce backs. When you ask the user to read the error to you they mention No space left on device.
- 54. Let's break down this scenario: A ticket from our monitoring solution has told us Apache is down We have also received reports from e-mail users with errors indicative of a file system being full Could all of this mean that Apache is down because the file system is full? Possibly. Should we investigate it? Absolutely! Is this something that I've encountered before? The above breakdown leads into the next technique for forming a hypothesis. It might sound simple but is often forgotten. "Have I seen something like this before?" With the previous scenario, the error reported from the e-mail bounce back was one that generally indicated that a file system was full. How do we know this? Well, simple, we have seen it before. Maybe we have seen that same error with e-mail bounce backs or maybe we have seen the error from other services. The point is, the error is familiar and the error generally means one thing. Remembering common errors can be extremely useful for the intuitive types such as the Educated Guesser and Adaptor; this is something they tend to naturally perform. For the Data Collector, a handy trick would be to keep a reference table of common errors handy. Tip From my experience, most Data Collectors tend to keep a set of notes that contain things such as common commands or steps for procedures. Adding common errors and the meaning behind those errors are a great way for systematic thinkers such as Data Collectors to establish a hypothesis faster. Overall, establishing a hypothesis is important for all types of troubleshooters. This is the area where the intuitive thinkers such as Educated Guessers and Adaptors excel. Generally, those types of troubleshooters will form a hypothesis sooner, even if sometimes those hypotheses are not always correct.
- 55. Exploring the Variety of Random Documents with Different Content
- 56. “Red wins,” quietly announced the croupier. “You observe how easy it is, I presume,” said Wiley, smiling. “I can’t help it. It’s as natural as breathing.” Frank saw that it was useless to argue with the sailor, and so he and Hodge left him still playing, while they strolled through the place. There was a dance hall connected, which provided amusement for them a while, although neither danced. Barely half an hour passed before Frank, who was somewhat anxious about Wiley, returned to note how Wiley was getting along. Luck had turned, and Wiley was losing steadily. Still he continued to bet with the same harebrained carelessness, apparently perfectly confident that his bad luck could not keep up. “He will go broke within twenty minutes if he sticks to it, Frank,” said Hodge. Merry nodded. “That’s right,” he agreed; “but he won’t listen to advice. If we attempt to get him away, we will simply kick up a disturbance and find ourselves in a peck of trouble. Even if he should cash in now and quit ahead of the game, he’d come back to it and lose all he’s won. Therefore we may as well let him alone.” They did so, and Bart’s prophecy came true. The sailor’s reckless betting lowered his pile so that it seemed to melt like dew before the sun. Finally he seemed to resolve on a grand stroke, and he bet everything before him on the red. The little ball clicked and whirred in the whirling wheel. The spectators seemed breathless as they watched for the result of that plunge. Slower and slower grew the revolutions of the wheel. The ball spun around on its rim like a cork on the water. At length it dropped. “He wins!” panted an excited man.
- 57. “No—see!” exclaimed another. The ball had bobbed out of its pocket and spun on again. “Lost!” was the cry, as it finally settled and rested securely in a pocket. Wiley swallowed down a lump in his throat as the man behind the table raked in the wager. “Excuse me,” said the sailor, rising. “I hope you will pardon me while I go drown myself. Can any one direct me to a tub of tanglefoot?” As he left the table, knowing now that it would cause no disturbance, Frank grasped his arm and again advised him to leave the place. “I admit to you,” said Wiley, “that I was mistaken when I stated that I had a mash on Dame Fortune. I have discovered that it was her daughter, Miss Fortune. Leave me—leave me to my fate! I shall now attempt to lap up all the liquids in the place, and in the morning I’ll have a large aching head.” Frank insisted, however, and his command led Wiley reluctantly to permit them to escort him from the place. “I might read you a lecture on the evils of gambling, cap’n,” said Merry; “but I shall not do so to-night. It strikes me that you have learned your lesson.” “It is only one of many such lessons,” sighed the sailor. “By this time I should have them by heart, but somehow I seem to forget them. I wish to tell you a secret that I have held buried in my bosom these many years. It is this: “Somewhere about my machinery there is a screw loose. In vain I have sought to find it. I know it is there just as well as I know that I am Cap’n Wiley. Now, you are a perfect piece of machinery, with everything tight, and firm, and well oiled, and polished. As an example you are the real thing. Perhaps to-morrow I may conclude
- 58. to follow in your footsteps. Just tuck me in my little bed and leave me to dreamy slumber.” After being left in his room, however, Wiley did not remain long in bed. Knowing they would not suspect such a thing of him, he arose, and dressed, and returned to the gambling house. When morning came he was not only broke, but he had pawned everything of value in his possession and was practically destitute. “Well,” said Merry, having discovered the cap’n’s condition, “I presume now you will return with us to the Mazatzals?” “No use,” was the answer; “I shall stay here in Prescott. I have my eye on a good thing. Don’t worry about me.” It was useless to urge him, for he persisted in his determination to stay there. And so before leaving Frank made some final arrangements with him. “I have wired for my mail to be forwarded here, Wiley,” he said. “If anything of importance comes, anything marked to be delivered in haste, I wish you would see that it reaches me. Cannot you do so?” “Depend upon me, Frank,” assured the sailor. “I will not fail you in this. But before departing it seems to me that you should make arrangements that any such message be delivered into my hands.” “I will do so,” said Merry. “Now, see here, cap’n, I don’t like to leave you strapped in this town. At the same time, I don’t care to let you have money of mine to gamble with. If I provide you with some loose change, will you give me your word not to use it in gambling?” “Your generosity is almost ignoble!” exclaimed Wiley. “However, I accept it in the same manner that it is tendered. I give you my word.” “Well, that goes with me,” nodded Merry. “Before leaving I shall see that you are fixed with ready money.”
- 60. CHAPTER VII. A STARTLING TELEGRAM. Sunset in the Enchanted Valley. Below the little waterfall which plunged down into the fissure at the southern end of the valley Frank and Bart had toiled hard all through the day. Their sleeves were rolled up and their clothes mud-bespattered. There they had worked in the sandy soil near the stream, and there they had found the shining stuff for which they sought. Every panful was carefully washed in the stream, showing dull yellow grains in the bottom when the last particles remained. Not far away, on the level of the valley above them, set near the stream, was their tent. In front of it little Abe was building a fire and was seeking to prepare supper for them, knowing they would be ravenously hungry when they quit work for the night. At intervals the cripple hobbled to the brink of the fissure and looked down at them as they toiled. No one had troubled them since their return to the valley. No longer did the place seem enchanted or mysterious. All the mysteries were solved, and it lay sleeping and silent amid that vast mountainous solitude. “Well, Bart,” said Frank, as he dropped his spade, “it seems to me that the thing is done to our satisfaction. At the northern end of the valley we have found Clark’s quartz claim, and the specimens we have taken from it seem decidedly promising. Here we have located this placer, and we know from what we have washed out that it is
- 61. rich and will prove extremely valuable while it lasts. Now it’s up to us to register our claims and open them for operation in the proper manner. We ought to be satisfied.” “Satisfied!” exclaimed Bart. “You bet I am satisfied! What if I had remained in Boston, Merry? Why, I would be plugging away to-day on a poor paying job, with decidedly poor prospects ahead of me. It was a most fortunate thing for me when I decided to stick by you and come West.” Frank smiled. “It was lucky, Hodge,” he agreed. “But I don’t forget that you came without a selfish thought on your part. You came to help me in my fight against Milton Sukes. I am far better pleased for your sake than for my own that we have had this streak of luck. Let’s knock off for the night, old man. There’s no reason why we should stick to it longer.” As they were climbing from the fissure by the narrow and difficult path, little Abe came rushing excitedly to the brink above and called to them. “Come quick! Come quick!” he cried. “What’s the matter, Abe?” asked Frank, alarmed by the boy’s manner. “Somebody’s coming,” said the hunchback; “a man on a horse. He is coming right this way. He has seen the tent!” “We may have some trouble after all, Merry,” said Hodge. Ere they could reach the head of the path near the waterfall they plainly heard the thudding hoofs of the horse coming rapidly in that direction. When they had reached the level ground above they beheld the horseman approaching. It seemed that he observed them at the same time, for he suddenly waved his hat in the air and gave a yell.
- 62. “By Jove!” exclaimed Merry, “I know him! It is Wiley!” “Right you are!” agreed Hodge. “What the dickens could have brought him here at this time?” “Perhaps he has some message for me. You know I made arrangements with him to bring any message of importance.” The sailor drew up his horse as he approached. “Ahoy there, mates!” he cried. “At last I have struck port, although I’d begun to wonder if I’d ever find it. This confounded old valley has moved since I was here last. I thought I knew just where it was, but I have spent two whole days cruising around in search of it.” “Hello, cap’n!” said Frank. “You’re just in time for supper.” “Supper!” exclaimed the sailor. “Say it again! Supper! Why, I have been living on condensed air for the last twenty-four hours. Look at me! I am so thin and emaciated that I can’t cast a shadow. Hungry! Mates, a bootleg stew would be a culinary luxury to me. I will introduce ravage and devastation among your provisions. This morning I found an empty tomato can and another that once contained deviled ham, and I lunched off them. They were rather hard to digest, but they were better than nothing.” He sprang down from his horse, which betrayed evidence of hard usage. “How did you happen to come?” asked Merry. Wiley fumbled in his pocket and brought forth a telegram. “I believe I made arrangements to deliver anything of importance directed to you,” he said. “This dispatch arrived in Prescott, and I lost no time in starting to fulfill my compact.” Merry took the telegram and quickly tore it open. There was a look of anxiety on his face when he had read its contents.
- 63. “Anything serious the matter?” asked Hodge. “It’s a message from my brother, Dick,” answered Frank. “You know I wired him to address his letters to Prescott. He didn’t stop to send a letter. Instead he sent this telegram. You know Felicia Delores, Dick’s cousin, with whom he was brought up? The climate of the East did not agree with her, therefore I provided a home for her in San Diego, California, where she could attend school. Dick has learned that she is ill and in trouble. He wants me to go to her at once.” “What will you do?” asked Hodge. “I must go,” said Frank quietly. Frank mounted the steps of a modern residence, standing on a palm-lined street in San Diego, and rang the bell. He was compelled to ring twice more before the door was opened by a sleepy-looking Mongolian. “I wish to see Mr. Staples at once,” said Merry. “Is he home?” “Mistal Staple not home,” was the serene answer, as the Chinaman moved to close the door. Frank promptly blocked this movement with a foot and leg. “Don’t be so hasty,” he said sharply. “If Mr. Staples is not home, where can I find him?” “No tellee. Velly solly.” “Then I must see Mrs. Staples,” persisted Merry. “She velly sick. Velly solly. She can’t slee anyblody.” “Well, you take her my card,” directed Merry, as he took out a card- case and tendered his card to the yellow-skinned servant. “No take cald. She tellee me no bothal her. Go ’way. Come bimeby— to-mollow.”
- 64. “Now, look here, you son of the Flowery Kingdom,” exclaimed Merry, “I am going to see Mrs. Staples immediately, if she’s in condition to see anyone. If you don’t take her my card, you will simply compel me to intrude without being announced.” “Bold, blad man!” chattered the Chinaman, with growing fear. “I callee police; have you ’lested.” “You’re too thick-headed for the position you hold!” exasperatedly declared Merry. “Take my card to Mrs. Staples instantly, and she will see me as soon as she reads my name, Frank Merriwell, upon it.” “Flank Mellowell!” almost shouted the Celestial. “You Flank Mellowell? Clome light in, quickee! Mladam, she expectee you.” The door was flung open now, and Frank entered. “Well, you have come to your senses at last!” he said. “You no undelstand. Blad men velly thick. Blad men make velly glate tloubal. Little glil she glone; mladam she cly velly much, velly much!” “Hustle yourself!” ordered Frank. “Don’t stand there chattering like a monkey. Hurry up!” “Hully velly flast,” was the assurance, as the Mongolian turned and toddled away at a snail’s pace, leaving Frank in the reception room. A few moments later there was a rustle of skirts, and a middle-aged woman, whose face was pale and eyes red and who carried a handkerchief in her hand, came down the stairs and found him waiting. “Oh, Mr. Merriwell!” she exclaimed, the moment she saw him. “So it’s really you! So you have come! We didn’t know where to reach you, and so we wired your brother. He wired back that he had dispatched you and that he thought you would come without delay.” Her agitation and distress were apparent.
- 65. “Felicia,” questioned Frank huskily; “what of her?” “Oh, I can’t tell you—I can’t tell you!” choked the woman, placing the handkerchief to her eyes. “It’s so dreadful!” “Tell me, Mrs. Staples, at once,” said Frank, immediately cool and self-controlled. “Don’t waste time, please. What has happened to Felicia? Where is she?” “She’s gone!” came in a muffled voice from behind the handkerchief. “Gone—where?” The agitated woman shook her head. “No one knows. No one can tell! Oh, it’s a terrible thing, Mr. Merriwell!” “Where is Mr. Staples?” questioned Frank, thinking he might succeed far better in obtaining the facts from the woman’s husband. “That I don’t know. He is searching for her. He, too, has been gone several days. I heard from him once. He was then in Warner, away up in the mountains.” Merry saw that he must learn the truth from the woman. “Mrs. Staples,” he said, “please tell me everything in connection with this singular affair. It’s the only way that you can be of immediate assistance. You know I am quite in the dark, save for such information as I received from my brother’s telegram. It informed me that Felicia was in trouble and in danger. What sort of trouble or what sort of danger threatens her, I was not told. In order for me to do anything I must know the facts immediately.” “It was nearly a month ago,” said Mrs. Staples, “that we first discovered anything was wrong. Felicia had not been very well for some time. She’s so frail and delicate! It has been my custom each night before retiring to look in upon her to see if she was comfortable and all right. One night, as I entered her room, light in
- 66. hand, I was nearly frightened out of my senses to see a man standing near her bed. He saw me or heard me even before I saw him. Like a flash he whirled and sprang out of the window to the veranda roof, from which he easily escaped to the ground. “I obtained barely a glimpse of him, and I was so frightened at the time that I could not tell how he looked. Felicia seemed to be sleeping soundly at the time, and didn’t awake until I gave a cry that aroused her and the whole house as well. I never had a thought then that the man meant her harm. She was so innocent and helpless it seemed no one would dream of harming her. I took him for a burglar who had entered the house by the way of her window. After that we took pains to have her window opened only a short space, and tightly locked in that position, so that it could not be opened further from the outside without smashing it and alarming some one. I was thankful we had escaped so easily, and my husband felt sure there would be no further cause for worry. He said that, having been frightened off in such a manner, the burglar was not liable to return. “Somehow it seemed to me that Felicia was still more nervous and pale after that. She seemed worried about something, but whenever I questioned her she protested she was not. The doctor came to see her several times, but he could give her nothing that benefited her. I continued my practice of looking in at her each night before retiring. One night, a week later, after going to bed, something—I don’t know what—led me to rise again and go to her room. Outside her door I paused in astonishment, for I distinctly heard her voice, and she seemed to be in conversation with some one. I almost fancied I heard another voice, but was not certain about that. I pushed open the door and entered. Felicia was kneeling by her partly opened window, and she gave a great start when I came in so quickly. A moment later I fancied I heard a sound as of some one or something dropping from the roof upon the ground. “I was so astonished that I scarcely knew what to say. ‘Felicia!’ I exclaimed. ‘What were you doing at that window?’
- 67. “‘Oh, I was getting a breath of the cool night air,’ she answered. ‘With my window partly closed it is almost stuffy in here. Sometimes I can’t seem to breathe.’ “‘But I heard you talking, child,’ I declared. ‘Who were you talking to?’ “‘I talk to myself sometimes, auntie, you know,’ she said, in her innocent way. She always called me auntie. I confess, Mr. Merriwell, that I was completely deceived. This came all the more natural because Felicia was such a frank, open-hearted little thing, and I’d never known her to deceive me in the slightest. I decided that my imagination had led me to believe I heard another voice than her own, and also had caused me to fancy that some one had dropped from the roof of the veranda. After that, however, I was uneasy. And my uneasiness was increased by the fact that the child seemed to grow steadily worse instead of better. “Often I dreamed of her and of the man I had seen in her room. One night I dreamed that a terrible black shadow was hanging over her and had reached out huge clawlike hands to clutch her. That dream awoke me in the middle of the night, and I could not shake off the impression that some danger menaced her. With this feeling on me I slipped out of bed, lighted a candle, and again proceeded to her room. This time I was astonished once more to hear her talking as if in conversation with some one. But now I knew that, unless I was dreaming or bewitched, I also heard another voice than her own —that of a man. My bewilderment was so great that I forgot caution and flung her door wide open. The light of the candle showed her sitting up in bed, while leaning on the footboard was a dark-faced man with a black-pointed mustache. I screamed, and, in my excitement, dropped the candle, which was extinguished. I think I fainted, for Mr. Staples found me in a dazed condition just outside Felicia’s door. She was bending over me, but when I told her of the man I had seen and when she was questioned, she behaved in a most singular manner. Not a word would she answer. Had she denied everything I might have fancied it all a grewsome dream. I
- 68. might have fancied I’d walked in my sleep and dreamed of seeing a man there, for he was gone when my husband reached the spot. “She would deny nothing, however, and what convinced us beyond question that some one had been in her room was the fact that the window was standing wide open. After that we changed her room to another part of the house and watched her closely. Although we persisted in urging her to tell everything, not a word could we get from her. Then it was that Mr. Staples wired Richard, your brother. “Three days later Felicia disappeared. She vanished in the daytime, when every one supposed her to be safe in the house. No one saw her go out. She must have slipped out without being observed. Of course we notified the police as soon as we were sure she was gone, and the city was searched for her. Oh! it is a terrible thing, Mr. Merriwell; but she has not been found! Mr. Staples believes he has found traces of her, and that’s why he is now away from home. That’s all I can tell you. I hope you will not think we were careless or neglected her. She was the last child in the world to do such a thing. I can’t understand it. I think she must have been bewitched.” Frank had listened quietly to this story, drinking in every word, the expression on his face failing to show how much it affected him. “I am sure it was no fault of yours, Mrs. Staples,” he said. “But what do you think has happened to her? She was too young to be led into an intrigue with a man. Still, I——” “You mustn’t suspect her of that, Mrs. Staples!” exclaimed Merry. “Whatever has happened, I believe it was not the child’s fault. When I placed her in your hands, you remember, I hinted to you of the fact that there was a mystery connected with her father’s life, and that he was an outcast nobleman of Spain. Where he is now I cannot say. I last saw him in Fardale. He was then hunted by enemies, and he disappeared and has never been heard from since. I believe it was his intention to seek some spot where he would be safe from annoyance and could lead his enemies to believe he was dead. I
- 69. believe this mystery which hung like a shadow over him has fallen at last on little Felicia. I would that I had known something of this before, that I might have arrived here sooner. I think Felicia would have trusted me—I am sure of it!” “But now—now?” “Now,” said Frank grimly, shaking his head, “now I must find her. You say you heard from your husband, who was then in a place called Warner?” “Yes.” “Then he may have tracked her thus far. It’s a start on the trail.” Mrs. Staples placed a trembling hand on Frank’s sleeve. “If you find her—the moment you find her,” she pleaded, “let me know. Remember I shall be in constant suspense until I hear from you.” “Depend upon me to let you know,” assured Frank. A moment later he was descending the steps. He walked swiftly along the palm-lined streets, revolving in his mind the perplexing problem with which he was confronted. Seemingly he was buried in deep thought and quite oblivious of his surroundings. As he passed around a corner into another street he glanced back without turning his head. Already he had noted that another man was walking rapidly in the same direction, and this sidelong glance gave him a glimpse of the man. Three corners he turned, coming at length to the main street of the city. There he turned about a moment later and was face to face with the man who had been following him. This chap would have passed on, but Frank promptly stepped out and confronted him. He saw a small, wiry, dark-skinned individual, on whose right cheek there was a triangular scar. “I beg your pardon,” said Merry.
- 70. “Si, señor,” returned the man with the scar, lifting his eyebrows in apparent surprise. “You seem very interested in me,” said Merry quietly. “But I wish to tell you something for your own benefit. It is dangerous for you to follow me, and you had better quit it. That’s all. Adios!” “Carramba!” muttered the man, glaring at Frank’s back as Merriwell again strode away.
- 72. CHAPTER VIII. FELIPE DULZURA. Frank did not find Rufus Staples at Warner. He had been there, however, and gone; but no one seemed to know where. The afternoon of a sunny day found Merry mounted on a fine horse, emerging from the mountains into a black valley that was shut in on either side by savage peaks. Through this valley lay a faint trail winding over the sand and through the forests of hideous cactus and yucca trees. He had not journeyed many miles along this trail ere he drew up. Turning his horse about, he took a powerful pair of field glasses from a case and adjusted them over his eyes. With their aid he surveyed the trail behind him as far as it could be seen. “I thought I was not mistaken,” he muttered, as his glasses showed him a mounted man coming steadily along from the foothills of the mountains. “I wonder if he is the gentleman with the scarred cheek. I think I will wait and see.” He dismounted and waited beside the trail for the horseman to approach. The man came on steadily and unhesitatingly and finally discovered Frank lingering there. Like Merry, the stranger was well mounted, and his appearance seemed to indicate that there was Spanish blood in his veins. He had a dark, carefully trimmed Van Dyke beard and was carelessly rolling a cigarette when he appeared in plain view. His clothing was plain and serviceable. Merry stood beside his horse and watched the stranger draw near. Frank’s hand rested lightly on his hip close to the butt of his
- 73. holstered revolver, but the unknown made no offensive move. Instead of that he called, in a pleasant, musical voice: “Good-day, sir. I have overtaken you at last. I saw you in advance, and I hastened somewhat.” “Did you, indeed?” retorted Merry, with a faint smile. “I fancied you were coming after me in a most leisurely manner. But, then, I suppose that’s what you call hurrying in this country.” “Oh, we never rush and exhaust ourselves after the manner of the East,” was the smiling declaration, as the handsome stranger struck a match and lighted the cigarette. Although Frank was confident the man was a Spaniard, he spoke with scarcely a hint of an accent. In his speech, if not in his manner, he was more like an American. “Seems rather singular,” questioned Frank, “that you should be traveling alone through this desolate region.” “The same question in reference to you has been troubling me, sir,” retorted the stranger, puffing lightly at his cigarette. “To me it seems altogether remarkable to find you here.” “In that case, we are something of a mystery to each other.” “Very true. As far as I am concerned, the mystery is easily solved. My name is Felipe Dulzura. I am from Santa Barbara. I own some vineyards there.” Having made this apparently frank explanation, the man paused and looked inquiringly at Merry, as if expecting at least as much in return. Frank did not hesitate. “My name is Frank Merriwell,” he said, “and I am a miner.” “A miner?”
- 74. “Yes, sir.” “You can’t have any mines in this vicinity.” “Possibly I am looking the country over for an investment.” “It’s possible,” nodded Dulzura. “But from your intelligent appearance, I should fancy it hardly probable.” “Thanks for the compliment. In regard to you, being a planter, it seems quite unlikely that you should be surveying this region in search of a vineyard. It seems to me that I have been fully as frank, sir, as you have.” Felipe Dulzura lifted an objecting hand. “I have not finished,” he protested. “I didn’t mean to give you the impression that I was seeking vineyards here. Far from it. On the contrary, having a little leisure, I am visiting the old missions in this part of the country. They interest me greatly. There was a time, long ago, you know, when this land belonged to my ancestors. My grandfather owned a vast tract of it. That was before gold was discovered and the great rush of ’forty-nine occurred. “I presume it is needless to state that my grandfather’s title to his lands was regarded as worthless after that and he lost everything. He died a poor man. My father was always very bitter about it, and he retired to Old Mexico where he spent his last days. I am happy to say that he did not transfer his bitterness toward the people of this country to me, and I have found it to my advantage to return here and engage in my present occupation. You should see my vineyard, Mr. Merriwell. I think I have one of the finest in the State.” The manner in which this statement was made seemed frankly open and aboveboard. To all appearances, Felipe Dulzura had nothing to conceal and was unhesitating in telling his business. “I, too,” declared Merry, “am interested in the old Spanish missions. They remind me of the days of romance, which seem so far
- 75. removed.” “Ah!” cried Dulzura, “then it may happen that we can journey a while in company. That will be agreeable to me. I confess that the trail has been lonely.” The planter was most agreeable and friendly in his manner, and his smile was exceedingly pleasant. In every way he seemed a most harmless individual, but experience had taught Merry the danger of always trusting to outward appearances. “Company of the right sort will not be disagreeable to me,” assured Frank. “Good!” laughed Dulzura. “I am sick of talking to myself, to my horse, or to the landscape. I am a sociable chap, and I like some one to whom I can talk. Do you smoke, Mr. Merriwell? I have tobacco and papers.” “Thank you; I don’t smoke.” “Ah, you miss one of the soothing friends of life. When I have no other company, my cigarette serves as one. This beastly valley is hot enough! The mountains shut it in and cut off all the cool breezes. However, ere nightfall we should get safely out of it and come to San Monica Mission. It lies yonder near the old Indian reservation. I have heard my father tell of it, and it has long been my object to see it.” For some little time they chatted, Dulzura seeming to be in the most communicative mood, but finally they prepared to go on together. When they were ready Frank suggested that his companion lead the way, as it was far more likely that he knew the trail better. “No, no, Mr. Merriwell,” was the protest. “There is but one trail here. Like you, I have never passed over it. You were in advance; it would scarcely be polite for me to take the lead.” Frank, however, had no thought of placing himself with his back turned on the self-styled planter, and, therefore, he insisted that
- 76. Dulzura should proceed in advance, to which the latter acquiesced. As they rode on through the somewhat stifling heat of the valley, the Spaniard continued to talk profusely, now and then turning his head and smiling back at Merry. “Next year,” he said, “I mean to visit Spain. I have never been there, you know. Years and years ago my ancestors lived there. I trust you will pardon the seeming egotism, Mr. Merriwell, if I say it’s not poor blood that runs in my veins. My ancestors far back were grandees. Did you ever hear of the Costolas? It’s likely not. There were three branches of the family. I am a descendant of one branch.” “Costola?” murmured Frank. “The name seems familiar to me, but I presume there are many who bear it.” “Quite true. As for our family, however, an old feud has nearly wiped it out. It started in politics, and it divided the Costolas against themselves. A divided house, you know, cannot stand. My grandmother was a Costola. She was compelled to leave Spain. At that time another branch of the family was in power. Since then things have changed. Since then that powerful branch of the family has declined and fallen. It was not so many years ago that the sole surviving member was compelled, like my grandmother, to escape secretly from Spain. He came to this country and here lived under another name, taking that of his mother’s family. I don’t even remember the name he assumed after reaching America; but I did know that the surviving Costolas hunted him persistently, although he managed to evade and avoid them. What has become of him now is likewise a mystery. Perhaps he is dead.” The speaker suddenly turned so that he could look fairly into Frank’s face, smiling a little, and said: “It’s not likely this interests you, sir.” “On the contrary,” Merry smiled back, “I find it quite interesting. To me Spain is a land of romance. Being a plain American, the tales of
- 77. those deadly feuds are fascinating to me. I presume the Costolas must have possessed large estates in Spain?” “Once they did.” “And the one you speak of—the one who was compelled to flee from the country—was he wealthy?” “I believe he was reckoned so at one time.” “And now,” said Frank, “if this feud were ended, if any offense of his were pardoned, could he not claim his property?” “That I don’t know,” declared Dulzura, shaking his head. “Well, then, if he has any descendants, surely they must be the rightful heirs to his estate.” “I doubt, sir, if they could ever possess it. It must eventually be divided among his living relatives.” “Ah!” cried Merry. “I understand, Mr. Dulzura, why you must have a particular interest in visiting Spain. It seems probable that you, being distantly related to this exiled nobleman, may finally come into possession of a portion of his property.” “It’s not impossible,” was the confession, as the man in advance rolled a fresh cigarette. “But I am not counting on such uncertainties. Although my grandfather and my father both died poor, I am not a pauper myself. To be sure, I am not immensely rich, but my vineyards support me well. I have lived in this country and in Mexico all my life. In fact, I feel that I am more American than anything else. My father could not understand the democracy of the Americans. He could not understand their disregard of title and royalty.” Frank laughed. “Had he lived in these days,” he said, “and associated with a certain class of degenerate Americans, he would have discovered that they
- 78. are the greatest worshipers of titles and royal blood in the whole world.” “I think that may be true,” agreed the Spaniard, puffing at his cigarette. “I have seen some of it. I know that many of your rich American girls sell themselves for the sake of titles to broken-down and rakish noblemen of other countries. I think most Americans are ashamed of this.” “Indeed they are,” seriously agreed Merry. “It makes them blush when a rich American girl is led to the altar by some broken-down old roué with a title, who has spent his manhood and wrecked his constitution in dissipation and licentiousness. Almost every week we read in the papers of some titled foreigner who is coming to America in search of a rich wife. We don’t hear of the scores and scores of American girls with wealthy parents who go abroad in search of titles. But we have forgotten the Costolas. Can you tell me anything more of them?” “You seem strangely interested in them,” said Dulzura, again glancing back. “It almost seems as if you had heard of them before.” “And it almost seems so to me,” confessed Frank. “I think I must have heard of them before. Sometime I shall remember when it was and what I have heard.” But, although they continued to talk, the Spaniard told Merry nothing more of interest in that line. Finally they relapsed into silence and rode on thus. Frank’s thoughts were busy when his tongue became silent. He remembered well that the most malignant and persistent enemy of little Felicia’s father was a man who called himself Felipe Costola. This man had made repeated efforts to get possession of Felicia, but had been baffled by Delores and had finally lost his life in Fardale. Beyond question, Felipe Costola was dead, and what had become of Juan Delores no man seemed to know.
- 79. Putting two and two together, Frank began to wonder if Delores might not be a Costola who had assumed the name of his mother’s family while living in Spain, thus arousing the everlasting enmity of all the Costolas, and who had finally been compelled to flee to America. In many respects the history of this man agreed with that told by Juan Delores himself. He had once told Frank the name and title by which he was known in Spain, but never had he explained the fierce enmity of Felipe Costola. Now Merry was speculating over the possibility that Delores must have once been a Costola. If this was true, then little Felicia was, by the statement of Dulzura, the rightful heir to the estate in Spain. Meditating on this possibility, Frank fancied he obtained a peep behind the curtain which hid the mystery of Felicia’s disappearance. With the child out of the way, a false heir might be substituted, and the schemers behind the plot would reap their reward. The shadows of evening were thickening in the mountain when Merry and his companion passed from the valley and reached the abrupt foothills. Here the trail was more clearly defined, and soon they were startled to see standing beside it an aged Indian, who regarded them with the stony gaze of the Sphinx. Dulzura drew up and asked the Indian in Spanish if the San Monica Mission was near. The reply was that it was less than half a mile in advance. They came to it, sitting on a little plateau, silent and sad in the purple twilight. It was worn and battered by the storms of years. On its ancient tower the cross stood tremblingly. A great crack showed in its wall, running from base to apex. In the dark opening of the tower a huge bell hung, silent and soundless. Merry drew up and sat regarding the ancient pile in almost speechless awe and reverence. It was a monument of other days in that sunny land. Here, long before the coming of the gold seekers, the Spanish priest had taught the Indian to bow his knee to the one true God. Here they had lived their calm and peaceful lives, which were devoted to the holy cause.
- 80. “Come,” urged Dulzura, “let’s get a peep within ere it becomes quite dark. There must be an Indian village somewhere near, and there, after looking into the mission, we may find accommodations.” Frank did not say that he was doubtful if such accommodations as they might find in an Indian village could satisfy him; but he followed his companion to the stone gate of the old mission, where Dulzura hastily dismounted. Even as Frank sprang from his horse he saw a dark figure slowly and sedately approaching the gate. It proved to be a bare-headed old monk in brown robes, who supported his trembling limbs with a short, stout staff. Dulzura saluted the aged guardian of the mission in a manner of mingled worship and respect. “What do ye here, my son?” asked the father, in a voice no less unsteady than his aged limbs. “We have come, father, to see the mission,” answered the Spaniard. “We have journeyed for that purpose.” “It’s now too late, my son, to see it to-night. On the morrow I will take you through it.” “You live here alone, father?” “All alone since the passing of Father Junipero,” was the sad answer, as the aged monk made the sign of the cross. Frank was deeply touched by the melancholy in the old man’s voice and in the lonely life he led there in the ruined mission. “What is the mission’s income?” questioned Merry. “Our lands are gone. We have very little,” was the reply. “Still Father Perez has promised to join me, and I have been looking for him. When I heard your horse approaching I thought it might be he. It was but another disappointment. Still, it matters not.”
- 81. “Let us take a peep inside,” urged Dulzura. “Just one peep to-night, father.” “You can see nothing but shadows, my son; but you shall look, if you wish.” He turned and moved slowly along the path, aided by the staff. They followed him through the gate and into the long stone corridor, where even then the twilight was thick with shadows. In the yard the foliage grew luxuriantly, but in sad neglect and much need of trimming and attention. At the mission door they paused. “Let’s go in,” urged Dulzura. “To-morrow will be time enough,” answered Frank, a sudden sensation of uneasiness and apprehension upon him. At this refusal Dulzura uttered a sudden low exclamation and took a swift step as if to pass Merry. Frank instantly turned in such a manner that he placed his back against the wall, with the door on his left and the old monk close at hand at his right. Suddenly, from beyond the shadows of the foliage in the yard, dark forms sprang up and came bounding into the corridor. Out from the door rushed another figure. Dulzura uttered a cry in Spanish and pointed at Frank. They leaped toward him. Merry’s hand dropped toward the holster on his hip, but with a gasp he discovered that it was empty. Instead of grasping the butt of his pistol, he found no weapon there with which to defend himself. For all of the shadows he saw the glint of steel in the hands of those men as they leaped toward him, and he knew his life was in frightful peril. How his pistol had escaped from the holster, whether it had slipped out by accident, or had in some inexplicable manner been removed
- 82. by human hands, Frank could not say. It was gone, however, and he seemed defenseless against his murderous assailants. In times of danger Frank’s brain moved swiftly, and on this occasion it did not fail him. With one sudden side-step, he snatched from the old monk’s hand the heavy staff. With a swift blow from this he was barely in time to send the nearest assailant reeling backward. The others did not pause, and during the next few moments Frank was given the liveliest battle of his career. “Cut him down! Cut him down!” cried Dulzura, in Spanish. They responded by making every effort to sink their knives in Frank. They were wiry, catlike little men, and in the gloom their eyes seemed to gleam fiercely, while their lips curled back from their white teeth. Merriwell’s skill as a swordsman stood him in good stead now. He took care not to be driven against the wall. He whirled, and cut, and struck in every direction, seeking ample room for evolutions. He knew full well that to be pressed close against the wall would put him at a disadvantage, for then he would not have room for his leaps, and swings, and thrusts, and jabs. The fighting American bewildered and astounded them. He seemed to have eyes in the back of his head. When one leaped at him from behind to sink a knife between his shoulders Frank suddenly whirled like lightning and smote the fellow across the wrist, sending the steel flying from his fingers to clang upon the stones. The old monk lifted his trembling hands in prayer and tottered away. What had happened seemed to him most astounding and appalling. “Come on, you dogs!” rang Frank’s clear voice. “Come on yourself, Felipe Dulzura, you treacherous cur! Why do you keep out of reach and urge your little beasts on?” The Spaniard uttered an oath in his own language.
- 83. “Close in! Close in!” he directed. “Press him from all sides! Don’t let one man beat you off like that!” “You seem to be taking good care of your own precious hide,” half laughed Frank. Then, as the opportunity presented, he made a sudden rush and reached Dulzura with a crack of the staff that caused the fellow to howl and stagger. It did not seem, however, that, armed only with that stick, Merry could long contend against such odds. Soon something must happen. Soon one of those little wretches would find the opportunity to come in and strike swift and sure with a glittering knife. The racket and uproar of the conflict startled the echoes of the mission building, and in that peaceful, dreamy spot such sounds seemed most appalling. Frank knew the end must come. Had he possessed a pistol he might have triumphed over them all in spite of the odds. Suddenly in the distance, from far down the trail toward the valley, came the sound of singing. As it reached Merry’s ears he started in the utmost amazement, for he knew that tune. Many a time had he joined in singing it in the old days. Although the words were not distinguishable at first, he could follow them by the sound of the tune. This is the stanza the unseen singers voiced: “Deep in our hearts we hold the love Of one dear spot by vale and hill; We’ll not forget while life may last Where first we learned the soldier’s skill; The green, the field, the barracks grim, The years that come shall not avail To blot from us the mem’ry dear Of Fardale—fair Fardale.” “Fair Fardale!”—that was the song. How often Frank had joined in singing it when a boy at Fardale Military Academy. No wonder Frank knew it well! By the time the stanza was finished the singers were
- 84. much nearer, and their words could be plainly distinguished. Dulzura and his tools were astounded, but the man urged them still more fiercely to accomplish their task before the singers could arrive. The singing of that song, however, seemed to redouble Merry’s wonderful strength and skill. He was now like a flashing phantom as he leaped, and dodged, and swung, and thrust with the heavy staff. His heart was beating high, and he felt that he could not be defeated then. Finally the baffled and wondering assailants seemed to pause and draw back. Frank retreated toward the wall and stood waiting, his stick poised. The musical voices of the unseen singers broke into the chorus, and involuntarily Frank joined them, his own clear voice floating through the evening air: “Then sing of Fardale, fair Fardale! Your voices raise in joyous praise Of Fardale—fair Fardale! Forevermore ’twixt hill and shore, Oh, may she stand with open hand To welcome those who come to her— Our Fardale—fair Fardale!” It was plain that, for some reason, Dulzura and his band of assassins had not wished to use firearms in their dreadful work. Now, however, the leader seemed to feel that there was but one course left for him. Merry saw him reach into a pocket and felt certain the scoundrel was in search of a pistol. He was right. Even as Dulzura brought the weapon forth, Frank made two pantherish bounds, knocking the others aside, and smote the chief rascal a terrible blow over the ear. Dulzura was sent whirling out between two of the heavy pillars to crash down into the shrubbery of the yard. That blow seemed to settle everything, for with the fall of their master the wretches who had been urged on by him took flight. Like
- 85. frightened deer they scudded, disappearing silently. Merry stood there unharmed, left alone with the old monk, who was still breathing his agitated prayers. From beyond the gate came a call, and the sound of that voice made Frank laugh softly with satisfaction. He leaped down from the corridor and ran along the path to the gate, outside which, in the shadows, were two young horsemen. “Dick—my brother!” exclaimed Merry. “Frank!” was the cry, as one of the two leaped from the horse and sprang to meet him.
- 86. Welcome to our website – the perfect destination for book lovers and knowledge seekers. We believe that every book holds a new world, offering opportunities for learning, discovery, and personal growth. That’s why we are dedicated to bringing you a diverse collection of books, ranging from classic literature and specialized publications to self-development guides and children's books. More than just a book-buying platform, we strive to be a bridge connecting you with timeless cultural and intellectual values. With an elegant, user-friendly interface and a smart search system, you can quickly find the books that best suit your interests. Additionally, our special promotions and home delivery services help you save time and fully enjoy the joy of reading. Join us on a journey of knowledge exploration, passion nurturing, and personal growth every day! ebookbell.com