![• Outages less than 30 seconds in duration
• Outages due to outside causes, such as power loss from utility or
network circuit failures caused by the provider
• Outages due to planned maintenance
The Cisco AVVID IP Telephony solution can achieve 99.999 percent
reliability per the BellCore FR-512 specification.
Command Syntax Conventions
The conventions used to present command syntax in this book are the same
conventions used in the IOS Command Reference. The Command Reference
describes these conventions as follows:
• Vertical bars | separate alternative, mutually exclusive elements.
• Square brackets [ ] indicate an optional element.
• Braces { } indicate a required choice.
• Braces within brackets [{ }] indicate a required choice within an optional
element.
• Boldface indicates commands and keywords that are entered literally as
shown. In actual configuration examples and output (not general command
syntax), boldface indicates commands that the user inputs (such as a show
command).
• Italic indicates arguments for which you supply actual values.
OSI Reference Model
Throughout the book, a few references are made to the OSI model. Table I-1
provides a brief primer on the OSI reference model layers and the functions of each.
You can learn more about the OSI model in any of the Cisco Press books that target
the CCNA certification.
Table I-1. OSI Reference Model Overview
OSI Layer
Name
Functional Description Examples
Physical
(Layer 1)
Responsible for moving bits of data between
devices. Also specifies characteristics such as
voltage, cable types, and cable pinouts.
EIA/TIA-232, V.35
Data link
(Layer 2)
Combines bytes of data into frames.
Provides access to the physical media using a
Media Access Control (MAC) address, which is
typically hard-coded into a network adapter. Also
performs error detection and recovery for the
data contained in the frame.
802.3/802.2,
HDLC
Network Uses logical addressing which routers use for IP, IPX](https://image.slidesharecdn.com/1084778-250607065928-7ce9f1d7/85/Troubleshooting-Cisco-Ip-Telephony-Networking-Technology-Paul-Giralt-13-320.jpg)
![The British Coffee-House was one of the first inns to take to itself
the newly imported title. It stood on the site of the granite building
numbered 66 State Street, and was, as its name implies, as
emphatically the headquarters of the out-and-out loyalists as the
Bunch of Grapes, over the way, was of the unconditional Whigs. A
notable thing about it was the performance there in 1750, probably
by amateurs, of Otway’s “Orphan,” an event which so outraged
public sentiment as to cause the enactment of a law prohibiting the
performance of stage plays under severe penalties.
Perhaps an even more notable occurrence was the formation in this
house of the first association in Boston taking to itself the distinctive
name of a Club. The Merchants’ Club, as it was called, met here
as early as 1751. Its membership was not restricted to merchants,
as might be inferred from its title, though they were possibly in a
majority, but included crown officers, members of the bar, military
and naval officers serving on the station, and gentlemen of high
social rank of every shade of opinion. No others were eligible to
membership.
Up to a certain time this club, undoubtedly, represented the best
culture, the most brilliant wit, and most delightful companionship
that could be brought together in all the colonies; but when the
political sky grew dark the old harmony was at an end, and a
division became inevitable, the Whigs going over to the Bunch of
Grapes, and thereafter taking to themselves the name of the Whig
Club.[1]
Under date of 1771, John Adams notes down in his Diary this item:
“Spent the evening at Cordis’s, in the front room towards the Long
Wharf, where the Merchants’ Club has met these twenty years. It
seems there is a schism in that church, a rent in that garment.”
Cordis was then the landlord.[2]
Social and business meetings of the bar were also held at the
Coffee-House, at one of which Josiah Quincy, Jr. was admitted. By
and by the word “American” was substituted for “British” on the](https://image.slidesharecdn.com/1084778-250607065928-7ce9f1d7/85/Troubleshooting-Cisco-Ip-Telephony-Networking-Technology-Paul-Giralt-62-320.jpg)
![him in the famous argument on the Writs of Assistance. Mention is
also made of a club of which Daniel Leonard (Massachusettensis),
John Lowell, Elisha Hutchinson, Frank Dana, and Josiah Quincy were
members. Similar clubs also existed in most of the principal towns in
New England.
The Sons of Liberty adopted the name given by Colonel Barré to
the enemies of passive obedience in America. They met in the
counting-room of Chase and Speakman’s distillery, near Liberty Tree.
[3] Mackintosh, the man who led the mob in the Stamp Act riots, is
doubtless the same person who assisted in throwing the tea
overboard. We hear no more of him after this. The “Sons” were an
eminently democratic organization, as few except mechanics were
members. Among them were men like Avery, Crafts, and Edes the
printer. All attained more or less prominence. Edes continued to print
the Boston Gazette long after the Revolution. During Bernard’s
administration he was offered the whole of the government printing,
if he would stop his opposition to the measures of the crown. He
refused the bribe, and his paper was the only one printed in America
without a stamp, in direct violation of an Act of Parliament. The
“Sons” pursued their measures with such vigor as to create much
alarm among the loyalists, on whom the Stamp Act riots had made a
lasting impression. Samuel Adams is thought to have influenced their
proceedings more than any other of the leaders. It was by no means
a league of ascetics, who had resolved to mortify the flesh, as punch
and tobacco were liberally used to stimulate the deliberations.](https://image.slidesharecdn.com/1084778-250607065928-7ce9f1d7/85/Troubleshooting-Cisco-Ip-Telephony-Networking-Technology-Paul-Giralt-72-320.jpg)
Troubleshooting Cisco Ip Telephony Networking Technology Paul Giralt
- 1. Troubleshooting Cisco Ip Telephony Networking Technology Paul Giralt download https://ebookbell.com/product/troubleshooting-cisco-ip-telephony- networking-technology-paul-giralt-2169556 Explore and download more ebooks at ebookbell.com
- 2. Here are some recommended products that we believe you will be interested in. You can click the link to download. Troubleshooting And Maintaining Cisco Ip Networks Tshoot Foundation Learning Guide Foundation Learning For The Ccnp Tshoot 642832 Amir Ranjbar https://ebookbell.com/product/troubleshooting-and-maintaining-cisco- ip-networks-tshoot-foundation-learning-guide-foundation-learning-for- the-ccnp-tshoot-642832-amir-ranjbar-21984388 Troubleshooting Ip Routing Protocols 1st Edition Faraz Shamim https://ebookbell.com/product/troubleshooting-ip-routing- protocols-1st-edition-faraz-shamim-2340902 Cisco Ip Routing Protocols Trouble Shooting Techniques 1st Edition Vijay Anand https://ebookbell.com/product/cisco-ip-routing-protocols-trouble- shooting-techniques-1st-edition-vijay-anand-917014 Troubleshooting Cisco Nexus Switches And Nxos 1st Edition Vinit Jain https://ebookbell.com/product/troubleshooting-cisco-nexus-switches- and-nxos-1st-edition-vinit-jain-10945304
- 3. Troubleshooting Cisco Nexus Switches And Nxos Networking Technology Vinit Jain Brad Edgeworth Richard Furr https://ebookbell.com/product/troubleshooting-cisco-nexus-switches- and-nxos-networking-technology-vinit-jain-brad-edgeworth-richard- furr-59723664 Deploying And Troubleshooting Cisco Wireless Lan Controllers Gress https://ebookbell.com/product/deploying-and-troubleshooting-cisco- wireless-lan-controllers-gress-21983832 Understanding And Troubleshooting Cisco Catalyst 9800 Series Wireless Controllers 1st Edition Simone Arena https://ebookbell.com/product/understanding-and-troubleshooting-cisco- catalyst-9800-series-wireless-controllers-1st-edition-simone- arena-43603762 Ccnp Cisco Internetwork Troubleshooting Study Guide 3rd Arthur Pfund https://ebookbell.com/product/ccnp-cisco-internetwork-troubleshooting- study-guide-3rd-arthur-pfund-995560 Troubleshooting Wan Protocols In Cisco Ios Software https://ebookbell.com/product/troubleshooting-wan-protocols-in-cisco- ios-software-1214962
- 5. About the Authors Paul Giralt, CCIE No. 4793, is an escalation engineer at the Cisco Systems Technical Assistance Center in Research Triangle Park, N.C., where he has worked since 1998. He has been troubleshooting complex IP Telephony networks since the release of CallManager 3.0 as a TAC engineer, a technical lead for the Enterprise Voice team, and now as an escalation engineer supporting the complete Cisco line of IP Telephony products. Paul has troubleshot problems in some of Cisco's largest IP Telephony deployments and has provided training for TAC teams around the globe. Prior to working on IP Telephony, he was a TAC engineer on the LAN Switching team. He holds a B.S. in computer engineering from the University of Miami. Addis Hallmark, CCNA, CIPT, is a senior technical marketing engineer with Cisco Systems. He has been installing, configuring, administering, and troubleshooting the Cisco IP Telephony solution since the 2.3 release of CallManager. He has contributed to numerous design guides, application notes, and white papers on a variety of IP Telephony subjects, including CallManager, IP Phones, and IP gateways. Anne Smith is a technical writer in the CallManager engineering group at Cisco Systems. She has written technical documentation for the Cisco IP Telephony solution since CallManager release 2.0 and was part of the Selsius Systems acquisition in 1998. Anne writes internal and external documents for CallManager, IP phones, and other Cisco IP Telephony products. She is a co-author of Cisco CallManager Fundamentals (ISBN: 1-58705-008-0) and Developing Cisco IP Phone Services (ISBN: 1-58705-060-9), both from Cisco Press. About the Technical Reviewers Shawn Armstrong is an IT engineer working in Cisco's Core Hosting group. She has been with Cisco for four years and is responsible for managing NT and Windows 2000 servers within Cisco's Information Technology group. Dave Goodwin, CCIE No. 4992, is a customer diagnostic engineer for Cisco's Advanced Engineering Services. He is responsible for discovering and resolving problems in new Cisco IP Telephony products while administering internal field trials for these systems. He also works closely with Cisco's development and TAC support teams to provide support for anything from troubleshooting to quality issues to tools. He has been at Cisco for almost five years and has worked as a network engineer for eight years. Christina Hattingh is a member of the Technical Marketing organization at Cisco Systems. In this role she works closely with product management and engineering. Christina focuses on helping Cisco sales engineers, partners, and customers design and tune enterprise and service provider Voice over Packet network infrastructures with particular focus on QoS. Prior to this she was a software engineer and engineering manager of PBX Call Center products at Nortel Networks. Her earlier software development experience in X.25 and network management systems provide background for the issues involved today in migrating customers' traditional data and voice networks to packet-based technologies. Christina has a graduate degree in computer science and mathematical statistics. Phil Jensen, CCIE No. 2065, is a consulting systems engineer for Cisco in the southeastern U.S. He has focused on helping Cisco's largest customers design and troubleshoot AVVID IP Telephony solutions for the past three years. He has worked as a network engineer for more than 14 years.
- 6. Ketil Johansen, CCIE No. 1145, is a business development manager with Cisco Systems, working with companies integrating their applications with Cisco CallManager. He has worked with networking technologies for more than 18 years and has been a CCIE since 1994. The last three years he has focused on IP Telephony technologies. Chris Pearce is a technical leader in the Cisco CallManager software group at Cisco Systems, Inc. He has ten years of experience in telecommunications. His primary areas of expertise include call routing, call control, and telephone features. He was a member of the team that developed and implemented the Cisco CallManager software from its early stages, and he was directly involved in developing the system architecture and design. Ana Rivas, CCIE No. 3877, is an escalation engineer in Cisco's EMEA region. She is one of the technical leaders for AVVID solutions in the Cisco TAC. She is responsible for technically leading the resolution of some of the most critical problems in voice and IP Telephony, spreading technical knowledge to other teams, and working with Cisco business units and the field to head IP Telephony solutions. She has been working as a network engineer for more than five years. Markus Schneider, CCIE No. 2863, is a diagnostic engineer for Cisco's Advanced Engineering Services. He is responsible for helping Cisco customers design, implement, and troubleshoot IP Telephony solutions in their environment. He has been working for Cisco as a network engineer for more than six years. Gert Vanderstraeten has been working as a telecom/datacom engineer for companies such as Alcatel, Bell, and Lucent Technologies since 1993. Since 1998 he has been an independent contractor for the Cisco Systems' IT department. During the course of his tenure, his main focus has been the design, implementation, and maintenance of VoIP, IP Telephony, voice and video applications, and the integration of AVVID technologies into solutions. He is currently operating within the Cisco Systems global Enterprise Architecture Solutions team. Liang Wu is a software engineer in the CallManager software group at Cisco Systems, Inc. For the last seven years, he has been focusing on PBX/Enterprise communication systems. He spent more than eight years in the Class 4/5/AIN telephone switching industry. Acknowledgments Paul Giralt I want to first thank Anne Smith for all her hard work and guidance throughout this entire project. There is no way this book would exist without her constant dedication and attention to detail. Thanks to Chris Cleveland for his excellent work as development editor on this book and for being so flexible when it comes to the unpredictable schedules of a TAC engineer. Thank you to the worldwide Enterprise Voice and AVVID TAC teams, especially the RTP Enterprise Voice team for being such a world-class group of engineers to work with. Thanks to the RTP Voice Network Team (VNT) for all the excellent VoX documentation. Special thanks to Gonzalo Salgueiro and Mike Whitley for the VoX boot camp material and to Steve Penna for knowing everything. Thanks to Dave Hanes for his excellent fax troubleshooting presentations and Andy Pepperell for his explanation of fax and modem passthrough.
- 7. Thanks to all the technical reviewers—Ana Rivas, Chris Pearce, Dave Goodwin, Ketil Johansen, Markus Schneider, Phil Jensen, Gert Vanderstraeten, Liang Wu, Shawn Armstrong, and especially Christina Hattingh—for always being on top of everything in the world of Cisco IOS gateways. Thanks to all the developers in Richardson and San Jose that I have worked with over the years. Your insight into the inner workings of CallManager has helped me understand how to better troubleshoot the product. Special thanks to Bill Benninghoff for always answering any question I throw his way and for always being so thorough in his explanations. Also thanks to Chris Pearce for his excellent grasp on the intricacies of call routing. Thank you to all the contributors to the VNT Voice University website as well as the AVVID TAC tips website on Cisco.com. Also thanks to all the other unnamed authors for the documentation scattered throughout various web pages. Thanks to all the customers I have worked with over the past several years on AVVID issues for being my teachers. Every customer I work with helps me understand a little more about IP Telephony. Addis Hallmark First, I'd like to thank Paul Hahn and Richard Platt for bringing me on at Cisco. Paul in particular spent a lot of time with me, bringing me up to speed on these technologies, and for that, I am indebted to him. I'd also like to thank all the brilliant development engineers who patiently helped me understand CallManager so well over the past few years. I'd like to thank Susan Sauter. She is a brilliant engineer, and so much of what I know about IP Phones came from her patient instruction. Chris Pearce has also helped me so much over the last few years in understanding dial plans. The chapter on applications is based on the hard work of Dave Bicknell. Without his efforts, that chapter would not be even close to what it should be. Manish Gupta and his team were a tremendous source of help on the LDAP Directory chapter. Stefano Giorcelli's excellent directory documentation also was so very helpful! The TAC is on the front lines of troubleshooting, and much of the help I received was from the experiences that only solid TAC engineers could provide. Also, the technical reviewers of this book were so helpful. Thank you so much to everyone for their hard work! I really believe this is a great book, and one of the biggest reasons for that is Paul Giralt's invaluable contribution and hard work on this project. I couldn't have done this without him! My manager, Shaik Kaleem, was very supportive of this project that I undertook on my own time, and I greatly appreciate that support. Finally, I'd like to thank Anne Smith. This project would never have happened without her tireless work and skillful help. I am so grateful for Anne's effort. She worked so very hard over this past year, and Paul Giralt and I would have been lost without her. Anne Smith My many thanks go to Paul Giralt and Addis Hallmark for making this book a reality with their knowledge, experience, hard work, and sacrifice. In particular, I thank Paul for a highly enjoyable working experience. Paul's dedication to the quality, accuracy, and comprehensiveness of this book was unsurpassed; he spent countless hours reviewing every page of technical information and his experience with the many components in the Cisco AVVID IP Telephony solution made his extensive contribution invaluable. At every turn, Paul's dedication, commitment to quality,
- 8. tireless drive for accuracy, and constant positive attitude made working with him a rewarding experience. As always, my thanks and great admiration go to Richard Platt and Scott Veibell. Without their continued support there would be no Cisco IP Telephony-related Cisco Press books. I would like to thank Chris Pearce for his help on the Call Routing chapter, Travis Amsler for his assistance on the Cisco CRA and extension mobility sections, and Brian Sedgley and Ken Pruski for their help with CCM and SDL tracing. Appreciation and recognition also go to the engineers who created and developed Dick Tracy: Rick Baugh, Jim Brasher, Long Huang, and David Patton. Foreword In November of 1998, Cisco Systems acquired a small startup called Selsius Systems. For over a year this small company had been shipping the world's first IP phones and Windows NT-based call management software consisting of close to a million lines of C++ code with a small development staff of about 40 engineers. Since the acquisition, the code base has evolved into many millions of lines of C++, XML, and Java code, and the development staff now has over 500 engineers. The level of sophistication and capability has increased dramatically and is a key component of the Cisco Architecture for Voice, Video, and Integrated Data (AVVID). Current deployments range from extremely distributed enterprises with hundreds of remote offices to small 50-person offices. Geographically, systems are deployed across the world, including exotic locations such as Antarctica and the International Space Station! AVVID's IP Telephony components (including the IP phones, gateways, and Cisco CallManager) comprise a telephony system that is both richer than and different from traditional TDM-based phone systems. For example, manageability and serviceability are achieved through either a browseable interface or an XML SOAP-based protocol for integration with existing IT systems. Geography disappears as a problem because telephony functions, manageability, and serviceability all traverse the IP network. Proprietary databases disappear in favor of standard SQL databases and LDAP directories. Nevertheless, this unification and standardization of telephony on IP networks also presents unique challenges. Voice quality can be impacted by poor IP network design. Capacity planning requires consideration of IP address numbering. Music on Hold as a multicast stream requires proper switch and router configuration. These are only a few examples of the unique considerations that must be given to IP Telephony deployments. This book incorporates the authors' real-life experiences in planning and troubleshooting IP Telephony within the AVVID solution. The wisdom contained herein has been gained over the course of thousands of real customer experiences. Paul Giralt and Addis Hallmark are two of the very best troubleshooters in the industry, and Anne Smith has written about and worked with the system since the earliest releases. Paul has been with Cisco's customer support organization for several years. His depth and breadth of knowledge across all Cisco products are legendary, including his most recent focus on IP Telephony. I have seen him in action at some very large and sensitive customer installations, where he resolved extremely difficult problems and provided excellent guidance during upgrades and installations. We were fortunate to get him back, inasmuch as our customers were loathe to let him leave! Addis has been involved in the development and testing of many AVVID products. He has been personally engaged with many key customers
- 9. during deployment and operation and has received numerous rave reviews from customers. Addis also has been instrumental in the security design aspects of Cisco CallManager. Anne is an author and the technical editor for this and several other AVVID books. She has been engaged with the technology since its inception at Selsius Systems. I highly recommend this book to any individual or organization involved in installing, operating, or troubleshooting one of the most exciting advances in the long history of telephony. Written by three of its pioneers, this book serves as a guide for the rest of the pioneers who aren't afraid to help their organization communicate in its own way, the better way, the IP way. Richard B. Platt Vice President for Enterprise Voice, Video Business Unit Cisco Systems, Inc. Introduction This book teaches you the troubleshooting skills you need to isolate and resolve IP telephony problems. IP telephony is a relatively new technology with many different components. The Cisco IP Telephony (CIPT) solution revolves around Cisco CallManager, the core call processing engine. CIPT includes many different endpoints, such as IP phones, various gateways, and various applications such as Cisco IP IVR, Cisco CallManager Attendant Console, Cisco IP SoftPhone, Cisco Conference Connection, extension mobility, and more. Additionally, the network infrastructure plays an important role in prioritizing voice packets to ensure quality of service (QoS). With all these components involved in transmitting voice across packet networks, it is essential that you be able to identify and resolve issues in the entire solution. This requires knowledge of the functionality of these components and how they interact with each other, as well as what tools are available to help you find the root cause when problems arise. This book educates you about the techniques, tools, and methodologies involved in troubleshooting an IP telephony system. Target CallManager Release This book is written to CallManager release 3.3. Updates to this book may be provided after publication. You should periodically check the ciscopress.com web site for updates (go to ciscopress.com and search for "Troubleshooting Cisco IP Telephony"). Goals and Methods This book intends to deliver a methodology you can follow when troubleshooting problems in an IP telephony network, particularly a Cisco IP Telephony solution. This book provides detailed troubleshooting information that applies to a variety of problems that can occur in any IP telephony deployment. "Best Practices" sections in each chapter provide tips and design considerations to help you avoid common configuration problems.
- 10. Who Should Read This Book? This book is designed to teach you how to isolate and correct problems in an IP telephony network. If you are a networking professional responsible for administering a Cisco IP Telephony (CIPT) system, this book is for you. Although this book's main focus is on CIPT, some concepts apply to IP telephony in general as well. You will best be able to assimilate the information in this book if you already have a working knowledge of a CIPT network. How This Book Is Organized Although you could read this book cover-to-cover, it is designed to help you find solutions to specific problems. The chapters are organized by the various components of a Cisco IP Telephony solution. Four appendixes provide reference information. • Chapter 1, "Troubleshooting Methodology and Approach"— You can troubleshoot even the most complex problems if you have a good methodology in place for finding the root cause. This chapter focuses on teaching that methodology: learning how to find clues and track down your "suspect" by breaking the problem into smaller pieces and tackling each piece individually. • Chapter 2, "IP Telephony Architecture Overview"— Cisco AVVID includes many different components that come together to form a comprehensive architecture for voice, video, and integrated data. This chapter covers the basic components of the IP Telephony architecture in order to provide a big-picture view of the system. • Chapter 3, "Understanding the Troubleshooting Tools"— To effectively troubleshoot problems in a Cisco IP Telephony network, you must be familiar with the many tools at your disposal. In addition, you need to know how to best use those tools to achieve maximum results. This chapter describes the various tools and their different uses. • Chapter 4, "Skinny Client Registration"— IP phone registration is a common source of problems. This chapter describes how Skinny protocol- based device registration works, including discussions of inline power, network connectivity, and potential TFTP and CallManager issues. • Chapter 5, "IP Phones"— IP phones can encounter various problems, from unexpected resets to directory and service problems, and more. This chapter explains proper IP phone behavior and examines problems that can occur after an IP phone successfully registers. • Chapter 6, "Voice Gateways"— Voice gateways are the interface that bridges the Voice over IP (VoIP) world with the Public Switched Telephone Network (PSTN). Voice gateways can be Cisco IOS Software gateways or modules within voice-enabled LAN switches. They can be analog or digital, and they can use a wide variety of signaling protocols. This chapter teaches you how to identify and resolve gateway problems by breaking these components into logical groups and following a methodical troubleshooting approach. • Chapter 7, "Voice Quality"— Voice quality is a broad term that covers the following conditions: delayed audio, choppy or garbled audio, static and noise, one-way or no-way audio, and echo. This chapter focuses on the information
- 11. you need to investigate and resolve voice quality problems in an IP Telephony network. • Chapter 8, "Fax Machines and Modems"— Fax machines and modems present unique challenges when carried over an IP Telephony network, primarily due to their unforgiving nature concerning any modification to the audio stream. This chapter discusses the effect of packet loss and jitter, fax passthrough, fax relay, and how to troubleshoot modems and faxes. • Chapter 9, "Call Routing"— Possessing a strong understanding of call routing is arguably one of the most important aspects of a smooth-operating CIPT solution. This chapter discusses closest-match routing, calling search spaces and partitions, transformations, and translation patterns as well as troubleshooting hold, transfer, park, and call pickup. • Chapter 10, "Call Preservation"— Call preservation is easier to predict when you understand the protocol interaction with CallManager. This chapter provides guidelines for determining call survivability based on endpoint type and protocol. • Chapter 11, "Conference Bridges, Transcoders, and Media Termination Points"— Conference bridges, transcoders, and media termination points are media resources. This chapter discusses the role of media resource groups and media resource group lists, codec selection, and troubleshooting transcoder and conference bridge resources. • Chapter 12, "Music on Hold"— The Music on Hold feature allows callers to hear streaming audio while on hold. This chapter describes this feature and provides steps to take if you encounter problems. • Chapter 13, "Call Admission Control"— Call admission control is used in situations where a limited amount of bandwidth exists between telephony endpoints such as phones and gateways. This chapter discusses the two types of call admission control—locations-based and gatekeeper—and the mechanisms available to reroute calls through the PSTN in the event of WAN congestion. • Chapter 14, "Voice Mail"— CallManager is compatible with a variety of voice mail systems that integrate with CallManager through various methods. This chapter focuses on troubleshooting the integration of CallManager and three types of voice mail systems: Cisco Unity, third-party voice mail systems integrated via Simple Message Desk Interface (SMDI), and Octel Voice Mail, integrated through Cisco DPA Voice Mail gateways. • Chapter 15, "Survivable Remote Site Telephony (SRST)"— SRST allows a router at a remote branch to assume call processing responsibilities in the event that phones at a remote site are unable to contact the central CallManager. This chapter describes SRST and provides detailed information about the various problems that can occur. • Chapter 16, "Applications"— Cisco AVVID allows for the creation of many different applications to interoperate within the converged network. This chapter discusses some of the primary applications in a Cisco AVVID IP Telephony solution, such as IP AA and IP IVR, extension mobility, Cisco IP SoftPhone, Personal Assistant, and Cisco CallManager Attendant Console. • Chapter 17, "SQL Database Replication"— The SQL relational database stores the majority of CallManager configuration information. This chapter discusses the Publisher-Subscriber model for database replication, name resolution, Enterprise Manager, Replication Monitor, broken subscriptions, and CDR database replication. • Chapter 18, "LDAP Integration and Replication"— User information is stored in a Lightweight Directory Access Protocol (LDAP) database. This
- 12. chapter describes directory integration versus directory access, using the CallManager embedded directory, and integrating with Active Directory and Netscape iPlanet. • Appendix A, "Cisco IP Telephony Protocol and Codec Information and Reference"— Cisco IP Telephony employs many different protocols and codecs. This appendix provides a list of applicable protocols and codecs with descriptions and the standards body corresponding to the protocol or the Request for Comments (RFC) number. Compression rates are given for each codec. • Appendix B, "NANP Call Routing Information"— CallManager provides a built-in dial plan for the North American numbering plan (NANP). This appendix provides information from the NANP file located in the C:Program FilesCiscoDial Plan directory. This file shows you how each part of an NANP number corresponds to a specific placeholder. It is particularly useful when you're learning how to apply route filters. • Appendix C, "Decimal to Hexadecimal and Binary Conversion Table"— This appendix provides a cheat sheet that shows you how to quickly convert between decimal, hexadecimal, and binary values. • Appendix D, "Performance Objects and Counters"— Microsoft Performance (PerfMon) and the Real-Time Monitoring Tool allow you to monitor your system through the use of performance counters. This appendix lists and describes the performance objects and counters in a Cisco IP Telephony network. Some pertinent Windows 2000 counters are also described. • Glossary— The glossary defines terms and acronyms used in this book. Best Practices In a perfect world, there would be no need for this book, because systems would always run perfectly. Unfortunately, in the real world, problems do arise, and they usually don't go away on their own. However, an administrator/installer can proactively take steps to ensure reliability and high availability and minimize the number of problems that arise. Best practices include not only design considerations but also monitoring and management. A properly monitored system can detect failures before they become service-affecting. Each chapter contains a section outlining best practices as they apply to the chapter topic. In a properly designed network, you can achieve 99.999 percent reliability—a rating that is expected of a telephone system. High Availability in an IP Telephony Environment High availability for IP telephony is based on distribution and core layers in the network and servers (call processing, application servers, and so on). BellCore Specification GR-512 defines what criteria must be met to achieve "five 9s" (99.999 percent) reliability. A careful examination of this document is recommended if you are interested in understanding 99.999 percent reliability. Note that many "events" are not counted against five 9s reliability. Some of these events include the following: • Outages of less than 64 devices
- 13. • Outages less than 30 seconds in duration • Outages due to outside causes, such as power loss from utility or network circuit failures caused by the provider • Outages due to planned maintenance The Cisco AVVID IP Telephony solution can achieve 99.999 percent reliability per the BellCore FR-512 specification. Command Syntax Conventions The conventions used to present command syntax in this book are the same conventions used in the IOS Command Reference. The Command Reference describes these conventions as follows: • Vertical bars | separate alternative, mutually exclusive elements. • Square brackets [ ] indicate an optional element. • Braces { } indicate a required choice. • Braces within brackets [{ }] indicate a required choice within an optional element. • Boldface indicates commands and keywords that are entered literally as shown. In actual configuration examples and output (not general command syntax), boldface indicates commands that the user inputs (such as a show command). • Italic indicates arguments for which you supply actual values. OSI Reference Model Throughout the book, a few references are made to the OSI model. Table I-1 provides a brief primer on the OSI reference model layers and the functions of each. You can learn more about the OSI model in any of the Cisco Press books that target the CCNA certification. Table I-1. OSI Reference Model Overview OSI Layer Name Functional Description Examples Physical (Layer 1) Responsible for moving bits of data between devices. Also specifies characteristics such as voltage, cable types, and cable pinouts. EIA/TIA-232, V.35 Data link (Layer 2) Combines bytes of data into frames. Provides access to the physical media using a Media Access Control (MAC) address, which is typically hard-coded into a network adapter. Also performs error detection and recovery for the data contained in the frame. 802.3/802.2, HDLC Network Uses logical addressing which routers use for IP, IPX
- 14. Table I-1. OSI Reference Model Overview OSI Layer Name Functional Description Examples (Layer 3) path determination. Can fragment and reassemble data if the upper-layer protocol is sending data larger than the data link layer can accept. Transport (Layer 4) Provides reliable or unreliable delivery of data packets. Allows for multiplexing of various conversations using a single network-layer address. Can also ensure data is presented to the upper layers in the same order it was transmitted. Can also provide flow control. TCP, UDP Session (Layer 5) Sets up, coordinates, and terminates network connections between applications. Also deals with session and connection coordination between network endpoints. Operating systems and application access scheduling Presentation (Layer 6) Defines how data is presented to the application layer. Can perform special processing, such as encryption, or can perform operations such as ensuring byte-ordering is correct. JPEG, ASCII Application (Layer 7) Interface between network and application software. Telnet, HTTP Comments for the Authors The authors are interested in your comments and suggestions about this book. Please send feedback to the following address: troubleshootingcipt@external.cisco.com Further Reading The authors recommend the following sources for more information. Cisco Documentation This book provides comprehensive troubleshooting information and methodology. However, details about common procedures might not be provided. You should be familiar with and regularly use the documentation that is provided with the Cisco IP Telephony system to supplement the information in this book. You can find Cisco IP Telephony documentation by searching for a specific product on Cisco.com or by starting at the following link:
- 15. www.cisco.com/univercd/cc/td/doc/product/voice/index.htm You can examine the following books at a technical bookseller near you or online by entering the title in the search box at www.ciscopress.com. Cisco CallManager Fundamentals: A Cisco AVVID Solution You can find detailed information about CallManager's inner workings in the book Cisco CallManager Fundamentals (ISBN 1-58705-008-0). Developing Cisco IP Phone Services: A Cisco AVVID Solution You can find instructions and tools for creating custom phone services and directories for Cisco IP Phones in the book Developing Cisco IP Phone Services (ISBN 1-58705- 060-9). Cisco IP Telephony You can find installation, configuration, and maintenance information for Cisco IP Telephony networks in the book Cisco IP Telephony (ISBN 1-58705-050-1). Integrating Voice and Data Networks You can find information on how to integrate and configure packetized voice networks in the book Integrating Voice and Data Networks (ISBN 1-57870-196-1). Cisco Router Configuration, Second Edition Cisco Router Configuration, Second Edition (ISBN 1-57870-241-0) provides example- oriented Cisco IOS Software configuration for the three most popular networking protocols used today—TCP/IP, AppleTalk, and Novell IPX. Icons Used in This Book Throughout this book, you will see a number of icons used to designate Cisco-specific and general networking devices, peripherals, and other items. The following icon legend explains what these icons represent.
- 16. IFC Active Directory domain name problems Active Directory integration troubleshooting and overview Active Directory schema modifications Active Directory—users added in Active Directory don't show up in CallManager Administration Adding a user fails Alarms (red and yellow) on a digital interface "Already in conference" message Attendant Console client configuration Attendant Console—fast busy on calls to a pilot point Attendant Console—line states won't update Attendant Console—lines are disabled Attendant Console—login failed Attendant Console—longest idle algorithm is not working properly Attendant Console—new user doesn't display
- 17. Attendant Console server configuration Attendant Console—some line states show Unknown status Attendant Console troubleshooting methodology Attendant Console—wrong directory list displays Audio problems Audio Translator problems Automated alternate routing (AAR) troubleshooting Busy signal not heard on an IP phone Calling name display problems Calling search spaces, overview CallManager Serviceability CallManager wildcard summary Call preservation, overview Call routing problems CCM traces—how to read CCM traces—how to read Skinny messages CDRs are not being written properly CDRs are not generated by Subscriber Choppy audio "CM Down, Features Disabled" message "CM Fallback Service Operating" message CMI—reading traces Codec selection between devices Conference bridge—out of resources Conference Connection doesn't work Corporate directory—add or delete users fails in CallManager Administration Corporate directory—Error: "The phone administrator is currently not allowed to add or delete users" CRA Administration page does not load CRA Application Engine problems CRA directory configuration troubleshooting CRA trace files (MIVR) Customer Directory Configuration Plugin troubleshooting Database Layer Monitor is not running properly Database replication problems D-channel won't establish on PRI DC Directory—reconfiguring in CallManager 3.3 DC Directory—reconfiguring in pre-3.3 CallManager Directory access troubleshooting Directory troubleshooting Delayed audio Delayed routing Dial peer matching in IOS, overview Dick Tracy Digit discard instructions (DDIs), overview directories button doesn't work Disconnected calls with cause code 0xE6, "Recovery on timer expiry." DPA 7610/7630—MWI problems Dropped calls Dropped packets DTMF relay, overview E1 interface troubleshooting Echo problems
- 18. "Exceeds maximum parties" message Extension mobility—common error messages Extension mobility problems on CallManager release 3.1 or 3.2 Extension mobility problems on CallManager release 3.3 Extension mobility troubleshooting methodology for CallManager release 3.1 or 3.2 Failover—phone behavior and causes Fax machine troubleshooting methodology Fax passthrough configuration Fax passthrough, overview Fax relay debugs, enabling Fax relay, overview Fax takes twice as long to complete FXO port will not disconnect a completed call Garbled audio Gatekeeper call admission control H.323 call flow (H.225 and H.245) Hold and resume problems Hold doesn't play music Intercluster trunk troubleshooting IOS gateway—call routing and dial peer debugs IOS gateway debugs IOS gateway—debugs and show commands IOS gateway—diagnosing the state of ports IOS gateway—TDM interfaces IOS gateway won't register with CallManager (MGCP) iPlanet integration troubleshooting ISDN cause codes (Q.850) ISDN messages, overview ISDN timers, overview Jitter Live audio source problems LMHOSTS file, overview Locations-based call admission control Masks, overview Methodology for troubleshooting MGCP overview Microsoft Performance (PerfMon) Modem passthrough configuration Modem passthrough, overview Modem troubleshooting methodology MOH—live audio source problems MOH—multicast and unicast problems MOH—no music when calls are on hold MOH—reading CCM traces MOH—troubleshooting methodology MWI problems (Personal Assistant) MWI problems (SMDI) MWI problems (Unity) MWI problems (VG248) "No conference bridge available" message No-way audio Octel integration One-way audio
- 19. Outside dial tone played at the wrong time Park problems Partitions, overview Personal Assistant is not intercepting calls Personal Assistant—MWI problems Phone—busy signal not heard Phone—failover and failback Phone—inline power problems Phone—network connectivity and Skinny registration Phone stuck in SRST mode Phone—switch port operation Phone—TFTP configuration file Phone—understanding the difference between restart and reset Phone—understanding the Skinny protocol Phone—VLAN configuration Phone won't register Pickup/group pickup problems PRI backhaul channel status PRI—CallManager sends the proper digits to the PSTN, but call won't route properly PRI signaling troubleshooting Publisher-Subscriber model, overview Q.931 Translator Registration problems on IP phone Replication problems Reset vs. restart Ringback problems Route filters, overview SDL traces—how to read Search for a user fails services button doesn't work Silence suppression—effect on voice quality SMDI—check configuration parameters SMDI—integration SMDI integration with VG248 SMDI—MWI problems SoftPhone has no lines SoftPhone—one-way audio over VPN SoftPhone shows line but won't go off-hook SQL database replication problems SQL—re-establishing a broken subscription SQL—reinitializing a subscription SRST and phone registration SRST—DHCP issues SRST—features lost during operation SRST—phones still registered after WAN connection is restored SRST—transfer problems SRST—voice mail and forwarding issues T1 CAS signaling troubleshooting T1 interface troubleshooting "Temporary Failure" message Time synchronization Toll fraud prevention Tone on hold plays instead of music
- 20. Transcoder—out of resources Transcoder—understanding codec selection between devices Transfer problems Transformation troubleshooting Transformations and masks, overview Transformations, overview Translation pattern troubleshooting Unity—MWI problems Unity—TSP configuration VAD—effect on voice quality VG248—MWI problems Voice mail—MWI problems (DPA 7610/7630) Voice mail—MWI problems (SMDI) Voice mail—MWI problems (Unity) Voice mail—MWI problems (VG248) Voice mail—Octel integration Voice quality problems WS-X6608/6624 gateway troubleshooting WS-X6608—D-channel is down WS-X6608—dropped calls WS-X6608—T1 CAS problems WS-X6608 T1/E1 configuration troubleshooting WS-X6608—unexpected resets WS-X6624 FXS analog gateway configuration Chapter 1. Troubleshooting Methodology and Approach It's 5:30 a.m. on a Monday and your pager goes off. You recognize the phone number— it's your CEO's administrative assistant. As the administrator of the company's 8000-phone IP Telephony network, you assume there's a big problem. You rush into work and find the CEO's administrative assistant, who states that several calls for the CEO have been disconnected in the middle of the call, including a call from a very important customer. Where do you start? Troubleshooting a Cisco IP Telephony network can be a daunting task. Rather than describing step-by-step how to solve specific problems (subsequent chapters provide that information), this chapter focuses on teaching a good troubleshooting methodology: learning how to find clues and track down your "suspect" by breaking the problem into smaller pieces and tackling each piece individually. A typical IP Telephony network consists of—at the very least—one or more of the following components: • Cisco CallManager servers • IP phones • Voice gateways These components are in addition to the data network infrastructure that supports voice over IP (VoIP) traffic. More-complex installations can have dozens of servers for different services and redundancy, each server running a variety of applications, as well as hundreds or thousands of IP phones and a large number of voice gateways.
- 21. Before exploring the myriad of tools, traces, and techniques available to you that aid in troubleshooting, you must develop a systematic method by which you can focus on the problem and narrow it down until you determine the root cause. In addition to the information in this book, you should become familiar with the various standard protocols that are used in an IP Telephony network, such as the following: • H.323 • Media Gateway Control Protocol (MGCP) • Telephony Application Programming Interface/Java Telephony Application Programming Interface (TAPI/JTAPI) You should also become familiar with the protocols used when interfacing with the traditional time-division multiplexing (TDM)-based Public Switched Telephone Network (PSTN), such as the following: • Q.931 (an ISDN protocol) • T1- or E1-Channel Associated Signaling (T1-CAS or E1-CAS) • Foreign Exchange Office (FXO) • Foreign Exchange Station (FXS) Additionally, because an IP Telephony network runs over a data network, it is important to understand the protocols that transport VoIP data, such as the following: • Internet Protocol (IP) • Transmission Control Protocol (TCP) • User Datagram Protocol (UDP) • Real-Time Transport Protocol (RTP) Later chapters cover some of these concepts. However, each of the mentioned protocols could take up an entire book on its own, so you should refer to the specifications and RFCs or to other materials that go into detail about these protocols. Appendix A "Cisco IP Telephony Protocol and Codec Information and References," provides references to where you can find additional information for each protocol discussed in this book. On the other hand, because the Skinny Client Control Protocol (SCCP or Skinny protocol, the Cisco-developed protocol that Cisco IP Phones use) is not the product of an industry-wide standards body, this book goes into additional detail about how this protocol works. Understanding the Skinny protocol is essential to understanding how the phone operates and how to troubleshoot problems with it. The Skinny protocol is covered in greater detail in Chapter 5, "IP Phones." Developing a Troubleshooting Methodology or Approach To track down a problem and resolve it quickly, you must assume the role of detective. First, you need to look for as many clues as you can find. Some clues lead you to additional clues, and others lead you to a dead end. As soon as you've got all the clues, you need to try to make sense of them and come up with a solution. This
- 22. book shows you where to look for these clues and track down the problem while trying to avoid as many dead ends as possible. Troubleshooting a problem can be broken down into two stages: data gathering and data analysis, although your analysis might lead you to collect additional data. The following list is a general guide for steps to take when troubleshooting an IP Telephony problem: Step 1. Gather data about the problem: a. Identify and isolate the problem. b. Use topology information to isolate the problem. c. Gather information from the end users. d. Determine the problem's timeframe. Step 2. Analyze the data you collected about the problem: a. Use deductive reasoning to narrow the list of possible causes. b. Verify IP network integrity. c. Determine the proper troubleshooting tool(s), and use them to find the root cause. Production Versus Nonproduction Outages Troubleshooting a problem can occur in one of two timeframes: • During a scheduled outage window, such as when you're installing a new system, adding components, or upgrading for new features or functionality • During production hours when the problem affects end users or service Although the methodology to troubleshoot problems in either of these two situations is similar, the focus on how to resolve the problem should be different. In the case of a service-affecting problem during production hours, the focus should be to quickly restore service by either resolving the problem or finding a suitable workaround. In contrast, when a problem is found during a new install or scheduled outage window, the focus should be on determining the root cause to ensure the problem is completely diagnosed and resolved so that it does not have the potential to become service-affecting. For example, if users are encountering a delayed dial tone or sluggish behavior on their phones, you might discover that a high-level process on CallManager is consuming 100 percent of the CPU on one of the servers. During a new install or scheduled outage window, it's a good idea to investigate what is causing the CPU consumption to ensure that the problem does not return during production hours. However, if this problem occurs during production hours, the best approach is to stop or restart the offending process and let the redundant systems take over to quickly restore service. After you restore service, perform a root-cause analysis to try to determine why that process was consuming the CPU. The downside of this approach is that you might not be able to further troubleshoot the problem when the process is restarted. Fortunately, CallManager provides many diagnostic traces (if they are enabled prior to the problem) that you can reference after a problem has occurred to see what was happening on CallManager at the time of the problem. Note that although 100 percent CPU of a high-level process can cause sluggish behavior or delayed dial tone, do not infer from this that 100 percent CPU is
- 23. necessarily always a bad thing. As of CallManager 3.3(1), low priority tasks (such as phone registrations) can consume 100 percent CPU without causing adverse effects to the ability to place or receive calls. Look at the 100 percent CPU as a possible symptom but not necessarily the root cause. In this case, you observe the symptoms of sluggish or delayed dial tone and 100 percent CPU utilization and make a correlation between the two. If you encounter an event where you are unable to determine the root cause due to insufficient information, it is a good idea to turn on the appropriate traces to ensure that if the problem reoccurs, you will have enough data to identify the root cause. Sometimes, several service-affecting problems occur simultaneously. In fact, this is not uncommon, because multiple problems often manifest themselves as symptoms of the same root cause. When multiple problems occur simultaneously, focus on the problem that has the greatest impact on users. For example, if some users are reporting dropped calls and others are reporting occasional echo, the two problems are probably unrelated. Troubleshoot the dropped-call problem first because keeping calls connected is more critical than removing the occasional echo on an active call. Step 1: Gathering Data About the Problem So you've just installed a new IP Telephony network, or you've been given the task of maintaining one—or maybe you've taken your first CallManager out of the box and are having problems getting it to run. You've encountered a problem. The first thing to do is gather as much information about the problem as possible. Identifying and Isolating the Problem Half the battle in troubleshooting a problem is determining which piece of the puzzle is the source of the problem. With so many different pieces composing an IP Telephony network, the first step is to isolate the problem and, if multiple problems are being reported, determine which of the problems might be related to each other and which should be identified as separate problems. You must also determine which parts of the problem are symptoms and which are the root cause of the problem. For example, if a user complains of a phone resetting itself, it might seem logical to first assume that something is wrong with the phone. However, the problem might lie with CallManager or one of the many routers and switches that make up the underlying data network. So although the symptom is a phone reset, the root cause could be a WAN network outage or CallManager failure. You must always remember to look at the big picture when searching for the root cause and not let the symptoms of the problem lead you in the wrong direction. To help you visualize the big picture, detailed topology information is essential. Using Topology Information to Isolate the Problem You can take many proactive steps to help make the troubleshooting process easier. One of the first lines of defense is possessing current topology information. One of the most important pieces of topology information is a detailed network diagram (usually created using Microsoft Visio or a similar application). The network diagram should include network addressing information and the names of all the devices. It should also clearly show how the devices are interconnected and the port numbers being used for these interconnections. This information will prove invaluable when you try to isolate which components are involved in a particular problem.
- 24. For medium- to larger-sized networks, you should have a high-level overview topology that gives you a general idea of how things are connected and then several more-detailed diagrams for each piece of the network that drill down to the interface level on your network devices. Figure 1-1 shows a typical high-level topology diagram for a large enterprise IP Telephony network. Notice that device names and IP addresses are listed in the diagram. This makes troubleshooting easier by allowing you to quickly look up devices to access them. Because Figure 1-1 is a high-level diagram, it does not get down to the interface level of each device. Figure 1-1. Sample High-Level Topology Diagram Most networks are not as large as the one shown in Figure 1-1. However, no matter the size of your network, a similar topology diagram is very useful for quickly sharing information about your network with others who might be assisting you in troubleshooting. In addition to the network diagram, you should use some method to store information such as IP address assignments, device names, password information, and so on. For a small network, you can use something as simple as a spreadsheet or even a plain text file. For larger deployments, some kind of database or network management application such as CiscoWorks is recommended. Many customers keep all this topology information on a web server as well, making it quickly and easily accessible to others when it is needed the most. Be sure to keep this information in a secure location. You also need documentation of your dial plan. Some deployments, especially those heavily utilizing toll-bypass, have very complex dial plans. Knowing where a call is
- 25. supposed to go just by knowing the phone number and from where it is dialed helps you quickly understand a problem. When your topology information is complete, it should include all the following information: • Interconnection information for all devices, including device names and port numbers. If any patch panels exist between devices, the port numbers should be listed. • IP addressing for all network devices (routers, switches, and so on) • IP addressing for all telephony and application servers and voice gateways (including data application servers) • IP addressing for endpoints (that is, scopes of a DHCP pool) • WAN and PSTN service provider names and Circuit IDs for each circuit • Spanning-tree topology, including root bridges for all VLANs and which ports should be forwarding and blocking • Dial plan information • Software version information for all devices If you are troubleshooting a network you didn't design, topology is one of the first pieces of information you should obtain, if it's available. If a topology drawing is not available, it is a good idea to spend time obtaining this information from someone who is familiar with the network and then making a quick sketch. A general topological understanding of the network or at least the piece of the network in question helps when you're trying to differentiate the problem from its symptoms. It's necessary when you're trying to isolate the problem to a particular part of the network. For example, if a user reports hearing choppy audio when making a conference call, it is essential to know exactly where in the network the conference bridge device is located in relation to the user's phone, including all the intermediate network devices. Without a network diagram, finding this information could waste precious time. Assume that the network you are troubleshooting looks like Figure 1-1. If the user's phone is connected to Access Switch 1A, the other conference participants are on Access Switch 1Z, and the conference bridge device is on Voice Switch 1A, you can see that the number of devices is greatly reduced from 100 or more switches and routers to four or five. What is worse than not having topology information? Having incorrect topology information can lead to countless hours heading down the wrong path. If you're going to keep topology information (highly recommended), make sure you keep it current. Use all the topology information you have to narrow down which pieces of the network might be involved in the problem you are trying to troubleshoot. To further isolate the problem, interview the end users who reported the problem to gather additional information. Gathering Information from the User Information the user provides can be vital to your ability to correct a problem. Try to gather as much detail as possible on exactly what the problem is. Often when troubleshooting a problem, you might realize that what you've been troubleshooting for hours is not really the problem the user encountered. The more detail about the problem you can gather before you begin troubleshooting, the easier it is to find a
- 26. resolution—and that means less frustration for you. Here is some general information to collect from users: • Details about exactly what the user experienced when the problem occurred. • Phone numbers for all parties involved in the problematic call or calls. You can use this as search criteria if you need to look through traces. • Actions performed by the user when the problem occurred. This includes what buttons were pressed and in what order. • User observations. This includes text messages displayed on the phone or recorded announcements. • Information about the user's device. For example, if the user experienced a problem while using a 7960 phone, get the phone's MAC address and IP address, along with registration information and any other statistics available from the phone. Sometimes the information provided by an end user is not enough to even begin troubleshooting. For example, if a user has trouble transferring calls, you should ask what steps the user took when the problem happened and, if possible, when the problem occurred so that you can examine traces. Sometimes the proper diagnostic tools are not enabled when the problem occurs, forcing you to ask the user to inform you the next time the problem occurs. Be sure to turn on tracing or debugs before making the request so that when the problem occurs again, you will have captured the data. Users can get quite irritated if you have to ask them for the same piece of information two or three times. Also point out to the user the importance of letting you know immediately after a problem occurs, as many of the diagnostic trace files overwrite themselves within several hours or days (depending on the amount of traffic on your system). Determining the Problem's Timeframe In addition to what the problem is, you should try to determine when the problem occurred. Determining the problem's earliest occurrence can help correlate the problem with other changes that might have been made to the system or other events that occurred around the same time. For example, assume that a regular workday begins at 9 a.m. and ends around 6 p.m. Many users report that they get a busy signal when dialing into their voice mail. It is important to know whether they are attempting to do this at 9:10 a.m., a time when the voice mail system is likely under attack from many users all trying to access the system at once. This might change the problem from a troubleshooting issue to a load-balancing or equipment- expansion issue. You check the voice mail system and notice that at the time the problem was reported, all the voice mail ports were in use. Clearly in this example you need more voice mail ports or servers to handle call volume. However, if the problem occurs at 10:30 p.m., capacity is likely not the problem, so it's time to start troubleshooting your network and voice mail system. As another example, if a user reports that her phone was not working for 10 minutes and you know there was a network outage in her part of the building at that time, you can be relatively sure that the problem was due to the network outage. When relying on end users to give "when" information about a problem, ask them to note the time on their phone when the problem occurred. The phone's time is synchronized with the clock on the CallManager to which the phone is registered. As long as you have the time on your CallManagers and network devices synchronized,
- 27. having a phone-based time from the user makes finding the proper trace files very easy. In some cases, the information about when a problem occurred might be the only piece of information you have other than a limited description of the problem at hand. If you have information about when, you might be able to look through trace files during that timeframe to search for anything abnormal. TIP Although it is important to use information about when the problem started happening, it is equally important to not assume that the problem was a direct result of an event. For example, if a user reports a problem the day after an upgrade was performed on CallManager, you might give some credence to the notion that the upgrade might have caused the problem, but don't automatically assume that this is the root cause. Step 2: Analyzing the Data Collected About the Problem Now that you have collected data from a variety of sources, you must analyze it to find the root cause and/or workaround for your problem. Using Deductive Reasoning to Narrow the List of Possible Causes The next part of your fact-finding mission is to identify the various components that might be involved and to eliminate as many components as possible. The more you can isolate the problem, the easier it is to find the root cause. For example, if a user complains about choppy voice quality, consider some of the following questions to help isolate the real problem, and think about how the answer will help narrow your focus: • Does the problem happen on only one phone? If so, you can probably eliminate hundreds or thousands of other phones as suspects. However, keep in mind a single user's perspective. He might think the problem happens only on his phone, so you'll have to ask other users to see if the problem is more widespread than a single phone. • What numbers are being called when the problem occurs? The answer to this question helps determine which parts of the system are being used when the problem occurs. For example, if the user never experiences poor audio quality when calling certain numbers but always experiences it when calling other numbers, this is a big clue. • Does the problem happen only between IP phones, only through one or more voice gateways, or both? The user probably won't know the answer, but you'll be able to answer this question yourself after you answer the preceding question about which numbers are being called when the problem happens. You will find more detailed questions similar to these throughout this book when troubleshooting particular problems.
- 28. Although not all of the following apply to every problem, where applicable, you must check all of the following pieces involved in the call. Use your topology information to help obtain this information. • CallManager nodes involved in the signaling • Network devices that signaling and/or voice traffic traverse • Gateways or phones involved in the call • Other devices involved, such as conference bridges or transcoders Concentrate your energy on the smallest subset of devices possible. For example, if all the users on a particular floor are having the same problem, concentrate on the problem a particular user is having. If you fix the problem for that one user, in most cases you fix it for all the affected users. Verifying IP Network Integrity One thing that people often forget is that your IP Telephony network is only as good as your IP network. A degraded network or a network outage can cause a wide range of problems, ranging from slight voice quality problems to a total inability to make or receive calls on one or more phones. The network is always a consideration when you encounter certain problems, so network health issues are covered throughout this book. Network health is especially important during the discussion of voice quality problems in Chapter 7, "Voice Quality," because most voice quality problems stem from packet delay and/or loss. Always remember to keep the IP network in mind and look at every layer in the OSI model, starting from Layer 1. Check your physical layer connectivity (cables, patch panels, fiber connectors, and so on). Then make sure you have Layer 2 connectivity by checking for errors on ports, ensuring that Layer 2 switches are functioning properly, and so forth. Continue working your way up the stack until you reach the application layer (Layer 7). As an example, two of the most common reasons for one-way audio (where one side of the conversation cannot hear the other) are the lack of an IP route from one phone to another and the lack of a default gateway being configured on a phone. Taking the layered approach, you would first check the cabling and switches to make sure that there are no errors on the ports. You would then check Layer 3, the network layer, by ensuring that IP routing is working correctly. When you reach this layer, you discover that for some reason the IP packets from one phone are unable to reach the other phone. Upon further investigation, you might discover that there was a missing IP route on one of the routers in the network or a missing default gateway on one of the end devices (such as an IP phone or voice gateway). Determining the Proper Troubleshooting Tool After you narrow down the appropriate component(s) causing a problem and have detailed information from the user(s) experiencing the problem, you must select the proper tool(s) to troubleshoot the problem. Most components have multiple troubleshooting tools available to help you. Chapter 3, "Understanding the Troubleshooting Tools," provides more details about some of the tools available for troubleshooting CallManager. You should use the tracing and debugging facilities available in CallManager and other devices to determine exactly what is happening. Additional tools and traces are covered in the chapter associated with diagnosing certain types of problems. For example, Chapter 6, "Voice Gateways," covers
- 29. debugging Cisco IOS Software voice gateways. Because CallManager is central to almost all problems, information about various portions of the CCM trace facilities appears throughout this book. This step is the most demanding on your troubleshooting skills because you analyze the detailed information provided in the various tools and use it to search for additional clues using other tools. Sometimes the problem description you have is not detailed enough to determine which tool to use. In this case, you should try various tools in search of anything that looks out of the ordinary. The following case study shows how this troubleshooting methodology works in a real-world scenario. Case Study: Resolving a Problem Using Proper Troubleshooting Methodology It is 6 a.m., and you have arrived at work to resolve your CEO's problem. The only data you have is the page you received at 5:30 a.m. that says "CEO's calls keep dropping. Please help ASAP!" You need a bit more information than that to fix the problem. This case study applies the methodology previously described. You must gather the data before you can begin the analysis. Gathering the Data As part of the data-gathering stage, you should do the following: • Identify and isolate the problem • Use topology information to isolate the problem • Gather data from the end users • Determine the problem's timeframe You find the CEO's administrative assistant and begin your fact-finding mission. He states that at various times during the previous day and one time this morning, the CEO is on the phone when, all of the sudden, the call is disconnected. Eager to resolve the problem, you ask the administrative assistant for the following information: • The exact date and times the problem occurred • Whether the dropped calls were incoming or outgoing • What number was dialed if it was an outbound call or what number the call came from if it was an inbound call The assistant states that the call was dropped around 5:15 a.m. because the CEO was in early to prepare for the stockholders meeting. This is the extent of the information he remembers. Most users do not pay attention to specifics like this unless they have been instructed to, but all is not lost. The CEO has a 7960 phone that stores information locally about missed calls, received calls, and placed calls. You head into the CEO's office and look at the list of received calls and placed calls for the morning. You notice that a call was received at 5:05 a.m. and a call placed at 5:25 a.m. You notice that the second call was placed to the same area code and prefix as the call that was received.
- 30. You ask the CEO about the two calls. She remembers that she was on the phone with a customer for about 15 minutes when the call was disconnected. She immediately called the customer back. She also confirms that the first call that was received was the dropped call. Now you know that the problematic call was received at approximately 5:05 a.m. and was dropped just before 5:25 a.m. While you are looking at the CEO's phone, you also go into the Settings menu (press the settings button > Network Configuration > CallManager 1) to see which CallManager the CEO's phone is registered to. This lets you isolate which CallManager in the cluster is involved in the signaling for this phone. Armed with this information, you can begin the task of isolating the problem. You refer to your topology diagram to isolate the components that are involved. Figure 1- 2 shows a high-level diagram of the network topology. Figure 1-2. High-Level Topology Diagram Reinforcing the topology in Figure 1-2, assume the following setup: • A cluster with eight CallManager nodes • 32 voice gateway connections to the PSTN for outgoing calls at your main site—16 for local calls and 16 for international and long distance • 32 more voice gateways at your main campus where all your inbound calls come in. The telephone company has set up the inbound calls so that the 32 gateways are redundant whereby if one of the gateways is down, all your incoming calls can still use any of the other remaining gateways. • Two gateways at each remote site used for both inbound and outbound calls. All outbound calls prefer the first gateway, and inbound calls prefer the
- 31. second gateway, although each can handle both inbound and outbound calls should one fail. As shown in Figure 1-2, the executive offices are at a remote site across the WAN. With just the information you have so far, you can eliminate a large portion of the network. So far you know that the problematic call was to the CEO. You also know that the problematic call was an inbound call. You ask the CEO and her admin if all the dropped calls were inbound calls. As far as they can remember, they were. You know that the call this morning was during a time of day where there is little phone activity. Remember that all inbound calls to the remote site come in through Primary Rate Interfaces (PRIs) connected to the remote voice gateways and that inbound calls to the site prefer the second gateway. It is unlikely that all the channels on the first PRI were in use during a time of low call volume, so you assume that the call probably came in through the second gateway, although you still keep it in the back of your mind that the call might have come in through the first gateway at the remote site. You then look at the configuration for the two gateways at Remote Site 2 and note that they are both configured to send incoming calls to CallManager Subscriber 3 as their preferred CallManager and CallManager Backup 1 in case CallManager Subscriber 3 fails. With the information you have so far, you can narrow down the possible suspect devices to the network shown in Figure 1-3. Figure 1-3. Network After You Narrow Down the Possible Suspects
- 32. Armed with this knowledge, you can immediately isolate the problem to the user's phone and the two gateways being used for inbound calls. Keep in mind that you haven't eliminated the possibility that the problem is on CallManager or is network- related. Now that you know the problem is related to inbound calls, it makes sense to try to understand the call flow for an inbound call to this user. Determine whether these calls all come directly to the user or if the call flow has any intermediate steps, such as Cisco IP Auto Attendant (Cisco IP AA) or an operator who transfers the call to the end user. For the sake of this example, assume that the user has a Direct Inward Dialing (DID) number, so the call comes straight from the PSTN through a gateway to the user, and a Cisco IP AA or operator is not involved. You have now eliminated Cisco IP AA from the picture, as well as the possibility that other phones or users are involved in this user's problems. This is not to say that other users are not experiencing similar problems, but the focus here is on solving this particular user's problem. If the problem is more widespread than this one user, you will probably find it as you continue to troubleshoot this user's problem. At this point, the problem has been isolated to the following culprits: • The CEO's phone
- 33. • CallManager Subscriber 3 • Site 2 Router/GW 1 and Site 2 Router/GW 2 • The underlying network connecting these devices It might seem like you haven't made much progress in this example, but in reality you have eliminated a large portion of the system as possible culprits. This concludes the data-gathering piece of your investigation. Now it is time to start analyzing the data. After you isolate the problem, you must break it into smaller pieces. Analyzing the Data As soon as you have a clear understanding of the problem you're trying to resolve, and you have isolated the piece or pieces of the network that are involved, the next step is to break the problem into pieces to find the root cause. As part of the data analysis stage, you should do the following: • Use deductive reasoning to narrow the list of possible causes • Verify IP network integrity • Determine the proper troubleshooting tools, and use them to find the root cause Continuing with the case study example, you now know the pieces involved in the puzzle, but you still don't know why the call is being dropped. For the sake of this example, this chapter keeps things general, but later chapters go into far greater detail on exactly what to look for. In this case, the problem is likely caused by the phone, CallManager, the gateway, the PSTN, or the IP network. So how do you determine which one is causing the problem? One important distinction to make that will become evident as you read through this book is that many problems can be narrowed down to being either signaling-related or voice packet-related. In this case, you are dealing with a signaling-related problem, because the problematic call is being torn down—a problem that must occur in the signaling path between devices. Because nearly all signaling for a call must go through one or more CallManager servers, the first tool you decide to use is a trace from CallManager Subscriber 3. You can then analyze the trace files to discover the device that disconnects the call from CallManager's perspective—in other words, "Who hung up first?" Using the information provided by the user, you must find the proper trace file and try to reconstruct the call from beginning to end. A call between the CEO's phone and the voice gateway has two distinct signaling connections. One is the communication between CallManager and the voice gateway. The other is the communication between CallManager and the phone. The phone and voice gateway never directly exchange signaling data. All signaling goes through CallManager. The trace includes all the messaging between CallManager and both the phone and the gateway. Chapter 3 provides more details on where to find these traces and how to read them. You know that the call in question was set up around 5:05 a.m., so you look through the traces during that timeframe, searching for the phone number you retrieved from the CEO's phone. After combing through the trace file, you determine that the gateway is sending a message to CallManager, telling it to disconnect the call. The CCM traces (discussed in Chapter 3) indicate which gateway the calls are coming from. This eliminates the CEO's phone as a cause of the problem because the
- 34. disconnect message is coming from the gateway. Because the user indicated that there were three drops, you can now go through the same process of looking through the CCM trace files for each instance of a dropped call and reconstructing those calls to see if the problem is isolated to one gateway. If you don't know the times that the other calls were dropped, you should just concentrate on the one call you do have data for. Because CallManager received a message from the gateway telling it to disconnect the call, it is unlikely that a network problem is causing the calls to disconnect. If there were a network problem, you would likely see an indication that there was a problem communicating between CallManager and the gateway. In this case, the gateway had no problem sending the disconnect message to CallManager. It would not hurt to look through the network devices between CallManager and the voice gateway to ensure that there are no network errors, but with a problem like this, the network is an unlikely culprit. At this point, you have narrowed down the problem to be originating from either the voice gateway or the PSTN. Figure 1-4 shows you've narrowed down the network to only a few devices. Figure 1-4. Network After You Continue Narrowing Down the Possible Suspects The next step is to go to the suspected gateway and try to determine why one of the calls was dropped. This involves turning on additional debugs on the gateway to determine if the gateway is disconnecting the call or just passing along information
- 35. from the PSTN about disconnecting the call. Unfortunately, it is unlikely that you had the debugs enabled at the time the problem occurred, so you need to enable the proper debugs and wait for the problem to happen again. This is why it is so important to narrow down the problem to a small subset of devices: You do not want to turn on debugs on dozens of gateways. Which debugs to use depends on the gateway model and the type of interface to the PSTN. Chapter 6 discusses these considerations in detail. While waiting for the problem to reoccur, you discover that a message to disconnect the call is coming from the PSTN. If you are using an ISDN voice circuit for connectivity to the PSTN, the disconnect message is accompanied by a cause code that provides a general reason why the call was disconnected. Depending on what you discover on the gateway debugs, the next step might be to contact the local service provider or perhaps debug the gateway further to find the root cause. Conclusions As this case study has demonstrated, the more information you can obtain about the problem, the easier it is to get to the root cause. For example, without the times the dropped calls occurred, it would have been almost impossible to find them in the trace files on a busy system. When deployed in a large enterprise, it is good to arm your help desk with a list of questions to ask depending on the problem being reported. The point of this example is not to teach you how to troubleshoot a specific problem or to find out exactly why the user's calls are being dropped. It is to show you how to approach a problem in order to isolate it and break it into more manageable pieces. The same principles can be applied to almost any problem you are troubleshooting. So remember, first put on your detective hat and gather enough information to isolate the problem to a few pieces of the system. Then dig deeper into each component by breaking the problem into more manageable pieces. Finally, apply your expertise to each of the smaller pieces until you find the resolution to your problem. Summary This chapter discussed the methodology you should employ to successfully troubleshoot problems in an IP Telephony network. You should become familiar with the methodologies discussed here. It is vital that you always follow a consistent approach to troubleshooting. Many basic problems can be avoided by using a consistent troubleshooting approach. Also, be sure that you understand the big picture of IP Telephony architecture. What areas are you unsure about? Are you strong in IP but weak in call processing skills? Are you familiar with the basic protocols that are used? Consider where you are now, and as you move forward, pay particular attention to strengthening your weak areas. As you begin this journey, hopefully this book can bring some illumination to the sometimes daunting task of troubleshooting an IP Telephony network. Chapter 2. IP Telephony Architecture Overview
- 36. Cisco AVVID (Architecture for Voice, Video and Integrated Data) includes many different components that come together to form a comprehensive architecture for voice, video, and integrated data. This chapter covers the basic components of the IP Telephony architecture in order to get a big-picture viewpoint of the system. With this overview as the starting point, the ensuing chapters address each of these components. Cisco AVVID IP Telephony can be characterized as having three primary layers: • Network infrastructure • IP Telephony infrastructure • Applications Network Infrastructure The network infrastructure is a key piece of the IP Telephony architecture. The infrastructure includes switches and routers, and it connects local-area networks (LANs), metropolitan-area networks (MANs), and wide-area networks (WANs). Your network design must be built for high availability, and the Cisco series of switches and routers provides that capability. A voice-enabled network is a quality of service (QoS)-enabled network that gives precedence to voice, call signaling, and data to ensure good voice quality and rapid call signaling. IP Telephony Infrastructure The IP Telephony infrastructure includes the Cisco CallManager call processing engine and the various endpoints that carry voice. This includes client endpoints and various voice gateways that are interfaces to the Public Switched Telephone Network (PSTN). Call Processing The CallManager software is the heart of Cisco AVVID IP Telephony that provides call processing features and capabilities to network devices in the enterprise. IP phones, voice gateways, media processing devices, and multimedia applications are just some of the network devices for which CallManager provides call processing. CallManager is installed on the Cisco Media Convergence server and other approved IBM and Compaq servers. CallManager is shipped with integrated voice applications and utilities such as Cisco CallManager Attendant Console (formerly Cisco WebAttendant), software conferencing, and the Bulk Administration Tool (BAT). Multiple CallManager servers are clustered and managed as a single entity. CallManager clustering yields scalability of up to 36,000 users per cluster with version 3.3. By interlinking multiple clusters, system capacity can be increased up to one million users in a 100-site system. Triple call processing server redundancy improves overall system availability. The benefit of this distributed architecture is improved system availability and scalability. Call admission control ensures that voice QoS is maintained across constricted WAN links and automatically diverts calls to alternative PSTN routes when WAN bandwidth is unavailable.
- 37. The four primary call processing models that are used to meet the needs of the enterprise are • Single-site deployment model • Multiple-site deployment model • Centralized deployment model • Distributed deployment model Single-Site Deployment Model In this deployment model, CallManager, applications, voice mail, and digital signal processor (DSP) resources are located at the same physical location. Figure 2-1 shows an example of these components located at a single site. Figure 2-1. Single-Site Deployment Model Multiple-Site Deployment Model In this deployment model, CallManager, applications, voice mail, and DSP resources are located at one physical location. Multiple sites exist, and they connect to each other via the PSTN. Figure 2-2 shows an example of each separate site connecting via the PSTN. Figure 2-2. Multiple-Site Deployment Model
- 38. Centralized Deployment Model A centralized call processing deployment model centrally locates CallManager, applications, voice mail, and DSP resources while many remote locations connect to the central site for all these services. Locations-based call admission control prevents over-subscription of the WAN. At each remote site, Survivable Remote Site Telephony (SRST) ensures that call processing continues in the event of a WAN outage. The centralized call processing model is really the same as the single-site deployment model with the addition of remote sites across the WAN. Figure 2-3 illustrates the centralized deployment model. Figure 2-3. Centralized Deployment Model
- 39. Distributed Deployment Model In a distributed deployment model, CallManager and applications are located at each site with up to 36,000 IP phones per cluster. One hundred or more sites could be interconnected via H.323 using a gatekeeper for call admission control and dial plan resolution. Transparent use of the PSTN is available if the WAN is down. Figure 2-4 depicts a distributed deployment model. Figure 2-4. Distributed Deployment Model
- 40. Cisco AVVID IP Telephony Infrastructure The Cisco AVVID IP Telephony infrastructure includes Cisco Media Convergence Servers and other certified servers running CallManager, Cisco Unity, or other applications, such as IP Auto Attendant, IP Interactive Voice Response (IVR), and IP Integrated Contact Distribution (ICD). Switches, routers, and voice gateways are all part of this infrastructure as well. Although infrastructure is not the primary focus of this book, it is an important part of the IP Telephony architecture, and you will see discussion of some infrastructure aspects when dealing with a Cisco AVVID IP Telephony deployment. Clients
- 41. Clients consist primarily of IP phones. Cisco offers several models with different functions, and these are deployed throughout the IP Telephony infrastructure. Additionally, several third-party companies throughout the world have developed IP phones for their markets. Figure 2-5 shows the Cisco family of IP Phones. Figure 2-5. Cisco Family of IP Phones Table 2-1 provides highlights of each phone and its features. Table 2-1. Descriptions of IP Phone Models Phone Model Description Cisco IP Phone 7960 A full-featured, six-line business set that supports the following features: • A help (i or ?) button • Six programmable line or speed dial buttons • Four fixed buttons for accessing voice mail messages, adjusting phone settings, and working with services and directories • Four soft keys for displaying additional call functionality, such as hold, transfer, conference, and so on • A large liquid crystal display (LCD) that shows call detail and soft key functions
- 42. Table 2-1. Descriptions of IP Phone Models Phone Model Description • An internal two-way speakerphone and microphone mute Cisco IP Phone 7940 A full-featured, two-line business set with all the same features as the Cisco IP Phone model 7960, except only two lines. Cisco IP Phone 7914 Expansion Module An expansion module for the Cisco IP Phone 7960 that provides 14 additional line or speed dial buttons. It has the following features: • An LCD to identify the function of the button and the line status • The capability to daisy-chain two Cisco IP Phone 7914 Expansion Modules to provide 28 additional line or speed dial buttons for a total of 34 line or speed dial buttons Cisco IP Phone 7910/7910+SW A single-line, basic feature phone designed primarily for common-use areas with medium telephone traffic, such as lobbies or break rooms. It includes the following features: • Four dedicated feature buttons for Line, Hold, Transfer, and Settings • Six programmable feature buttons that you can configure through phone button templates in Cisco CallManager Administration. Available features include call park, redial, speed dial, call pickup, conference, forward all, group call pickup, message waiting, and Meet-Me conference • A two-line LCD (24 characters per line) that indicates the directory number, call status, date, and time • An internal speaker designed to be used for hands-free dialing • A handset cord jack that can also be used for a headset • (7910+SW only) A Cisco two-port switch with 10/100BaseT interface Cisco IP Conference Station 7935 A full-featured, IP-based, full-duplex, hands-free conference station for use on desktops, in offices, and in small- to-medium- sized conference rooms. It includes the following features: • Three soft keys and menu navigation keys that guide a user through call features and functions. Available features include call park, call pickup, group call pickup, transfer, conference, and Meet-Me conference • An LCD that indicates the date and time, calling party name, calling party number, digits dialed, and feature and line status • A digitally-tuned speaker and three microphones
- 43. Table 2-1. Descriptions of IP Phone Models Phone Model Description allowing conference participants to move around while speaking • Microphone mute Cisco IP Phone Models 7960 and 7940 The 7960/7940 phones are the most common clients in a Cisco IP Telephony network. These phones feature a large pixel-based display that allows for XML-based applications on the phone. Because these phones are largely soft key-based, new features can easily be added via software upgrades instead of requiring the purchase of new hardware. The internal three-port Ethernet switch allows for a direct connection to a 10/100BaseT Ethernet network via an RJ-45 interface with single LAN connectivity for both the phone and a co-located PC. The system administrator can designate separate VLANs (802.1Q) for the PC and Cisco IP Phone. The 7960/7940 phones can also receive power down the line from any of the Cisco inline power-capable switches or the Cisco inline power patch panel. A dedicated headset port eliminates the need for a separate, external amplifier when you use a headset, consequently reducing desk clutter. The 7960/7940 phones feature a high-quality, full-duplex speakerphone, as well as a speaker on/off button and microphone mute buttons. Cisco IP Phone Expansion Module 7914 The Cisco IP Phone Expansion Module 7914 extends the capabilities of the Cisco IP Phone 7960 with additional buttons and an LCD. The Expansion Module lets you add 14 buttons to the existing six buttons on the 7960 phone, increasing the total number of buttons to 20 with one module or 34 when you add two 7914s. Up to two Cisco 7914s can be connected to a 7960. The 14 buttons on each 7914 Expansion Module can be programmed as directory numbers or speed dial buttons, just like the 7960. Multicolor button illumination allows you to identify which lines are ringing, on hold, or in use. Cisco IP Phone 7910 This low-end phone features on-hook dialing and call monitor mode but does not include speakerphone capability. The phone provides a mute button for the handset and headset microphones. You can attach a headset by removing the handset and using the port into which the handset cord was attached. The 7910 plugs into a standard RJ-45 Ethernet connection. A second version of the 7910 phone, the 7910+SW, provides a Cisco two-port switch with a 10/100BaseT interface. The 7910+SW phone model provides a single RJ-45 connection at the desktop for the phone and an additional LAN device, such as a PC. The 7910 phones can also receive power down the line from any of the Cisco inline power-capable switches or the Cisco inline power patch panel.
- 44. Cisco IP Conference Station 7935 The Cisco IP Conference Station 7935 is a conference room speakerphone utilizing speakerphone technologies from Polycom with the Cisco AVVID voice communication technologies. The 7935 is an IP-based, full-duplex, hands-free conference station for use on desktops and offices and in small- to-medium-sized conference rooms. Although the 7935 does not accept inline power from a Cisco inline power-capable switch, it does feature a power interface module (PIM) that provides power interface and network connectivity. Voice Gateways Many Cisco voice gateways are available for use in the IP Telephony network. These gateways interoperate with CallManager using various protocols. They interface with the PSTN using different TDM-based protocols such as T1/E1-PRI, T1-CAS, FXO, FXS, and so on. One of the primary protocols used by Cisco voice gateways is MGCP. Gateways that support MGCP as of CallManager Release 3.1 include the following: • Cisco VG200 • Cisco 3700, 3600 and 2600 • Cisco Catalyst 6000 E1/T1 Voice and Service modules • Cisco Catalyst 4000 Access Gateway Module • Cisco DE-30+ • Cisco DE-24+ When MGCP is used, CallManager controls routing and tones and provides supplementary services to the gateway. MGCP provides the following services: • Call preservation (calls are maintained during switchover to a backup CallManager and switch back to the primary CallManager) • Redundancy • Dial plan simplification (no dial-peer configuration is required on the gateway) • Hookflash transfer • Tone/music on hold MGCP-controlled gateways do not require a media termination point (MTP) to enable supplementary services such as hold, transfer, call pickup, and call park. H.323 is another protocol used by Cisco voice gateways. Cisco IOS Software integrated router gateways can also use the H.323 protocol to communicate with CallManager. Intercluster trunks for connecting remote CallManagers across the IP WAN are also configured as H.323 gateways. Compared to MGCP, H.323 requires more configuration on the gateway because the gateway must maintain the dial plan and route patterns. Two gateways, the VG248 and the ATA-186, use the Skinny Client Control Protocol (SCCP or Skinny protocol), the same protocol used by IP phones, to communicate with CallManager. Each port on these gateways registers individually as a phone device, so from the CallManager perspective, it does not treat the gateway as a single entity but rather treats each port the same way it treats an IP phone. Using Skinny on these gateways allows the gateway to provide features such as message waiting indicators (MWI), conferencing, and transfer to non-IP phones that would not be possible using any of the other supported protocols.
- 45. Finally, some older Cisco voice gateways use the Skinny Gateway Control Protocol (SGCP), such as the Cisco Analog Access AT-2, AT-4, AT-8, AS-2, AS-4, and AS-8. Cisco AVVID IP Telephony Applications The following list describes some voice and video applications in the application layer of Cisco AVVID IP Telephony: • Cisco Unity— A messaging application that provides voice and/or unified messaging to enterprise communications. • Video IP/TV and IP videoconferencing products— Applications that enable distance learning and workgroup collaboration. • Cisco IP IVR— An IP-powered interactive voice response (IVR) solution, the IP IVR, combined with Cisco IP Auto Attendant, provides an open and feature- rich foundation for delivering IVR solutions over an IP network. Cisco IP IVR and IP AA are built on the Cisco Customer Response Applications (CRA) platform. • Cisco IP ICD— An automatic call distributor (ACD) for enterprise organizations. Cisco IP ICD is built on the Cisco Customer Response Applications (CRA) platform. • Cisco CallManager Attendant Console— A flexible and scalable application that replaces the traditional PBX manual attendant console. • Cisco IP SoftPhone— A software computer-based phone for Windows 98, Windows NT, Windows 2000, and Windows XP that provides communication capabilities that increase efficiency and promote collaboration. • Personal Assistant— A software application that selectively handles calls and helps you make outgoing calls. Personal Assistant provides rule-based call routing, speech-enabled directory dialing, voice mail browsing, and simple Ad Hoc conferencing. • Cisco Conference Connection— A Meet-Me audio conference server that provides integrated operation with CallManager. • Cisco Emergency Responder— An application that allows emergency agencies to identify the location of 911 callers and eliminates the need for any administration when phones or people move from one location to another. Cisco ER features a real-time location-tracking database and improved routing capabilities to direct emergency calls to the appropriate Public Safety Answering Point (PSAP) based on the caller's location. Summary This chapter provided an overview of the primary components that are involved in the Cisco IP Telephony architecture, including the deployment models, clients, voice gateways, and Cisco AVVID IP Telephony applications. When troubleshooting, it is important that you have the big picture of all the architecture components to implement the troubleshooting methodology covered in Chapter 1, "Troubleshooting Methodology and Approach."
- 46. Chapter 3. Understanding the Troubleshooting Tools To effectively troubleshoot problems in a Cisco IP Telephony (CIPT) network, you must be familiar with the many tools at your disposal. In addition, you need to know how best to use those tools to achieve maximum results. This chapter describes the various tools and their different uses. Depending on the problem you encounter and your particular skill set, you might find certain tools more helpful than others. Nevertheless, knowing what tools are available and how they can help you solve the problem is essential to a successful resolution. Also, some tools might be more useful to you based on your past experience. If you are strong in IP, a sniffer trace might be your preferred tool when applicable. However, if you are stronger in call processing-related traces, the Cisco CallManager (CCM, also sometimes called SDI) traces might prove helpful once you understand how they work. Later chapters demonstrate the use of these tools in different scenarios you might encounter. This chapter covers the following topics: • Time synchronization— Explains how to synchronize the clocks on all devices in an IP telephony network to ensure that timestamps on your trace files and debugs are synchronized with each other. • Reading CCM (or SDI) traces— Describes one of the most important trace files you use to troubleshoot CallManager-related problems. CCM trace files provide information about call processing events and all messages exchanged between Skinny, MGCP, and H.323 endpoints. • Reading SDL traces— Discusses the components of the less-used Signal Distribution Layer (SDL) trace files in CallManager. SDL traces describe the events occurring in the CallManager software at a code level. These traces are usually reserved for Cisco development engineering use; however, there are a few key pieces of information you can use to your advantage when troubleshooting CallManager problems. • Microsoft Performance (PerfMon)— Details the capabilities of PerfMon, a built-in Windows 2000 utility that helps you troubleshoot CallManager. • CCEmail— Details the third-party alerting tool that can be used in conjunction with PerfMon to configure alerts for the performance counters. • CallManager Serviceability— Discusses various web-enabled tools provided with CallManager for reading alarms and XML-based tracing. • Real Time Monitoring Tool (RTMT)— Describes the Cisco web-based monitoring application that allows you to view CallManager cluster details, monitor performance objects (much like PerfMon), and monitor devices and CTI applications. • Call detail records (CDR) and the CDR Analysis and Reporting (CAR) Tool— Describes the CAR tool that helps you analyze the raw data that comprises the CDR database and create reports based on your search criteria. • CDR Time Converter— Describes how to use this small utility that allows you to convert the UNIX Epoch-based date and time format stored in a CDR to standard date and time format. • Event Viewer— Briefly explains the function of another built-in Windows 2000 tool that plays a key role in troubleshooting CallManager.
- 47. • Q.931 Translator and Enhanced Q.931 Translator— Describes two tools that read a CCM trace file and analyze all Q.931 and H.225 messages. The various information elements are decoded for ease of reading. The Enhanced Q.931 Translator also adds additional search and filtering options and decodes more information elements than the original Q.931 Translator. • Dick Tracy— Describes an important tool used to troubleshoot the WS-X6608 and WS-X6624 voice gateways for the Catalyst 6000 series switches. • Sniffer traces— Discusses when and why to use a network packet-capture tool. • Voice Codec Bandwidth Calculator— Describes how to use the Voice Codec Bandwidth Calculator to determine the bandwidth used by different codecs with various voice protocols over different media. • Cisco Bug Toolkit (formerly Bug Navigator)— Describes the web-based tool that allows you to find known bugs based on software version, feature set, and keywords. The resulting matrix shows when each bug was integrated or fixed if applicable. • Remote Access Tools— Describes applications like Terminal Services and Virtual Network Computing, which allow you to access a server from a remote location. • Websites and Further Reading— Provides URLs for websites that contain additional troubleshooting information. Also points you to a section in the "Introduction" with a recommended reading list. Time Synchronization Time synchronization is simply making sure that all the participating CallManager servers and network devices have the same exact time. Time synchronization is critical. A large CallManager cluster can have eight or more separate servers, not including any voice mail servers or application servers. This distributed architecture creates a highly available and scalable system. It also makes the troubleshooting process more involved because you have to collect traces from all participating servers to see the full picture of what happened. As endpoints such as IP phones and voice gateways call each other, signaling occurs between CallManager and the endpoint device. Signaling also occurs between the respective CallManagers of each endpoint device. If a problem occurs, you need to consolidate trace files from all involved servers. CallManager Serviceability, discussed later in this chapter, can collect this information into one file. However, if the timestamps of each file are mismatched, it can be impossible to tell what the real series of call processing events is. All the CallManager servers can be time-synched using the Network Time Protocol (NTP). When you synchronize the time on all involved servers, all the trace files are timestamped the same. When trace files are collected for analysis, you can follow the true series of call processing events with accuracy. In addition to CallManager servers, you should ensure all network devices such as switches, routers, and voice gateways are synchronized to the same time source as the CallManager servers. This ensures consistent timestamps regardless of which device you are looking at. Configuring Automatic Time Synchronization on CallManager Servers
- 48. Use the following steps to configure the CallManager server to automatically synchronize—and stay synchronized—with a Time Server. Step 1. Verify that the NetworkTimeProtocol service is configured to launch automatically upon startup. Right-click My Computer and select Manage. Step 2. Expand the Services and Applications section, and select Services. Step 3. Double-click the NetworkTimeProtocol service, and ensure that Startup Type is set to Automatic. Step 4. Configure the C:WINNTntp.conf file. This file contains the list of time servers that CallManager will synchronize with. You can configure CallManager to point to specific time servers (see Example 3-1), or you can configure it to receive NTP broadcasts (see Example 3-2) on the local LAN segment from the router (as long as the router is configured to do so). Example 3-1. Sample ntp.conf File Using Static Time Servers server 10.0.0.10 server 10.1.0.10 driftfile %windir%ntp.drift Example 3-2. Sample ntp.conf File Using an NTP Broadcast Router broadcastclient driftfile %windir%ntp.drift Step 5. Go to the Services Control Panel and stop/start the NetworkTimeProtocol service. Allow several minutes for the update to take place. Synchronizing Time Manually on CallManager Servers Use the following steps to manually configure time synchronization. Step 1. Stop the NetworkTimeProtocol service in the Services Control Panel. Step 2. Synchronize the clock by using one of the following commands from a command prompt. To synchronize with a remote Time Server, use the following command where x.x.x.x is the IP address of the Time Server: ntpdate x.x.x.x To synchronize with a Broadcast Router, use the command where x.x.x.x is the IP address of the Ethernet port of the router:
- 49. ntpdate x.x.x.x Step 3. Restart the NetworkTimeProtocol service in the Services Control Panel. Synchronizing Time on Cisco IOS Devices In addition to your CallManager servers, you should also ensure the time is synchronized on all your network devices, including IOS voice gateways, switches, and routers. For Cisco IOS devices, you can use the Network Time Protocol (NTP) to synchronize the time. To enable NTP, you must configure the time zone the IOS device is in along with the IP address of the NTP server. You should configure the IOS device to take daylight savings time into account as well if you live in a time zone that observes daylight savings time. First you must configure the time zone. Use the command clock timezone name_of_time_zone offset_from_GMT to configure the time zone. For example, the following command configures the IOS device for Eastern Standard Time (EST): clock timezone EST -5 If you are in a time zone that observes daylight savings time, use the command clock summer-time name_of_time_zone recurring to enable automatic daylight savings time. For example, the following configures Eastern Daylight Savings Time: clock summer-time EDT recurring You can determine time zones by performing a search at Google.com; search for a string such as "time zone GMT." Any one of the many hits should point you to a table showing the various time zones around the world, including daylight savings time. Once your time zone is configured properly, you can enable NTP. If you do not have a device on the network running NTP, you can make an IOS device into an NTP master clock. You can do this by configuring the command ntp master on the IOS device. If you are not making the device an NTP master, you should configure the IP address of the NTP server using the command ntp server NTP_server_IP_address. For example, the following tells the IOS device to synchronize its clock with the NTP server at IP address 172.18.109.1: ntp server 172.18.109.1 Once you have enabled NTP, all IOS devices should synchronize their clocks with the central NTP server. You should also synchronize the clocks on non-IOS network devices such as switches running CatOS software. Synchronizing Time on CatOS Devices
- 50. You can use NTP to synchronize the clocks on switches running CatOS software. The configuration is similar to configuring NTP on a device running Cisco IOS Software. First you must configure the time zone the switch is in using the command set timezone name_of_time_zone offset_from_GMT. For example, the following sets the clock to Eastern Standard Time: set timezone EST -5 You can also configure daylight savings time using the command set summertime enable name_of_time_zone. For example, the following enables Eastern Daylight Savings Time: set summertime enable EDT You can determine time zones by performing a search at Google.com; search for a string such as "time zone GMT." Any one of the many hits should point you to a table showing the various time zones around the world, including daylight savings time. Once you have the time zone configured properly, you can set the NTP server IP address and enable the NTP client on the switch. First set the NTP server IP address using the command set ntp server NTP_server_IP_address. Then enable the NTP client using the command set ntp client enable. The following shows the configuration to use the NTP server with IP address 172.18.109.1: set ntp server 172.18.109.1 set ntp client enable Once you have enabled proper time synchronization in your network, you can move on to reading the variety of trace files available to you for troubleshooting problems in an IP Telephony network. Reading CCM (or SDI) Traces CCM (also known as System Diagnostic Interface or SDI) traces are the user- friendliest call processing trace files you have available to you. After you learn a few tricks, it is easy to follow the call flows and find the potential problem. Although you might see "SDI" and "CCM" used interchangeably to refer to this type of trace, in this book we primarily refer to this type of trace as a CCM trace. Some pages in CallManager Serviceability refer to CCM traces as SDI traces. Because CallManager is at the heart of a CIPT network, CCM traces are usually the first place to look when troubleshooting most problems. You can analyze problems related to device registration, call flow, digit analysis, and related devices such as IP phones, gateways, gatekeepers, and more. By the end of this section, you should be able to follow some basic call flows in the CCM trace. Future chapters continue to show CCM trace examples as you learn how to troubleshoot more problems. Later in this chapter you also learn about the Q.931 Translator, which is useful for quickly troubleshooting a variety of gateway problems. It is not a substitute for learning how to read the CCM trace, but it helps you quickly examine a trace in a graphical format
- 51. without having to wade through irrelevant debugs. See the later "Q.931 Translator and Enhanced Q.931 Translator" section for more information. Setting the Appropriate Trace Level and Flags CallManager allows you to select from a variety of different options that adjust which events are logged to the CCM trace files. If you know exactly what you are looking for, you can configure CallManager to log only specific events in the CCM trace file. If you configure the trace to collect too much information, the trace becomes more difficult to analyze. However, if you do not configure the trace to collect enough information, you might miss the problem you are trying to find. When you are beginning to learn how to read CCM trace files, it is best to enable more debugs than less. As you learn which trace settings are required for specific problems, you can enable just the settings you need. Unfortunately you can never predict what problems will come up, so if you do not have a high level of tracing enabled, you may have to wait for a problem to happen a second time after enabling the appropriate trace settings before you can troubleshoot. For this reason, it is usually best to leave a majority of the trace flags enabled during normal operating con-ditions so you have trace data available if a problem occurs. You configure a CCM trace in Cisco CallManager Serviceability (Trace > Configuration). Figure 3-1 shows the top half of the Trace Configuration page, and this section discusses the relevant fields and settings on that page. Figure 3-1. Trace Configuration (Part 1) in CallManager Serviceability
- 52. First thing to note is the Trace On checkbox. This must be selected for any of the trace settings to be available. The Apply to All Nodes checkbox allows you to apply the specified trace settings to all CallManager servers in the cluster. This is useful when you are troubleshooting a problem that is occurring on more than one CallManager server. You generally want to keep the same level of tracing enabled on all the servers in the cluster unless you are absolutely certain the problem is isolated to a single server in the cluster. Because of the distributed nature of CallManager, you may think a process only involves a single server when in reality part of the processing for a particular call is occuring on another server in the cluster. The Trace Filter Settings area allows you to specify the exact parameters of your trace. First, you set the level of tracing you want to perform in the Debug Trace Level field. CallManager Serviceability provides six different levels, but for nearly every kind of problem that you'll want traces for, the recommended levels are either Detailed or Arbitrary:
- 53. • Detailed— Provides detailed debug information and highly repetitive messages that are primarily used for debugging, including KeepAlives and responses. For CallManager releases 3.1 and earlier, be cautious about using this level during normal production hours on a heavily loaded system. It can cause performance degradation on CallManager. In release 3.2 and later, CallManager uses asynchronous tracing to reduce the impact that trace file generation has on call processing. • Arbitrary— Provides low-level debug traces. This level is best suited for debugging difficult problems. This level includes nearly everything that is included in Detailed with the exception of KeepAlives. If you are not troubleshooting problems related to missed KeepAlives, this level is good for day-to-day troubleshooting. Other trace levels are provided, but generally, they don't offer as much information as the Detailed and Arbitrary levels. The other levels are • Error— Provides traces generated in abnormal conditions, such as coding errors or other errors that normally should not occur. • Special— Provides traces for all informational, non-repetitive messages such as process startup messages, registration messages, and so on. All system and device initialization traces are at this level. • State Transition— Provides traces for call processing events or normal events traced for the subsystem (signaling layers). • Significant— Provides traces for media layer events. Second, make sure the Enable CallManager Trace Fields checkbox is selected, which gives you the opportunity to select which specific traces you want to run and to choose your traces. Table 3-1 describes the trace fields to choose from. Most are reasonably self-explanatory. For example, if you're having problems with an H.323 device, you would enable the H245 Message Trace and Enable H225 & Gatekeeper Trace options. Likewise, for music on hold (MOH) issues, the Enable Music on Hold option would display trace relating to all MOH activity. Table 3-1. Cisco CallManager Trace Fields Field Name Description Enable H245 Message Trace Debugs H.245 signaling for H.323 calls, including the media processing messages. Enable DT- 24+/DE-30+ Trace Activates the logging of events related to the legacy gateways, Cisco Digital Access DT-24+/DE-30+. Enable PRI Trace Activates a trace of Primary Rate Interface (PRI) devices. Enable ISDN Translation Trace Activates a Layer 3 trace of Q.931 (ISDN messages). Enable H225 & Gatekeeper Trace Activates a trace showing H.225 signaling messaging for H.323 calls. Enable Miscellaneous Trace Activates a trace of miscellaneous devices. Enable Conference Bridge Trace Activates a trace of the conference bridges. Use this level to trace conference bridge statuses such as
- 54. Table 3-1. Cisco CallManager Trace Fields Field Name Description • Registered with CallManager • Unregistered with CallManager • Resource allocation processed successfully • Resource allocation failed Enable Music on Hold Trace Activates a trace of MOH devices. Use this level to trace MOH device statuses such as • Registered with CallManager • Unregistered with CallManager • Resource allocation processed successfully • Resource allocation failed Enable CM Real- Time Information Server Trace Activates CallManager real-time information traces used by the real-time information server. Enable CDR Trace Enables tracing of call detail record (CDR) processing. However, this trace flag does not provide much information about CDRs because CDR processing is mostly handled by the Database Layer Monitor and CDR Insert services discussed in Chapter 17, "SQL Database Replication." Enable Analog Trunk Trace Activates a trace of all MGCP-based devices using an analog interface. Enable All Phone Device Trace Activates a trace of phone devices, including Cisco IP SoftPhones, and shows events such as on-hook, off-hook, key presses, and so on. Enable MTP Trace Activates a trace of media termination point devices and transcoders. Use this level to trace MTP device statuses such as • Registered with CallManager • Unregistered with CallManager • Resource allocation processed successfully • Resource allocation failed Enable All Gateway Trace Activates a trace of all analog and digital gateways. Enable Forward and Miscellaneous Trace Activates a trace of call forwarding and all subsystems not covered by another checkbox. Enable MGCP Trace Activates a trace showing media gateway control protocol (MGCP) messages for MGCP-based devices. Enable Media Activates a trace for media resource manager (MRM) activities.
- 55. Other documents randomly have different content
- 56. only sixpence and Converse not a penny to bless himself with, so they were compelled to forego this pledge of friendship and part with thirsty lips. Going on to Roxbury, Church luckily found an old neighbor of his, who generously lent him money enough to get home with. He tells the anecdote in order to show to what straits the parsimony of the Massachusetts rulers had reduced him, their great captain, to whom the colony owed so much. The Red Lion, in North Street, was one of the oldest public houses, if not the oldest, to be opened at the North End of the town. It stood close to the waterside, the adjoining wharf and the lane running down to it both belonging to the house and both taking its name. The old Red Lion Lane is now Richmond Street, and the wharf has been filled up, so making identification of the old sites difficult, to say the least. Nicholas Upshall, the stout-hearted Quaker, kept the Red Lion as early as 1654. At his death the land on which tavern and brewhouse stood went to his children. When the persecution of his sect began in earnest, Upshall was thrown into Boston jail, for his outspoken condemnation of the authorities and their rigorous proceedings toward this people. He was first doomed to perpetual imprisonment. A long and grievous confinement at last broke Upshall’s health, if it did not, ultimately, prove the cause of his death. The Ship Tavern stood at the head of Clark’s Wharf, or on the southwest corner of North and Clark streets, according to present boundaries. It was an ancient brick building, dating as far back as 1650 at least. John Vyal kept it in 1663. When Clark’s Wharf was built it was the principal one of the town. Large ships came directly up to it, so making the tavern a most convenient resort for masters of vessels or their passengers, and associating it with the locality itself. King Charles’s commissioners lodged at Vyal’s house, when they undertook the task of bringing down the pride of the rulers of the colony a peg. One of them, Sir Robert Carr, pummeled a constable who attempted to arrest him in this house. He afterward refused to obey a summons to answer for the assault before the
- 57. magistrates, loftily alleging His Majesty’s commission as superior to any local mandate whatever. He thus retaliated Governor Leverett’s affront to the commissioners in keeping his hat on his head when their authority to act was being read to the council. But Leverett was a man who had served under Cromwell, and had no love for the cavaliers or they for him. The commissioners sounded trumpets and made proclamations; but the colony kept on the even tenor of its way, in defiance of the royal mandate, equally regardless of the storm gathering about it, as of the magnitude of the conflict in which it was about to plunge, all unarmed and unprepared.
- 58. III. IN REVOLUTIONARY TIMES. uch thoroughfares as King Street, just before the Revolution, were filled with horsemen, donkeys, oxen, and long-tailed trucks, with a sprinkling of one-horse chaises and coaches of the kind seen in Hogarth’s realistic pictures of London life. To these should be added the chimney-sweeps, wood- sawyers, market-women, soldiers, and sailors, who are now quite as much out of date as the vehicles themselves are. There being no sidewalks, the narrow footway was protected, here and there, sometimes by posts, sometimes by an old cannon set upright at the corners, so that the traveller dismounted from his horse or alighted from coach or chaise at the very threshold. Next in the order of antiquity, as well as fame, to the taverns already named, comes the Bunch of Grapes in King, now State Street. The plain three-story stone building situated at the upper corner of Kilby Street stands where the once celebrated tavern did. Three gilded clusters of grapes dangled temptingly over the door before the eye of the passer-by. Apart from its palate-tickling suggestions, a pleasant aroma of antiquity surrounds this symbol, so dear to all devotees of Bacchus from immemorial time. In Measure for Measure the clown says, “’Twas in the Bunch of Grapes, where indeed you have a delight to sit, have you not?” And Froth answers, “I have so, because it is an open room and good for winter.”
- 59. THE BUNCH OF GRAPES This house goes back to the year 1712, when Francis Holmes kept it, and perhaps further still, though we do not meet with it under this title before Holmes’s time. From that time, until after the Revolution, it appears to have always been open as a public inn, and, as such, is feelingly referred to by one old traveller as the best punch-house to be found in all Boston. When the line came to be drawn between conditional loyalty, and loyalty at any rate, the Bunch of Grapes became the resort of the High Whigs, who made it a sort of political headquarters, in which patriotism only passed current, and it was known as the Whig tavern. With military occupation and bayonet rule, still further intensifying public feeling, the line between Whig and Tory houses was drawn at the threshold. It was then kept by Marston. Cold welcome awaited the appearance of scarlet regimentals or a Tory phiz there; so gentlemen of that side of politics also formed cliques of their own at other houses, in which the talk and the toasts were more to their liking, and where they could abuse the Yankee rebels over their port to their heart’s content.
- 60. But, apart from political considerations, one or two incidents have given the Bunch of Grapes a kind of pre-eminence over all its contemporaries, and, therefore, ought not to be passed over when the house is mentioned. On Monday, July 30, 1733, the first grand lodge of Masons in America was organized here by Henry Price, a Boston tailor, who had received authority from Lord Montague, Grand Master of England, for the purpose. Again, upon the evacuation of Boston by the royal troops, this house became the centre for popular demonstrations. First, His Excellency, General Washington, was handsomely entertained there. Some months later, after hearing the Declaration read from the balcony of the Town-house, the populace, having thus made their appeal to the King of kings, proceeded to pull down from the public buildings the royal arms which had distinguished them, and, gathering them in a heap in front of the tavern, made a bonfire of them, little imagining, we think, that the time would ever come when the act would be looked upon as vandalism on their part. General Stark’s timely victory at Bennington was celebrated with all the more heartiness of enthusiasm in Boston because the people had been quaking with fear ever since the fall of Ticonderoga sent dismay throughout New England. The affair is accurately described in the following letter, written by a prominent actor, and going to show how such things were done in the times that not only tried men’s souls, but would seem also to have put their stomachs to a pretty severe test. The writer says:— “In consequence of this news we kept it up in high taste. At sundown about one hundred of the first gentlemen of the town, with all the strangers then in Boston, met at the Bunch of Grapes, where good liquors and a side-table were provided. In the street were two brass field-pieces with a detachment of Colonel Craft’s regiment. In the balcony of the Town-house all the fifes and drums of my regiment were stationed. The ball opened with a discharge of
- 61. thirteen cannon, and at every toast given three rounds were fired and a flight of rockets sent up. About nine o’clock two barrels of grog were brought out into the street for the people that had collected there. It was all conducted with the greatest propriety, and by ten o’clock every man was at his home.” Shortly after this General Stark himself arrived in town and was right royally entertained here, at that time presenting the trophies now adorning the Senate Chamber. On his return from France in 1780 Lafayette was also received at this house with all the honors, on account of having brought the news that France had at last cast her puissant sword into the trembling balance of our Revolutionary contest. But the important event with which the Bunch of Grapes is associated is, not the reception of a long line of illustrious guests, but the organization, by a number of continental officers, of the Ohio Company, under which the settlement of that great State began in earnest, at Marietta. The leading spirit in this first concerted movement of New England toward the Great West was General Rufus Putnam, a cousin of the more distinguished officer of Revolutionary fame. Taking this house as a sample of the best that the town could afford at the beginning of the century, we should probably find a company of about twenty persons assembled at dinner, who were privileged to indulge in as much familiar chat as they liked. No other formalities were observed than such as good breeding required. Two o’clock was the hour at which all the town dined. The guests were called together by the ringing of a bell in the street. They were served with salmon in season, veal, beef, mutton, fowl, ham, vegetables, and pudding, and each one had his pint of Madeira set before him. The carving was done at the table in the good old English way, each guest helping himself to what he liked best. Five shillings per day was the usual charge, which was certainly not an exorbitant one. In half an hour after the cloth was removed the table was usually deserted.
- 62. The British Coffee-House was one of the first inns to take to itself the newly imported title. It stood on the site of the granite building numbered 66 State Street, and was, as its name implies, as emphatically the headquarters of the out-and-out loyalists as the Bunch of Grapes, over the way, was of the unconditional Whigs. A notable thing about it was the performance there in 1750, probably by amateurs, of Otway’s “Orphan,” an event which so outraged public sentiment as to cause the enactment of a law prohibiting the performance of stage plays under severe penalties. Perhaps an even more notable occurrence was the formation in this house of the first association in Boston taking to itself the distinctive name of a Club. The Merchants’ Club, as it was called, met here as early as 1751. Its membership was not restricted to merchants, as might be inferred from its title, though they were possibly in a majority, but included crown officers, members of the bar, military and naval officers serving on the station, and gentlemen of high social rank of every shade of opinion. No others were eligible to membership. Up to a certain time this club, undoubtedly, represented the best culture, the most brilliant wit, and most delightful companionship that could be brought together in all the colonies; but when the political sky grew dark the old harmony was at an end, and a division became inevitable, the Whigs going over to the Bunch of Grapes, and thereafter taking to themselves the name of the Whig Club.[1] Under date of 1771, John Adams notes down in his Diary this item: “Spent the evening at Cordis’s, in the front room towards the Long Wharf, where the Merchants’ Club has met these twenty years. It seems there is a schism in that church, a rent in that garment.” Cordis was then the landlord.[2] Social and business meetings of the bar were also held at the Coffee-House, at one of which Josiah Quincy, Jr. was admitted. By and by the word “American” was substituted for “British” on the
- 63. Coffee-House sign, and for some time it flourished under its new title of the American Coffee-House. But before the clash of opinions had brought about the secession just mentioned, the best room in this house held almost nightly assemblages of a group of patriotic men, who were actively consolidating all the elements of opposition into a single force. Not inaptly they might be called the Old Guard of the Revolution. The principals were Otis, Cushing, John Adams, Pitts, Dr. Warren, and Molyneux. Probably no minutes of their proceedings were kept, for the excellent reason that they verged upon, if they did not overstep, the treasonable. His talents, position at the bar, no less than intimate knowledge of the questions which were then so profoundly agitating the public mind, naturally made Otis the leader in these conferences, in which the means for counteracting the aggressive measures then being put in force by the ministry formed the leading topic of discussion. His acute and logical mind, mastery of public law, intensity of purpose, together with the keen and biting satire which he knew so well how to call to his aid, procured for Otis the distinction of being the best- hated man on the Whig side of politics, because he was the one most feared. Whether in the House, the court-room, the taverns, or elsewhere, Otis led the van of resistance. In military phrase, his policy was the offensive-defensive. He was no respecter of ignorance in high places. Once when Governor Bernard sneeringly interrupted Otis to ask him who the authority was whom he was citing, the patriot coldly replied, “He is a very eminent jurist, and none the less so for being unknown to your Excellency.” It was in the Coffee-House that Otis, in attempting to pull a Tory nose, was set upon and so brutally beaten by a place-man named Robinson, and his friends, as to ultimately cause the loss of his reason and final withdrawal from public life. John Adams says he was “basely assaulted by a well-dressed banditti, with a commissioner of customs at their head.” What they had never been able to compass by fair argument, the Tories now succeeded in
- 64. accomplishing by brute force, since Otis was forever disqualified from taking part in the struggle which he had all along foreseen was coming,—and which, indeed, he had done more to bring about than any single man in the colonies. Connected with this affair is an anecdote which we think merits a place along with it. It is related by John Adams, who was an interested listener. William Molyneux had a petition before the legislature which did not succeed to his wishes, and for several evenings he had wearied the company with his complaints of services, losses, sacrifices, etc., always winding up with saying, “That a man who has behaved as I have should be treated as I am is intolerable,” with much more to the same effect. Otis had said nothing, but the whole club were disgusted and out of patience, when he rose from his seat with the remark, “Come, come, Will, quit this subject, and let us enjoy ourselves. I also have a list of grievances; will you hear it?” The club expected some fun, so all cried out, “Ay! ay! let us hear your list.” “Well, then, in the first place, I resigned the office of advocate- general, which I held from the crown, which produced me—how much do you think?” “A great deal, no doubt,” said Molyneux. “Shall we say two hundred sterling a year?” “Ay, more, I believe,” said Molyneux. “Well, let it be two hundred. That, for ten years, is two thousand. In the next place, I have been obliged to relinquish the greater part of my business at the bar. Will you set that at two hundred pounds more?” “Oh, I believe it much more than that!” was the answer. “Well, let it be two hundred. This, for ten years, makes two thousand. You allow, then, I have lost four thousand pounds sterling?”
- 65. “Ay, and more too,” said Molyneux. Otis went on: “In the next place, I have lost a hundred friends, among whom were men of the first rank, fortune, and power in the province. At what price will you estimate them?” “D—n them!” said Molyneux, “at nothing. You are better off without them than with them.” A loud laugh from the company greeted this sally. “Be it so,” said Otis. “In the next place, I have made a thousand enemies, among whom are the government of the province and the nation. What do you think of this item?” “That is as it may happen,” said Molyneux, reflectively. “In the next place, you know I love pleasure, but I have renounced pleasure for ten years. What is that worth?” “No great matter: you have made politics your amusement.” A hearty laugh. “In the next place, I have ruined as fine health as nature ever gave to man.” “That is melancholy indeed; there is nothing to be said on that point,” Molyneux replied. “Once more,” continued Otis, holding down his head before Molyneux, “look upon this head!” (there was a deep, half-closed scar, in which a man might lay his finger)—“and, what is worse my friends think I have a monstrous crack in my skull.” This made all the company look grave, and had the desired effect of making Molyneux who was really a good companion, heartily ashamed of his childish complaints.
- 66. Larger Image Another old inn of assured celebrity was the Cromwell’s Head, in School Street. This was a two-story wooden building of venerable appearance, conspicuously displaying over the footway a grim likeness of the Lord Protector, it is said much to the disgust of the ultra royalists, who, rather than pass underneath it, habitually took the other side of the way. Indeed, some of the hot-headed Tories were for serving Cromwell’s Head as that man of might had served their martyr king’s. So, when the town came under martial law, mine host Brackett, whose family kept the house for half a century or more, had to take down his sign, and conceal it until such time as
- 67. the “British hirelings” should have made their inglorious exit from the town. After Braddock’s crushing defeat in the West, a young Virginian colonel, named George Washington, was sent by Governor Dinwiddie to confer with Governor Shirley, who was the great war governor of his day, as Andrew was of our own, with the difference that Shirley then had the general direction of military affairs, from the Ohio to the St. Lawrence, pretty much in his own hands. Colonel Washington took up his quarters at Brackett’s, little imagining, perhaps, that twenty years later he would enter Boston at the head of a victorious republican army, after having quartered his troops in Governor Shirley’s splendid mansion. Major-General the Marquis Chastellux, of Rochambeau’s auxiliary army, also lodged at the Cromwell’s Head when he was in Boston in 1782. He met there the renowned Paul Jones, whose excessive vanity led him to read to the company in the coffee-room some verses composed in his own honor, it is said, by Lady Craven.
- 68. From the tavern of the gentry we pass on to the tavern of the mechanics, and of the class which Abraham Lincoln has forever distinguished by the title of the common people. Among such houses the Salutation, which stood at the junction of Salutation with North Street, is deserving of a conspicuous place. Its vicinity to the shipyards secured for it the custom of the sturdy North End shipwrights, caulkers, gravers, sparmakers, and the like,—a numerous body, who, while patriots to the backbone, were also quite clannish and independent in their feelings and views, and consequently had to be managed with due regard to their class prejudices, as in politics they always went in a body. Shrewd politicians, like Samuel Adams, understood this. Governor Phips owed his elevation to it. As a body, therefore, these mechanics were extremely formidable, whether at the polls or in carrying out the plans of their leaders. To their meetings the origin of the word caucus is usually referred, the word itself undoubtedly having come into familiar use as a short way of saying caulkers’ meetings. The Salutation became the point of fusion between leading Whig politicians and the shipwrights. More than sixty influential mechanics attended the first meeting, called in 1772, at which Dr. Warren drew up a code of by-laws. Some leading mechanic, however, was always chosen to be the moderator. The “caucus,” as it began to be called, continued to meet in this place until after the destruction of the tea, when, for greater secrecy, it became advisable to transfer the sittings to another place, and then the Green Dragon, in Union Street, was selected. The Salutation had a sign of the sort that is said to tickle the popular fancy for what is quaint or humorous. It represented two citizens, with hands extended, bowing and scraping to each other in the most approved fashion. So the North-Enders nicknamed it “The Two Palaverers,” by which name it was most commonly known. This house, also, was a reminiscence of the Salutation in Newgate Street, London, which was the favorite haunt of Lamb and Coleridge.
- 69. The Green Dragon will probably outlive all its contemporaries in the popular estimation. In the first place a mural tablet, with a dragon sculptured in relief, has been set in the wall of the building that now stands upon some part of the old tavern site. It is the only one of the old inns to be so distinguished. Its sign was the fabled dragon, in hammered metal, projecting out above the door, and was probably the counterpart of the Green Dragon in Bishopsgate Street, London. THE GREEN DRAGON TAVERN As a public house this one goes back to 1712, when Richard Pullen kept it; and we also find it noticed, in 1715, as a place for entering horses to be run for a piece of plate of the value of twenty-five pounds. In passing, we may as well mention the fact that Revere Beach was the favorite race-ground of that day. The house was well situated for intercepting travel to and from the northern counties.
- 70. THE GREEN DRAGON. To resume the historical connection between the Salutation and Green Dragon, its worthy successor, it appears that Dr. Warren continued to be the commanding figure after the change of location; and, if he was not already the popular idol, he certainly came little short of it, for everything pointed to him as the coming leader whom the exigency should raise up. Samuel Adams was popular in a different way. He was cool, far-sighted, and persistent, but he certainly lacked the magnetic quality. Warren was much younger, far more impetuous and aggressive,—in short, he possessed all the more brilliant qualities for leadership which Adams lacked. Moreover, he was a fluent and effective speaker, of graceful person, handsome, affable, with frank and winning manners, all of which added no little to his popularity. Adams inspired respect, Warren confidence. As Adams himself said, he belonged to the “cabinet,” while Warren’s whole make-up as clearly marked him for the field. In all the local events preliminary to our revolutionary struggle, this Green Dragon section or junto constituted an active and positive force. It represented the muscle of the Revolution. Every member was sworn to secrecy, and of them all one only proved recreant to his oath.
- 71. These were the men who gave the alarm on the eve of the battle of Lexington, who spirited away cannon under General Gage’s nose, and who in so many instances gallantly fought in the ranks of the republican army. Wanting a man whom he could fully trust, Warren early singled out Paul Revere for the most important services. He found him as true as steel. A peculiar kind of friendship seems to have sprung up between the two, owing, perhaps, to the same daring spirit common to both. So when Warren sent word to Revere that he must instantly ride to Lexington or all would be lost, he knew that, if it lay in the power of man to do it, the thing would be done. Besides the more noted of the tavern clubs there were numerous private coteries, some exclusively composed of politicians, others more resembling our modern debating societies than anything else. These clubs usually met at the houses of the members themselves, so exerting a silent influence on the body politic. The non- importation agreement originated at a private club in 1773. But all were not on the patriot side. The crown had equally zealous supporters, who met and talked the situation over without any of the secrecy which prudence counselled the other side to use in regard to their proceedings. Some associations endeavored to hold the balance between the factions by standing neutral. They deprecated the encroachments of the mother-country, but favored passive obedience. Dryden has described them: “Not Whigs nor Tories they, nor this nor that, Nor birds nor beasts, but just a kind of bat,— A twilight animal, true to neither cause, With Tory wings but Whiggish teeth and claws.” It should be mentioned that Gridley, the father of the Boston Bar, undertook, in 1765, to organize a law club, with the purpose of making head against Otis, Thatcher, and Auchmuty. John Adams and Fitch were Gridley’s best men. They met first at Ballard’s, and subsequently at each other’s chambers; their “sodality,” as they called it, being for professional study and advancement. Gridley, it appears, was a little jealous of his old pupil, Otis, who had beaten
- 72. him in the famous argument on the Writs of Assistance. Mention is also made of a club of which Daniel Leonard (Massachusettensis), John Lowell, Elisha Hutchinson, Frank Dana, and Josiah Quincy were members. Similar clubs also existed in most of the principal towns in New England. The Sons of Liberty adopted the name given by Colonel Barré to the enemies of passive obedience in America. They met in the counting-room of Chase and Speakman’s distillery, near Liberty Tree. [3] Mackintosh, the man who led the mob in the Stamp Act riots, is doubtless the same person who assisted in throwing the tea overboard. We hear no more of him after this. The “Sons” were an eminently democratic organization, as few except mechanics were members. Among them were men like Avery, Crafts, and Edes the printer. All attained more or less prominence. Edes continued to print the Boston Gazette long after the Revolution. During Bernard’s administration he was offered the whole of the government printing, if he would stop his opposition to the measures of the crown. He refused the bribe, and his paper was the only one printed in America without a stamp, in direct violation of an Act of Parliament. The “Sons” pursued their measures with such vigor as to create much alarm among the loyalists, on whom the Stamp Act riots had made a lasting impression. Samuel Adams is thought to have influenced their proceedings more than any other of the leaders. It was by no means a league of ascetics, who had resolved to mortify the flesh, as punch and tobacco were liberally used to stimulate the deliberations.
- 73. THE LIBERTY TREE No important political association outlived the beginning of hostilities. All the leaders were engaged in the military or civil service on one or the other side. Of the circle that met at the Merchants’ three were members of the Philadelphia Congress of 1774, one was president of the Provincial Congress of Massachusetts, the career of two was closed by death, and that of Otis by insanity.
- 75. IV. SIGNBOARD HUMOR. nother tavern sign, though of later date, was that of the Good Woman, at the North End. This Good Woman was painted without a head. Still another board had painted on it a bird, a tree, a ship, and a foaming can, with the legend,— “This is the bird that never flew, This is the tree which never grew, This is the ship which never sails, This is the can which never fails.”
- 76. The Dog and Pot, Turk’s Head, Tun and Bacchus, were also old and favorite emblems. Some of the houses which swung these signs were very quaint specimens of our early architecture. So, also, the signs themselves were not unfrequently the work of good artists. Smibert or Copley may have painted some of them. West once offered five hundred dollars for a red lion he had painted for a tavern sign. DOG AND POT. Not a few boards displayed a good deal of ingenuity and mother-wit, which was not without its effect, especially upon thirsty Jack, who could hardly be expected to resist such an appeal as this one of the Ship in Distress: “With sorrows I am compass’d round; Pray lend a hand, my ship’s aground.” We hear of another signboard hanging out at the extreme South End of the town, on which was depicted a globe with a man breaking through the crust, like a chicken from its shell. The man’s nakedness was supposed to betoken extreme poverty.
- 77. Welcome to our website – the perfect destination for book lovers and knowledge seekers. We believe that every book holds a new world, offering opportunities for learning, discovery, and personal growth. That’s why we are dedicated to bringing you a diverse collection of books, ranging from classic literature and specialized publications to self-development guides and children's books. More than just a book-buying platform, we strive to be a bridge connecting you with timeless cultural and intellectual values. With an elegant, user-friendly interface and a smart search system, you can quickly find the books that best suit your interests. Additionally, our special promotions and home delivery services help you save time and fully enjoy the joy of reading. Join us on a journey of knowledge exploration, passion nurturing, and personal growth every day! ebookbell.com