SDN Software Defined Networks 1st Edition Thomas Nadeau D.

SDN Software Defined Networks 1st Edition Thomas
Nadeau D. download
https://ebookgate.com/product/sdn-software-defined-networks-1st-
edition-thomas-nadeau-d/
Get Instant Ebook Downloads – Browse at https://ebookgate.com

Get Your Digital Files Instantly: PDF, ePub, MOBI and More
Quick Digital Downloads: PDF, ePub, MOBI and Other Formats
Autonomous Software Defined Radio Receivers for Deep
Space Applications 1st Edition Jon Hamkins
https://ebookgate.com/product/autonomous-software-defined-radio-
receivers-for-deep-space-applications-1st-edition-jon-hamkins/
Instrument engineers handbook Process Software and
Digital Networks 4th ed Edition Eren
https://ebookgate.com/product/instrument-engineers-handbook-
process-software-and-digital-networks-4th-ed-edition-eren/
Democracy Defined The Manifesto 2nd Edition Kenn
D'Oudney
https://ebookgate.com/product/democracy-defined-the-
manifesto-2nd-edition-kenn-doudney/
Social Networks and Health Models Methods and
Applications 1st Edition Thomas W. Valente
https://ebookgate.com/product/social-networks-and-health-models-
methods-and-applications-1st-edition-thomas-w-valente/

New Cancer Research Developments 1st Edition Thomas D.
Ford
https://ebookgate.com/product/new-cancer-research-
developments-1st-edition-thomas-d-ford/
Variation and Reconstruction 1st Edition Thomas D.
Cravens (Ed.)
https://ebookgate.com/product/variation-and-reconstruction-1st-
edition-thomas-d-cravens-ed/
Mental Illness Defined Continuums Regulation and
Defense 1st Edition Brad Bowins
https://ebookgate.com/product/mental-illness-defined-continuums-
regulation-and-defense-1st-edition-brad-bowins/
Thomas Calculus with Differential Equations 11th
Edition Maurice D. Weir
https://ebookgate.com/product/thomas-calculus-with-differential-
equations-11th-edition-maurice-d-weir/
The Revenge of Thomas Eakins First Edition Sidney D.
Kirkpatrick
https://ebookgate.com/product/the-revenge-of-thomas-eakins-first-
edition-sidney-d-kirkpatrick/

Thomas D. Nadeau and Ken Gray
SDN: Software Defined Networks

SDN: Software Defined Networks
by Thomas D. Nadeau and Ken Gray
Copyright © 2013 Thomas D. Nadeau, Ken Gray. All rights reserved.
Printed in the United States of America.
Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472.
O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are
alsoavailableformosttitles(http://my.safaribooksonline.com).Formoreinformation,contactourcorporate/
institutional sales department: 800-998-9938 or corporate@oreilly.com.
Editors: Mike Loukides and Meghan Blanchette
Production Editor: Kristen Borg
Copyeditor: Jasmine Kwityn
Proofreader: Amanda Kersey
Indexer: Judith McConville
Cover Designer: Karen Montgomery
Interior Designer: David Futato
Illustrator: Rebecca Demarest and Kara Ebrahim
August 2013: First Edition
Revision History for the First Edition:
2013-08-07: First release
See http://oreilly.com/catalog/errata.csp?isbn=9781449342302 for release details.
Nutshell Handbook, the Nutshell Handbook logo, and the O’Reilly logo are registered trademarks of O’Reilly
Media, Inc. SDN: Software Defined Networks, the image of a goosander duck, and related trade dress are
trademarks of O’Reilly Media, Inc.
Many of the designations used by manufacturers and sellers to distinguish their products are claimed as
trademarks. Where those designations appear in this book, and O’Reilly Media, Inc., was aware of a trade‐
mark claim, the designations have been printed in caps or initial caps.
While every precaution has been taken in the preparation of this book, the publisher and authors assume
no responsibility for errors or omissions, or for damages resulting from the use of the information contained
herein.
ISBN: 978-1-449-34230-2
[LSI]

Table of Contents
Foreword by David Meyer. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix
Foreword by David Ward. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xi
Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xvii
1. Introduction. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1
2. Centralized and Distributed Control and Data Planes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
Introduction 9
Evolution versus Revolution 10
What Do They Do? 11
The Control Plane 11
Data Plane 16
Moving Information Between Planes 18
Why Can Separation Be Important? 20
Distributed Control Planes 28
IP and MPLS 29
Creating the IP Underlay 30
Convergence Time 32
Load Balancing 33
High Availability 34
Creating the MPLS Overlay 34
Replication 37
Centralized Control Planes 37
Logical Versus Literal 38
ATM/LANE 39
Route Servers 42
Conclusions 44
3. OpenFlow. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
iii

Introduction 47
Wire Protocol 50
Replication 53
FAWG (Forwarding Abstraction Workgroup) 54
Config and Extensibility 57
Architecture 62
Hybrid Approaches 63
Ships in the Night 64
Dual Function Switches 65
Conclusions 69
4. SDN Controllers. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
Introduction 71
General Concepts 72
VMware 75
Nicira 79
VMware/Nicira 83
OpenFlow-Related 83
Mininet 85
NOX/POX 87
Trema 89
Ryu 92
Big Switch Networks/Floodlight 93
Layer 3 Centric 95
L3VPN 96
Path Computation Element Server 101
Plexxi 109
Plexxi Affinity 111
Cisco OnePK 111
Relationship to the Idealized SDN Framework 113
Conclusions 113
5. Network Programmability. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
Introduction 117
The Management Interface 118
The Application-Network Divide 118
The Command-Line Interface 122
NETCONF and NETMOD 124
SNMP 126
Modern Programmatic Interfaces 132
Publish and Subscribe Interfaces 132
XMPP 135
iv | Table of Contents

Google’s Protocol Buffers 137
Thrift 140
JSON 142
I2RS 143
Modern Orchestration 146
OpenStack 147
CloudStack 151
Puppet 153
Conclusions 156
6. Data Center Concepts and Constructs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
Introduction 157
The Multitenant Data Center 160
The Virtualized Multitenant Data Center 163
Orchestration 167
Connecting a Tenant to the Internet/VPN 168
Virtual Machine Migration and Elasticity 169
Data Center Interconnect (DCI) 175
Fallacies of Data Center Distributed Computing 176
Data Center Distributed Computing Pitfalls to Consider 177
SDN Solutions for the Data Center Network 184
The Network Underlay 185
VLANs 186
EVPN 188
Locator ID Split (LISP) 191
VxLan 192
NVGRE 195
OpenFlow 197
Network Overlays 199
Network Overlay Types 201
Conclusions 205
7. Network Function Virtualization. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207
Introduction 207
Virtualization and Data Plane I/O 208
Data Plane I/O 210
I/O Summary 213
Services Engineered Path 214
Service Locations and Chaining 217
Metadata 219
An Application Level Approach 220
Scale 222
Table of Contents | v

NFV at ETSI 223
Non-ETSI NFV Work 228
Middlebox Studies 229
Embrane/LineRate 231
Platform Virtualization 233
Conclusions 238
8. Network Topology and Topological Information Abstraction. . . . . . . . . . . . . . . . . . . . . 241
Introduction 241
Network Topology 242
Traditional Methods 244
LLDP 248
BGP-TE/LS 252
BGP-LS with PCE 253
ALTO 254
BGP-LS and PCE Interaction with ALTO 255
I2RS Topology 256
Conclusions 259
9. Building an SDN Framework. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261
Introduction 261
Build Code First; Ask Questions Later... 262
The Juniper SDN Framework 265
IETF SDN Framework(s) 268
SDN(P) 268
ABNO 270
Open Daylight Controller/Framework 271
API 274
High Availability and State Storage 275
Analytics 276
Policy 279
Conclusions 279
10. Use Cases for Bandwidth Scheduling, Manipulation, and Calendaring. . . . . . . . . . . . . 281
Introduction 281
Bandwidth Calendaring 284
Base Topology and Fundamental Concepts 285
OpenFlow and PCE Topologies 286
Example Configuration 287
OpenFlow Provisioned Example 287
Enhancing the Controller 289
Overlay Example Using PCE Provisioning 290
vi | Table of Contents

Expanding Your Reach: Barbarians at the Gate 294
Big Data and Application Hyper-Virtualization for Instant CSPF 295
Expanding Topology 297
Conclusions 298
11. Use Cases for Data Center Overlays, Big Data, and Network Function Virtualization. . 299
Introduction 299
Data Center Orchestration 299
Creating Tenant and Virtual Machine State 302
Forwarding State 304
Data-Driven Learning 305
Control-Plane Signaling 306
Scaling and Performance Considerations 306
Puppet (DevOps Solution) 308
Network Function Virtualization (NFV) 311
NFV in Mobility 312
Optimized Big Data 315
Conclusions 319
12. Use Cases for Input Traffic Monitoring, Classification, and Triggered Actions. . . . . . . . 321
Introduction 321
The Firewall 321
Firewalls as a Service 324
Network Access Control Replacement 326
Extending the Use Case with a Virtual Firewall 330
Feedback and Optimization 333
Intrusion Detection/Threat Mitigation 333
Conclusions 335
13. Final Thoughts and Conclusions. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 337
What Is True About SDN? 337
Economics 339
SDN Is Really About Operations and Management 340
Multiple Definitions of SDN 341
Are We Making Progress Yet? 342
Index. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 345
Table of Contents | vii

Foreword by David Meyer
Although the ideas underlying software-defined networking (SDN) have only recently
come into the public consciousness, a few of us who are active in the research, operator,
and vendor communities immediately saw the applicability of SDN-like techniques to
data center and service provider environments (and beyond). In addition to the explo‐
sion of innovative thinking going on in the research community, we also saw SDN as a
programmatic way to optimize, monetize, and scale networks of all kinds.
In 2011, the first organization dedicated to the growth and success of SDN began with
the Open Networking Foundation (ONF). Among its stated missions was to evolve the
OpenFlow protocol from its academic roots to a commercially viable substrate for
building networks and networking products. Within two years, the ONF’s membership
had grown to approximately 100 entities, representing the diverse interest and expect‐
ationsforSDN.Againstthisbackdrop,manyofuswerelookingatthewiderimplications
of the ideas underlying SDN, and in the process, generalized SDN to include not only
OpenFlow but other forms of network programmability as well.
Early on in this process, both Tom Nadeau and Ken Gray realized that SDN was really
about general network programmability and the associated interfaces, protocols, data
models, and APIs. Using this insight, they helped to organize the SDN Birds of a Feather
session at IETF 82, in Taipei, to investigate this more general SDN model. At that meet‐
ing, Tom presented a framework for software-defined networks that envisioned SDN
as a generalized mechanism for network programmability. This work encouraged the
community to take a more general view of SDN and eventually led to the formation of
the Interface to the Routing System Working Group in the IETF.
Since that time, in addition to their many contributions to Internet technologies, Tom
and Ken have become well-respected senior members of the SDN community. They are
activeparticipantsinthecoreSDNindustryactivitiesanddevelopproductsfortheSDN
market. Some of the key industry activities that Tom and Ken drive include the ONF,
IETF, ETSI, industry events such as SDN Summit 2012/2013, as well as open source
consortia such as the Open Daylight Project. This book draws on their deep
ix

understanding and experience in the field and offers a unique perspective on SDN. It
will help you understand not only the technology but also how it is being developed,
standardized, and deployed.
Tom and Ken are eminently qualified to give you a lucid understanding of the technol‐
ogy and the common-sense use and deployment of network programmability techni‐
ques. In particular, their book is an excellent and practical introduction to the
fundamentals of SDN and is filled with innumerable anecdotes explaining the ideas and
the background behind the development of SDN. So if you are interested in writing
SDN applications, building SDN capable networks, or just understanding what SDN is,
this book is for you!
—David Meyer
CTO and Chief Scientist, Brocade Communications
x | Foreword by David Meyer

Foreword by David Ward
Technological shifts that affect how developers and engineers build and design their
business architectures are monumental. These shifts are not applicable to Moore’s law
and tend to be transformations that affect not only the IT landscape but the business
landscape as well. These shifts tend to occur every 8 to 10 years and have a long-lasting
impact on how people build, consume, and distribute technologies. They also force
people to frame their business opportunities in new ways.
In 1996, Gartner coined the term “service-oriented architecture.” By 2000, it had taken
centerstagewiththecorepurposeofallowingfortheeasycooperationofalargenumber
of computers connected over a network to exchange information via services without
human interaction. There was no need to make underlying changes to the program or
application itself. Essentially, it took on the same role as a single operating system on
one machine and applied it to the entire infrastructure of servers, allowing for more
usable, flexible, and scalable applications and services to be built, tested, deployed, and
managed. It introduced web services as the de facto way to make functional building
blocks accessible over standard Internet protocols independent of platforms and lan‐
guages—allowing for faster and easier development, testing, deployment, and manage‐
ability of IT infrastructures. SOA drastically changed the way developers, their man‐
agers, and the business looked at technology.
When you look at software-defined networking, you see similarities. The network is the
cornerstone of IT in that it can enable new architectures that in turn create new business
opportunities.Inessence,itallowsITtobecomemorerelevantthaneverandtheenabler
of new business. The network is now the largest business enabler if architected and
utilized in the correct way—allowing for the network, server, and storage to be tied
together to enable the principles of SOA to be executed at the network layer. SDN and
APIs to the network change the accessibility to programming intent and receiving state
from the network and services, thus overcoming the traditional view that the network
has to be built and run by magicians. However, when SOA principles become applied
to the networking layer, the network becomes more accessible, programmable, and
xi

flexible, allowing organizations to actually shift IT at the speed that the business moves,
all while adding increased value to the business in new ways.
But what is a software-defined network? There are many camps that have varying def‐
initions. When broken down into simple terms, it needs to be looked at as an approach
or architecture to not only simplify your network but also to make it more reactive to
the requirements of workloads and services placed in the network. IT infrastructure
needs to move at the speed of business opportunities and must enable new ways to do
business quickly, flexibly, and faster than before. A pragmatic definition is this: SDN
functionally enables the network to be accessed by operators programmatically, allow‐
ing for automated management and orchestration techniques; application of configu‐
ration policy across multiple routers, switches, and servers; and the decoupling of the
application that performs these operations from the network device’s operating system.
As SDN becomes increasingly the buzzword of multiple industries, it’s worthwhile to
take a look at why SDN came about. Historically, network configuration state has re‐
mained largely static, unchanged, and commonly untouchable. Manual configuration
and CLI-based configuration on a device-by-device basis was the norm, and network
management constituted the basic “screen scraping” or use of Expect scripts as a way
to solve manageability problems and core scalability issues (cut-and-paste methodol‐
ogy). The highest end of programmatic interfaces included XML interfaces and on-
board Perl, Tk/Tcl, and Expect. However, when you’re dealing with multiple routers,
switches, and servers working as a system (and services that are routing traffic across
multiple domains with different users, permissions, and policies), control and man‐
agement state needs to be applied across the network as an operation. Element-by-
element management simply doesn’t provide enough flexibility and agility or the notion
ofdynamicorephemeraldata(configurationandstatenotpersistentlyheldintheconfig
file). But as service-oriented architecture principles started to shift southbound down
the stack and the realization of their application at the networking layer was recognized,
new architectures—coupled with advancements in networking—allowed for software-
defined networking to emerge and users to realize the power that the network was
capable of in new ways.
Yes, it’s true that there is a history of protocol interfaces to routers, switches, servers,
gateways, and so on. Decades of deployment of the current Internet that program dy‐
namic data associated with subscribers, sessions, and applications does currently exist
and is widely deployed. These protocol servers (e.g., Radius, Diameter, PCMM, COPS,
3GPP) all could be considered early forms of SDN, so why aren’t they? What’s a bit
different now is that one major functionality of the SDN architecture is the ability to
write applications on top of a platform that customizes data from different sources or
data bases into one network-wide operation.
SDN is also an architecture that allows for a centrally managed and distributed control,
management, and data plane, where policy that dictates the forwarding rules is
xii | Foreword by David Ward

centralized, while the actual forwarding rule processing is distributed among multiple
devices. In this model, application policy calculation (e.g., QoS, access control lists, and
tunnel creation) happens locally in real time and the quality, security, and monitoring
of policies are managed centrally and then pushed to the switching/routing nodes. This
allows for more flexibility, control, and scalability of the network itself, and the use of
templates,variables,multipledatabasesofusers,andpoliciesallworkingincombination
to derive or compile the desired configuration and state to be downloaded to the routers
and switches. What’s key to understand is that SDN doesn’t replace the control plane
on the router or switch. It augments them. How? By having a view of the entire network
all at once versus only from one position in the topology (e.g., the router or switch).
The marriage of dynamic routing and signaling and a centralized view is incredibly
powerful. It enables the fastest possible protection in the event of a failure, the greatest
resiliency, and the ability to place services into a network in one command. The two
technologies working together are really a major step forward that wasn’t previously in
our toolbox.
There are a few variations on the SDN theme and some oft spoken components to be
considered. OpenFlow is one, which architecturally separates the control and manage‐
ment planes from the data plane on the networking device. This allows for a centralized
controller to manage the flows in the forwarding nodes. However, OpenFlow is only
one protocol and one element of SDN. There are many other protocols now. Some
examplesincludeI2RS,PCE-P,BGP-LS,FORCES,OMI,andNetConf/Yang.Allofthese
are also open standards. What’s important to remember is that SDN is not a protocol;
it’s an operational and programming architecture.
What do we get from SDN? The architecture brings the network and networking data
closer to the application layer and the applications closer to the networking layer. As
practicedinSOA,nolongeristheretheneedforahumanelementorscriptinglanguages
to act as humans to distribute data and information bidirectionally because APIs and
tooling now have evolved in a way that this can be delivered in a secure and scalable
way via open interfaces and interoperability. The data in the network (e.g., stats, state,
subscriber info, service state, security, peering, etc.) can be analyzed and used by an
application to create policy intent and program the network into a new configuration.
It can be programmed this way persistently or only ephemerally.
Programmability (i.e., the ability to access the network via APIs and open interfaces) is
central to SDN. The notion of removing the control and management planes to an off-
switch/router application connected to the networking device by SDN protocols is
equally important. This off-box application is really what software developers would
call a “platform,” as it has its own set of APIs, logic, and the ability for an application to
make requests to the network, receive events, and speak the SDN protocols. What’s key
here is that programmers don’t need to know the SDN protocols because they write to
the controller’s APIs. Programmers don’t need to know the different configuration syn‐
tax or semantics of different networking devices because they program to a set of APIs
Foreword by David Ward | xiii

on the controller that can speak to many different devices. Different vendors, eras of
equipment, and classes of equipment (e.g., transport, simple switches, wireless base
stations, subscriber termination gateways, peering routers, core routers, and servers)
all are on the trajectory to be able to be programmed by the SDN protocols that plug
into the bottom of the controller. The programmer only uses the APIs on the top of the
controller to automate, orchestrate, and operate the network. This doesn’t necessarily
mean there is a grand unification theory of controllers and one to serve all layers and
functions of networking, but what it does mean is that the network now has been ab‐
stracted and is being programmed off box. Thus, when integrated into an IaaS (Infra‐
structureasaService)layerinastack,OSS,orITsystem,thenetworkisbeingautomated
and orchestrated as fast as users log onto the net and as fast as workloads are being spun
up on servers.
The use of new tooling practices typically utilized by system administrators and new
available to network operators are related to the whole SDN movement. Tools such as
Puppet, Chef, CFEngine, and others are being used to automate and orchestrate the
network in new ways as plug-ins can now be created to utilize the network data via the
open interfaces of the network. Controller APIs also allow for easier and faster ways to
build and apply policy across the network in multiple languages and with integration
into existing tools such as IDEs (NetBeans, Eclipse, et al.). This allows for a better user
experience for network engineers versus the traditionally used CLI model.
Before we dig into examples, it’s important to understand what SDN actually solves and
why there is a shift to this particular architecture. As networks evolve and new services
are deployed, it’s critical to implement new ways for users to more easily provision and
orchestrate network resources in real time. By implementing this, cost can be reduced
bytheautomationofmovingresourcesaroundfasterandmorereliably,andbyallowing
thenetworktoresponddirectlytoarequestfromanapplication(versustheintervention
by a human). This allows for operators to use programmatic (scalable) control versus
manual to create and apply these services in a way that is simpler than a command-line
interface. Additionally, it enables the ability to utilize new resources from the network
(user data, traffic path information, etc.) and create new types of applications that can
control policy for the network in a scalable fashion. It also allows for the optimization
of infrastructure, services, and applications by allowing for new network data and ca‐
pabilitiestobeextendedandappliedintotheaforementionedarchitecture,creatingnew
ways to not only optimize existing applications but also to insert new services or offer‐
ings that can provide a better user experience or create a new offering or advanced
feature that could be monetized.
As SDN evolves, it’s important to look at some implementations to understand why it’s
so critical for multiple industries (e.g., video delivery, user services and mobile, cable
and broadband, security, and provider edge) to embrace. Where SDN reaches its po‐
tential, however, is when you look at it for not just programming the network functions
and scaling those across your infrastructure, but also for actually tying server, storage,
xiv | Foreword by David Ward

and the network together for new use cases. In this case, systems can actually interact
with each other, allowing for more infrastructure flexibility, whether physical, virtual,
or hybrid.
Traffic policy and rerouting based on network conditions and/or regulation shifts are
also common applications, as are the insertion of new services or data into applications
that may be able to more clearly prioritize bandwidth for a user that pays a premium
amount for faster connection speeds. When you apply SDN and a centralized manage‐
ment plane that is separate from the data plane, you can more quickly make decisions
on where data traffic can be rerouted, as this can occur programmatically with software
interfaces (APIs), versus on-the-box CLI methodology.
One advanced use case is the hybrid cloud. In this case, an application may run in a
private cloud or data center yet utilize the public cloud when the demand for computing
capacity spikes or cost can be reduced. Historically, cloud bursting was typically used
only in environments with non-mission critical applications or services, but with the
network tie-in and software principles applied, the use case shifts. Applications now
remain in compliance with the IT organizations’ policies and regulations. The applica‐
tion can also retain its dependency model if it is reliant on different data or information
that it typically has on premises versus off, or in the public cloud environment. It also
allows for the application to run across different platforms regardless of where the ap‐
plication was built.
As we look at SDN, we must also consider Network Functions Virtualization and how
this ties into the broader infrastructure and virtualization picture. The transition from
physical to virtual is one that is leading many of these changes in the industry. By tying
the hardware (physical) to software (virtual), including network, server, and storage,
there’s the opportunity to virtualize network services and have them orchestrated as fast
as any other workload. Tie this via programmatic interfaces to the WAN, and you can
absolutely guarantee service delivery. SDN coupled with NFV is a pivotal architectural
shift in both computing and networking. This shift is marked by dynamic changes to
infrastructure to closely match customer demand, analytics to assist in predicting per‐
formance requirements, and a set of management and orchestration tools that allow
network functions and applications to scale up, down, and out with greater speed and
less manual intervention. This change affects how we build cloud platforms for appli‐
cations and at the most basic level must provide the tools and techniques that allow the
network to respond to changing workload requirements as quickly as the platforms that
leverage them. It also allows workload requirements to include network requirements
and have them satisfied.
It’s important to note that not all networks are the same, and that’s why it’s critical to
understand the importance of the underlying infrastructure when abstracting control
from the network—either from physical or virtual devices. Network Functions Virtu‐
alization is simply the addition of virtual or off-premises devices to augment traditional
Foreword by David Ward | xv

infrastructure. However, the tie to both the on- and off-premises offerings must be
considered when running applications and services to ensure a seamless experience not
just for the organization running the applications or services but also for the consumer
of the services (whether they be enterprise and in-house users or external customers).
So why should you care? From a technical perspective, SDN allows for more flexibility
and agility as well as options for your infrastructure. By allowing data to be controlled
centrally and tied into not just the network, but also the storage and server, you get a
morecohesiveviewonperformance,speed,trafficoptimization,andserviceguarantees.
With programmatic interfaces (APIs) that can be exposed in multiple languages and
utilized with tools, your operators and administrators can more quickly respond to the
demand of the business side of the house or external customer needs. They can now
apply policies for other development organizations in-house to allow them network
data to more effectively spin up server farms or even build applications with network
intelligence built in for faster, better performing applications. By allowing for the data
to be exposed in a secure and scalable way, the entire IT organization benefits, and with
faster development and deployment cycles and easier delivery of new services, so too
does the business. The promise that SOA gave developers—write once, run anywhere
—can now be fully realized with the underlying network’s ability to distribute infor‐
mation across the enterprise, access, WAN, and data center (both physical and virtual).
This allows for applications to break free from the boundaries of the OSS and manage‐
mentplatformsthathadpreviouslylimitedtheirabilitytorunindifferentenvironments.
The IT industry is going through a massive shift that will revolutionize the way users
build,test,deploy,andmonetizetheirapplications.WithSDN,thenetworkisnowcloser
to applications (and vice versa), allowing for a new breed of smarter, faster, and better
performing applications. It enables the network to be automated in new ways, providing
more flexibility and scalability for users, and unleashes the potential for business cost
savings and revenue-generating opportunities. It’s a new era in networking and the IT
industry overall, and it will be a game-changing one. Check out this book—it’s required
reading.
—David Ward
CTO, Cisco Systems
xvi | Foreword by David Ward

1. The real answer is that one of the authors has a fondness for ducks, as he raises Muscovy Ducks on his family
farm.
Preface
The first question most readers of an O’Reilly book might ask is about the choice of the
cover animal. In this case, “why a duck?” Well, for the record, our first choice was a
unicorn decked out in glitter and a rainbow sash.
That response always gets a laugh (we are sure you just giggled a little), but it also brings
to the surface a common perception of software-defined networks among many expe‐
riencednetworkprofessionals.Althoughwethinkthereissometruthtothisperception,
there is certainly more meat than myth to this unicorn.
So, starting over, the better answer to that first question is that the movement of a
duck1
is not just what one sees on the water; most of the action is under the water, which
xvii

2. http://www.gartner.com/technology/research/methodologies/hype-cycle.jsp
you can’t easily see. Under the waterline, some very muscular feet are paddling away to
move that duck along. In many ways, this is analogous to the progress of software-
defined networks.
The surface view of SDN might lead the casual observer to conclude a few things. First,
defining what SDN is, or might be, is something many organizations are frantically
trying to do in order to resuscitate their business plans or revive their standards-
developing organizations (SDOs). Second, that SDN is all about the active rebranding
of existing products to be this mythical thing that they are not. Many have claimed that
products they built four or five years ago were the origins of SDN, and therefore ev‐
erything they have done since is SDN, too.
Along these lines, the branding of seemingly everything anew as SDN and the expected
hyperbole of the startup community that SDN has been spawning for the past three or
four years have also contributed negatively toward this end.
If observers are predisposed by their respective network religions and politics to dismiss
SDN, it may seem like SDN is an idea adrift.
Now go ahead and arm yourself with a quick pointer to the Gartner hype-cycle.2
We
understand that perspective and can see where that cycle predicts things are at.
Some of these same aspects of the present SDN movement made us lobby hard for the
glitter-horned unicorn just to make a point—that we see things differently.
For more than two years, our involvement in various customer meetings, forums, con‐
sortia, and SDOs discussing the topic, as well as our work with many of the startups,
converts, and early adopters in the SDN space, leads us to believe that something worth
noting is going on under the waterline. This is where much of the real work is going on
to push the SDN effort forward toward a goal of what we think is optimal operational
efficiency and flexibility for networks and applications that utilize those networks.
There is real evidence that SDN has finally started a new dialogue about network pro‐
grammability, control models, the modernization of application interfaces to the net‐
work, and true openness around these things.
In that light, SDN is not constrained to a single network domain such as the data center
—although it is true that the tidal wave of manageable network endpoints hatched via
virtualizationisaprimemoverofSDNatpresent.SDNisalsonotconstrainedtoasingle
customer type (e.g., research/education), a single application (e.g., data center orches‐
tration), or even a single protocol/architecture (e.g., OpenFlow). Nor is SDN constrain‐
ed to a single architectural model (e.g., the canonical model of a centralized controller
and a group of droid switches). We hope you see that in this book.
xviii | Preface

At the time of writing of the first edition of this book, both Thomas Nadeau and Ken
Gray work at Juniper Networks in the Platform Systems Division Chief Technologist’s
Office. We both also have extensive experience that spans roles both with other vendors,
such as Cisco Systems, and service providers, such as BT and Bell Atlantic (now Veri‐
zon). We have tried our best to be inclusive of everyone that is relevant in the SDN space
without being encyclopedic on the topic still providing enough breadth of material to
cover the space. In some cases, we have relied on references or examples that came from
our experiences with our most recent employer (Juniper Networks) in the text, only
because they are either part of a larger survey or because alternative examples on the
topic are net yet freely available for us to divulge. We hope the reader finds any bias to
be accidental and not distracting or overwhelming. If this can be corrected or enhanced
in a subsequent revision, we will do so. We both agree that there are likely to be many
updates to this text going forward, given how young SDN still is and how rapidly it
continues to evolve.
Finally, we hope the reader finds the depth and breadth of information presented herein
tobeinterestingandinformative,whileatthesametimeevocative.Wegiveouropinions
about topics, but only after presenting the material and its pros and cons in as unbiased
a manner as possible.
We do hope you find unicorns, fairy dust, and especially lots of paddling feet in this
book.
Assumptions
SDN is a new approach to the current world of networking, but it is still networking.
As you get into this book, we’re assuming a certain level of networking knowledge. You
don’t have to be an engineer, but knowing how networking principles work—and
frankly, don’t work—will aid your comprehension of the text.
You should be familiar with the following terms/concepts:
OSI model
The Open Systems Interconnection (OSI) model defines seven different layers of
technology: physical, data link, network, transport, session, presentation, and ap‐
plication. This model allows network engineers and network vendors to easily dis‐
cuss and apply technology to a specific OSI level. This segmentation lets engineers
divide the overall problem of getting one application to talk to another into discrete
parts and more manageable sections. Each level has certain attributes that describe
it and each level interacts with its neighboring levels in a very well-defined manner.
Knowledge of the layers above layer 7 is not mandatory, but understanding that
interoperability is not always about electrons and photons will help.
Preface | xix

Switches
These devices operate at layer 2 of the OSI model and use logical local addressing
to move frames across a network. Devices in this category include Ethernet in all
its variations, VLANs, aggregates, and redundancies.
Routers
These devices operate at layer 3 of the OSI model and connect IP subnets to each
other. Routers move packets across a network in a hop-by-hop fashion.
Ethernet
These broadcast domains connect multiple hosts together on a common infra‐
structure. Hosts communicate with each other using layer 2 media access control
(MAC) addresses.
IP addressing and subnetting
Hosts using IP to communicate with each other use 32-bit addresses. Humans often
use a dotted decimal format to represent this address. This address notation in‐
cludes a network portion and a host portion, which is normally displayed as
192.168.1.1/24.
TCP and UDP
These layer 4 protocols define methods for communicating between hosts. The
Transmission Control Protocol (TCP) provides for connection-oriented commu‐
nications, whereas the User Datagram Protocol (UDP) uses a connectionless para‐
digm. Other benefits of using TCP include flow control, windowing/buffering, and
explicit acknowledgments.
ICMP
Network engineers use this protocol to troubleshoot and operate a network, as it is
the core protocol used (on some platforms) by the ping and traceroute programs.
In addition, the Internet Control Message Protocol (ICMP) is used to signal error
and other messages between hosts in an IP-based network.
Data center
A facility used to house computer systems and associated components, such as
telecommunications and storage systems. It generally includes redundant or back‐
up power supplies, redundant data communications connections, environmental
controls (e.g., air conditioning and fire suppression), and security devices. Large
data centers are industrial-scale operations that use as much electricity as a small
town.
MPLS
Multiprotocol Label Switching (MPLS) is a mechanism in high-performance net‐
works that directs data from one network node to the next based on short path
labels rather than long network addresses, avoiding complex lookups in a routing
table. The labels identify virtual links (paths) between distant nodes rather than
xx | Preface

endpoints. MPLS can encapsulate packets of various network protocols. MPLS
supports a range of access technologies.
Northbound interface
An interface that conceptualizes the lower-level details (e.g., data or functions) used
by, or in, the component. It is used to interface with higher-level layers using the
southbound interface of the higher-level component(s). In architectural overview,
thenorthboundinterfaceisnormallydrawnatthetopofthecomponentitisdefined
in, hence the name northbound interface. Examples of a northbound interface are
JSON or Thrift.
Southbound interface
An interface that conceptualizes the opposite of a northbound interface. The south‐
bound interface is normally drawn at the bottom of an architectural diagram.
Examples of southbound interfaces include I2RS, NETCONF, or a command-line
interface.
Network topology
The arrangement of the various elements (links, nodes, interfaces, hosts, etc.) of a
computer network. Essentially, it is the topological structure of a network and may
be depicted physically or logically. Physical topology refers to the placement of the
network’s various components, including device location and cable installation,
while logical topology shows how data flows within a network, regardless of its
physical design. Distances between nodes, physical interconnections, transmission
rates, and/or signal types may differ between two networks, yet their topologies
may be identical.
Application programming interfaces
A specification of how some software components should interact with each other.
In practice, an API is usually a library that includes specification for variables,
routines, object classes, and data structures. An API specification can take many
forms, including an international standard (e.g., POSIX), vendor documentation
(e.g., the JunOS SDK), or the libraries of a programming language.
What’s in This Book?
Chapter 1, Introduction
This chapter introduces and frames the conversation this book engages in around
the concepts of SDN, where they came from, and why they are important to discuss.
Chapter 2, Centralized and Distributed Control and Data Planes
SDN is often framed as a decision between a distributed/consensus or centralized
network control-plane model for future network architectures. In this chapter, we
visit the fundamentals of distributed and central control, how the data plane is
Preface | xxi

3. Yes, we have had centralized control models in the past!
generated in both, past history with both models,3
some assumed functionality in
the present distributed/consensus model that we may expect to translate into any
substitute, and the merits of these models.
Chapter 3, OpenFlow
OpenFlow has been marketed either as equivalent to SDN (i.e., OpenFlow is SDN)
or a critical component of SDN, depending on the whim of the marketing of the
Open Networking Foundation. It can certainly be credited with sparking the dis‐
cussion of the centralized control model. In this chapter, we visit the current state
of the OpenFlow model.
Chapter 4, SDN Controllers
Forsome,thediscussionofSDNtechnologyisallaboutthemanagementofnetwork
state, and that is the role of the SDN controller. In this chapter, we survey the con‐
trollers available (both open source and commercial), their structure and capabil‐
ities, and then compare them to an idealized model (that is developed in Chapter 9).
Chapter 5, Network Programmability
This chapter introduces network programmability as one of the key tenets of SDN.
It first describes the problem of the network divide that essentially boils down to
older management interfaces and paradigms keeping applications at arm’s length
from the network. In the chapter, we show why this is a bad thing and how it can
be rectified using modern programmatic interfaces. This chapter firmly sets the
tone for what concrete changes are happening in the real world of applications and
network devices that are following the SDN paradigm shift.
Chapter 6, Data Center Concepts and Constructs
This chapter introduces the reader to the notion of the modern data center through
an initial exploration of the historical evolution of the desktop-centric world of the
late 1990s to the highly distributed world we live in today, in which applications—
as well as the actual pieces that make up applications—are distributed across mul‐
tiple data centers. Multitenancy is introduced as a key driver for virtualization in
the data center, as well as other techniques around virtualization. Finally, we explain
why these things form some of the keys to the SDN approach and why they are
driving much of the SDN movement.
Chapter 7, Network Function Virtualization
In this chapter, we build on some of the SDN concepts that were introduced earlier,
such as programmability, controllers, virtualization, and data center concepts. The
chapter explores one of the cutting-edge areas for SDN, which takes key concepts
and components and puts them together in such a way that not only allows one to
xxii | Preface

virtualize services, but also to connect those instances together in new and inter‐
esting ways.
Chapter 8, Network Topology and Topological Information Abstraction
This chapter introduces the reader to the notion of network topology, not only as
it exists today but also how it has evolved over time. We discuss why network top‐
ology—its discovery, ongoing maintenance, as well as an application’s interaction
with it—is critical to many of the SDN concepts, including NFV. We discuss a
number of ways in which this nut has been partially cracked and how more recently,
the IETF’s I2RS effort may have finally cracked it for good.
Chapter 9, Building an SDN Framework
This chapter describes an idealized SDN framework for SDN controllers, applica‐
tions, and ecosystems. This concept is quite important in that it forms the archi‐
tectural basis for all of the SDN controller offerings available today and also shows
a glimpse of where they can or are going in terms of their evolution. In the chapter,
we present the various incarnations and evolutions of such a framework over time
and ultimately land on the one that now forms the Open Daylight Consortium’s
approach. This approach to an idealized framework is the best that we reckon exists
today both because it is technically sound and pragmatic, and also because it very
closely resembles the one that we embarked on ourselves after quite a lot of trial
and error.
Chapter 10, Use Cases for Bandwidth Scheduling, Manipulation, and Calendaring
This chapter presents the reader with a number of use cases that fall under the areas
of bandwidth scheduling, manipulation, and bandwidth calendaring. We demon‐
strate use cases that we have actually constructed in the lab as proof-of-concept
trials, as well as those that others have instrumented in their own lab environments.
These proof-of-concept approaches have funneled their way into some production
applications, so while they may be toy examples, they do have real-world applica‐
bility.
Chapter 11, Use Cases for Data Center Overlays, Big Data, and Network Function Vir‐
tualization
This chapter shows some use cases that fall under the areas of data centers. Specif‐
ically, we show some interesting use cases around data center overlays, and network
function virtualization. We also show how big data can play a role in driving some
SDN concepts.
Chapter 12, Use Cases for Input Traffic Monitoring, Classification, and Triggered Ac‐
tions
This chapter presents the reader with some use cases in the input traffic/triggered
actions category. These uses cases concern themselves with the general action of
receiving some traffic at the edge of the network and then taking some action. The
action might be preprogrammed via a centralized controller, or a device might need
Preface | xxiii

to ask a controller what to do once certain traffic is encountered. Here we present
two use cases to demonstrate these concepts. First, we show how we built a proof
of concept that effectively replaced the Network Access Control (NAC) protocol
and its moving parts with an OpenFlow controller and some real routers. This
solved a real problem at a large enterprise that could not have been easily solved
otherwise. We also show a case of how a virtual firewall can be used to detect and
trigger certain actions based on controller interaction.
Chapter 13, Final Thoughts and Conclusions
This chapter brings the book into the present tense—re-emphasizing some of our
fundamentalopinionsonthecurrentstateofSDN(asofthiswriting)andproviding
a few final observations on the topic.
Conventions Used in This Book
The following typographical conventions are used in this book:
Italic
Indicates new terms, URLs, email addresses, filenames, file extensions, pathnames,
directories, and Unix utilities.
Constant width
Indicates commands, options, switches, variables, attributes, keys, functions, types,
classes, namespaces, methods, modules, properties, parameters, values, objects,
events, event handlers, XML tags, HTML tags, macros, the contents of files, and the
output from commands.
Constant width bold
Shows commands and other text that should be typed literally by the user, as well
as important lines of code.
Constant width italic
Shows text that should be replaced with user-supplied values.
This icon signifies a tip, suggestion, or general note.
This icon indicates a warning or caution.
xxiv | Preface

Using Code Examples
Supplemental material (code examples, exercises, etc.) is available for download at
http://oreil.ly/SDN_1e. This page hosts a .txt file of the complete configurations used in
Chapter 10’s use case. You may download the configurations for use in your own lab.
This book is here to help you get your job done. In general, if this book includes code
examples, you may use the code in your programs and documentation. You do not need
to contact us for permission unless you’re reproducing a significant portion of the code.
For example, writing a program that uses several chunks of code from this book does
not require permission. Selling or distributing a CD-ROM of examples from O’Reilly
books does require permission. Answering a question by citing this book and quoting
example code does not require permission. Incorporating a significant amount of ex‐
ample code from this book into your product’s documentation does require permission.
We appreciate, but do not require, attribution. An attribution usually includes the title,
author, publisher, and ISBN, for example: “SDN: Software-Defined Networks by Thomas
D. Nadeau and Ken Gray. Copyright 2013 Thomas D. Nadeau and Ken Gray,
978-1-449-34230-2.”
If you feel your use of code examples falls outside fair use or the permission given above,
feel free to contact us at permissions@oreilly.com.
Safari® Books Online
Safari Books Online (www.safaribooksonline.com) is an on-
demand digital library that delivers expert content in both book and
video form from the world’s leading authors in technology and busi‐
ness.
Technology professionals, software developers, web designers, and business and crea‐
tive professionals use Safari Books Online as their primary resource for research, prob‐
lem solving, learning, and certification training.
Safari Books Online offers a range of product mixes and pricing programs for organi‐
zations, government agencies, and individuals. Subscribers have access to thousands of
books, training videos, and prepublication manuscripts in one fully searchable database
from publishers like O’Reilly Media, Prentice Hall Professional, Addison-Wesley Pro‐
fessional, Microsoft Press, Sams, Que, Peachpit Press, Focal Press, Cisco Press, John
Wiley & Sons, Syngress, Morgan Kaufmann, IBM Redbooks, Packt, Adobe Press, FT
Press, Apress, Manning, New Riders, McGraw-Hill, Jones & Bartlett, Course Technol‐
ogy, and dozens more. For more information about Safari Books Online, please visit us
online.
Preface | xxv

How to Contact Us
Please address comments and questions concerning this book to the publisher:
O’Reilly Media, Inc.
1005 Gravenstein Highway North
Sebastopol, CA 95472
800-998-9938 (in the United States or Canada)
707-829-0515 (international or local)
707-829-0104 (fax)
We have a web page for this book, where we list errata, examples, and any additional
information. You can access this page at http://oreil.ly/SDN_1e. The authors also have
created a blog and discussion forum about SDN and network programmability at http://
sdnprogrammability.net.
To comment or ask technical questions about this book, send email to bookques
tions@oreilly.com.
For more information about our books, courses, conferences, and news, see our website
at http://www.oreilly.com.
Find us on Facebook: http://facebook.com/oreilly
Follow us on Twitter: http://twitter.com/oreillymedia
Watch us on YouTube: http://www.youtube.com/oreillymedia
Acknowledgments from Thomas Nadeau
I would like to first thank my wonderful wife, Katie, and two sons, Thomas Peter and
Henry Clifford. I can’t imagine being happy without you guys. Life is a journey, and I
am glad you guys are walking the road with me. I would also like to thank my parents,
Clement and Janina. Without your support and encouragement, I would likely have
never made it as an engineer—or at least without Dad’s instruction at a young age, I
wouldn’t be so adept at soldering now. Thank you to my many colleagues present and
past who pushed me to stretch my imagination in the area of SDN. These folks include
but are not limited to David Ward, Dave Meyer, Jan Medved, Jim Guichard, Ping Pan,
Alia Atlas, Michael Beesley, Benson Scliesser, Chris Liljenstolpe, Dan Backman, Nils
Swart, and Michael Bushong. Also, I will never forget how George Swallow took me on
as his young Padawan and gave me the Jedi training that helped me be where I am today.
Without that, I would likely not have achieved the accomplishments I have in the net‐
working industry. There are many others from my journey at Cisco, CA, and my current
employer, Juniper Networks, who are too numerous to mention. I would like to thank
thelargerSDNcommunity,includingthoseatStanford,whoweretrulyontosomething
xxvi | Preface

in the early days of this work, and my colleagues at the IETF, ONF, and Open Daylight
Project. Thank you to Meghan Blanchette and the rest of the staff at O’Reilly. And, of
course, Patrick Ames, our editor who held the course when we strayed and helped us
express the best, most articulate message we could convey.
Last, but surely not least, I would like to give my heartfelt thanks to Ken Gray, my
coauthor on this book. Without you grabbing the other oar of this boat, I am not sure
I would have been able to row it myself to the end. Your contributions truly enhanced
this book beyond anything I would have imagined myself.
Acknowledgments from Ken Gray
I would like to thank my amazing wife, Leslie. You patiently supported me through this
project and all that went with it and provided much needed balance and sanity.
For my children, Lilly and Zane, I hope my daring to write this first book may provide
inspiration for you to start your own great work (whatever it may be).
The space here can’t contain the list of customers, colleagues, and friends whose con‐
versations over the last two years have shaped my views on this topic.
It’s no coincidence that my acknowledgments list of colleagues, standards bodies, and
(of course) those who assisted in this publication would look exactly like that of my
coauthor. I would particularly like to reiterate the thanks to my past Juniper Networks
colleagues (many now with SDN startups) who got started in SDN with both of us over
two years ago, when the word that described SDN theorists and strategists was not
“visionary,” and who helped shape my views. And, if another redundancy can be spared,
I’d extend a special thanks to a present Juniper colleague, Benson Scliesser, for the same
reasons.
I’d finally like to give great thanks to my coauthor, Thomas Nadeau. We share a common
view on this topic that we developed from two different but complementary perspec‐
tives. Putting those two views together, first in our numerous public engagements over
the past year and finally in print, has been a great experience for me, has helped me
personally refine the way I talk about SDN, and hopefully has resulted in a great book.
Preface | xxvii

CHAPTER 1
Introduction
Up until a few years ago, storage, computing, and network resources were intentionally
kept physically and operationally separate from one another. Even the systems used to
manage those resources were separated—often physically. Applications that interacted
with any of these resources, such as an operational monitoring system, were also kept
at arm’s length significantly involved access policies, systems, and access procedures all
in the name of security. This is the way IT departments liked it. It was really only after
the introduction of (and demand for) inexpensive computing power, storage, and net‐
working in data center environments that organizations were forced to bring these dif‐
ferent elements together. It was a paradigm shift that also brought applications that
manage and operate these resources much, much closer than ever before.
Data centers were originally designed to physically separate traditional computing el‐
ements (e.g., PC servers), their associated storage, and the networks that interconnected
them with client users. The computing power that existed in these types of data centers
became focused on specific server functionality—running applications such as mail
servers, database servers, or other such widely used functionality in order to serve
desktop clients. Previously, those functions—which were executed on the often thou‐
sands (or more) of desktops within an enterprise organization—were handled by de‐
partmental servers that provided services dedicated only to local use. As time went on,
the departmental servers migrated into the data center for a variety of reasons—first
and foremost, to facilitate ease of management, and second, to enable sharing among
the enterprise’s users.
It was around 10 years ago that an interesting transformation took place. A company
called VMware had invented an interesting technology that allowed a host operating
system such as one of the popular Linux distributions to execute one or more client
operating systems (e.g., Windows). What VMware did was to create a small program
that created a virtual environment that synthesized a real computing environment (e.g.,
1

virtual NIC, BIOS, sound adapter, and video). It then marshaled real resources between
the virtual machines. This supervisory program was called a hypervisor.
Originally, VMware was designed for engineers who wanted to run Linux for most of
their computing needs and Windows (which was the corporate norm at the time) only
for those situations that required that specific OS environment to execute. When they
were finished, they would simply close Windows as if it were another program, and
continue on with Linux. This had the interesting effect of allowing a user to treat the
client operating system as if it were just a program consisting of a file (albeit large) that
existed on her hard disk. That file could be manipulated as any other file could be (i.e.,
it could be moved or copied to other machines and executed there as if it were running
on the machine on which it was originally installed). Even more interestingly, the op‐
erating system could be paused without it knowing, essentially causing it to enter into
a state of suspended animation.
Withtheadventofoperatingsystemvirtualization,theserversthattypicallyranasingle,
dedicated operating system, such as Microsoft Windows Server, and the applications
specifically tailored for that operating system could now be viewed as a ubiquitous
computing and storage platform. With further advances and increases in memory,
computing, and storage, data center compute servers were increasingly capable of ex‐
ecutingavarietyofoperatingsystemssimultaneouslyinavirtualenvironment.VMware
expanded its single-host version to a more data-center-friendly environment that was
capable of executing and controlling many hundreds or thousands of virtual machines
from a single console. Operating systems such as Windows Server that previously oc‐
cupied an entire “bare metal” machine were now executed as virtual machines, each
runningwhateverapplicationsclientusersdemanded.Theonlydifferencewasthateach
was executing in its own self-contained environment that could be paused, relocated,
cloned, or copied (i.e., as a backup). Thus began the age of elastic computing.
Within the elastic computing environment, operations departments were able to move
servers to any physical data center location simply by pausing a virtual machine and
copying a file. They could even spin up new virtual machines simply by cloning the
same file and telling the hypervisor to execute it as a new instance. This flexibility al‐
lowed network operators to start optimizing the data center resource location and thus
utilization based on metrics such as power and cooling. By packing together all active
machines, an operator could turn down cooling in another part of a data center by
sleepingoridlingentirebanksorrowsofphysicalmachines,thusoptimizingthecooling
load on a data center. Similarly, an operator could move or dynamically expand com‐
puting, storage, or network resources by geographical demand.
As with all advances in technology, this newly discovered flexibility in operational de‐
ployment of computing, storage, and networking resources brought about a new prob‐
lem: one not only of operational efficiency both in terms of maximizing the utilization
of storage and computing power, but also in terms of power and cooling. As mentioned
2 | Chapter 1: Introduction

earlier, network operators began to realize that computing power demand in general
increased over time. To keep up with this demand, IT departments (which typically
budget on a yearly basis) would order all the equipment they predicted would be needed
for the following year. However, once this equipment arrived and was placed in racks,
it would consume power, cooling, and space resources—even if it was not yet used! This
was the dilemma discovered first at Amazon. At the time, Amazon’s business was grow‐
ing at the rate of a “hockey stick” graph—doubling every six to nine months. As a result,
growth had to stay ahead of demand for its computing services, which served its retail
ordering, stock, and warehouse management systems, as well as internal IT systems. As
a result, Amazon’s IT department was forced to order large quantities of storage, net‐
work, and computing resources in advance, but faced the dilemma of having that
equipment sit idle until the demand caught up with those resources. Amazon Web
Services (AWS) was invented as a way to commercialize this unused resource pool so
that it would be utilized at a rate closer to 100%. When internal resources needed more
resources, AWS would simply push off retail users, and when it was not, retail compute
users could use up the unused resources. Some call this elastic computing services, but
this book calls it hyper virtualization.
ItwasonlythenthatcompanieslikeAmazonandRackspace,whichwerebuyingstorage
andcomputinginhugequantitiesforpricingefficiency,realizedtheywerenotefficiently
utilizingalloftheircomputingandstorageandcouldreselltheirsparecomputingpower
and storage to external users in an effort to recoup some of their capital investments.
This gave rise to a multitenant data center. This of course created a new problem, which
washowtoseparatethousandsofpotentialtenants,whoseresourcesneededtobespread
arbitrarily across different physical data centers’ virtual machines.
Another way to understand this dilemma is to note that during the move to hyper
virtualized environments, execution environments were generally run by a single en‐
terprise or organization. That is, they typically owned and operated all of the computing
and storage (although some rented co-location space) as if they were a single, flat local
area network (LAN) interconnecting a large number of virtual or physical machines
and network attached storage. (The exception was in financial institutions where reg‐
ulatory requirements mandated separation.) However, the number of departments in
these cases was relatively small—fewer than 100—and so this was easily solved using
existing tools such as layer 2 or layer 3 MPLS VPNs. In both cases, though, the network
components that linked all of the computing and storage resources up until that point
were rather simplistic; it was generally a flat Ethernet LAN that connected all of the
physical and virtual machines. Most of these environments assigned IP addresses to all
of the devices (virtual or physical) in the network from a single network (perhaps with
IP subnets), as a single enterprise owned the machines and needed access to them. This
also meant that it was generally not a problem moving virtual machines between dif‐
ferent data centers located within that enterprise because, again, they all fell within the
same routed domain and could reach one another regardless of physical location.
Introduction | 3

In a multitenant data center, computing, storage, and network resources can be offered
in slices that are independent or isolated from one another. It is, in fact, critical that they
are kept separate. This posed some interesting challenges that were not present in the
single tenant data center environment of the past. Keep in mind that their environment
allowed for the execution of any number of operating systems and applications on top
of those operating systems, but each needed a unique network address if it was to be
accessed by its owner or other external users such as customer. In the past, addresses
could be assigned from a single, internal block of possibly private addresses and routed
internally easily. Now, however, you needed to assign unique addresses that are exter‐
nally routable and accessible. Furthermore, consider that each virtual machine in ques‐
tion had a unique layer 2 address as well. When a router delivers a packet, it ultimately
has to deliver a packet using Ethernet (not just IP). This is generally not an issue until
you consider virtual machine mobility (VM mobility). In these cases, virtual machines
are relocated for power, cooling, or computing compacting reasons. In here lies the rub
because physical relocation means physical address relocation. It also possibly means
changes to layer 3 routing in order to ensure packets previously destined for that ma‐
chine in its original location can now be changed to its new location.
At the same time data centers were evolving, network equipment seemed to stand still
in terms of innovations beyond feeds and speeds. That is, beyond the steady increase
in switch fabric capacities and interface speeds, data communications had not evolved
much since the advent of IP, MPLS, and mobile technologies. IP and MPLS allowed a
network operator to create networks and virtual network overlays on top of those base
networksmuchinthewaythatdatacenteroperatorswereabletocreatevirtualmachines
to run over physical ones with the advent of computing virtualization. Network virtu‐
alization was generally referred to as virtual private networks (VPN) and came in a
number of flavors, including point-to-point (e.g., a personal VPN as you might run on
yourlaptopandconnecttoyourcorporatenetwork);layer3(virtualizinganIPorrouted
network in cases such as to allow a network operator to securely host enterprise in a
manner that isolated their traffic from other enterprise); and layer 2 VPNs (switched
network virtualization that isolates similarly to a layer 3 VPN except that the addresses
used are Ethernet).
Commercialroutersandswitchestypicallycomewithmanagementinterfacesthatallow
a network operator to configure and otherwise manage these devices. Some examples
of management interfaces include command line interfaces, XML/Netconf, graphical
user interfaces (GUIs), and the Simple Network Management Protocol (SNMP). These
options provide an interface that allows an operator suitable access to a device’s capa‐
bilities, but they still often hide the lowest levels of details from the operator. For ex‐
ample, network operators can program static routes or other static forwarding entries,
but those ultimately are requests that are passed through the device’s operating system.
This is generally not a problem until one wants to program using syntax or semantics
of functionality that exists in a device. If someone wishes to experiment with some new

routing protocol, they cannot on a device where the firmware has not been written to
support that protocol. In such cases, it was common for a customer to make a feature
enhancement request of a device vendor, and then typically wait some amount of time
(several years was not out of the ordinary).
At the same time, the concept of a distributed (at least logically) control plane came
back onto the scene. A network device is comprised of a data plane that is often a switch
fabric connecting the various network ports on a device and a control plane that is the
brains of a device. For example, routing protocols that are used to construct loop-free
paths within a network are most often implemented in a distributed manner. That is,
each device in the network has a control plane that implements the protocol. These
communicate with each other to coordinate network path construction. However, in a
centralized control plane paradigm, one single (or at least logical) control plane would
exist. This über brain would push commands to each device, thus commanding it to
manipulate its physical switching and routing hardware. It is important to note that
although the hardware that executed data planes of devices remained quite specialized,
and thus expensive, the control plane continued to gravitate toward less and less ex‐
pensive, general-purpose computing, such as those central processing units produced
by Intel.
All of these aforementioned concepts are important, as they created the nucleus of mo‐
tivation for what has evolved into what today is called software-defined networking
(SDN). Early proponents of SDN saw that network device vendors were not meeting
their needs, particularly in the feature development and innovation spaces. High-end
routing and switching equipment was also viewed as being highly overpriced for at least
the control plane components of their devices. At the same time, they saw the cost of
raw, elastic computing power diminishing rapidly to the point where having thousands
of processors at one’s disposal was a reality. It was then that they realized that this pro‐
cessing power could possibly be harnessed to run a logically centralized control plane
and potentially even use inexpensive, commodity-priced switching hardware. A few
engineers from Stanford University created a protocol called OpenFlow that could be
implemented in just such a configuration. OpenFlow was architected for a number of
devices containing only data planes to respond to commands sent to them from a (log‐
ically) centralized controller that housed the single control plane for that network. The
controller was responsible for maintaining all of the network paths, as well as program‐
ming each of the network devices it controlled. The commands and responses to those
commands are described in the OpenFlow protocol. It is worth noting that the Open
Networking Foundation (ONF) commercially supported the SDN effort and today re‐
mains its central standardization authority and marketing organization. Based on this
basic architecture just described, one can now imagine how quickly and easily it was to
devise a new networking protocol by simply implementing it within a data center on
commodity priced hardware. Even better, one could implement it in an elastic com‐
puting environment in a virtual machine.
Introduction | 5

A slightly different view of SDN is what some in the industry refer to as software-driven
networks, as opposed to software-defined networks. This play on words is not meant to
completely confuse the reader, but instead highlight a difference in philosophy of ap‐
proaches. In the software-driven approach, one views OpenFlow and that architecture
as a distinct subset of functionality that is possible. Rather than viewing the network as
being comprised of logically centralized control planes with brainless network devices,
one views the world as more of a hybrid of the old and the new. More to the point, the
reality is that it is unrealistic to think that existing networks are going to be dismantled
wholesale to make way for a new world proposed by the ONF and software-defined
networks. It is also unrealistic to discard all of the advances in network technology that
exist today and are responsible for things like the Internet. Instead, there is more likely
a hybrid approach whereby some portion of networks are operated by a logically cen‐
tralized controller, while other parts would be run by the more traditional distributed
control plane. This would also imply that those two worlds would need to interwork
with each other.
ItisinterestingtoobservethatatleastoneofthemajorpartsofwhatSDNandOpenFlow
proponents are trying to achieve is greater and more flexible network device pro‐
grammability. This does not necessarily have anything to do with the location of the
network control and data planes; however, it is concerned with how they are program‐
med. Do not forget that one of the motivations for creating SDN and OpenFlow was
the flexibility of how one could program a network device, not just where it is pro‐
grammed. If one observes what is happening in the SDN architecture just described,
both of those questions are solved. The question is whether or not the programmability
aspect is the most optimal choice.
To address this, individuals representing Juniper, Cisco, Level3, and other vendors and
service providers have recently spearheaded an effort around network programmability
called the Interface to the Routing System (I2RS). A number of folks from these sources
have contributed to several IETF drafts, including the primary requirements and frame‐
work drafts to which Alia Atlas, David Ward, and Tom have been primary contributors.
In the near future, at least a dozen drafts around this topic should appear online. Clearly
there is great interest in this effort. The basic idea around I2RS is to create a protocol
and components to act as a means of programming a network device’s routing infor‐
mation base (RIB) using a fast path protocol that allows for a quick cut-through of
provisioning operations in order to allow for real-time interaction with the RIB and the
RIB manager that controls it. Previously, the only access one had to the RIB was via the
device’s configuration system (in Juniper’s case, Netconf or SNMP).
The key to understanding I2RS is that it is most definitely not just another provisioning
protocol; that’s because there are a number of other key concepts that comprise an entire
solution to the overarching problem of speeding up the feedback loop between network
elements, network programming, state and statistical gathering, and post-processing

analytics. Today, this loop is painfully slow. Those involved in I2RS believe the key to
the future of programmable networks lies within optimizing this loop.
To this end, I2RS provides varying levels of abstraction in terms of programmability of
network paths, policies, and port configuration, but in all cases has the advantage of
allowing for adult supervision of said programming as a means of checking the com‐
mands prior to committing them. For example, some protocols exist today for pro‐
gramming at the hardware abstraction layer (HAL), which is far too granular or detailed
for the network’s efficiency and in fact places undue burden on its operational systems.
Another example is providing operational support systems (OSS) applications quick
and optimal access to the RIB in order to quickly program changes and then witness
the results, only to be able to quickly reprogram in order to optimize the network’s
behavior. One key aspect around all of these examples is that the discourse between the
applications and the RIB occur via the RIB manager. This is important, as many oper‐
ators would like to preserve their operational and workflow investment in routing pro‐
tocol intelligence that exists in device operating systems such as Junos or IOS-XR while
leveraging this new and useful programmability paradigm to allow additional levels of
optimization in their networks.
I2RS also lends itself well to a growing desire to logically centralize routing and path
decisions and programmability. The protocol has requirements to run on a device or
outside of a device. In this way, distributed controller functionality is embraced in cases
where it is desired; however, in cases where more classic distributed control is desired,
we are able to support those as well.
Finally, another key subcomponent of I2RS is normalized and abstracted topology.
Defining a common and extensible object model will represent this topology. The ser‐
vice also allows for multiple abstractions of topological representation to be exposed. A
key aspect of this model is that nonrouters (or routing protocol speakers) can more
easily manipulate and change the RIB state going forward. Today, nonrouters have a
major difficulty getting at this information at best. Going forward, components of a
network management/OSS, analytics, or other applications that we cannot yet envision
will be able to interact quickly and efficiently with routing state and network topology.
So, to culminate these thoughts, it is appropriate that we define SDN for what we think
it is and will become:
Software-defined networks (SDN): an architectural approach that optimizes and sim‐
plifies network operations by more closely binding the interaction (i.e., provisioning,
messaging,andalarming)amongapplicationsandnetworkservicesanddevices,wheth‐
er they be real or virtualized. It often is achieved by employing a point of logically
centralized network control—which is often realized as an SDN controller—which then
orchestrates, mediates, and facilitates communication between applications wishing to
interact with network elements and network elements wishing to convey information
Introduction | 7

to those applications. The controller then exposes and abstracts network functions and
operationsviamodern,application-friendlyandbidirectionalprogrammaticinterfaces.
So, as you can see, software-defined, software-driven, and programmable networks
come with a rich and complex set of historical lineage, challenges, and a variety of
solutions to those problems. It is the success of the technologies that preceded software-
defined, software-driven, and programmable networks that makes advancing technol‐
ogy based on those things possible. The fact of the matter is that most of the world’s
networks—includingtheInternet—operateonthebasisofIP,BGP,MPLS,andEthernet.
Virtualization technology today is based on the technologies started by VMware years
ago and continues to be the basis on which it and other products are based. Network
attached storage enjoys a similarly rich history.
I2RShasasimilarfutureaheadofitinsofarassolvingtheproblemsofnetwork,compute,
and storage virtualization as well as those of the programmability, accessibility, location,
and relocation of the applications that execute within these hyper virtualized environ‐
ments.
Although SDN controllers continue to rule the roost when it comes to press, many other
advances have taken place just in the time we have been writing this book. One very
interesting and bright one is the Open Daylight Project. Open Daylight’s mission is to
facilitateacommunity-led,industry-supportedopensourceframework,includingcode
and architecture, to accelerate and advance a common, robust software-defined net‐
working platform. To this end, Open Daylight is hosted under the Linux Foundation’s
umbrella and will facilitate a truly game changing, and potentially field-leveling effort
around SDN controllers. This effort will also spur innovation where we think it matters
most in this space: applications. While we have seen many advances in controllers over
the past few years, controllers really represent the foundational infrastructure for SDN-
enabled applications. In that vein, the industry has struggled to design and develop
controllers over the past few years while mostly ignoring applications. We think that
SDN is really about operational optimization and efficiency at the end of the day, and
the best way to achieve this is through quickly checking off that infrastructure and
allowing the industry to focus on innovating in the application and device layers of the
SDN architecture.
This book focuses on the network aspects of software-defined, software-driven, and
programmable networks while giving sufficient coverage to the virtualization, location,
and programming of storage, network, and compute aspects of the equation. It is the
goal of this book to explore the details and motivations around the advances in network
technology that gave rise to and support of hyper virtualization of network, storage, and
computing resources that are now considered to be part of SDN.

CHAPTER 2
Centralized and Distributed Control
and Data Planes
One of the tenets expressed early in the introduction of SDN is the potential advantage
in the separation of a network device’s control and data planes. This separation affords
a network operator certain advantages in terms of centralized or semi-centralized pro‐
grammatic control. It also has a potential economic advantage based on the ability to
consolidate in one or a few places what is often a considerably complex piece of software
to configure and control onto less expensive, so-called commodity hardware.
Introduction
The separation of the control and data planes is indeed one of the fundamental tenets
of SDN—and one of its more controversial, too. Although it’s not a new concept, the
contemporary way of thinking has some interesting twists on an old idea: how far away
the control plane can be located from the data plane, how many instances are needed
toexisttosatisfyresiliencyandhigh-availabilityrequirements,andwhetherornot100%
of the control plane can be, in fact, relocated further away than a few inches are all
intensely debated. The way we like to approach these ideas is to think of them as a
continuum of possibilities stretching between the simplest, being the canonical fully
distributed control plane, to the semi- or logically centralized control plane, to finally
the strictly centralized control plane. Figure 2-1 illustrates the spectrum of options
available to the network operator, as well as some of the pros and cons of each approach.
9

Figure 2-1. Spectrum of control and data plane distribution options
Evolution versus Revolution
At one end of the spectrum of answers to the question of where to put the control plane
lies the revolutionary proponents, who propose a clean slate approach in which the
control plane of a network is completely centralized. In most cases, this extreme ap‐
proach has been tempered to be, in reality, a logically centralized approach due to either
scaleorhighavailabilityrequirementsthatmakeastrictlycentralizedapproachdifficult.
In this model, no control plane functions effectively exist at a device; instead, a device
is a dumb (albeit fast) switching device under the total control of the remotely located,
centralized control plane. We shall explore this in detail later in the chapter and show
why it generally applies best to newly deployed networks rather than existing ones.
Toward the middle of the spectrum, the evolutionary proponents see domains within
the general definition of networks in which a centralized control paradigm provides
some new capabilities, but does not replace every capability nor does it completely re‐
move the control plane from the device. Instead, this paradigm typically works in con‐
junction with a distributed control plane in some fashion, meaning that the device
retains some classical control plane functions (e.g., ARP processing or MAC address
10 | Chapter 2: Centralized and Distributed Control and Data Planes

1. As part of its evolution, the Open Networking Foundation has alternately bound the definition of SDN to
OpenFlow tightly (i.e., OpenFlow = SDN) and loosely (i.e., OpenFlow is a critical component of SDN).
Regardless, it’s undeniable that the existence of OpenFlow and the active marketing of the ONF triggered the
market/public discussion and interest in SDN.
2. The management plane is responsible for element configuration that may affect local forwarding decisions
(forwarding features) like access control lists (ACLs) or policy-based routing (PBR).
learning), while allowing a centralized controller to manipulate other areas of func‐
tionality more convenient for that operational paradigm. This view is often character‐
ized as the hybrid operation or as part of the underlay/overlay concept in which the
distributed control plane provides the underlay and the centralized control plane pro‐
vides a logical overlay that utilizes the underlay as a network transport.
Finally, at the other end of the spectrum is the classic use of control planes: completely
distributed. In this model, every device runs a complete instance of a control plane in
addition to at least one data plane. Also in this model, each independent control plane
must cooperate with the other control planes in order to support a cohesive and op‐
erational network. The approach obviously presents nothing new and is neither revo‐
lutionary nor evolutionary.
This chapter will not present the reader with a comprehensive discussion of control/
dataplanedesignordevelopment,asthiscouldbethetopicofanentirebook.Therefore,
we will discuss general concepts as they pertain to the SDN space and refer the reader
to other references, when possible, for further detailed investigation.1
Instead, we will
explore each of the places on the spectrum of control plane distribution and operation
that were just introduced. These will include some past and present examples of cen‐
tralization of control, hybrid, and fully distributed operation.
What Do They Do?
Let’sfirstdiscussthefundamentalcomponentsandbehaviorsofcontrolanddataplanes,
why they differ, and how they might be implemented.
The Control Plane
At a very high level, the control plane establishes the local data set used to create the
forwarding table entries, which are in turn used by the data plane to forward traffic
between ingress and egress ports on a device.2
The data set used to store the network
topology is called the routing information base (RIB). The RIB is often kept consistent
(i.e., loop-free) through the exchange of information between other instances of control
planes within the network. Forwarding table entries are commonly called the forward‐
ing information base (FIB) and are often mirrored between the control and data planes
ofatypicaldevice.TheFIBisprogrammedoncetheRIBisdeemedconsistentandstable.
To perform this task, the control entity/program has to develop a view of the network
What Do They Do? | 11

topology that satisfies certain constraints. This view of the network can be programmed
manually, learned through observation, or built from pieces of information gathered
through discourse with other instances of control planes, which can be through the use
of one or many routing protocols, manual programming, or a combination of both.
The mechanics of the control and data planes is demonstrated in Figure 2-2, which
represents a network of interconnected switches. At the top of the figure, a network of
switches is shown, with an expansion of the details of the control and data planes of two
of those switches (noted as A and B). In the figure, packets are received by switch A on
the leftmost control plane and ultimately forwarded to switch B on the righthand side
of the figure. Inside each expansion, note that the control and data planes are separated,
with the control plane executing on its own processor/card and the data plane executing
on a separate one. Both are contained within a single chassis. We will discuss this and
other variations on this theme of physical location of the control and data planes later
in the chapter. In the figure, packets are received on the input ports of the line card
where the data plane resides. If, for example, a packet is received that comes from an
unknown MAC address, it is punted or redirected (4) to the control plane of the device,
where it is learned, processed, and later forwarded onward. This same treatment is given
to control traffic such as routing protocol messages (e.g., OSPF link-state advertise‐
ments). Once a packet has been delivered to the control plane, the information con‐
tained therein is processed and possibly results in an alteration of the RIB as well as the
transmission of additional messages to its peers, alerting them of this update (i.e., a new
route is learned). When the RIB becomes stable, the FIB is updated in both the control
plane and the data plane. Subsequently, forwarding will be updated and reflect these
changes. However, in this case, because the packet received was one of an unlearned
MAC address, the control plane returns the packet (C) to the data plane (2), which
forwards the packet accordingly (3). If additional FIB programming is needed, this also
takes place in the (C) step, which would be the case for now the MAC addresses source
has been learned. The same algorithm for packet processing happens in the next switch
to the right.
The history of the Internet maps roughly to the evolution of control schemes for man‐
aging reachability information, protocols for the distribution of reachability informa‐
tion, and the algorithmic generation of optimized paths in the face of several challenges.
In the case of the latter, this includes an increasing growth of the information base used
(i.e., route table size growth) and how to manage it. Not doing so could result in the
possibility of a great deal of instability in the physical network. This in turn may lead to
high rates of change in the network or even nonoperation. Another challenge to over‐
come as the size of routing information grows is the diffusion of responsibility for
advertising reachability to parts of the destination/target data, not only between local
instances of the data plane but also across administrative boundaries.

Figure 2-2. Control and data planes of a typical network
In reality, the control plane for the Internet that was just discussed is some combination
of layer 2 or layer 3 control planes. As such, it should be no surprise then that the same
progression and evolution has taken place for both layer 2 and layer 3 networks and the
protocols that made up these control planes. In fact, the progression of the Internet
happened because these protocols evolved both in terms of functionality and hardware
vendors learned how to implement them in highly scalable and highly available ways.
A layer 2 control plane focuses on hardware or physical layer addresses such as IEEE
MAC addresses. A layer 3 control plane is built to facilitate network layer addresses such
as those of the IP protocol. In a layer 2 network, the behaviors around learning MAC
addresses, the mechanisms used to guarantee an acyclic graph (familiar to most readers
through the Spanning Tree Protocol), and flooding of BUM (broadcast, unicast un‐
known, and multicast) traffic create their own scalability challenges and also reveal their
scalability limitations. There have been several iterations or generations of
standards-based layer 2 control protocols whose goals were to address these and other

issues. Most notably, these included SPB/802.1aq from the IEEE and TRILL from the
IETF.
As a generalization, though, layer 2 and layer 3 scaling concerns and their resulting
control plane designs eventually merge or hybridize because layer 2 networks ultimately
do not scale well due to the large numbers of end hosts. At the heart of these issues is
dealing with end hosts moving between networks, resulting in a massive churn of for‐
warding tables—and having to update them quickly enough to not disrupt traffic flow.
In a layer 2 network, forwarding focuses on the reachability of MAC addresses. Thus,
layer 2 networks primarily deal with the storage of MAC addresses for forwarding pur‐
poses. Since the MAC addresses of hosts can be enormous in a large enterprise network,
themanagementoftheseaddressesisdifficult.Worse,imaginemanagingalloftheMAC
addresses across multiple enterprises or the Internet!
In a layer 3 network, forwarding focuses on the reachability of network addresses. Layer
3 network reachability information primarily concerns itself with the reachability of a
destinationIPprefix.Thisincludesnetworkprefixesacrossanumberofaddressfamilies
forbothunicastandmulticast.Inallmoderncases,layer3networkingisusedtosegment
or stitch together layer 2 domains in order to overcome layer 2 scale problems. Specif‐
ically, layer 2 bridges that represent some sets of IP subnetworks are typically connected
together with a layer 3 router. Layer 3 routers are connected together to form larger
networks—or really different subnetwork address ranges. Larger networks connect to
other networks via gateway routers that often specialize in simply interconnecting large
networks. However, in all of these cases, the router routes traffic between networks at
layer 3 and will only forward packets at layer 2 when it knows the packet has arrived at
the final destination layer 3 network that must then be delivered to a specific host.
Some notable blurring of these lines occurs with the Multiprotocol Label Switching
(MPLS) protocol, the Ethernet Virtual Private Network (EVPN) protocol, and the Lo‐
cator/ID Separation Protocol (LISP). The MPLS protocol—really a suite of protocols—
was formed on the basis of combining the best parts of layer 2 forwarding (or switching)
with the best parts of layer 3 IP routing to form a technology that shares the extremely
fast-packet forwarding that ATM invented with the very flexible and complex path
signaling techniques adopted from the IP world. The EVPN protocol is an attempt to
solve the layer 2 networking scale problems that were just described by effectively tun‐
neling distant layer 2 bridges together over an MPLS (or GRE) infrastructure—only
then is layer 2 addressing and reachability information exchanged over these tunnels
and thus does not contaminate (or affect) the scale of the underlying layer 3 networks.
ReachabilityinformationbetweendistantbridgesisexchangedasdatainsideanewBGP
address family, again not contaminating the underlying network. There are also other
optimizations that limit the amount of layer 2 addresses that are exchanged over the
tunnels, again optimizing the level of interaction between bridges. This is a design that
minimizes the need for broadcast and multicast. The other hybrid worth mentioning is
LISP (see RFC 4984). At its heart, LISP attempts to solve some of the shortcomings of

the general distributed control plane model as applied to multihoming, adding new
addressing domains and separating the site address from the provider in a new map
and encapsulation control and forwarding protocol.
Ataslightlylowerlevel,thereareadjunctcontrolprocessesparticulartocertainnetwork
types that are used to augment the knowledge of the greater control plane. The services
provided by these processes include verification/notification of link availability or qual‐
ity information, neighbor discovery, and address resolution.
Because some of these services have very tight performance loops (for short event de‐
tectiontimes),theyarealmostinvariablylocaltothedataplane(e.g.,OAM)—regardless
of the strategy chosen for the control plane. This is depicted in Figure 2-3 by showing
the various routing protocols as well as RIB-to-FIB control that comprises the heart of
the control plane. Note that we do not stipulate where the control and data planes reside,
only that the data plane resides on the line card (shown in Figure 2-3 in the LC box),
and the control plane is situated on the route processor (denoted by the RP box).
Figure 2-3. Control and data planes of a typical network device

Other documents randomly have
different content

Christ Jesus." (Gal. III; 27-28). "And let the peace of God rule in your
hearts, to the which also ye are called in one body, and be ye thankful."
(Col. III; 15). "For as the body is one, and hath many members, and all the
members of that one body, being many, are one body so also is Christ. For
by one spirit we are all baptized into one body, whether we be Jews or
Gentiles, whether we be bond or free; and have been all made to drink into
one spirit." (1 Cor. XII; 12, 13). In His prayer to the Father that all who
believed in Him might be one, Jesus spoke of this unity as proof to the
world that God had sent Him. (John XVII; 21). The great purpose of the gift
of the Holy Ghost was to guide into all truth, and bring its possessors to
"the unity of the faith and the knowledge of the Son of God." Strife,
contention, division, are not the fruits of the Holy Spirit, but come from
beneath. "For where envying and strife is, there is confusion and every evil
work." (James III: 16).
The presence and inspiration of the Holy Ghost, with its gifts,
manifestations and divine light are the signs of spiritual life and divine
acceptance. Without the Holy Ghost there is no true, living Church of Christ
on earth. It can be obtained in no other way than that which God has
appointed. Following the birth of water, the birth of the Holy Spirit makes
man a new creature, and initiates him into the Church or Kingdom of God.
Its various gifts are within his reach according to his faith and diligence in
seeking after them. They are as obtainable in this age as at any former
period. By the Holy Ghost mankind may come to the knowledge of God. In
its light the sayings and writings of inspired men may be clearly
understood. The Bible is no longer a sealed book. The heavens are not
closed against mortals. Darkness flees before it and mysteries vanish. It
brings peace and comfort to the soul. It awakens and thrills the spiritual
sense. It unfolds the things of eternity and the glories of immortality. It links
earth and heaven. It fills the soul with joy unspeakable, and he who gains
and keeps it has boundless wealth and everlasting life!

RAYS OF LIVING LIGHT.
No. 5.
BY CHARLES W. PENROSE
The ordinances of the Gospel referred to in previous tracts of this series,
cannot be effectually administered without divine authority. That authority
does not and cannot originate in man. It may be assumed, it is true, and
presumptuous men may claim to be called of God without communication
from Him. But their performances will be without avail and will not be
recognized in heaven, either in time or in eternity. When there is no
revelation from God there can be no divine authority on earth. Baptism,
even if solemnized according to the form and pattern followed by the
Savior and his appointed servants, will be of no avail and will not bring
remission of sins, unless the officiating minister has received authority from
Deity to act in the name of the Father and of the Son and of the Holy Ghost.
Men may lay their hands on the baptized believer in the form of
confirmation, but if they have not been divinely appointed to do so, the
Holy Ghost will not flow to the convert, and the performance will be void
in the sight of heaven. Those who have the temerity to act in that manner
will be counted guilty of taking the name of the Lord in vain. No council,
convocation, conference, synod, or presbytery, composed of any number of
learned, devout, and venerable persons, without divine communication can
confer the smallest amount of divine authority. Their power is only human,
their decisions, their commissions and their creeds are equally valueless in
the plan of salvation.
Whenever the Almighty desired to communicate with man on earth, he
selected His own representatives and endowed them with authority to speak
and act in His name. What they uttered by the power of the Holy Ghost, and
what they administered as He directed, was recognized by Him as if
performed and spoken by Deity in person. When He gave them authority to
call and ordain others to the same duties, their administrations were also

accepted by the Lord, and were fully efficacious. This divine authority was
called the Holy Priesthood. It was bestowed in the earliest ages. It existed
among the Patriarchs, was exercised in the Mosaic dispensation, was held
by many of the Prophets, and was established in the Christian Church by
the Savior himself. There were two orders or branches, of that Priesthood.
The higher, which includes the lower, came to be known as the Melchisedek
Priesthood. This was because Melchisedek, the King of Salem, who lived in
the time of Abraham and from whom, "the father of the faithful" received
his blessing, obtained a great power in that Priesthood. It is referred to in
the Epistle to the Hebrews, 7th chapter. Much controversy has arisen over
the meaning of the third verse, which says: "Without father, without mother,
without descent, having neither beginning of days nor end of life, but made
like unto the Son of God; abideth a Priest continually." The difficulty has
arisen through the application of these remarks to the individual instead of
to the Priesthood which he held. The higher, or Melchisedek Priesthood was
not limited, as the Levitical Order subsequently was, to a special lineage. It
did not depend upon parentage or descent, and it was an eternal Priesthood,
those who possessed it worthily retaining it through life, and being Kings
and priests unto God forever.
The Lesser Priesthood was held notably by Aaron and his sons, in the line
of the first born, and has therefore been called by his name. It had authority
to administer in the lesser ordinances and in temporal affairs, but not in the
higher and more spiritual concerns of the Kingdom of God. But no man
could take this honor unto himself. He must be called of God as Aaron was,
or he could not hold that Priesthood. (Heb. V; 4.) Aaron was called by
revelation through Moses the Prophet, and ordained under his hands.
This being so, as a matter of course, no man can take unto himself the
higher, or Melchisedek Priesthood. Unless called of God by revelation and
properly ordained, he could not obtain that authority. Even Jesus of
Nazareth, though he was the Son of God, did not assume that Priesthood.
He was "called of God, a High Priest after the order of Melchisedek." It is
written further: "So also Christ glorified not himself to be made a High
Priest but He that said unto him thou art my Son, this day have I begotten
thee." (Heb. V; 3, 10.)

It has been erroneously taught among the Christian sects of the present age
that this Priesthood, in both of its branches or orders, was done away in
Christ. That it has not been on earth for several centuries may be true, and
therefore the authority to administer in the name of the Lord has not been
enjoyed among men. But the authority held by Jesus Christ as "a Priest
forever after the order of Melchisedek" was conferred by him upon his
Apostles, to whom he gave the keys of that power and authority, so that
what they sealed on earth should be sealed in heaven, and what they loosed
on earth should be loosed in heaven. (Matt. XVIII; 18.) He said to them:
"As my Father hath sent me, even so send I you." (John XX; 21.) Again he
said: "Ye have not chosen me, but I have chosen you, and ordained you; that
ye should go and bring forth fruit and that your fruit should remain." (John
XV; 16.) The Apostles thus authorized had power to call others to this
Priesthood and ministry, when directed by the Holy Ghost, as Moses called
and ordained his brother Aaron.
The law of carnal commandments in which the lesser or Levitical
Priesthood administered was fulfilled in Jesus Christ, but the Priesthood or
authority to administer in the name of the Lord was not then abolished, the
higher, or Melchisedek Priesthood was restored. That was the change in the
Priesthood referred to in Heb. VII; 12: "For the Priesthood being changed
there is made of necessity a change also of the law." From this it is evident
that the Priesthood was not abolished, but the law of the Gospel being
introduced by Christ in place of the Mosaic Code, the higher Priesthood
was also introduced, for the Gospel is a higher law than that of Moses. The
sacrifice of animals in which the lesser Priesthood administered was no
longer required, after the great sacrifice of the Son of God of which they
were typical, so that function of the lesser, or Aaronic Priesthood was
discontinued. But the administration of the ordinances of the Gospel was
necessary, and could not be rightfully performed without divine authority.
Therefore, the Priesthood of God held by Jesus Christ, and by his Apostles
and by others called of God through them, was a part of and essential to the
Christian dispensation.
The term "called of God" appears to be as much misunderstood as is the
subject of the Priesthood of God. Men assume to act in the name of Jesus
Christ, either because they feel or imagine they have a call in their hearts to

this ministry, or because they have been called by some person or conclave
having no more divine communication and authority than they had
themselves. In contrast to their assumption let us view the case of Saul of
Tarsus, afterwards called Paul the Apostle. In the narration of his case as
given in Acts XXII he says that on his way to Damascus the Lord Jesus
Christ appeared to him in glory, and he was stricken blind thereby. He
received his sight by miracle and was informed: "The God of our Fathers
hath chosen thee that thou shouldst know his will, and see that Just One,
and shouldst hear the voice of His mouth. For thou shalt be His witness
unto all men of what thou hast seen and heard. And now why tarriest thou?
Arise and be baptized and wash away thy sins, calling on the name of the
Lord." Paul subsequently received another divine communication,
informing him that the Lord would send him unto the Gentiles. (Verses 12-
21.) After all this he was not authorized to act as a minister of the Gospel,
because he had not yet been properly called and ordained.
It was ten years after this, according to the chronology of the New
Testament, that Paul was ordained to the Priesthood or authority to act in
the name of the Lord. It is stated that certain Prophets and Teachers were in
the Church at Antioch, and "As they ministered to the Lord and fasted, the
Holy Ghost said, 'Separate me Barnabas and Saul for the work whereunto I
have called them.' And when they had fasted and prayed and laid their
hands upon them they sent them away." (Acts XIII; 2, 3; see also Acts IX;
15-18.) Paul in his epistles invariably declared that he was not called by the
will of man; and he taught that no man of himself could rightfully assume
the authority to administer in the name of the Lord. To the Galatians he
wrote: "Paul an Apostle (not of men, neither by man, but by Jesus Christ
and God the Father who raised him from the dead)." (Gal. I; 1.) Writing to
Titus, Paul said: "For this cause left I thee in Crete. That thou shouldst set in
order the things that are wanting, and ordain Elders in every city as I had
appointed thee." (Titus I; 5.) Writing to Timothy, Paul says: "Neglect not the
gift that is in thee, which was given thee by prophecy, with the laying on of
the hands of the presbytery." (I Tim. IV; 14.) It was thus that the seven
Deacons were ordained, as recorded in Acts VI; 6.
That there was a divinely appointed ministry in the Church established by
our Savior, must be evident to every mind open to the truth, on reading the

New Testament; also that these were essential to the Church, and that
without them there can be no true Church of Christ on earth. Explaining this
subject and stating the order of the Christian ministry given by Christ, Paul
says: "And he gave some Apostles, and some Prophets, and some
Evangelists, and some Pastors and Teachers." (Eph. IV; 11.) These inspired
men were, as we have seen, called of God, not of men, and were appointed
and ordained to their respective callings by divine authority. It is claimed
that these were necessary only in the first days of the Church of Christ on
earth, and that they are no longer needed. But the succeeding verses of the
scripture we have quoted show most positively to the contrary. They were
given Paul says, "For the perfecting of the Saints, for the work of the
ministry, for the edifying of the body of Christ; till we all come in the unity
of the faith, and of the knowledge of the Son of God, unto a perfect man,
unto the measure of the stature of the fulness of Christ; that we henceforth
be no more children, tossed to and fro and carried about with every wind of
doctrine, by the sleight of men, and cunning craftiness whereby they lie in
wait to deceive." (Verses 12-14.) Without these divinely ordained and
inspired men, holding this Holy Priesthood, the work of the ministry cannot
be performed acceptable to God, neither can the Church be perfected. They
are absolutely necessary until all shall come to the unity of the faith and a
knowledge of the Son of God. The absence of that divine authority, and of
the gift of the Holy Ghost, has caused the division and dissension that now
exist among professing Christians, who are, "tossed to and fro and carried
about with every wind of doctrine," led hither and thither by unauthorized
and uninspired men, and by the "cunning craftiness" whereby hirelings who
preach for money, "lie in wait to deceive" and "make merchandise of the
souls of men."
All the ministrations, ordinances, baptisms, confirmations, performances
and ceremonies that have been instituted by men and conducted under
merely human authority, whether devoutly, sincerely, and piously, or with
wilful intent to impose upon the ignorance and credulity of mankind, are
void in the sight of heaven, are not recognized of God, and have no virtue
or effect as aids to salvation. God's house is a house of order, and He will
accept only that which He has authorized and ordained. However startling
this may appear, it is the eternal truth, which will stand the test of both

reason and revelation. Truth is mighty and will prevail. The remedy for
these tremendous evils will be pointed out in succeeding pamphlets.

No. 6.
That there has been a great departure from the doctrines, ordinances and
discipline of the Church as it existed in the days of Christ and His Apostles,
must be evident to every unbiased enquirer into religious truth. This has
been demonstrated to some extent in tracts already presented to the reader.
But the full measure of the apostasy that has taken place would take
volumes to represent in detail. The proofs are ample that it has been
universal
When Jesus Christ commenced His ministry on earth He found the people
who claimed to be the special subjects of divine blessing and approbation,
with all their Priests and ministers and learned divines, entirely out of the
way of life and salvation. None were acceptable unto God. He denounced
the most pious, respectable, devout and educated among them as hypocrites
and "whited sepulchres." Their foreign missionary enterprises he declared
obnoxious to the Almighty, and informed them that when they compassed
sea and land to make one proselyte they made him "two fold more the child
of hell." (Matt. XXIII; 15). He pronounced them blind guides who made
clean the outside, but within were full of extortion and excess. The spirit of
the Lord had departed from those who honored His name with their lips, but
who had departed from His ways, and who, in place of the word of God,
"taught for doctrine the commandments of men." They were without
authority from God, although they claimed to have it by descent and
ordination through a long line of predecessors and prophets. It should not be
deemed impossible that a similar universal apostasy could take place after
the establishment of the Church of Christ by Him and His Apostles. But
whether so considered or not, the facts are too patent to be denied when
they confront the honest and enlightened mind.

It has been shown that the Gospel as taught and administered by Christ and
His Apostles required first, faith in God and Jesus Christ; second,
repentance, which included reform of conduct; third, baptism by immersion
for the remission of sins; fourth, the reception of the Holy Ghost by the
laying on of the hands of divinely authorized men; and that obedience to
these brought the gifts of the spirit, including love, joy, peace, patience,
brotherly kindness, charity, healings, tongues, interpretations, discerning of
spirits, miracles, prophecy, revelation, and the unity in one body of all who
were baptized into the Church, no matter what had been their previous
beliefs. Also that the ordinances of the Gospel were administered by men
inspired of God, who were in communion with Him, and who were
ordained to act for and in behalf of Deity, so that what they performed by
that authority on earth was acknowledged and sealed in heaven. And that in
the Church of Christ there were Apostles, Prophets, Evangelists, Pastors,
Teachers, Elders, and other officers, who were constituent parts of the body
of Christ. This may be further seen by a careful reading of 1st Cor. XII,
from which it clearly appears that God placed these in the Church, that they
were all essential to its existence, and that one of them could not say to any
of the others, "I have no need of thee."
Look at the condition of so-called Christendom today! There are no inspired
Apostles, Prophets, Evangelists, Pastors and Teachers, administering by
divine authority and in the power and demonstration of the Holy Ghost. In
their place there are contending Priests and Teachers guided by the wisdom
of men, the learning of the schools and the traditions of the Fathers, not
even claiming that there is any direct communication between them and
God, but persuading mankind that revelation has ceased, and the voice of
prophecy is hushed forever. Not one of the clashing, jarring and discordant
sects of the day proclaim the Gospel as it was preached by Peter on the day
of Pentecost, and as taught by all the duly authorized servants of God in the
primitive Christian Church. The gifts and signs which Christ promised to
true believers, and which were enjoyed by the members of His Church
according to their needs and their faith, are not only absent from the
churches of these degenerate times, but are pronounced needless and "done
away." There is no "unity of the faith," no actual "knowledge of the Son of
God," no manifestations of His divine acceptance nor of the power and
glory of the Holy Ghost.

What is the reason of this transformation? Has God changed? Is Christ
divided? Is the Holy Spirit dead? Or, have not men changed the order,
ordinances, discipline, doctrines, and spirit of the Church of Christ? Is not
the prediction of Isaiah the Prophet concerning these times literally
fulfilled? "The earth also is defiled under the inhabitants thereof, because
they have transgressed the laws, changed the ordinance, broken the
everlasting covenant." He said it should be "As with the people, so with the
priest; as with the servant, so with his master; as with the maid, so with her
mistress; as with the buyer, so with the seller; as with the lender, so with the
borrower; as with the taker of usury, so with the giver of usury to him."
(Isaiah XXIV; 2-5).
The deplorable condition of affairs in modern Christendom was foreseen
and predicted by the Apostles of Jesus Christ, whose forebodings have
come down to us in the New Testament. Paul, writing to Timothy, spoke in
this wise: "This know also, that in the last days perilous times shall come.
For men shall be lovers of their own selves, covetous, boasters, proud,
blasphemers, disobedient to parents, unthankful, unholy, without natural
affection, trucebreakers, false accusers, incontinent, fierce, despisers of
those that are good, traitors, heady, high-minded, lovers of pleasures more
than lovers of God; having a form of godliness, but denying the power
thereof; from such turn away." (2nd Tim. III; 1-5). Also: "Now the spirit
speaketh expressly, that in the latter times some shall depart from the faith,
giving heed to seducing spirits, and doctrines of devils; speaking lies in
hypocrisy; having their conscience seared with a hot rod." (lst Tim. IV; 1,
2). Paul further said: "I charge thee therefore before God, and the Lord
Jesus Christ, who shall judge the quick and the dead at His appearing and
His kingdom; preach the word; be instant in season, out of season; reprove,
rebuke, exhort with all long-suffering and doctrine. For the time will come
when they will not endure sound doctrine; but after their own lusts shall
they heap to themselves teachers, having itching ears; and they shall turn
away their ears from the truth, and shall be turned unto fables." (lst Tim. IV;
1-4). Paul also said they should be "ever learning and never able to come to
a knowledge of the truth." Writing to the Thessalonians he said: "Now we
beseech you brethren by the coming of our Lord Jesus Christ, and by our
gathering together unto Him, that ye be not soon shaken in mind or be
troubled, neither by spirit, nor by word, nor by letter as from us, as that the

day of Christ is at hand. Let no man deceive you by any means, for that day
shall not come except there come a falling away first." (2nd Thess. II; 1-3).
The Apostle Peter also foresaw this great apostasy, and spoke of it in this
wise: "But there were false prophets also among the people, even as there
shall be false teachers among you, who privily shall bring in damnable
heresies, even denying the Lord that brought them, and bring upon
themselves swift destruction. And many shall follow their pernicious ways,
by reason of whom the way of truth shall be evil spoken of. And through
covetousness they shall with feigned words make merchandise of you,
whose judgment now of a long time lingereth not and their damnation
slumbereth not." (II Peter; 1-3.)
The "falling away" commenced in the time of the Apostles, and hence their
numerous warnings and exhortations to the Saints, rebuking schisms and
divisions, and counseling unity, showing that the Spirit of the Lord
promoted union and led people to the knowledge of the truth, while
dissension and strife came from that Evil One, and led to darkness and
death. That the great apostasy commenced at a very early period is shown
by the words of Paul, "for the mystery of iniquity doth already work. Only
He that now letteth will let until he be taken out of the way." (II Thess. II;
7.) By the time the Apostles were taken out of the way, most of them slain
by the hands of wicked men, the apostacy had assumed such proportions
that only seven of the Churches were deemed worthy of a divine
communication through the Apostle John, who had been banished to the
island of Patmos. And in that revelation most of them were denounced by
the Lord because they had "left their first love," and were commanded to
repent or he would remove them out of their place. Some of them were
"neither cold or hot," others had given away to seducing spirits, and had
committed abominations and imbibed false doctrines. (See Rev., chapters I,
II, and III.) In that same vision John the beloved saw the Church in the form
of a woman, clothed with the sun, the moon under her feet, and a crown of
twelve stars on her head taken away into the wilderness, to remain for a
lengthened period, and in her place he saw "a woman sitting upon a scarlet
colored beast, full of names of blasphemy," and though decked with gold
and precious stones, she held in her hand a golden cup full of abominations,
and the name upon her head was Mystery. He saw further that all nations

were made to drink out of that golden cup, by which they were made
drunken. (See Rev. XII; 1-6; XVII; 1-5; XVIII; 2, 3.)
It is clear from these predictions in the New Testament, and others that
might be cited, that the departure from the purity, simplicity and unity of the
Gospel of Christ was to be universal; and that these prophecies were
fulfilled we have the testimony of the Church of England. In her Homily on
the Perils of Idolatry she declares: "Clergy and laity, learned and unlearned,
men, women and children, of all ages, sects and degrees, of whole
Christendom, a most horrible and dreadful thing to think, have been at once
buried in the most abominable idolatry, and that for eight hundred years or
more." That being true, how is it possible to believe that the Church of
Christ had any existence on earth after that long continued darkness and
apostacy? How could there be any remnant left of the divine authority held
by the Apostles and Priesthood of the original Christian Church? If the
Romish Church, from which the Church of England seceded, had no divine
authority, then the Church of England could have none, for all she had she
obtained from that Church. If the Romish Church possessed that authority,
still the Church of England could have none, for Rome excommunicated
her with all her priests and ministers. The Church of England being without
divine authority, all the various contending sects that have sprung from her
are of necessity in a similar condition, for none of them even claim to have
received any revelation from God restoring that authority and re-
establishing the Church of Christ.
From the Pope of Rome down to the latest minister presuming to act in the
name of the Lord, there is not and cannot be one who holds the Holy
Apostleship or any portion of that sacred Priesthood which God placed in
the Church, and which Paul declared essential to its existence. Good men,
learned men, devout men, there have been by millions; noble, pious, and
blessed women also, with them, have done the best they could according to
their light and opportunities; but darkness "has covered the earth and gross
darkness the people," and the apostacy from primitive Christianity, as
foretold by its founders, has been awful and universal!
But thank God, the restoration was also predicted, and it will be a pleasing
task in further tracts to set this forth, as revealed and brought about by

revelation from God the Eternal Father, through Jesus Christ His Son and
the Holy Angels sent from their presence, to usher in the last and greatest of
all dispensations.

No. 7.
"And I saw another angel fly in the midst of heaven, having the everlasting
Gospel to preach unto them that dwell on the earth, and to every nation, and
kindred, and tongue, and people, Saying with a loud voice, Fear God and
give glory to Him; for the hour of His judgment is come; and worship Him
that made heaven, and earth, and the sea, and the fountains of waters. And
there followed another angel, saying Babylon is fallen, is fallen, that great
city, because she made all nations drink of the wine of the wrath of her
fornication." (Rev. XVI; 6-8.) In these inspired words John the beloved
Apostle predicted the restoration of the Gospel to the earth, and the
subsequent destruction of that power which had filled the earth with the
darkness of spiritual inebriety and wickedness. That these events were not
revelations of the past, but prophecies of the future manifested to the
Apostle John, is made certain by what he says in Chapter IV, verse 1: "After
this I looked and behold, a door was opened in heaven; and the first voice
which I heard was as it were of a trumpet talking with me, which said, come
up hither, and I will show thee things which must be hereafter." The angels
spoken of in the XIV chapter, quoted above, were among the things which
John was told "must be hereafter." It should be observed that when the
angel should fly to the earth bearing the everlasting Gospel, it was to be at a
time when every nation, and kindred, and tongue, and people would be
without that Gospel in its fullness. That this has been the condition of the
world for a long time has already been demonstrated to the reader.
In predicting events that would occur previous to his coming and "the end
of the world," Christ declared, "And this Gospel of the kingdom shall be
preached in all the world for a witness unto all nations, and then shall the
end come." (Matt. XXIV; 14.) From this we learn that the Gospel as
preached by Christ and delivered by Him to the Apostles, is to be preached

in all the world as a witness of His second advent and a sign of the
approaching end. (See verse 3.)
The foregoing predictions correspond with the prophecy of Isaiah:
"Wherefore the Lord said, forasmuch as this people draw near me with their
mouth, and with their lips do honor me, but have removed their heart far
from me, and their fear toward me is taught by the precepts of men;
Therefore I will proceed to do a marvelous work among this people, even a
marvelous work and a wonder; for the wisdom of their wise men shall
perish and the understanding of their prudent men shall be hid." (Isaiah
XXIX; 13, 14.) All the Prophets whose writings have been collected in the
sacred volume called the Bible, have proclaimed the glory of the latter days
and the final triumph of truth over error, and of the power of God over the
deceptions of that Evil One.
Thus not only the restoration of the Gospel was foretold by holy men of
God, after the great apostacy that was to take place, but the manner of its
revelation was also explained. It was to be by the coming of an angel from
heaven. To whom might it be expected that this angel should appear? To the
learned divines and contending sectaries of modern Christendom? Do they
not all declare that revelation ceased when John received his vision,
recorded in the Book of Revelation? Do they not teach that though angels
once ministered to men, the day of their coming has long since passed?
Have they any faith to call on God for a divine communication? And will
the Almighty reveal anything except to those who call upon Him in faith?
God's ways are not as man's ways. Therefore, as Paul expressed it, "Not
many wise men after the flesh, not many mighty, not many noble are called,
but God has chosen the foolish things of the world to confound the wise.
And God hath chosen the weak things of the world to confound the things
which are mighty, that no flesh should glory in His presence." (I Cor. I; 26-
29.) And as quoted above, the Lord determined that in bringing forth His
latter-day work, "a marvelous work and a wonder," "the wisdom of the wise
should perish and the understanding of the prudent should be hid."
It was in the year 1823 that the angel spoken of by John the Revelator came
with the everlasting Gospel to a young man scarcely eighteen years of age,
of obscure, though respectable parentage, and without the learning of the

schools. His name, too, was common, and his occupation that of a farmer's
boy. Joseph Smith, whom the Lord raised up to receive His word, establish
His Church, and prepare the way for the Redeemer's second coming, was
led to enquire of the Lord through reading the scriptures for the purpose of
finding out which of all the disputing religions was right. Coming to the
Epistle of James, 1st chapter and 5th verse, he read: "If any of you lack
wisdom let him ask of God, that giveth to all men liberally and upbraideth
not; and it shall be given him. But let him ask in faith, nothing wavering."
Relying on this word, he went into the woods to pray, and in the simplicity
of his heart called on God for the wisdom which he felt he greatly needed.
He was then but fourteen years of age, but his faith was strong and wavered
not. His prayers were heard, and in a heavenly vision in open daylight, the
Father and the Son revealed themselves to his astonished gaze. The Father,
pointing to the Son, proclaimed, "This is my beloved Son, hear Him." Our
Savior spoke to the boy, and in answer to His question as to which of all the
religious sects was right, he was told that they had all gone out of the way,
and was commanded to go after none of them, but was promised that in due
time the true Gospel of Christ should be revealed to him.
When the Angel appeared to him, three years later, it was in his chamber,
just as he had retired for the night. Coming in glory, the Angel showed to
Joseph the place where an ancient record was hidden in the side of a hill,
containing the history of the former inhabitants of the American continent,
including an account of a visit made to them by Jesus Christ after His
resurrection from the dead, when He declared to them the same Gospel that
he had preached in Palestine, and also established His Church among them
after the same pattern as that organized on the eastern hemisphere. He was
informed that this record should be subsequently placed in his hands to
translate by the gift and power of God to be given to him through means
which the Lord had prepared for that purpose. This manifestation was thrice
repeated, that Joseph might be fully assured of its reality. Under the
inspiration of Almighty God, the young man was able to obtain possession
of this precious record, inscribed in small and curious characters upon
metallic plates. The Gospel is there set forth in plain and simple language,
and no one who reads the book, which is called the Book of Mormon, with
a prayerful and unprejudiced heart, will fail to be impressed with its divine
origin.

After being thus favored of the Lord, Joseph Smith received a visitation
from John the Baptist, who held authority in ancient times to preach and
administer baptism by immersion for the remission of sins. He came as a
ministering angel, and ordained Joseph Smith and his companion Oliver
Cowdery, to that Priesthood and authority. Thus endowed, these young men
baptized each other, and at a later date were ministered to by the Apostles
Peter, James and John, who ordained them to the Apostleship, with
authority to lay hands on baptized believers and confer the gift of the Holy
Ghost, also to build up and organize the Church of Christ according to the
original pattern.
On the sixth day of April, 1830, the Church of Jesus Christ was organized in
the state of New York, with six members, Latter-day Saints who had been
baptized for the remission of sins and had been confirmed by the laying on
of hands. The Holy Ghost was manifested unto them, and as the Church
grew in numbers the gifts of the spirit were imparted, and the organization
was eventually made complete with Apostles, Prophets, Seventies, Elders,
Priests, Teachers and Deacons, also Bishops and other officers that were in
the primitive Christian Church; indeed all the grades of the Melchisedek
and Aaronic Priesthood, with their keys, powers and endowments, and all
the ordinances, ministrations and divine manifestations necessary to the true
Church of Christ. Men thus divinely authorized, were sent out into the
world to preach the Gospel like the Apostles of old, without purse or scrip,
without salary and without pay of any kind, depending upon the Lord and
friends whom He might rise up to minister to their temporal wants.
Wherever they went and people received their testimony and were baptized
for the remission of sins, the Holy Ghost was poured out upon them through
the laying on of hands, and they invariably obtained a testimony from God
that they were accepted of Him, and that He had in very deed reestablished
His Church on earth. There are now many thousands of living witnesses to
the truth of these things. They are natives of various countries, speaking
different languages, reared in divers religions; they are now brought to the
unity of the faith; they have come to a knowledge of the truth. Doubt has
fled and darkness has been dispersed; the light of heaven shines in their
souls. They are in the strait and narrow way. They are members of the body
of Christ, and His spirit, which searcheth all things, yea the deep things of

God, is the abiding witness from on high and shows them things past,
present, and to come.
This is the latter-day work spoken of by the Holy Prophets. It is the
dispensation of the fulness of times, in the which "God will gather together
in one all things in Christ, both which are in heaven and which are on earth,
even in Him." (Eph. I; 9, 10.) It is the last and greatest of dispensations. In
it will be accomplished the "restitution of all things, which God hath spoken
by the mouth of all His Holy Prophets since the world began." (Acts III;
21.) It is to prepare the way for the second advent of our Lord Jesus Christ,
who will come "in the clouds of heaven with power and great glory," and
"in flaming fire, taking vengeance on them that know not God and that obey
not the Gospel of our Lord Jesus Christ, when He shall come to be glorified
in his Saints." (II Thess. I; 7-10.) In this dispensation, after all people have
been warned and the Gospel has been preached for a witness to all nations,
and the elect are gathered together from the four winds, namely East, West,
North and South, the great tribulations and judgments will be poured out,
the end of the world, that is, the end of the rule of Satan and of the wicked
will come, the kingdoms of this world will become the kingdom of our God
and His Christ, and He will reign over them forever.
"The times of ignorance God hath winked at, but He now commands all
men everywhere to repent." Therefore, oh! ye inhabitants of the earth,
hearken to the voice of the Lord, which is unto all people, Christian and
Pagan, preachers and hearers, Papists, Protestants, infidels, secularists and
agnostics, rich and poor, kings, presidents, rulers, peasants and men and
women of all race, religions and degrees, saying, repent of your sins, of
your false creeds, of your dead forms, and of all your unbelief and
iniquities, and come unto me, and be baptized by my servants, on whom I
have placed my authority, and receive the laying on of their hands, and you
shall have the remission of your sins and the gift of the Holy Ghost, and
shall know that I am God, and that I have set my hand to accomplish my
great work in the earth, and if you abide in me you shall inherit the earth
when it is cleansed and glorified, and shall be crowned with eternal life!

No. 8.
"Truth shall spring out of the earth, and righteousness shall look down from
heaven." So prophesied the Psalmist, (Ps. LXXXV; 11). This may be
viewed as a figurative expression, but it has been literally fulfilled in the
19th century. In the midst of the disputations over the meaning of many
parts of the Bible, which have caused so many heart-burnings and bitter
feelings among preachers and professors of religion, out of the earth has
come forth a sacred record containing divine truth in such plainness and
simplicity as to settle in the minds of believers those controversies which
have agitated the world of theology. When the American continent was
discovered by Columbus and others, who were led to cross the great waters
in search of unknown lands, a dark-skinned race, composed of many
different tribes but evidently of a common origin, were found in possession
of the Western Continent. Varying in their characteristics from the white,
the black, the yellow, and all the European, Asiatic and Ethiopian branches
of the human family, their origin became a cause of wonder and scientific
investigation. The general conclusion arrived at was, that at some remote
period their ancestors had migrated from some portion of the Eastern
Hemisphere, but when, or how, or why this emigration had taken place was
a profound mystery.
But in the year 1829 a book was published in the state of New York,
claiming to have been translated from metallic plates found in a hill-side in
that State, by a young man who was directed to their place of deposit by an
Angel of God, and who was inspired in the work of translation to decipher
the hieroglyphics inscribed on those plates, being aided in the work by an
instrument, discovered with them, called the Urim and Thummin. The
plates had the appearance of gold, were not quite so thick as common tin,
were about six inches by seven in size, were engraved on both sides, and

were fastened together in the shape of a book by three rings at the back.
Acting under instructions of the heavenly messenger the young man, Joseph
Smith, proceeded as quietly as possible to perform the arduous task
required of him. As he was but a poor scholar, he obtained the assistance of
a scribe to write, as he dictated word by word. The news of the discovery,
however, became noised around, and ridicule from both preachers and
people was followed by attempts at violence, so that the plates had to be
concealed, and, with their translator, removed from place to place.
A farmer, named Martin Harris, who had become interested in the work,
received from Joseph Smith a copy of some of the hieroglyphics with their
translation. These he carried to New York and submitted them to some
learned linguists, among them Prof. Anthon, who after examining them,
pronounced them true characters and the translation, so far as he could
determine, to be correct. He wrote a certificate to this effect, and gave it to
Martin Harris. But questioning him as to how the young man had obtained
the record containing these characters, he was informed that it was revealed
to him by an Angel of God. He then requested Mr. Harris to let him look at
the certificate he had given him. On receiving it he tore it up, declaring that
there was no such thing as angels from heaven now-a-days, but said if the
book was brought to him he would endeavor to translate it. A portion of the
record being sealed, Martin Harris informed him of that fact, when he
exclaimed, "I cannot read a sealed book." As will be seen subsequently, he
was, though unwittingly, fulfilling a scriptural prophecy.
That portion of the record which was not sealed was finally translated into
the English language by Joseph Smith, and formed a volume of about 600
pages, which was published as the Book of Mormon. This title was given to
it because a Prophet named Mormon, by command of God, about four
hundred years after Christ, compiled and abridged the records of Prophets
who ministered on the American continent, back to about 600 years before
Christ, when a colony of Israelites was led from Palestine across the waters
and became a numerous people, the ancestors of the present race of
American Indians. The account of their travels, their establishment on the
Western Hemisphere, the revelations of God to them, their division through
wickedness into separate tribes, the manner in which the hue of their
complexion was changed, their wars, their works, their buildings, their

customs, their language, the words of their prophets, are all given in great
plainness in the Book of Mormon. An account is also given of the visit of
our Lord Jesus Christ to this people after His resurrection, fulfilling His
own prediction recorded in John X; 16: "And other sheep I have which are
not of this fold. Them also I must bring, and they shall hear my voice, and
there shall be one fold and one shepherd." That these "other sheep" were not
the Gentiles, as popularly supposed, is clear from Christ's statement, "I am
not sent but unto the lost sheep of the House of Israel." (Matt. XV; 24.) He
established His Church among them, ordaining Twelve Apostles, and giving
them the same Gospel, authority, gifts, powers, ordinances and blessings as
He gave to His "sheep" on the Eastern Hemisphere. Thus the fulness of the
Gospel is contained in the Book of Mormon, which stands as a witness of
the truth of the Bible. The two records supporting each other, and both
united bearing testimony to an unbelieving world that Jesus of Nazareth is
the Christ, the Son of the Eternal God and the Savior of the world.
This record also contains an account of a colony directed of the Lord to the
Western Continent at the time of the scattering of the people from the land
of Shinar and the confusion of tongues, at the stoppage of the building of
the Tower of Babel. The ruins of their cities and temples and fortifications,
discovered by travelers and archaeologists since the publication of the Book
of Mormon, are silent but potent witnesses of the truth of the record. Each
succeeding year brings forth further evidences of this character, that form a
cloud of witnesses to the divine mission of the Prophet, Seer, and
Translator, Joseph Smith. The Book of Mormon has since been published in
many languages and submitted to the scrutiny of the religious and scientific
world, and no one as yet has been able to point out wherein it disagrees
with the Jewish Scriptures or with the facts developed by antiquarian
research and scientific investigation. Yet it was brought forth in this age by
an unlearned youth, not acquainted with the world, reared in rural simplicity
without access to the literature of the time, and without even the ordinary
acquirements of the schoolboy of the present.
According to the Book of Mormon, the people who journeyed from
Jerusalem to the American Continent, taking with them the genealogy of
their fathers and writings of the Law and the Prophets, were of the tribe of
Joseph through Ephraim and Manasseh, and were led out of Palestine when

Zedekiah was King of Judah. In keeping the record which was subsequently
abridged by the Prophet Mormon, they used the learning of the Jews and
the language of the Egyptians. Their hieroglyphs and symbols, however,
were changed and modified, so that the characters upon the plates revealed
to Joseph Smith, where they had lain hidden for about 1,400 years, was a
reformed Egyptian. How this uneducated youth was able to bring forth a
work of such magnitude and importance, unless by inspiration of Almighty
God, and by the means explained, remains a mystery to unbelievers. For a
long time it was pretended by enemies of the work that one Solomon
Spaulding wrote a Manuscript story, which in some unexplained manner
fell into the hands of Joseph Smith, who worked it over into the Book of
Mormon. But that foolish tale has signally failed of its purpose, for in recent
years the Spaulding manuscript has come to light, and is now deposited in
the Library of Oberlin College, Ohio, and proves to be as unlike the Book
of Mormon as Jack the Giant Killer is dissimilar to the Bible.
The colonization of America by the seed of Joseph, who was sold into
Egypt, fulfills the blessing pronounced on the head of Joseph and his sons
by the Patriarch Jacob. (See Gen. XLVII; also XLIX; 22-26;) also the
blessing pronounced by the Prophet Moses, (Deut. XXXIII; 13-17). The
historical portion of the Book of Mormon shows that the American
Continent, possessed by a "multitude of nations," the seed of Ephraim and
Manasseh, is the "blessed land" bestowed on Joseph in addition to his
portion in Canaan. There are to be found the "everlasting hills" and the
"ancient mountains," "the precious things of heaven, and the precious things
of the earth," and all of the characteristics of the country unto which the
branches of the "fruitful bough," were to "run over the wall," as Jacob
predicted. That the word of the Lord was to be given to the seed of Ephraim
may be seen from Hosea VIII; 11, 12: "Because Ephraim hath made many
altars to sin, altars shall be made unto him to sin. I have written to him the
great things of my law, but they were counted as a strange thing." The
coming forth of the Book of Mormon is foreshadowed by Isaiah the
Prophet, Chapter XXIX; 4-9. It is the voice of a fallen people whispering
"out of the dust." It has come at a time when the world is "drunken, but not
with wine," staggering under the influence of false doctrine, and without
Prophets and Seers. It is the "marvelous work and the wonder," which the
Lord was to bring to pass for the confounding of those who had turned

things upside down, and who worshipped Him with their mouths while their
hearts were far from Him.
The words of the book, Isaiah said, were to be presented to the learned,
saying, "Read this I pray thee," and he was to say, "I cannot for it is sealed."
The book itself was to be "delivered to him that is not learned;" and that it
was to be read is clear from verse 18: "And in that day shall the deaf hear
the words of the book, and the eyes of the blind shall see out of obscurity
and out of darkness, the meek also shall increase their joy in the Lord, and
the poor among men shall rejoice in the Holy One of Israel." The coming
forth of the Book of Mormon as the "stick of Joseph," is also predicted in
Ezekial XXXVII; 15-22. The interview of Martin Harris with Prof. Anthon,
related above, fulfilled one portion of Isaiah's prophecy, the other portions
have come to pass in the translation of the book by the unlearned youth and
its reception by the meek and poor among men, and by the restoration of
sight to the blind and hearing to the deaf, who have seen and heard the
words of the book and bear testimony to its divine origin. The "Stick of
Judah"—the Bible, is now joined with the "Stick of Joseph"—the Book of
Mormon—and, as Ezekial foretold, they have become one in the hand of
the Lord, as a witness for Him and His Son Jesus Christ in the latter days.
As a preface to the Book of Mormon the testimony of three witnesses,
namely, Oliver Cowdery, David Whitmer and Martin Harris, is published,
declaring "with words of soberness" that an angel of God came down from
heaven and brought and laid before their eyes the plates from which the
book was translated; that the voice of God from heaven declared that it had
been translated by the gift and power of God, and commanded them to bear
record of it. Also the testimony of eight witnesses is given, who saw the
plates naturally, handled them, inspected the engraving thereon, and turned
over the leaves that had been translated. In addition to these witnesses,
chosen of the Lord to bear record of these facts, thousands of people, of
various nationalities, have received divine testimony that the book is true,
and that Joseph Smith, who translated it by the gift of God, was a true
Prophet, called of God to usher in the dispensation of the fulness of times
proclaim anew the everlasting Gospel, the one plan of salvation, re-
establish the Church of Christ on earth, and prepare the way for the coming
of Him whose right it is to reign, and for the final redemption of the earth

from sin and satan, from darkness and death. And every person who will
read the Book of Mormon with an unprejudiced mind and will ask God in
faith, in the name of Jesus Christ, concerning it, shall surely receive a
witness of its truth, and be guided in the way of eternal salvation.

No. 9.
In proclaiming the great truths that the silence of centuries has been broken;
that the voice of God has again been heard from heaven; that Jesus Christ
His Son has manifested Himself in these latter days; that Angels from the
courts of glory have ministered to man on earth in the present age; that a
sacred record has been brought forth from the ground disclosing the history
of a hemisphere; and bearing the same truths as those recorded in the Bible;
that a Prophet, Seer and Revelator has been raised up to bring in the last
dispensation; that Apostles and other inspired servants of God now minister
among them; that the Church of Christ with all its former organization,
ordinances, gifts, signs and spiritual power has been reorganized on earth;
and that communications may be had with Deity by men and women of
faith now, as at any period in the world's history, the servants of God are
met with the assertion that the day of revelation has long since passed, and
that they must of necessity be either impostors or deluded, because there is
to be no more scripture, prophecy, miracles, angelic ministrations, visions
or actual communication from heaven to earth. This popular error is
fostered and propagated by the ministers of various so-called Christian
denominations, and is accepted by the masses of the people as a settled and
foregone conclusion.
On what ground is such an irrational position assumed? Is not the Almighty
declared in scripture to be unchangeable? Has not His work on earth always
been conducted by men divinely chosen, appointed and inspired? Is there
not as much need of divine revelation to settle religious feuds and doctrinal
differences in the 19th century, as at any previous period? Would not the
word of the Lord be of much more value to mankind than the varied
opinions of uninspired men, no matter how great be their human learning?

Ought not the inhabitants of the earth to be not only willing, but eager to
receive a message from the eternal worlds?
"Ah!" exclaims the objector, "but there were to be no more Prophets after
Christ. He finished the divine plan and completed the revelation of God to
the earth. He warned His disciples against false prophets and false Christs,
and said if it were possible they would deceive the very elect." Does not the
very fact Christ said there would be false prophets, convey the idea that
there would be true Prophets also? If there were to be no more true
Prophets, it would have been easy for the Savior to plainly say so, and thus
there would be no place left for deceivers. But He declared emphatically:
"Wherefore, behold I send unto you Prophets and wise men and scribes, and
some of them ye shall kill and crucify, and some of them ye shall scourge in
your synagogues and persecute them from city to city." (Matt. XXIII; 34.)
Were not Prophets established in the Church of Christ as members of His
body? Read I Cor. XII; 28: "And God hath set some in the Church; first
Apostles, secondarily Prophets, thirdly, Teachers, after that miracles, then
gifts of healing, helps, governments, diversities of tongues." Did not Christ
promise His disciples that after He went away the Comforter should come?
And was not one of the offices of that spirit to show them "things to come?"
(John XVI; 13.) Was not the gift of prophecy bestowed upon members of
the Church of Christ as one of the manifestations of the Holy Spirit? (I Cor.
XII; 10.) And can anybody possess the true testimony of Jesus without that
spirit? The angel that appeared to John the Apostle said: "The testimony of
Jesus is the spirit of prophecy." (Rev. XIX; 10.) Paul prayed for the
Ephesians: "That the God of our Lord Jesus Christ, the Father of glory, may
give unto you the spirit of wisdom and revelation in the knowledge of
Him." (Eph. I; 17.) If revelation and prophecy ceased with Christ, what
about the New Testament, all written after His death and resurrection, by
men now believed to be inspired? Did not the Apostle John behold a
glorious vision and receive a grand revelation, when banished to the Island
of Patmos?
Here again the objection will be raised: "But that revelation was the last
communication from heaven, and its closing chapter forbids any further
revelation." That is also a popular error promulgated by men professing to
be ministers of Christ, and finding themselves destitute of divine power and

Welcome to Our Bookstore - The Ultimate Destination for Book Lovers
Are you passionate about books and eager to explore new worlds of
knowledge? At our website, we offer a vast collection of books that
cater to every interest and age group. From classic literature to
specialized publications, self-help books, and children’s stories, we
have it all! Each book is a gateway to new adventures, helping you
expand your knowledge and nourish your soul
Experience Convenient and Enjoyable Book Shopping Our website is more
than just an online bookstore—it’s a bridge connecting readers to the
timeless values of culture and wisdom. With a sleek and user-friendly
interface and a smart search system, you can find your favorite books
quickly and easily. Enjoy special promotions, fast home delivery, and
a seamless shopping experience that saves you time and enhances your
love for reading.
Let us accompany you on the journey of exploring knowledge and
personal growth!
ebookgate.com

SDN Software Defined Networks 1st Edition Thomas Nadeau D.

More Related Content

Similar to SDN Software Defined Networks 1st Edition Thomas Nadeau D. (20)

Recently uploaded (20)

SDN Software Defined Networks 1st Edition Thomas Nadeau D.